Real domino: open-loop robot execution at test time (Phase 3 + bridge client)#62
Merged
Conversation
Adds a robot-side bridge client and wires the real-bench env to drive the Franka in TEST mode. The env rolls the WHOLE episode out in sim, then flushes the buffered joint trajectory to the robot in one call (env.flush_real_execution), where it is split into move / gripper segments. The split must see the whole episode in ONE call: it emits a gripper segment on a finger transition, tracked from the start of the call. Splitting per option restarts that tracking, so a Place -- which begins already holding the domino from the Pick -- re-emits a leading 'close', i.e. a second force-grasp on the already-clamped domino. That leaves the Franka Hand stuck and the later release is silently dropped. Shipping whole-episode makes the gripper commands alternate cleanly (one grasp per pick, one release per place). Execution is gated behind domino_real_execute_real (default False = dry-run, pure sim). Re-perception at option boundaries is deferred; this is open-loop.
…o LLM) Grounds a solved plan's Pick/Place/Push/Wait options with their exact parameters and rolls them through the real-bench env. Dry-run by default (pure sim, optional MP4); --execute drives the Franka via the bridge.
yichao-liang
approved these changes
Jul 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Builds on #54 (the
pybullet_domino_realenv). Adds open-loop execution on the real Franka at test time — the predicators (robot-ml) half of the bridge. The robot-side server lives in the babyrobot repo (branchreal-phase5, PR BasisResearch/BabyRobotPredicator#42), not here. (Phases refer to the plan at REAL_PLAN.md.)What
pybullet_domino_real.py): TEST-modereset/stepwrap the rollout with a real-executor hook. The internal sim rolls each option out (predicators' own BiRRT-planned skill policy is the trajectory generator); the env buffers the joint actions per option and, at the option boundary, ships the buffered trajectory to the robot. Gated bydomino_real_execute_real(defaultFalse= dry-run == pure sim). TRAIN mode is untouched.go_homes the arm there first, so each option's streamed trajectory starts where the robot actually is (else the robot-side drift guard trips).envs/pybullet_domino/real_bridge_client.py): length-prefixed JSON-over-TCP (stdlib only, connects per RPC);execute_actionssplits the buffered 9-vec joint-target Actions into move/gripper segments (the shapeRobotInterface.execute_segmentsconsumes) and ships them;go_homeRPC. Imports no babyrobot / droid / pyzed.scripts/domino_debug/replay_plan.py: run an exact solved option plan deterministically (no LLM) through the env — dry-run, or--executeto drive the robot.Open-loop: no option-boundary re-perception — the env state stays the sim's prediction. Closed-loop perception is a deliberate follow-up.
Verification (robot-ml, dry-run — no hardware)
Start the bridge in one terminal, then in another terminal run: