From simulated picks
to a learned policy.
Bottle-picking demonstrations, ACT training and held-out simulation results using the existing robot, blue camera mount and hybrid Gaussian-background scene.
Simulation results. The final benchmark uses expert-feasible, in-range starts. Separate domain conditions scored 5–8/10; broad robustness and real-robot transfer are not established.
Data collection · three episodes
Varied approach, an empty-grasp recovery, and a motion-extrema example, joined into a 36-second video. Each episode shows the robot overview beside its recorded wrist view.
Playback follows virtual simulation time at 20 FPS. The overview restores recorded states; wrist images are original collection frames. Presentation captions are outside the images. These three examples illustrate behavior, not a random sample or a collection acceptance-rate estimate.
The selected data contains 249 training and 65 validation episodes, split by whole seed: 75,360 transitions with 1,488 fault frames masked out of supervised action targets. Hermite paths vary motion and grasp parameters, with explicit retries. Matching measured human demonstration statistics has not been established.
Final ACT policy · eight rollouts
All eight videos saved by the frozen 100-trial test runner, concatenated into 96 seconds. Each clip identifies its seed. All eight saved clips succeeded; the complete benchmark also contains ten failures, reported below.
The policy receives current 320×200 wrist RGB and 16 causal proprioception frames. Actions are six measured-TCP pose deltas plus gripper opening, 0 closed / 1 open. Arm control is position-only; no force action or wrench input. ACT predicts 20 actions, replanning after five around predicted gripper transitions.
Training and validation
Actual logged metrics for seed 1: initial training through step 15,000, then the 16-frame history fine-tune through step 23,000. The exported policy is step 18,000, selected using validation rollouts before final testing.
Success rates and failure modes
The final combined-DR result is 90/100 (95% interval 82.6–94.5%); the stricter legacy-style metric is 83/100. Of ten failed trials, seven missed the lift/hold deadline and four exceeded penetration limits, with one overlap. There were zero reset errors, IK failures, nonfinite episodes or unintended-contact episodes in that final suite.
Primary success requires at least 10 cm bottle clearance for two continuous seconds within 12 seconds, plus contact, penetration, speed and numerical-validity checks. Maximum penetration across all trials was 4.135 mm, including failures; successful trials must stay below 3 mm. No thresholds were relaxed.
The domain suite retains wider starts with nominal assets/appearance, then varies individual families. It is a separate seed set and distribution from the combined test. Bottle/table/base conditions include 3/1/1 reset errors. Expert feasibility is reported separately in the downloadable diagnostics; failures are not silently removed. The expanded-camera condition uses 1.5× training deviations.
What these results establish
An end-to-end simulation pipeline now collects diverse demonstrations, trains ACT and evaluates a frozen policy. Isolated inference is fast enough for the 50 ms decision interval in this benchmark; hybrid rendering averages about 10.3 FPS, so offline simulation runs slower than its virtual 20 Hz clock.
ServoJ dynamics, TCP/hand-eye and gripper response remain approximate. Mount external collisions are disabled, sensor noise is not a calibrated temporal D455 model, and Gaussian lighting remains baked with some incomplete views. Gaussian appearance DR, additional policy families and hardware transfer remain future work.