Tube bin picking: sim2real progress

1. Setup and domain randomization

done 2026-09-24 D415 on mount v4, 320×180 policy image, DR in four ablatable groups. Found and fixed: overlapping box spawns, pile towers that ejected tubes, idle-arm drift from gain DR. Stability numbers below are from the final configuration.

Wrist camera: D415 on mount v4

Domain randomization gallery

DR ranges (read from the gallery run)

Gains are multipliers; distances in metres, angles in degrees; ranges are the full range at DR scale 1 (10% of worlds are nominal). A disabled group keeps all random draws, so ablations use identical starts.

Stability checks under the new DR

2. Data collection

done 2026-09-25 2-box demonstrations: 413 accepted episodes (D 174, E 239; expert 91-92% picked, 51-53% cleared per run). DAgger round 0 (policy-driven states labelled by the expert) started and stopped with partial data.

Expert baseline on 7–10 tubes

Expert summary JSON

10 s expert clip with the calibrated look (26 September)

Tube, box and table colours and the lighting calibrated against the real D415 wrist video; pedestal turned 90°, smooth action noise, 2 mm near plane.

10 s expert clip, current setup (26 September)

External and wrist views after the pedestal, action-noise and near-plane fixes.

20 s expert clip after three fixes (26 September)

Stand (pedestal) turned 90° about z; action noise now smooth (gripper no longer jitters: direction reversals in slow motion 37% -> 10%); camera near plane 2 mm so the finger linkage close to the D415 is drawn instead of see-through. 8 tubes, 1 placed in 20 s.

Scripted expert with the new robot setup (26 September)

8 tubes (messy pile): 7 picked and placed in the right box in 98 s, valid. EE = closed-fingertip midpoint, robot base rotated 180° with home shoulder pan 0, D415 optical centre on the front glass.

Expert ablations (same starts, full time cap)

3-box task (paired with the 3-box baseline, 91.5% picked). One DR group switched off at a time; every random draw is kept, so the piles are identical to the full-DR run on those starts.

Demonstration dataset

Accepted = at least 80% of the tubes picked and valid (box contacts ≤ 2 mm). Rejections so far: {'invalid': 528, 'picked fraction below 0.8': 128}. Frames are 320×180 JPEG q85 wrist images with the 16-value state and the expert's 7-value action.

Pilot check, end to end

An ACT model trained for 6,000 steps on the 40 pilot episodes (validation action L1 0.076) was exported, verified and evaluated in the simulator on 32 paired starts. As expected at this data size it does not solve the task: it approaches and grasps but carries the tube away from the boxes. Measured: bundle delivered 0/283 tubes in 150 s episodes; 6 episodes toppled a box. Pipeline: dataset → training → export → verified bundle → simulator evaluation loading only the bundle.

3. Training

in progress 2026-09-25 v2 = D+E, 100k steps, validation action L1 0.055 (preliminary D-only model: 0.061 at 40k).

4. Sim evaluation

in progress 2026-09-25 v2 on the first 32 expert-baseline starts: picked 50/283 (17.7%, CI 12.3–24.2) vs 37/283 for the D-only model and 88.6% for the expert on the same starts; delivered 27/283; 4 box topples; 1/32 cleared. Full stage-4 evaluation (ablations) not yet run.

ACT policy v2, 2-box task (413 episodes, 100k steps)

Picked 50/283 tubes (17.7%) on 32 paired starts; lifts 1.69 per episode. In this episode it delivers 4 of 7 tubes in about 3 minutes, then drifts up and out of the workspace with 3 tubes left.

ACT policy, 2-box task, collection D only (174 episodes, 40k steps)

Picked 37/283 (13.1%). After the first placement it lines the pads up over a tube and freezes for minutes.

Preliminary ACT policy, 3-box task (324 episodes, 30k steps)

Picked 31/283 tubes (11%). Slow grasps, carried tube drifts away (the 3-box destination ambiguity).

5. Export

not started

Known problems

• The ACT policy is far below the expert: v2 picks 17.7% of tubes vs 88.6% (same 32 starts). Offline it imitates every phase closely, so this is closed-loop drift into states the demonstrations do not cover (coverage problem / covariate shift): it freezes above the pile or drifts out of the workspace when few tubes are left. Planned fix: DAgger rounds.
• The scripted expert cannot work with action delay: at 2 steps (100 ms) it picks 4% of tubes in 150 s. Demonstrations are collected without delay; delay is only used to test the policy.
• Action noise (σ 0.05 of a step) costs the expert: 92.6% picked and 31/64 bins cleared with it vs 96.1% and 49/64 without, on the same starts. It is kept in the demonstrations on purpose (recovery states for imitation).
• About half of the expert's episodes fail the validity gate, mostly tube–tube penetration above 1.5 mm (40 of 128); those episodes are rejected from the dataset.
• Resting tube piles creep: in 10–20% of worlds a tube still moves 1–3 cm/s after 15 s (CG-80, CG-200 and MuJoCo-Newton behave the same), so the strict 5 s settling audit passes only 2–12 of 32 worlds per case. No ejections, NaN or penetration above 0.6 mm; the expert waits at most 6 s.
• Tube size is not randomized: the visual mesh cannot be scaled per world.
• Sim renders the 320×180 image directly while the real image is a 4×4 area average: measured mean difference 4.9–5.7/255 at full DR; training adds blur augmentation.
• The real RG2 width mapping, TCP offset relative to the real flange, camera latency and hand-eye calibration are not measured; the control contract leaves them null.
• Fixed in stage 1: box pose DR of ±20 mm/±8° let neighbouring boxes spawn overlapping (5/192 toppled); loose 7–10 tube drops towered and ejected tubes; arm gain DR with nominal gravity compensation let an idle arm drift up to 0.2 m in 30 s.