superdex_gs RL teacher

# ppo_v1_seed42 — stopped at iteration 85 (696,320 env steps)

Reward `teacher-v1` (SuperDex YAM task v2 shaping), teacher DR scale 0.1, 256 envs.
Rolling training success 0/512; mean return rose from −1.8 to 4.0 and plateaued.

Deterministic rollout of `model_50.pt` (16 envs, 200 ticks):
TCP settled 5.7 cm from the grasp point — 3.8 cm off the bottle axis and 4.3 cm below it (beside the bottle);
mean raw gripper action +1.8 (beyond the +1 clip, fully open), no bilateral contact in any env.

Cause: the closing bonus only paid within ~2.5 cm while premature closure was penalized everywhere else,
the reach term rewarded descending beside the bottle, and unbounded raw gripper means saturated open, removing exploration.
Revision `teacher-v2`: above-then-descend reach target, centring term, closing/contact reward ×5/×3, premature-closure penalty only
beyond 5 cm, raw-action bound penalty. Reward ordering check (8 envs): scripted pick 200.0, hover open at grasp 13.7, beside bottle 1.2.

# ppo_v2_seed42 — stopped at iteration ~180 (≈1.47 M env steps)

Reward `teacher-v2`, teacher DR scale 0.1, 256 envs, from random weights. Mean training return rose from 0.2 to 92.7 and plateaued;
rolling strict success 0/512 (4 bottles dropped after lifting during the whole run).

Deterministic rollout of `model_150.pt` (16 envs, 200 ticks): bilateral grasp in 16/16 envs from tick ~40, but the z action stayed at −1
(pressing down) and no bottle was lifted (max clearance 0 cm). Local optimum: steady contact/closing reward while pressing down.
Reward ordering was already correct (scripted pick 200 vs scripted grasp-and-press 36.7), so the obstacle was exploration of lifting.

Revision `teacher-v3` (continues from this run's final weights): contact pays mostly once the bottle is above 1 cm, lift-progress reward
(20 × clearance gain per tick while grasped), push-down penalty while grasped below 10 cm. Ordering: scripted pick 197.4, grasp-and-press 17.7.

# ppo_v3_seed42 — continued from ppo_v2_seed42 (iteration 179); stopped after 173 more iterations (≈1.42 M env steps)

Reward `teacher-v3`. Mean training return rose from ~30 to ~162 and plateaued; lifts began (≈110 bottles lifted then dropped) but
rolling strict success stayed 0/512. Forced-lift probe on `model_200.pt`: the learned grasp is centred within 4 mm at the planned height
and lifts every bottle past 10 cm with bilateral contact when z = +1 is commanded, so physics and grasp quality were not the obstacle.

Deterministic rollout of `model_300.pt`: bilateral grasp and lift in 16/16 envs, then hover at 4.4–5.7 cm clearance with z ≈ 0;
no tick above 10 cm. Scripted pick vs scripted hover-at-5 cm returned 197 vs 151 under v3 — too small a margin, and success terminated the
episode, cutting off further reward. Revision `teacher-v4`: lift reward ×1.5, lift progress ×2 below 13 cm, holding reward 1.0 per tick
when grasped, stable and ≥10 cm, and episodes no longer end at success (hold keeps paying).

# ppo_v4_seed42 — continued from ppo_v3_seed42; stopped after 35 iterations (286,720 env steps)

Reward `teacher-v4`, episodes continue after success. Return rose quickly (106 → 273). Deterministic `model_200.pt` (rsl iteration count)
lifted all 16 bottles within ~3.5 s but settled at 10.0–10.2 cm clearance, exactly the success threshold; small dips reset the strict
2 s hold timer (max 0.2 s). Cause: step holding reward at 10 cm and lift reward saturating at 12 cm gave no margin above the threshold.
Revision `teacher-v5`: lift saturates at 15 cm, holding reward ramps from 9 to 13 cm. Scripted pick-and-hold return under v5: 515.6 (8/8 success).

# ppo_v5b_seed42 — continued from ppo_v5_seed42 (v5 reward, eval-gated curriculum); stopped after ~90 iterations

Deterministic validations (64 envs, scale 0.1): model_50 0/64 and model_75 0/64 strict successes; all 64 lifted but held at
10.1–10.2 cm max clearance. Per-tick v5 reward for the scripted hold rose from 1.76 at 9.9 cm to 2.87 at 13.4 cm, so the reward favoured
higher holds. Forced-lift probe from the policy's hover state: z = +1 with the policy's own gripper output dropped the bottles within 0.5 s
(its gripper command drifted from −0.84 toward −0.6 as the state left the training distribution); z = +1 with gripper −1 lifted all 16
bottles to 29 cm. Cause: partial gripper closure. Revision: teacher uses a binary gripper (action < 0 fully closed), same reward.

# ppo_v6_seed42 — continued from ppo_v5b_seed42 with a binary gripper; stopped after ~45 iterations

Switching the gripper to binary (action < 0 fully closed) removed the partial-closure slip, but the inherited policy's gripper command
crossed zero during approach, so it closed early and never grasped (deterministic validation 0/64 "never bilaterally grasped" at
model_100 and model_125; training return fell from ~12 to ~9). PPO did not re-discover grasp timing in 45 iterations.
Decision: stop reward iteration from scratch-trained weights and bootstrap the teacher from demonstrations of the privileged scripted
expert (binary gripper commands, orientation locked), then fine-tune with PPO under the evaluation-gated curriculum.