Perception + decisions
SAM3, FoundationStereo, geometric grasp selection, cached tube estimates, the task FSM and recovery. No simulator object-state input.
TUBE MANIPULATION · IMPLEMENTATION MAP
Inside the
scripted expert.
Wrist images become tube geometry. A finite-state policy turns that geometry into goals. A small local controller closes the robot motion loop.
30 Hzphysical RGB stream
up to 5 Hzasynchronous A100 requests
20 Hzshared local controller
125 Hzrobot joint stream
FOLLOW THE SIGNAL
Inputs, decisions, outputs.
Choose a view. Click any node for its exact inputs, outputs, guard and source function.
From observation to motion
Loading the implementation map…
Diagram could not load. Use the SVG link above to open it directly.
Scroll or drag to pan · + / − to zoom · click a node to inspectGraph source ↓
Reading an FSM guard: “Arrived” checks measured EE error, not elapsed command time. If an arrival or wait guard is false, the state retains its current goal. The focused views spell out failure branches and exceptions; the full map shows their connections.
One controller for both policies
Observation capture, API client, command validation, persistent local reference, watchdogs and robot drivers. An NN adapter feeds the same motion contract.
Where the loop closes
Measured pose and gripper feedback close the local motion loop. Object perception refreshes during inspection. The stereo path’s refine state does not rerun the models.
THE BOUNDARY
What crosses the API?
Observation → A100
RGB: 320 × 180 × 3 q: 6 measured joint angles (rad) gap: measured finger opening (m) EE: position (m) + two rotation columns Tool: busy, contact, valid, feedback age Sequence + observation time + frame ID
K, hand-eye and workcell priors are fixed session configuration. Rotation’s third column is reconstructed by a cross product.
A100 → local controller
T_base_EE: absolute 4 × 4 target pose open_width_mm: physical opening; 0 = closed Translation cap: m/s · rotation cap: rad/s Session + sequence + inference time Phase + events + terminal outcome
The client resolves expiry on its own clock: request start + 350 ms. A delayed result does not extend the deadline. The real driver clips large widths to its physical maximum.
THE REAL-RUN CORRECTION
Why placement v2 exists.
The latest recording exposed three distinct behaviors. The diagrams keep the previous v1 separate from the deployed replacement.
| Decision | Previous v1 | Placement v2 · deployed |
|---|---|---|
| What gets centred? | The gripper targets one fixed destination XY. Its carried tube can be off-centre. | The tube centre targets a slot. The EE target compensates the perceived object’s offset in the measured grasp frame. |
| Where does tube two go? | Same EE destination as tube one. | Separate object-centre slots, 30 mm apart across the configured short axis. |
| When does the arm retreat? | After release age > 0.7 s, even if opening is busy. | After fresh opening completion, no grip detected, sufficient gap and 0.4 s continuously clear feedback. |
| When does the task end? | No count limit. Retreat → return → inspect. | Two confirmed releases → clearance retreat → completed → normal stop. |
| What does completed prove? | No explicit completed state. | The requested release cycles finished. Final containment is not visually verified. |
24 mmsecond grasp’s axial offset; previously uncompensated
2.20 ssecond gripper opening, versus 0.8 s until retreat
56 mmmeasured upward wrist travel while that opening completed
Grasp-offset reconstruction uses saved perceived geometry and measured EE transforms. It is not an independent measurement of the final tube position. The recording ended while returning; it did not contain a third grasp.
CURRENT LIMITS
What is still assumed?
Intrinsics, hand-eye, table height, source/destination locations and tube dimensions remain configured priors. Tray boundaries are not checked continuously. Grasp geometry can change if the tube slips. Nudge pressing still has a time-based transition. The controller is bounded DLS IK, not an arm collision planner or force controller.
The nominal Gaussian check for v2 delivered 2/2 validly and reached completed after 944 steps (47.2 simulated seconds). This is one nominal simulation, not a DR result or a new real-robot success claim. Ten-tube work remains parked.