lab_v2
session
work
/ihub/homedirs/svs_ald/sudhir/real2sim/captures/session_20260901_180746work
/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_v2Restructured pipeline (segmentation-first), 8-frame test session, COLMAP poses.
Stage order changed: segmentation now runs BEFORE the ground fit, so the plane
is fitted to pixels labelled "floor" rather than to whichever plane happens to
dominate a pooled cloud of walls, furniture and floor.
10 pose -> 20 scale -> 30 segment -> 40 ground -> 50 splat
-> 60 mesh -> 70 objects -> 80 scene -> 90 verify
Every geometric stage now consumes labels from stage 30 instead of running its
own SAM3 pass with its own prompt. One model load for all five labels: 49 s,
down from ~3-4 min.
Labels found (median score): floor 0.99, support 0.99, robot 0.74,
green cup 0.96, small bottle 0.93. See lab_v2_labels_overlay.png.
CLAIM TO CHECK WITH A TAPE MEASURE: the pipeline puts the stool top at
660 mm above the floor plane. That is its estimate, not a validated number --
an earlier "0 mm error" line in the log was circular, because the hand-measured
value passed in had been copied from this same output.
Fixed on the way here:
* scale was baked into the rotation block of the world transform, so six
stages were transforming camera poses with a non-orthonormal matrix
(det(R) = 0.011). Rendering hid it by being self-consistent.
* world.json was written into the segmentation stage's directory.
* SAM3 was loading once per label, not once per session.
Known issues:
* the "support" label catches both stools, not just the workspace one
* only 32.4% of floor-labelled points lie on one plane (the label includes
the raised mat and cable runs); the fit is still 8.4 mm rms
--- update: object sizes now measured, and honestly qualified ---
Object reconstruction had two separate faults, found by asking each view what
size it thought the object was:
1. MASK BLEED ONTO BACKGROUND. A segmentation mask boundary is never
pixel-exact. Where it spilled a few pixels past the cup it landed on the
wall metres behind, and those depths were lifted into the object cloud.
Three of eight views reported the cup as 194-258 cm deep. Fixed by keeping
only the dominant depth mode inside each mask (common/geometry.py,
mask_depth_inliers).
2. MERGING SMEARS. With that fixed, every view agrees:
green cup 8.8 x 8.7 x 19.3 cm (near-circular, as a mug should be)
small bottle 3.7 x 3.8 x 9.9 cm
but the merged cloud is 1.9x and 2.5x larger. Global poses carry ~73 mm of
error, comparable to the whole width of a mug, so eight views of it do not
land on top of each other.
Per-object ICP was tried as a fix and made it WORSE (cup 20.0 -> 26.3 cm,
and not one view of the bottle registered). Cups and bottles are surfaces of
revolution: point-to-plane ICP between two partial views slides freely
around the axis and along it. It is off by default; --icp re-enables it.
The pipeline now reports per-view size alongside merged size and warns when
they disagree by more than 1.5x. Per-view is the trustworthy number here.
The real fix is the dedicated dense object orbit in CAPTURE.md, where
consecutive views are close enough that merging is well-posed. This is the
clearest evidence so far that the separate object captures are necessary, not
optional.
--- update: depth supervision measured, and the scene now simulates ---
SPLAT ABLATION (same data, same held-out views, 6 train / 2 held out)
variant PSNR SSIM LPIPS depth err gaussians
photometric only 14.10 0.561 0.413 221.3 mm 600k
+ depth 15.19 0.573 0.415 111.8 mm 600k
+ depth + free-space 15.00 0.569 0.418 101.1 mm 600k
+ depth + free-space, 150k 14.64 0.590 0.404 105.7 mm 150k
Depth supervision is the clear win: +1.1 dB held-out PSNR and it HALVES the
depth error. The second number is the one that matters -- it separates
"reproduces the training photographs" from "reconstructs the surface".
Read the rest cautiously: with two held-out views, 0.2-0.5 dB is noise.
Free-space carving gives the best depth error but no PSNR gain. The 150k
variant is worse on PSNR yet best on SSIM and LPIPS at a quarter the size,
which suggests PSNR here partly rewards memorisation. Re-run the ablation on
the dense capture before setting defaults from it.
THE SCENE NOW SIMULATES. It compiled and rendered correctly long before it
simulated correctly -- objects fell straight through the stool. Four fixes:
1. support mesh spanned 1.7 m: SAM3 correctly returns EVERY stool in the
room. A geometric filter now keeps the component nearest the objects.
1.71 m -> 0.53 x 0.57 x 0.70 m, one stool.
2. objects placed at their cloud centroid started inside the stool (the
cloud is a 2x smear) and were ejected. Now placed resting on the support.
3. objects tunnelled through the seat: a TSDF mesh is a shell, CoACD makes
thin slabs, and an object falling 15 cm moves 3.4 mm per 2 ms step. A
solid slab is now fitted to the seat only, so the space under the stool
stays open for the robot.
4. that slab first fitted to noise -- max(z) was set by 181 stray vertices
spanning 5 x 31 cm while the real seat sat 6 cm lower with 18,000
vertices over 50 x 53 cm. Seat is now found by vertex density: 44 x 45 cm.
Both objects settle at |v| ~ 0. Nothing falls through.
--- production splat, depth-supervised (7000 iters) ---
held-out PSNR 17.81 (was 8.4 photometric on the first lab_colmap run)
held-out depth err 59.0 mm (was 221 mm photometric in the ablation)
end-to-end verify 19.23 PSNR / 0.775 SSIM against the real photographs
59 mm of depth error is now comparable to the pose consistency of this capture
(~73 mm) and to the D455's own noise at 2 m, so the splat is no longer the
limiting factor -- the capture is.
Video
Stages
10_pose — Pose
Camera poses from unposed images (VGGT / COLMAP / GLOMAP).
How to read it. `registered` should equal `requested`. `n_models` > 1 means the capture split into disconnected pieces -- not enough overlap. A near-zero baseline means the camera barely moved.
| backend | colmap |
|---|---|
| views | 8 |
| baseline_recon_units | 8.449 |
| points | 867 |
| has_depth | False |
| seconds | 9.400 |
| matcher | exhaustive |
| registered | 8 |
| requested | 8 |
| n_models | 1 |
| split_warning | False |
| sparse_points | 867 |
raw json
{
"backend": "colmap",
"views": 8,
"baseline_recon_units": 8.4488,
"points": 867,
"has_depth": false,
"seconds": 9.4,
"matcher": "exhaustive",
"registered": 8,
"requested": 8,
"n_models": 1,
"split_warning": false,
"sparse_points": 867
}20_scale — Metric scale
Converts the scale-free reconstruction to metres using the D455 depth map. This is what replaces Re3Sim's ArUco marker.
How to read it. `spread` is agreement between independent per-frame estimates: <5% consistent, 5-15% loose, >15% do not trust. `baseline_m` should match the real extent of the capture.
| backend | colmap |
|---|---|
| method | sparse_points |
| scale_m_per_unit | 0.211 |
| spread | 0.023 |
| verdict | consistent |
| frames_used | 1 |
| frames_total | 1 |
| baseline_m | 1.783 |
raw json
{
"backend": "colmap",
"method": "sparse_points",
"scale_m_per_unit": 0.21108233672992835,
"spread": 0.022845669051798717,
"verdict": "consistent",
"frames_used": 1,
"frames_total": 1,
"baseline_m": 1.783401260853507,
"per_frame": [
{
"scale": 0.21108233672992835,
"median_scale": 0.21134794175103508,
"n": 3452,
"rel_mad": 0.022845669051798717,
"trusted": true,
"method": "sparse_points"
}
]
}30_ground — Ground plane / world frame
Fits the table plane and builds a world frame with +Z up and the surface at z=0, so MuJoCo gravity and object placement mean something.
not run
40_mask — Robot mask
SAM3 text-prompted segmentation of the robot, which must be kept out of the background splat (MuJoCo renders it from the URDF).
not run
50_splat — Background splat
Gaussian splat trained on the masked images, initialised from the measured depth cloud rather than sparse SfM points.
How to read it. `eval` is on HELD-OUT views -- the honest number. Training-view PSNR only measures overfitting. `eval.depth.bias_mm` is the early warning for a wrong scale factor.
| backend | gsplat |
|---|---|
| ply | /ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_v2/50_splat/background.ply |
| n_gaussians | 600,000 |
| train_seconds | 448.500 |
| peak_vram_gb | 0.960 |
| iters | 7,000 |
| cap_max | 600,000 |
| sh_degree | 3 |
| init_points | 300,000 |
| train_views | 7 |
| masked | True |
| depth_weight | 0.500 |
| freespace_weight | 0.100 |
| n_views | 1 |
|---|---|
| lpips | 0.385 |
| psnr | 17.814 |
| ssim | 0.607 |
| bias_mm | -17.997 |
|---|---|
| inlier_frac_10mm | 0.106 |
| mean_abs_mm | 112.097 |
| median_abs_mm | 59.025 |
| n | 181,789.000 |
| p95_abs_mm | 355.947 |
| rms_mm | 223.266 |
| valid_frac | 0.710 |
raw json
{
"backend": "gsplat",
"ply": "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_v2/50_splat/background.ply",
"n_gaussians": 600000,
"train_seconds": 448.5,
"peak_vram_gb": 0.96,
"eval": {
"n_views": 1,
"lpips": 0.3848234713077545,
"psnr": 17.81375632588254,
"ssim": 0.6067019104957581,
"depth": {
"bias_mm": -17.996898340496514,
"inlier_frac_10mm": 0.10593050184554621,
"mean_abs_mm": 112.09723858564217,
"median_abs_mm": 59.024810791015625,
"n": 181789.0,
"p95_abs_mm": 355.94739913940447,
"rms_mm": 223.26620945331206,
"valid_frac": 0.71011328125
}
},
"iters": 7000,
"cap_max": 600000,
"sh_degree": 3,
"init_points": 300000,
"train_views": 7,
"masked": true,
"depth_weight": 0.5,
"freespace_weight": 0.1
}60_mesh — Static geometry
TSDF fusion of the depth maps into a mesh with real measured thickness, plus convex parts for collision.
How to read it. `extent_m` must match the real object. MuJoCo hulls a mesh whole, so the convex parts are what make concave shapes collide right.
| visual_mesh | /ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_v2/60_mesh/static.obj |
|---|---|
| views_fused | 8 |
| label | support |
| voxel_m | 0.005 |
| vertices | 55,260 |
| faces | 99,501 |
raw json
{
"visual_mesh": "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_v2/60_mesh/static.obj",
"views_fused": 8,
"label": "support",
"seat_slab": {
"centre": [
-0.02753481009643549,
0.155,
0.6523519083180233
],
"half_extent": [
0.2199651899035645,
0.2225,
0.02
]
},
"voxel_m": 0.005,
"vertices": 55260,
"faces": 99501,
"extent_m": [
0.528844455927069,
0.5688891211162406,
0.7020732835476742
],
"collision_parts": [
"/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_v2/60_mesh/collision/part_000.obj",
"/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_v2/60_mesh/collision/part_001.obj",
"/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_v2/60_mesh/collision/part_002.obj",
"/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_v2/60_mesh/collision/part_003.obj",
"/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_v2/60_mesh/collision/part_004.obj",
"/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_v2/60_mesh/collision/part_005.obj",
"/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_v2/60_mesh/collision/part_006.obj",
"/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_v2/60_mesh/collision/part_007.obj",
"/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_v2/60_mesh/collision/part_008.obj",
"/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_v2/60_mesh/collision/part_009.obj",
"/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_v2/60_mesh/collision/part_010.obj",
"/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_v2/60_mesh/collision/part_011.obj",
"/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_v2/60_mesh/collision/part_012.obj",
"/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_v2/60_mesh/collision/part_013.obj",
"/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_v2/60_mesh/collision/part_014.obj",
"/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_v2/60_mesh/collision/part_015.obj",
"/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_v2/60_mesh/collision/part_016.obj",
"/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_v2/60_mesh/collision/part_017.obj",
"/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_v2/60_mesh/collision/part_018.obj",
"/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_v2/60_mesh/collision/part_019.obj",
"/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_v2/60_mesh/collision/part_020.obj",
"/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_v2/60_mesh/collision/part_021.obj",
"/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_v2/60_mesh/collision/part_022.obj",
"/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_v2/60_mesh/collision/part_023.obj"
]
}70_objects — Objects
Per-object mesh and physics. Mass is hand-tuned; inertia follows from the mesh at that mass.
How to read it. `extents_m` is the reality check. A 4 mm slab means one-sided observation (the RoboSnap failure); anything over ~60 cm means the views are not aligning and the poses are too imprecise.
| object | size (cm) | mass | parts | watertight |
|---|---|---|---|---|
| green cup | 20.0 x 16.7 x 25.4 | 0.15 kg | 1 | True |
| small bottle | 11.5 x 9.4 x 16.8 | 0.3 kg | 1 | False |
raw json
{
"objects": [
{
"name": "green cup",
"slug": "green_cup",
"mesh": "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_v2/70_objects/green_cup/mesh.obj",
"collision_parts": [
"/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_v2/70_objects/green_cup/collision/hull.obj"
],
"position": [
-0.016382989997952,
0.07750294579513561,
0.7707729848436063
],
"mass_kg": 0.15,
"friction": 1.0,
"com": [
-0.015660912607581505,
-0.006785399262649235,
-0.0201965142355957
],
"inertia": [
[
0.0007908505742181527,
-6.070527154578289e-05,
-0.00015651043702924827
],
[
-6.070527154578289e-05,
0.0008493387267219038,
-4.365015817495558e-05
],
[
-0.00015651043702924827,
-4.365015817495558e-05,
0.00042615898458198294
]
],
"extents_m": [
0.1996621,
0.16714489999999999,
0.254438
],
"watertight": true,
"single_view_extents_m": [
0.08776857976012052,
0.08745041733004033,
0.1928439028584596
],
"merge_inflation": 1.9113104900253413,
"n_points": 39523,
"mode": "in_scene"
},
{
"name": "small bottle",
"slug": "small_bottle",
"mesh": "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_v2/70_objects/small_bottle/mesh.obj",
"collision_parts": [
"/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_v2/70_objects/small_bottle/collision/hull.obj"
],
"position": [
-0.06686909916851103,
0.2763267989502063,
0.7142762481554252
],
"mass_kg": 0.3,
"friction": 1.0,
"com": [
0.01918599437086801,
-0.02354451712883716,
0.06034538072290146
],
"inertia": [
[
-0.0007544934420253513,
-0.00017985388234106165,
0.00042241465973592656
],
[
-0.00017985388234106165,
-0.0004997210064117483,
-0.00038089516238770123
],
[
0.00042241465973592656,
-0.00038089516238770123,
-0.0001857729819577093
]
],
"extents_m": [
0.1154804,
0.094484,
0.1679716
],
"watertight": false,
"single_view_extents_m": [
0.03664423932767006,
0.03784992066145032,
0.09917739052503571
],
"merge_inflation": 2.49628000135363,
"n_points": 8711,
"mode": "in_scene"
}
]
}80_scene — MuJoCo scene
Assembles the MJCF: static geometry, objects, cameras at the real capture poses, robot.
How to read it. `compiles` must be true. The splat is deliberately absent -- MuJoCo cannot render gaussians; it is composited in stage 90.
| xml | /ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_v2/80_scene/scene.xml |
|---|---|
| bodies | 4 |
| geoms | 31 |
| nq | 14 |
| cameras | 8 |
| objects | 2 |
| collision_parts | 24 |
| compiles | True |
raw json
{
"xml": "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_v2/80_scene/scene.xml",
"bodies": 4,
"geoms": 31,
"nq": 14,
"cameras": 8,
"objects": 2,
"collision_parts": 24,
"compiles": true
}90_verify — Verification vs real photographs
Renders the sim, composites the splat behind it, and diffs against the actual photograph from that pose. The end-to-end check.
How to read it. This catches the silent failures: wrong scale, rotated background, objects floating. Compare held-out views against training views -- a large gap means the capture was too sparse.
| views | 3 |
|---|---|
| splat | True |
| frame | 3.000 |
|---|---|
| lpips | 0.237 |
| psnr | 19.234 |
| sim_pixel_frac | 0.085 |
| ssim | 0.775 |
raw json
{
"summary": {
"frame": 3.0,
"lpips": 0.2368802030881246,
"psnr": 19.2338222454058,
"sim_pixel_frac": 0.08511718750000001,
"ssim": 0.7750827074050903
},
"per_view": [
{
"psnr": 16.275255065798802,
"ssim": 0.5468048453330994,
"lpips": 0.43973368406295776,
"frame": 0,
"sim_pixel_frac": 0.0768828125
},
{
"psnr": 21.91877127581748,
"ssim": 0.9004730582237244,
"lpips": 0.10499389469623566,
"frame": 3,
"sim_pixel_frac": 0.08189453125
},
{
"psnr": 19.507440394601115,
"ssim": 0.8779702186584473,
"lpips": 0.16591303050518036,
"frame": 6,
"sim_pixel_frac": 0.09657421875
}
],
"views": 3,
"splat": true
}
