lab58
session
work
/ihub/homedirs/svs_ald/sudhir/real2sim/captures/session_20260902_171546work
/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab5858-frame tripod capture, IR stereo pass, depth from FoundationStereo.
This is the first run on a real capture rather than the 8-frame plumbing test,
and the first where depth comes from the stereo pair instead of the camera's
own ASIC.
DEPTH: FOUNDATIONSTEREO vs THE ASIC
valid pixels 88.9% (the D455 ASIC manages 65% on the same scene)
speed 0.6 s/frame, 58 frames in 35 s
before warping 100% valid in the IR frame; the loss is honest occlusion
when reprojecting into the colour camera
The pair needs no rectification -- the recorded calibration proves the D455
delivers it already rectified: rotation exactly identity, translation
(-94.814, 0, 0) mm, zero distortion, identical fx/fy/cx/cy on both eyes. So
depth = fx*B/disp is exact rather than approximate.
The depth is reprojected from the left IR camera into the colour camera using
the recorded extrinsics. The two sensors sit 59 mm apart; skipping that step
shifts the entire reconstruction by that much. It is then written back into the
session in the same 16-bit millimetre convention the camera would have used, so
every later stage consumes it unchanged.
WHAT 58 FRAMES BOUGHT, AGAINST THE 8-FRAME SESSION
8 frames 58 frames
COLMAP registered 8/8 barely 58/58, one model
consecutive inliers 51 median 220 median
held-out PSNR 17.8 25.8
held-out depth error 59.0 mm 16.7 mm
floor plane inliers 32% 53% (rms 8.1 mm)
end-to-end vs photos 19.5 PSNR 21.7 PSNR / 0.776 SSIM
novel-view flythrough shards near-photographic
The flythrough was the honest complaint about the old data and it is now
answered: lab58_splat_only.mp4 is a smooth novel-view path, not a smear. No
change to the method produced that -- the earlier attempt to fix it by tuning
the splat (150k gaussians, double depth weight) moved nothing. It was the
capture.
GEOMETRY AND PHYSICS
support (stool) 0.64 x 0.64 x 0.68 m, top 627 mm above the floor
seat slab 45.6 x 44.6 cm
green cup 7.3 x 7.2 x 17.9 cm (agreed across 56 views)
small bottle 7.5 x 9.0 x 9.0 cm (53 views)
physics both objects REST, |v| ~ 0
Object meshes still need the per-view rescale -- merging inflated them by 3-4x
even at 58 frames, because the views around each small object are still widely
spaced. The dedicated dense object orbits in CAPTURE.md remain the fix.
STILL UNVALIDATED: nobody has put a tape measure on that stool. The pipeline
says its top is 627 mm above the floor. That is its claim, not a checked fact.
Video
Stages
10_pose — Pose
Camera poses from unposed images (VGGT / COLMAP / GLOMAP).
How to read it. `registered` should equal `requested`. `n_models` > 1 means the capture split into disconnected pieces -- not enough overlap. A near-zero baseline means the camera barely moved.
| backend | colmap |
|---|---|
| views | 58 |
| baseline_recon_units | 10.393 |
| points | 28,152 |
| has_depth | False |
| seconds | 119.200 |
| matcher | exhaustive |
| registered | 58 |
| requested | 58 |
| n_models | 1 |
| split_warning | False |
| sparse_points | 28,152 |
raw json
{
"backend": "colmap",
"views": 58,
"baseline_recon_units": 10.3934,
"points": 28152,
"has_depth": false,
"seconds": 119.2,
"matcher": "exhaustive",
"registered": 58,
"requested": 58,
"n_models": 1,
"split_warning": false,
"sparse_points": 28152
}20_scale — Metric scale
Converts the scale-free reconstruction to metres using the D455 depth map. This is what replaces Re3Sim's ArUco marker.
How to read it. `spread` is agreement between independent per-frame estimates: <5% consistent, 5-15% loose, >15% do not trust. `baseline_m` should match the real extent of the capture.
| backend | colmap |
|---|---|
| method | sparse_points |
| scale_m_per_unit | 0.277 |
| spread | 0.058 |
| verdict | usable but loose |
| frames_used | 1 |
| frames_total | 1 |
| baseline_m | 2.875 |
raw json
{
"backend": "colmap",
"method": "sparse_points",
"scale_m_per_unit": 0.2766237898375192,
"spread": 0.057505452312191986,
"verdict": "usable but loose",
"frames_used": 1,
"frames_total": 1,
"baseline_m": 2.8750624602689334,
"per_frame": [
{
"scale": 0.2766237898375192,
"median_scale": 0.28009389277040797,
"n": 467150,
"rel_mad": 0.057505452312191986,
"trusted": true,
"method": "sparse_points"
}
]
}30_ground — Ground plane / world frame
Fits the table plane and builds a world frame with +Z up and the surface at z=0, so MuJoCo gravity and object placement mean something.
not run
40_mask — Robot mask
SAM3 text-prompted segmentation of the robot, which must be kept out of the background splat (MuJoCo renders it from the URDF).
not run
50_splat — Background splat
Gaussian splat trained on the masked images, initialised from the measured depth cloud rather than sparse SfM points.
How to read it. `eval` is on HELD-OUT views -- the honest number. Training-view PSNR only measures overfitting. `eval.depth.bias_mm` is the early warning for a wrong scale factor.
| backend | gsplat |
|---|---|
| ply | /ihub/homedirs/svs_ald/sudhir/real2sim/work/lab58/50_splat/background.ply |
| n_gaussians | 1,000,000 |
| train_seconds | 186.700 |
| peak_vram_gb | 1.730 |
| iters | 7,000 |
| cap_max | 1,000,000 |
| sh_degree | 3 |
| init_points | 499,960 |
| train_views | 50 |
| masked | True |
| depth_weight | 0.500 |
| freespace_weight | 0.100 |
| n_views | 8 |
|---|---|
| lpips | 0.164 |
| psnr | 25.804 |
| ssim | 0.853 |
| bias_mm | 27.321 |
|---|---|
| inlier_frac_10mm | 0.348 |
| mean_abs_mm | 68.907 |
| median_abs_mm | 16.670 |
| n | 228,664.625 |
| p95_abs_mm | 375.310 |
| rms_mm | 221.006 |
| valid_frac | 0.893 |
raw json
{
"backend": "gsplat",
"ply": "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab58/50_splat/background.ply",
"n_gaussians": 1000000,
"train_seconds": 186.7,
"peak_vram_gb": 1.73,
"eval": {
"n_views": 8,
"lpips": 0.16358349844813347,
"psnr": 25.80442753530149,
"ssim": 0.8531841263175011,
"depth": {
"bias_mm": 27.321293334988802,
"inlier_frac_10mm": 0.3478857270642104,
"mean_abs_mm": 68.90685689016665,
"median_abs_mm": 16.670353710651398,
"n": 228664.625,
"p95_abs_mm": 375.31017884612004,
"rms_mm": 221.00640869570128,
"valid_frac": 0.89322119140625
}
},
"iters": 7000,
"cap_max": 1000000,
"sh_degree": 3,
"init_points": 499960,
"train_views": 50,
"masked": true,
"depth_weight": 0.5,
"freespace_weight": 0.1
}60_mesh — Static geometry
TSDF fusion of the depth maps into a mesh with real measured thickness, plus convex parts for collision.
How to read it. `extent_m` must match the real object. MuJoCo hulls a mesh whole, so the convex parts are what make concave shapes collide right.
| visual_mesh | /ihub/homedirs/svs_ald/sudhir/real2sim/work/lab58/60_mesh/static.obj |
|---|---|
| views_fused | 58 |
| label | support |
| voxel_m | 0.006 |
| vertices | 41,029 |
| faces | 77,166 |
raw json
{
"visual_mesh": "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab58/60_mesh/static.obj",
"views_fused": 58,
"label": "support",
"seat_slab": {
"centre": [
0.09300000220537186,
0.09800713427364827,
0.6088884634068653
],
"half_extent": [
0.2280000075697899,
0.22299287550151348,
0.02
]
},
"voxel_m": 0.006,
"vertices": 41029,
"faces": 77166,
"extent_m": [
0.642,
0.6439709028888707,
0.6780076517936796
],
"collision_parts": [
"/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab58/60_mesh/collision/part_000.obj",
"/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab58/60_mesh/collision/part_001.obj",
"/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab58/60_mesh/collision/part_002.obj",
"/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab58/60_mesh/collision/part_003.obj",
"/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab58/60_mesh/collision/part_004.obj",
"/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab58/60_mesh/collision/part_005.obj",
"/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab58/60_mesh/collision/part_006.obj",
"/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab58/60_mesh/collision/part_007.obj",
"/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab58/60_mesh/collision/part_008.obj",
"/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab58/60_mesh/collision/part_009.obj",
"/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab58/60_mesh/collision/part_010.obj",
"/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab58/60_mesh/collision/part_011.obj"
]
}70_objects — Objects
Per-object mesh and physics. Mass is hand-tuned; inertia follows from the mesh at that mass.
How to read it. `extents_m` is the reality check. A 4 mm slab means one-sided observation (the RoboSnap failure); anything over ~60 cm means the views are not aligning and the poses are too imprecise.
| object | size (cm) | mass | parts | watertight |
|---|---|---|---|---|
| green cup | 7.3 x 7.2 x 17.9 | 0.15 kg | 1 | True |
| small bottle | 7.5 x 9.0 x 9.0 | 0.3 kg | 1 | False |
raw json
{
"objects": [
{
"name": "green cup",
"slug": "green_cup",
"mesh": "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab58/70_objects/green_cup/mesh.obj",
"collision_parts": [
"/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab58/70_objects/green_cup/collision/hull.obj"
],
"position": [
0.16698321826610973,
0.09705993230296543,
0.71204601758497
],
"mass_kg": 0.15,
"friction": 1.0,
"com": [
-0.004019278740899325,
-0.0026659383504090786,
-0.01447237895446348
],
"inertia": [
[
0.0003188087502834143,
-1.1886264799955154e-05,
-1.4155670857521712e-05
],
[
-1.1886264799955154e-05,
0.0003183630273169395,
-1.3783084506670337e-05
],
[
-1.4155670857521712e-05,
-1.3783084506670337e-05,
5.4325009907752914e-05
]
],
"extents_m": [
0.07333039999999999,
0.0716236,
0.1787455
],
"watertight": true,
"single_view_extents_m": [
0.0733303045861719,
0.07319739745970857,
0.17874546215557147
],
"merge_inflation": 1.0000002117224576,
"n_points": 503239,
"mode": "in_scene"
},
{
"name": "small bottle",
"slug": "small_bottle",
"mesh": "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab58/70_objects/small_bottle/mesh.obj",
"collision_parts": [
"/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab58/70_objects/small_bottle/collision/hull.obj"
],
"position": [
-0.02440141104376278,
0.07720262926955085,
0.6572448085437447
],
"mass_kg": 0.3,
"friction": 1.0,
"com": [
-0.02031156014135594,
-0.011503268474867857,
0.2188096232974717
],
"inertia": [
[
-0.014451127591343045,
0.00014004600100449202,
-0.001338773757399298
],
[
0.00014004600100449202,
-0.014548139647182648,
-0.0008151136211940545
],
[
-0.001338773757399298,
-0.0008151136211940545,
-0.000276438341108814
]
],
"extents_m": [
0.0753424,
0.0901573,
0.0898481
],
"watertight": false,
"single_view_extents_m": [
0.07534234510188659,
0.09826594106858508,
0.09022457145483465
],
"merge_inflation": 0.9958273954781475,
"n_points": 111790,
"mode": "in_scene"
}
]
}80_scene — MuJoCo scene
Assembles the MJCF: static geometry, objects, cameras at the real capture poses, robot.
How to read it. `compiles` must be true. The splat is deliberately absent -- MuJoCo cannot render gaussians; it is composited in stage 90.
| xml | /ihub/homedirs/svs_ald/sudhir/real2sim/work/lab58/80_scene/scene.xml |
|---|---|
| bodies | 4 |
| geoms | 16 |
| nq | 14 |
| cameras | 58 |
| objects | 2 |
| collision_parts | 9 |
| compiles | True |
raw json
{
"xml": "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab58/80_scene/scene.xml",
"bodies": 4,
"geoms": 16,
"nq": 14,
"cameras": 58,
"objects": 2,
"collision_parts": 9,
"compiles": true
}90_verify — Verification vs real photographs
Renders the sim, composites the splat behind it, and diffs against the actual photograph from that pose. The end-to-end check.
How to read it. This catches the silent failures: wrong scale, rotated background, objects floating. Compare held-out views against training views -- a large gap means the capture was too sparse.
| views | 8 |
|---|---|
| splat | True |
| frame | 28.000 |
|---|---|
| lpips | 0.221 |
| psnr | 22.763 |
| sim_pixel_frac | 0.064 |
| ssim | 0.798 |
raw json
{
"summary": {
"frame": 28.0,
"lpips": 0.22094916552305222,
"psnr": 22.762799780115966,
"sim_pixel_frac": 0.06368994140625,
"ssim": 0.7982228845357895
},
"per_view": [
{
"psnr": 27.205265295883336,
"ssim": 0.9006074070930481,
"lpips": 0.15371127426624298,
"frame": 0,
"sim_pixel_frac": 0.04297265625
},
{
"psnr": 19.988006392916297,
"ssim": 0.740713894367218,
"lpips": 0.29143771529197693,
"frame": 8,
"sim_pixel_frac": 0.06985546875
},
{
"psnr": 23.240440567750717,
"ssim": 0.7686419486999512,
"lpips": 0.20257964730262756,
"frame": 16,
"sim_pixel_frac": 0.02478515625
},
{
"psnr": 20.075073883271575,
"ssim": 0.7040772438049316,
"lpips": 0.2919166684150696,
"frame": 24,
"sim_pixel_frac": 0.0925625
},
{
"psnr": 23.912487584657708,
"ssim": 0.8599441647529602,
"lpips": 0.17146238684654236,
"frame": 32,
"sim_pixel_frac": 0.059578125
},
{
"psnr": 21.7922159025198,
"ssim": 0.7722250819206238,
"lpips": 0.22508922219276428,
"frame": 40,
"sim_pixel_frac": 0.09004296875
},
{
"psnr": 23.547633830123843,
"ssim": 0.8074946403503418,
"lpips": 0.23526531457901,
"frame": 48,
"sim_pixel_frac": 0.039125
},
{
"psnr": 22.341274783804455,
"ssim": 0.8320786952972412,
"lpips": 0.19613109529018402,
"frame": 56,
"sim_pixel_frac": 0.09059765625
}
],
"views": 8,
"splat": true
}
