lab58_sam3d
session
work
/ihub/homedirs/svs_ald/sudhir/real2sim/captures/session_20260902_171546work
/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab58_sam3dVideo
Stages
10_pose — Pose
Camera poses from unposed images (VGGT / COLMAP / GLOMAP).
How to read it. `registered` should equal `requested`. `n_models` > 1 means the capture split into disconnected pieces -- not enough overlap. A near-zero baseline means the camera barely moved.
| backend | colmap |
|---|---|
| views | 58 |
| baseline_recon_units | 10.393 |
| points | 28,152 |
| has_depth | False |
| seconds | 119.200 |
| matcher | exhaustive |
| registered | 58 |
| requested | 58 |
| n_models | 1 |
| split_warning | False |
| sparse_points | 28,152 |
raw json
{
"backend": "colmap",
"views": 58,
"baseline_recon_units": 10.3934,
"points": 28152,
"has_depth": false,
"seconds": 119.2,
"matcher": "exhaustive",
"registered": 58,
"requested": 58,
"n_models": 1,
"split_warning": false,
"sparse_points": 28152
}20_scale — Metric scale
Converts the scale-free reconstruction to metres using the D455 depth map. This is what replaces Re3Sim's ArUco marker.
How to read it. `spread` is agreement between independent per-frame estimates: <5% consistent, 5-15% loose, >15% do not trust. `baseline_m` should match the real extent of the capture.
| backend | colmap |
|---|---|
| method | sparse_points |
| scale_m_per_unit | 0.277 |
| spread | 0.058 |
| verdict | usable but loose |
| frames_used | 1 |
| frames_total | 1 |
| baseline_m | 2.875 |
raw json
{
"backend": "colmap",
"method": "sparse_points",
"scale_m_per_unit": 0.2766237898375192,
"spread": 0.057505452312191986,
"verdict": "usable but loose",
"frames_used": 1,
"frames_total": 1,
"baseline_m": 2.8750624602689334,
"per_frame": [
{
"scale": 0.2766237898375192,
"median_scale": 0.28009389277040797,
"n": 467150,
"rel_mad": 0.057505452312191986,
"trusted": true,
"method": "sparse_points"
}
]
}30_ground — Ground plane / world frame
Fits the table plane and builds a world frame with +Z up and the surface at z=0, so MuJoCo gravity and object placement mean something.
not run
40_mask — Robot mask
SAM3 text-prompted segmentation of the robot, which must be kept out of the background splat (MuJoCo renders it from the URDF).
not run
50_splat — Background splat
Gaussian splat trained on the masked images, initialised from the measured depth cloud rather than sparse SfM points.
How to read it. `eval` is on HELD-OUT views -- the honest number. Training-view PSNR only measures overfitting. `eval.depth.bias_mm` is the early warning for a wrong scale factor.
| backend | gsplat |
|---|---|
| ply | /ihub/homedirs/svs_ald/sudhir/real2sim/work/lab58/50_splat/background.ply |
| n_gaussians | 1,000,000 |
| train_seconds | 186.700 |
| peak_vram_gb | 1.730 |
| iters | 7,000 |
| cap_max | 1,000,000 |
| sh_degree | 3 |
| init_points | 499,960 |
| train_views | 50 |
| masked | True |
| depth_weight | 0.500 |
| freespace_weight | 0.100 |
| n_views | 8 |
|---|---|
| lpips | 0.164 |
| psnr | 25.804 |
| ssim | 0.853 |
| bias_mm | 27.321 |
|---|---|
| inlier_frac_10mm | 0.348 |
| mean_abs_mm | 68.907 |
| median_abs_mm | 16.670 |
| n | 228,664.625 |
| p95_abs_mm | 375.310 |
| rms_mm | 221.006 |
| valid_frac | 0.893 |
raw json
{
"backend": "gsplat",
"ply": "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab58/50_splat/background.ply",
"n_gaussians": 1000000,
"train_seconds": 186.7,
"peak_vram_gb": 1.73,
"eval": {
"n_views": 8,
"lpips": 0.16358349844813347,
"psnr": 25.80442753530149,
"ssim": 0.8531841263175011,
"depth": {
"bias_mm": 27.321293334988802,
"inlier_frac_10mm": 0.3478857270642104,
"mean_abs_mm": 68.90685689016665,
"median_abs_mm": 16.670353710651398,
"n": 228664.625,
"p95_abs_mm": 375.31017884612004,
"rms_mm": 221.00640869570128,
"valid_frac": 0.89322119140625
}
},
"iters": 7000,
"cap_max": 1000000,
"sh_degree": 3,
"init_points": 499960,
"train_views": 50,
"masked": true,
"depth_weight": 0.5,
"freespace_weight": 0.1
}60_mesh — Static geometry
TSDF fusion of the depth maps into a mesh with real measured thickness, plus convex parts for collision.
How to read it. `extent_m` must match the real object. MuJoCo hulls a mesh whole, so the convex parts are what make concave shapes collide right.
| visual_mesh | /ihub/homedirs/svs_ald/sudhir/real2sim/work/lab58/60_mesh/static.obj |
|---|---|
| views_fused | 58 |
| label | support |
| voxel_m | 0.006 |
| vertices | 41,029 |
| faces | 77,166 |
raw json
{
"visual_mesh": "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab58/60_mesh/static.obj",
"views_fused": 58,
"label": "support",
"seat_slab": {
"centre": [
0.09300000220537186,
0.09800713427364827,
0.6088884634068653
],
"half_extent": [
0.2280000075697899,
0.22299287550151348,
0.02
]
},
"voxel_m": 0.006,
"vertices": 41029,
"faces": 77166,
"extent_m": [
0.642,
0.6439709028888707,
0.6780076517936796
],
"collision_parts": [
"/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab58/60_mesh/collision/part_000.obj",
"/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab58/60_mesh/collision/part_001.obj",
"/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab58/60_mesh/collision/part_002.obj",
"/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab58/60_mesh/collision/part_003.obj",
"/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab58/60_mesh/collision/part_004.obj",
"/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab58/60_mesh/collision/part_005.obj",
"/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab58/60_mesh/collision/part_006.obj",
"/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab58/60_mesh/collision/part_007.obj",
"/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab58/60_mesh/collision/part_008.obj",
"/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab58/60_mesh/collision/part_009.obj",
"/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab58/60_mesh/collision/part_010.obj",
"/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab58/60_mesh/collision/part_011.obj"
]
}70_objects — Objects
Per-object mesh and physics. Mass is hand-tuned; inertia follows from the mesh at that mass.
How to read it. `extents_m` is the reality check. A 4 mm slab means one-sided observation (the RoboSnap failure); anything over ~60 cm means the views are not aligning and the poses are too imprecise.
| object | size (cm) | mass | parts | watertight |
|---|---|---|---|---|
| green cup | 7.3 x 7.2 x 17.9 | 0.15 kg | 24 | True |
| small bottle | 7.5 x 9.0 x 9.0 | 0.3 kg | 2 | False |
raw json
{
"objects": [
{
"name": "green cup",
"slug": "green_cup",
"mesh": "/ihub/homedirs/svs_ald/sudhir/real2sim/sceneforge/runs/sam3d_lab58/sim_assets/green_cup/visual.obj",
"collision_parts": [
"/ihub/homedirs/svs_ald/sudhir/real2sim/sceneforge/runs/sam3d_lab58/sim_assets/green_cup/collision/hull00.obj",
"/ihub/homedirs/svs_ald/sudhir/real2sim/sceneforge/runs/sam3d_lab58/sim_assets/green_cup/collision/hull01.obj",
"/ihub/homedirs/svs_ald/sudhir/real2sim/sceneforge/runs/sam3d_lab58/sim_assets/green_cup/collision/hull02.obj",
"/ihub/homedirs/svs_ald/sudhir/real2sim/sceneforge/runs/sam3d_lab58/sim_assets/green_cup/collision/hull03.obj",
"/ihub/homedirs/svs_ald/sudhir/real2sim/sceneforge/runs/sam3d_lab58/sim_assets/green_cup/collision/hull04.obj",
"/ihub/homedirs/svs_ald/sudhir/real2sim/sceneforge/runs/sam3d_lab58/sim_assets/green_cup/collision/hull05.obj",
"/ihub/homedirs/svs_ald/sudhir/real2sim/sceneforge/runs/sam3d_lab58/sim_assets/green_cup/collision/hull06.obj",
"/ihub/homedirs/svs_ald/sudhir/real2sim/sceneforge/runs/sam3d_lab58/sim_assets/green_cup/collision/hull07.obj",
"/ihub/homedirs/svs_ald/sudhir/real2sim/sceneforge/runs/sam3d_lab58/sim_assets/green_cup/collision/hull08.obj",
"/ihub/homedirs/svs_ald/sudhir/real2sim/sceneforge/runs/sam3d_lab58/sim_assets/green_cup/collision/hull09.obj",
"/ihub/homedirs/svs_ald/sudhir/real2sim/sceneforge/runs/sam3d_lab58/sim_assets/green_cup/collision/hull10.obj",
"/ihub/homedirs/svs_ald/sudhir/real2sim/sceneforge/runs/sam3d_lab58/sim_assets/green_cup/collision/hull11.obj",
"/ihub/homedirs/svs_ald/sudhir/real2sim/sceneforge/runs/sam3d_lab58/sim_assets/green_cup/collision/hull12.obj",
"/ihub/homedirs/svs_ald/sudhir/real2sim/sceneforge/runs/sam3d_lab58/sim_assets/green_cup/collision/hull13.obj",
"/ihub/homedirs/svs_ald/sudhir/real2sim/sceneforge/runs/sam3d_lab58/sim_assets/green_cup/collision/hull14.obj",
"/ihub/homedirs/svs_ald/sudhir/real2sim/sceneforge/runs/sam3d_lab58/sim_assets/green_cup/collision/hull15.obj",
"/ihub/homedirs/svs_ald/sudhir/real2sim/sceneforge/runs/sam3d_lab58/sim_assets/green_cup/collision/hull16.obj",
"/ihub/homedirs/svs_ald/sudhir/real2sim/sceneforge/runs/sam3d_lab58/sim_assets/green_cup/collision/hull17.obj",
"/ihub/homedirs/svs_ald/sudhir/real2sim/sceneforge/runs/sam3d_lab58/sim_assets/green_cup/collision/hull18.obj",
"/ihub/homedirs/svs_ald/sudhir/real2sim/sceneforge/runs/sam3d_lab58/sim_assets/green_cup/collision/hull19.obj",
"/ihub/homedirs/svs_ald/sudhir/real2sim/sceneforge/runs/sam3d_lab58/sim_assets/green_cup/collision/hull20.obj",
"/ihub/homedirs/svs_ald/sudhir/real2sim/sceneforge/runs/sam3d_lab58/sim_assets/green_cup/collision/hull21.obj",
"/ihub/homedirs/svs_ald/sudhir/real2sim/sceneforge/runs/sam3d_lab58/sim_assets/green_cup/collision/hull22.obj",
"/ihub/homedirs/svs_ald/sudhir/real2sim/sceneforge/runs/sam3d_lab58/sim_assets/green_cup/collision/hull23.obj"
],
"position": [
0.2170755133450821,
0.02508587668221862,
0.76107428707065
],
"mass_kg": 0.15,
"friction": 1.0,
"com": [
-0.004019278740899325,
-0.0026659383504090786,
-0.01447237895446348
],
"inertia": [
[
0.0003188087502834143,
-1.1886264799955154e-05,
-1.4155670857521712e-05
],
[
-1.1886264799955154e-05,
0.0003183630273169395,
-1.3783084506670337e-05
],
[
-1.4155670857521712e-05,
-1.3783084506670337e-05,
5.4325009907752914e-05
]
],
"extents_m": [
0.07333039999999999,
0.0716236,
0.1787455
],
"watertight": true,
"single_view_extents_m": [
0.0733303045861719,
0.07319739745970857,
0.17874546215557147
],
"merge_inflation": 1.0000002117224576,
"n_points": 503239,
"mode": "in_scene",
"quat_wxyz": [
0.8468563350764406,
0.015530792443348942,
-0.00015794622716819385,
0.5315948807879695
],
"source": "sam3d + place_objects (ICP fitness 1.000, bbox IoU 0.879)"
},
{
"name": "small bottle",
"slug": "small_bottle",
"mesh": "/ihub/homedirs/svs_ald/sudhir/real2sim/sceneforge/runs/sam3d_lab58/sim_assets/small_bottle/visual.obj",
"collision_parts": [
"/ihub/homedirs/svs_ald/sudhir/real2sim/sceneforge/runs/sam3d_lab58/sim_assets/small_bottle/collision/hull00.obj",
"/ihub/homedirs/svs_ald/sudhir/real2sim/sceneforge/runs/sam3d_lab58/sim_assets/small_bottle/collision/hull01.obj"
],
"position": [
-0.03328346294580517,
0.07785516803299378,
0.7216823862280474
],
"mass_kg": 0.3,
"friction": 1.0,
"com": [
-0.02031156014135594,
-0.011503268474867857,
0.2188096232974717
],
"inertia": [
[
-0.014451127591343045,
0.00014004600100449202,
-0.001338773757399298
],
[
0.00014004600100449202,
-0.014548139647182648,
-0.0008151136211940545
],
[
-0.001338773757399298,
-0.0008151136211940545,
-0.000276438341108814
]
],
"extents_m": [
0.0753424,
0.0901573,
0.0898481
],
"watertight": false,
"single_view_extents_m": [
0.07534234510188659,
0.09826594106858508,
0.09022457145483465
],
"merge_inflation": 0.9958273954781475,
"n_points": 111790,
"mode": "in_scene",
"quat_wxyz": [
0.7110255785316437,
-0.029772823248472947,
-0.028461510235985126,
0.7019587937369608
],
"source": "sam3d + place_objects (ICP fitness 1.000, bbox IoU 0.000)"
}
]
}80_scene — MuJoCo scene
Assembles the MJCF: static geometry, objects, cameras at the real capture poses, robot.
How to read it. `compiles` must be true. The splat is deliberately absent -- MuJoCo cannot render gaussians; it is composited in stage 90.
| xml | /ihub/homedirs/svs_ald/sudhir/real2sim/work/lab58_sam3d/80_scene/scene.xml |
|---|---|
| bodies | 4 |
| geoms | 16 |
| nq | 14 |
| cameras | 58 |
| objects | 2 |
| collision_parts | 9 |
| compiles | True |
raw json
{
"xml": "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab58_sam3d/80_scene/scene.xml",
"bodies": 4,
"geoms": 16,
"nq": 14,
"cameras": 58,
"objects": 2,
"collision_parts": 9,
"compiles": true
}90_verify — Verification vs real photographs
Renders the sim, composites the splat behind it, and diffs against the actual photograph from that pose. The end-to-end check.
not run