sceneforge results

/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_e2e
← back to the capture

lab_e2e

session /ihub/homedirs/svs_ald/sudhir/real2sim/captures/session_20260901_180746
work /ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_e2e
Full pipeline from scratch via run_pipeline.sh -- the exact command the lab capture will use. 8-frame session, COLMAP poses, depth-supervised splat. ./run_pipeline.sh <work> <session> colmap "green cup,small bottle" RESULTS metric scale 1.794 m baseline (1.783 / 1.787 m on two earlier runs) support top 659 mm (654 / 660 mm on earlier runs) seat 44.4 x 46.2 cm green cup 8.8 x 8.7 x 19.3 cm (mesh now matches the measurement) small bottle 3.6 x 3.8 x 9.9 cm end-to-end verify 19.65 PSNR / 0.782 SSIM against the real photographs physics both objects REST, zero drift Independent full-pipeline runs agree on the stool top to +-6 mm and reproduce object sizes exactly, so the tape-measure comparison is a sharp test. STILL UNVALIDATED: nobody has measured that stool. 659 mm is the pipeline's claim about the world, not a checked fact. WHY THE OBJECTS NOW REST The scene compiled, rendered and scored well long before it simulated. Objects fell through the stool, and the cause moved every time it was chased: * the support mesh spanned 1.7 m -- SAM3 correctly returns EVERY stool in the room. Fixed with a geometric filter anchored on the objects. * objects were placed at their cloud centroid, which for a 2x smear starts them inside the table. * they tunnelled through the seat: a TSDF mesh is a shell, CoACD turns a shell into thin slabs. Fixed with a solid slab fitted to the seat only. * that slab first fitted to noise: max(z) was set by 181 stray vertices, the real seat sat 6 cm lower with 18,000. Now found by vertex density. * coarse convex hulls bulged above the seat and LAUNCHED the bottle sideways at 0.7 m/s. Hulls are now kept only below the slab, which owns the seat. * and finally: the bottle slid off. Its collider was the convex hull of a one-sided reconstruction, so its base is a wedge, not a flat face. It tipped, slid 25 cm across the seat, and fell. The last one is not fixable by tuning collision geometry -- the mesh is wrong. Objects now use a CYLINDER collider sized from the PER-VIEW measurements, which are trustworthy where the merged mesh is a smear, and the mesh stays as the visual. Re3Sim reached the same conclusion for the same reason (a hull of a revolved mesh gives a line contact; a cylinder gives a stable patch) -- see reference/re3sim_port_notes.md. Placement is then settled by MuJoCo itself: objects are dropped from 10 cm up and where they come to rest is baked into the XML. The simulator is the authority on what its own collision geometry supports. Objects that do NOT come to rest are left as placed so the failure stays visible rather than being quietly frozen in mid-air. Every one of these was found by the physics check that now ends each run. None of them was visible in any render, PSNR or SSIM. OBJECT GEOMETRY -- what is measured, what is corrected, what is assumed Merging sparse wide-baseline views inflates an object. Reconstructing from the single densest view instead gives correct dimensions but a mesh Poisson cannot close: a 19 cm cup came back as a 4 cm disc, the exact slab failure this pipeline exists to avoid. So neither raw option is usable. What is done instead: MEASURED per-view size. Every view is asked independently how big the object is, after depth outliers inside the mask are rejected. Eight views agree on the cup to within a few millimetres. CORRECTED the merged mesh is scaled per axis onto that measurement. The inflation is anisotropic -- (0.46, 0.47, 0.75) for the cup -- so a single factor fixes the footprint and leaves it 7 cm too short. That the correction is ~0.46 laterally and 0.75 vertically is independent confirmation that the smear comes from lateral pose error, which is where the ~73 mm pose inconsistency lives. ASSUMED mass, and that a cylinder approximates the object for contact. The collider is a cylinder sized from the measurement, not the mesh hull. The hull of a partially-observed object has a wedge for a base: it tips, slides and falls off (measured: 25 cm across the seat, then onto the floor). Re3Sim uses a primitive for its manipulated item for the same reason. None of this is a substitute for the dedicated dense object orbit in CAPTURE.md. It is what can honestly be built from eight wide-baseline views. --- the flythrough problem, and what actually fixes it --- The splat looks excellent AT the eight capture poses and falls apart between them. Measured: 11.9 mm depth error on training views, 60.7 mm one view away -- and visually far worse than 60.7 mm sounds, because a splat fails catastrophically (shards, floaters) rather than gracefully. Tried and FAILED to fix it by tuning: 150k gaussians instead of 1M, double depth weight, triple free-space. Held-out PSNR 18.09 vs 17.81, depth 58.7 vs 59.0 mm. Unchanged. Eight wide-baseline photos do not constrain the space between them and no regulariser invents that information. What works instead: render the POINT CLOUD. A splat must synthesise appearance for angles nothing observed; a point cloud never synthesises anything, so moving the camera makes it sparser rather than wrong. pipeline_points_flythrough.mp4 same camera path as the splat flythrough, 5.2M points from MEASURED depth pipeline_points_orbit.mp4 same path as the splat orbit recon_vggt_* VGGT's predicted-geometry equivalent, 934k pts Ours has 5.6x more points than VGGT's and they are measured rather than predicted. For LOOKING at a sparse capture, points are the better artifact; the splat is still what gets composited behind the robot in the sim, where the camera sits at a real capture pose. This is a sparse-capture symptom, not a verdict on splats. At 250 frames with 70-80% overlap there is real information between viewpoints and the failure mode largely goes away.

Video

lab_e2e_compare.jpg0.2 MB ↓
lab_e2e_compare.mp42.2 MB ↓
lab_e2e_flythrough.jpg0.1 MB ↓
lab_e2e_flythrough.mp46.6 MB ↓
lab_e2e_holds.jpg0.1 MB ↓
lab_e2e_holds.mp41.1 MB ↓
lab_e2e_labels_overlay.png1.3 MB ↓
lab_e2e_mujoco_only.jpg0.0 MB ↓
lab_e2e_mujoco_only.mp40.4 MB ↓
lab_e2e_orbit.jpg0.1 MB ↓
lab_e2e_orbit.mp45.4 MB ↓
lab_e2e_physics.jpg0.1 MB ↓
lab_e2e_physics.mp41.2 MB ↓
lab_e2e_splat_only.jpg0.1 MB ↓
lab_e2e_splat_only.mp41.0 MB ↓

Stages

10_pose — Pose

Camera poses from unposed images (VGGT / COLMAP / GLOMAP).

How to read it. `registered` should equal `requested`. `n_models` > 1 means the capture split into disconnected pieces -- not enough overlap. A near-zero baseline means the camera barely moved.
backendcolmap
views8
baseline_recon_units8.562
points873
has_depthFalse
seconds9.200
matcherexhaustive
registered8
requested8
n_models1
split_warningFalse
sparse_points873
raw json
{
  "backend": "colmap",
  "views": 8,
  "baseline_recon_units": 8.5619,
  "points": 873,
  "has_depth": false,
  "seconds": 9.2,
  "matcher": "exhaustive",
  "registered": 8,
  "requested": 8,
  "n_models": 1,
  "split_warning": false,
  "sparse_points": 873
}

20_scale — Metric scale

Converts the scale-free reconstruction to metres using the D455 depth map. This is what replaces Re3Sim's ArUco marker.

How to read it. `spread` is agreement between independent per-frame estimates: <5% consistent, 5-15% loose, >15% do not trust. `baseline_m` should match the real extent of the capture.
backendcolmap
methodsparse_points
scale_m_per_unit0.210
spread0.023
verdictconsistent
frames_used1
frames_total1
baseline_m1.794
raw json
{
  "backend": "colmap",
  "method": "sparse_points",
  "scale_m_per_unit": 0.20955538549380287,
  "spread": 0.02280070765796717,
  "verdict": "consistent",
  "frames_used": 1,
  "frames_total": 1,
  "baseline_m": 1.7941930749750432,
  "per_frame": [
    {
      "scale": 0.20955538549380287,
      "median_scale": 0.2099375286724237,
      "n": 3458,
      "rel_mad": 0.02280070765796717,
      "trusted": true,
      "method": "sparse_points"
    }
  ]
}

30_ground — Ground plane / world frame

Fits the table plane and builds a world frame with +Z up and the surface at z=0, so MuJoCo gravity and object placement mean something.

not run

40_mask — Robot mask

SAM3 text-prompted segmentation of the robot, which must be kept out of the background splat (MuJoCo renders it from the URDF).

not run

50_splat — Background splat

Gaussian splat trained on the masked images, initialised from the measured depth cloud rather than sparse SfM points.

How to read it. `eval` is on HELD-OUT views -- the honest number. Training-view PSNR only measures overfitting. `eval.depth.bias_mm` is the early warning for a wrong scale factor.
backendgsplat
ply/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_e2e/50_splat/background.ply
n_gaussians150,000
train_seconds354.300
peak_vram_gb0.330
iters7,000
cap_max150,000
sh_degree3
init_points75,000
train_views7
maskedTrue
depth_weight1.000
freespace_weight0.300
eval
n_views1
lpips0.369
psnr18.086
ssim0.632
eval.depth
bias_mm5.305
inlier_frac_10mm0.108
mean_abs_mm129.638
median_abs_mm58.665
n181,464.000
p95_abs_mm443.833
rms_mm291.682
valid_frac0.709
raw json
{
  "backend": "gsplat",
  "ply": "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_e2e/50_splat/background.ply",
  "n_gaussians": 150000,
  "train_seconds": 354.3,
  "peak_vram_gb": 0.33,
  "eval": {
    "n_views": 1,
    "lpips": 0.36938703060150146,
    "psnr": 18.085691559285728,
    "ssim": 0.6322409510612488,
    "depth": {
      "bias_mm": 5.305069283531892,
      "inlier_frac_10mm": 0.10842370938588371,
      "mean_abs_mm": 129.63787646615117,
      "median_abs_mm": 58.66450071334839,
      "n": 181464.0,
      "p95_abs_mm": 443.83326768875145,
      "rms_mm": 291.682377655126,
      "valid_frac": 0.70884375
    }
  },
  "iters": 7000,
  "cap_max": 150000,
  "sh_degree": 3,
  "init_points": 75000,
  "train_views": 7,
  "masked": true,
  "depth_weight": 1.0,
  "freespace_weight": 0.3
}

60_mesh — Static geometry

TSDF fusion of the depth maps into a mesh with real measured thickness, plus convex parts for collision.

How to read it. `extent_m` must match the real object. MuJoCo hulls a mesh whole, so the convex parts are what make concave shapes collide right.
visual_mesh/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_e2e/60_mesh/static.obj
views_fused8
labelsupport
voxel_m0.006
vertices35,288
faces62,691
raw json
{
  "visual_mesh": "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_e2e/60_mesh/static.obj",
  "views_fused": 8,
  "label": "support",
  "seat_slab": {
    "centre": [
      -0.02099999999999999,
      0.14400000000000002,
      0.638582082663031
    ],
    "half_extent": [
      0.222,
      0.23099999999999998,
      0.02
    ]
  },
  "voxel_m": 0.006,
  "vertices": 35288,
  "faces": 62691,
  "extent_m": [
    0.5291961748692864,
    0.599821374331846,
    0.747731470733928
  ],
  "collision_parts": [
    "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_e2e/60_mesh/collision/part_000.obj",
    "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_e2e/60_mesh/collision/part_001.obj",
    "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_e2e/60_mesh/collision/part_002.obj",
    "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_e2e/60_mesh/collision/part_003.obj",
    "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_e2e/60_mesh/collision/part_004.obj",
    "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_e2e/60_mesh/collision/part_005.obj",
    "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_e2e/60_mesh/collision/part_006.obj",
    "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_e2e/60_mesh/collision/part_007.obj",
    "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_e2e/60_mesh/collision/part_008.obj",
    "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_e2e/60_mesh/collision/part_009.obj",
    "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_e2e/60_mesh/collision/part_010.obj",
    "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_e2e/60_mesh/collision/part_011.obj"
  ]
}

70_objects — Objects

Per-object mesh and physics. Mass is hand-tuned; inertia follows from the mesh at that mass.

How to read it. `extents_m` is the reality check. A 4 mm slab means one-sided observation (the RoboSnap failure); anything over ~60 cm means the views are not aligning and the poses are too imprecise.
objectsize (cm)masspartswatertight
green cup8.8 x 8.7 x 19.30.15 kg1True
small bottle3.6 x 3.8 x 9.90.3 kg1False
raw json
{
  "objects": [
    {
      "name": "green cup",
      "slug": "green_cup",
      "mesh": "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_e2e/70_objects/green_cup/mesh.obj",
      "collision_parts": [
        "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_e2e/70_objects/green_cup/collision/hull.obj"
      ],
      "position": [
        -0.015368462743123125,
        0.06947933506881507,
        0.7668919331015226
      ],
      "mass_kg": 0.15,
      "friction": 1.0,
      "com": [
        -0.006750852244965649,
        -0.0009331117405819147,
        -0.015346871997462326
      ],
      "inertia": [
        [
          0.00037832905752053554,
          -1.78318684122811e-05,
          -4.230934107543993e-05
        ],
        [
          -1.78318684122811e-05,
          0.00038258691067671066,
          -2.1379351938854727e-05
        ],
        [
          -4.230934107543993e-05,
          -2.1379351938854727e-05,
          0.00010026270368234642
        ]
      ],
      "extents_m": [
        0.0875211,
        0.0874485,
        0.1927818
      ],
      "watertight": true,
      "single_view_extents_m": [
        0.08752116924011547,
        0.08744846591563823,
        0.1927820384747566
      ],
      "merge_inflation": 0.9999992088757947,
      "n_points": 39523,
      "mode": "in_scene"
    },
    {
      "name": "small bottle",
      "slug": "small_bottle",
      "mesh": "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_e2e/70_objects/small_bottle/mesh.obj",
      "collision_parts": [
        "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_e2e/70_objects/small_bottle/collision/hull.obj"
      ],
      "position": [
        -0.06807690904335877,
        0.2683518509393675,
        0.7058546162463515
      ],
      "mass_kg": 0.3,
      "friction": 1.0,
      "com": [
        0.020669364959979578,
        -0.024543537119834574,
        0.07047984561544224
      ],
      "inertia": [
        [
          -0.001239219025333764,
          -0.000179142776310758,
          0.0004919486789987552
        ],
        [
          -0.000179142776310758,
          -0.0011392764199068595,
          -0.0004337546116989692
        ],
        [
          0.0004919486789987552,
          -0.0004337546116989692,
          -0.00031679515090074023
        ]
      ],
      "extents_m": [
        0.036147,
        0.0375739,
        0.0989589
      ],
      "watertight": false,
      "single_view_extents_m": [
        0.03614704054705972,
        0.03757383599590186,
        0.09895885339816812
      ],
      "merge_inflation": 1.0000004709213,
      "n_points": 8711,
      "mode": "in_scene"
    }
  ]
}

80_scene — MuJoCo scene

Assembles the MJCF: static geometry, objects, cameras at the real capture poses, robot.

How to read it. `compiles` must be true. The splat is deliberately absent -- MuJoCo cannot render gaussians; it is composited in stage 90.
xml/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_e2e/80_scene/scene.xml
bodies4
geoms18
nq14
cameras8
objects2
collision_parts11
compilesTrue
raw json
{
  "xml": "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_e2e/80_scene/scene.xml",
  "bodies": 4,
  "geoms": 18,
  "nq": 14,
  "cameras": 8,
  "objects": 2,
  "collision_parts": 11,
  "compiles": true
}

90_verify — Verification vs real photographs

Renders the sim, composites the splat behind it, and diffs against the actual photograph from that pose. The end-to-end check.

How to read it. This catches the silent failures: wrong scale, rotated background, objects floating. Compare held-out views against training views -- a large gap means the capture was too sparse.
views3
splatTrue
summary
frame3.000
lpips0.227
psnr19.525
sim_pixel_frac0.075
ssim0.784
raw json
{
  "summary": {
    "frame": 3.0,
    "lpips": 0.22685198982556662,
    "psnr": 19.524993588519013,
    "sim_pixel_frac": 0.0752890625,
    "ssim": 0.784220019976298
  },
  "per_view": [
    {
      "psnr": 16.253631655053088,
      "ssim": 0.5518948435783386,
      "lpips": 0.44270581007003784,
      "frame": 0,
      "sim_pixel_frac": 0.0671640625
    },
    {
      "psnr": 22.624678669932372,
      "ssim": 0.9132885932922363,
      "lpips": 0.09102070331573486,
      "frame": 3,
      "sim_pixel_frac": 0.071453125
    },
    {
      "psnr": 19.696670440571584,
      "ssim": 0.8874766230583191,
      "lpips": 0.14682945609092712,
      "frame": 6,
      "sim_pixel_frac": 0.08725
    }
  ],
  "views": 3,
  "splat": true
}
000000.png1.3 MB ↓
000003.png1.1 MB ↓
000006.png0.8 MB ↓