> Historical integration report. For the current domain randomization, curriculum, rewards and metrics, see [current report](domain-report.md) and [task specification](task.md). The fixed-scene and starter-reward limitations below describe the earlier task version.

# MJLab Gaussian simulator integration

Created September 12, 2026. Canonical workspace: `/home/sudhir/mjlab_gs`.
Task: `Mjlab-GS-Bottle-UR5`. Upstream MJLab commit:
`8ee51fbcf806a7419189f706d9e394cbeb7790fa` (1.6.0).

## Result

The copied UR5/RG2/D455 bottle scene now runs as a real MJLab task. BC recording,
BC evaluation, PPO, standard MJLab train/play and browser visualization share
`ManagerBasedRlEnv`, the task managers and the same versioned action/observation
contract. The local MJLab clone and task code are separate within the workspace.

All 203 asset/calibration files (395,581,344 bytes) are physically copied and
hash-verified. An OS file-access trace of construction, reset, action, physics
and RGB found **zero accesses to old simulator/training workspaces**.
System drivers/Python remain ordinary system dependencies; the application and
assets have no dependency on GSim, real2sim, policy_training or rl_training.

## Measured full-task performance

One NVIDIA RTX A6000 (48 GB), 128×128 calibrated SH3 RGB, shadows, visual LOD,
packed gsplat with 128-camera chunks. Each environment action includes base-frame
Cartesian IK, 50 × 1 ms physics steps, managers, rewards, termination/reset
bookkeeping and image observations. Five warm-up steps, then 30 timed steps,
with CUDA synchronization at timing boundaries. The hold-open policy returns
constant actions; learned-policy inference and optimizer work are excluded.

| Configuration | Worlds | Environment steps/s | Sampled process GPU GiB |
|---|---:|---:|---:|
| Initial integration | 64 | 624 | 8.38 |
| Optimized | 64 | 872 | 2.04 |
| Optimized | 1024 | 2,842 | 5.03 |

The optimized 64-world run is 1.40× faster than the
initial integration. CUDA graph capture removes repeated IK operator launches.
`solve_ex` avoids a CPU solver-status check each physics substep. Expandable
PyTorch allocation segments prevent packed gsplat's pose-dependent allocation
sizes from accumulating large cached blocks. At 64 worlds, process memory fell
from 8.38 to 2.04 GiB.

These are short integration measurements, not repeated statistical benchmark
trials or maximum-capacity results. Long, varied camera trajectories may have
different visibility/memory peaks. GPU 0 was used; unrelated jobs on the machine
were not stopped. The earlier GSim 1,024-world held-control measurement was about
3,256 env steps/s at 5.18 GiB. Its workload omits Cartesian IK and MJLab managers;
that historical figure is context, not an equal-workload speed comparison.

**Use the MJLab task as the canonical simulator.** Start RL development at 64
worlds/128×128, then scale after profiling the actual policy and rollout storage.
The measured 1,024-world task is practical for batched evaluation and is a useful
training target, but this run did not measure 1,024-world PPO memory. A 24-step
float32 RGB rollout alone is about 4.5 GiB at that scale, before simulator,
networks, activations and optimizer state.

## Correctness checks

- Composed model: nq=19, nv=18, nu=7, 172 geoms. Unused mesh assets are pruned;
  128 meshes remain. Source/composed world geometry positions match exactly.
  Collision size/friction/solver parameters, actuator gains and mass matrix match.
- Six nominal-calibration views: RGB MAE
  1.224–1.508/255.
  The legacy seed-7 frozen images use camera/material/scene randomization, so their
  poses are re-rendered with the nominal reference model for this comparison.
  This does not validate the old randomized visual settings.
- CUDA actor/camera observations have expected dimensions and remain finite.
  Partial reset leaves the other world's qpos unchanged. Manual terminal
  observations and training auto-reset behavior pass.
- Captured/uncaptured IK: across 20 steps in two worlds, maximum arm difference
  7.15e-07 rad and bottle position difference
  0.038 mm. Not bit-identical physics.
- PPO: a real optimizer update changes weights, checkpoint reload reproduces
  policy actions exactly. The normal MJLab `train` executable also completes an
  iteration with this registered task.
- BC: 32 recorded transitions, three optimizer updates, checkpoint loading and
  finite evaluation steps. Hold-open data validates plumbing, not demonstrations.
- Browser check: **passed**. Actual Viser/mjviser client, 3D mesh scene,
  Gaussian wrist feed and play/pause/step/reset controls. See `browser/result.json`.
- Seven relevant upstream entity tests and two contract tests pass.

## GPU boundaries

Physics state, IK arithmetic/Jacobians, camera transforms, Gaussian/foreground
rendering, compositing, observations and rewards use CUDA. Asset loading and
compilation happen on CPU at startup. Packed gsplat still needs small host-side
allocation counts; MJLab also synchronizes some reset/termination bookkeeping.
The pipeline is GPU-resident for bulk computation, **not completely host-free**.
Browser streaming copies selected world state/images to CPU. Disk recording
intentionally transfers data to CPU; neither is part of headless throughput.

## Scope and next qualification work

This is a usable simulator/task foundation, not a trained manipulation policy.
The starter reaching/lifting reward is not the original task's strict success
criterion. MuJoCo Warp's contact-count differences remain relevant to grasp
robustness; long stable grasps, unintended contacts and real-robot performance
still need dedicated evaluation.

The base/room Gaussian field is fixed, with per-world bottle x/y randomization.
Multiple policy cameras and paired camera/material/Gaussian domain randomization
are not implemented. The sensor rejects unsupported multi-camera/intrinsics
contexts. Legacy CPU IK policies require an explicit migration study; matching
16-value proprioception and action units does not make controllers equivalent.

The simple BC loader is for datasets fitting GPU memory. A production dataset
pipeline should stream batches. PPO already uses MJLab's visual actor, privileged
state critic, normalization, checkpoints and TensorBoard logger.

## Running and remote viewing

```bash
cd ~/mjlab_gs
source env.sh
mjlab-gs view --num-envs 4
# On your laptop: ssh -L 8080:127.0.0.1:8080 <GPU-host>
# Open http://127.0.0.1:8080

mjlab-gs train --num-envs 64 --output logs/rl/bottle
mjlab-gs evaluate --checkpoint logs/rl/bottle/final.pt
```

The viewer starts paused and binds localhost. The public `/mjlab-gs/` page reports
results; it does not expose live robot/simulator controls. See the workspace
README for standard MJLab commands, Python API, BC and recording commands.

## Maintained upstream changes

Four small hooks/config options in the local MJLab clone preserve upstream
defaults: renderer function, sensor-context rebinding, optional welded fixed
entities, and optional asynchronous differential-IK solve. Task-specific graph
capture, Gaussian logic, assets, rewards and workflows remain outside the clone.
A fifth change corrects the Viser initial camera elevation sign to match MuJoCo.
`git -C mjlab diff` shows all modifications. Package versions and asset hashes
are recorded locally; setup never needs the older simulator folders.