# Visual task: 320×200 and an ACT-style network

The canonical visual task, CLI and matrix evaluator default to **320×200** wrist
RGB. Observations are `[N,3,200,320]` float32 in [0,1]; sensor pixels are uint8
`[N,200,320,3]`. The 16-value proprioception, separate 3-value commanded goal,
62-value critic and seven 20-Hz actions are preserved.

## Similarity to the earlier policy-training network

The visual actor uses **ResNet18 + a 256-wide, 8-head transformer with 3 encoder
and 2 decoder layers**, matching the main architecture choices in the earlier
ACT export. Its PPO output head has one hidden layer of 256 units and emits one
7D action per tick. The current actor has **15,847,630 parameters**.
Visual BC uses the same ACT-style features and a deterministic action head.
The state teacher remains the rendering-free 37-state, 256–128–64 MLP task.

This is a PPO-compatible architecture, not an unchanged ACT checkpoint. There is
one action query, no CVAE/action chunk, zero dropout and frozen BatchNorm statistics
for consistent rollout/minibatch outputs. ImageNet channel normalization is used;
weights start untrained without pretrained-weight downloads. The older policy's
16-frame history is not yet implemented here: current proprioception uses one
frame, with goal conditioning supplied explicitly. Full temporal/servo matching
remains part of student qualification.

The implementation is standalone under `src/mjlab_gs/models/visual.py`; it does not
import the old policy-training project. Task spec `bottle-lift-v5` records its
profile. Image dimensions are part of the existing simulator contract. Old small
CNN or differently sized image checkpoints need explicit migration; BC format
is now `mjlab-gs-bc-v3`.

## Validation (no optimizer run)

- Actual two-world GPU reset and step return 320×200 pixels and finite rewards.
- The RSL-RL Gaussian action-distribution path produces finite actions and
  nonzero finite gradients in image, proprioception and goal feature projections.
- Batch versus single-world action difference was
  6.12e-05; train/eval output consistency passes
  with absolute tolerance 1e-4 and relative tolerance 1e-3.
- A scripted deterministic three-input export (`state16`, `goal3`, RGB NCHW)
  reproduces the actor outputs. Visual BC forward uses the shared feature network.
- The teacher's MLP configuration is unchanged. This validation did not perform
  policy optimization, learned evaluation, or a fresh throughput benchmark.

PyTorch/TorchScript export is supported. ONNX attention export is explicitly left
for qualification rather than returning an incorrect MLP-only export. Historical
128×128 task throughput remains labeled with its measured settings. At 256 worlds
and 24 rollout steps, 320×200 float32 RGB storage alone is about **4.39 GiB**;
this excludes the model, optimizer, activations, physics and other rollout data.

## Updated experiment handoff

`~/rl_training/context.md` and `goal.md` now reflect these defaults. The ordinary
success rate remains the reported metric, and unreachable starts remain in its
denominator. Overall SR below 90% is acceptable when that reflects unreachable
starts. The intended outcome is strong accuracy throughout reachable conditions,
using existing graphs, videos and labeled restricted-DR tests where diagnosis
requires them. There is no rigid >90% promotion gate or separate reachable-only
success metric. Unreachable starts are not an excuse for poor learnable behavior.

`results/` is for training/testing runs. Other artifacts may use organized
`configs/`, `datasets/`, `insights/`, `exports/` and `site_publish/` folders. The
handoff favors organization, not forcing all generated files below results.

[Current task specification](task.md) · [Teacher/student workflow](teacher.md).