Where the faults lived¶
Every correction on the register, with the layer it looked like and the layer it was. The scoreboard says how many gates failed; this page says where the faults were found, and by what. A fault that presents at one layer and lives at another is the normal case.
12 faults traced · 11 of them lived at a different layer than they presented · 7 layers of the stack
Lived at:
| # | Presented as | Lived at | Found by | Changed | Source |
|---|---|---|---|---|---|
| Lived at: platform (1) | |||||
| F-01 2026-09-09 | platform "LeRobot won't install on the Orin": pip install lerobot fails with an error that says almost nothing | platform No channel ships Python ≥ 3.12 + torchcodec ≥ 0.11 with CUDA on aarch64; the stack has to be built from source | Building every piece from source in a container and timing it: 5 h 33 min, 92 % of it PyTorch | The lerobot-jetson container image, upstreamed as LeRobot #4599 Cost: a full source build before the first rollout | post 2026-09-09, LeRobot on Jetson: no pip install torch Post |
| Lived at: engine (1) | |||||
| F-09 2026-10-25 | model The policy jerks at every chunk boundary | engine Each new chunk starts from fresh noise and is sampled d steps before it runs; nothing ties its first actions to the queued ones | Measuring the jump at 254 boundaries per delay, offline, on recorded episodes | Prefix overwrite in vla.cpp: jump cut 71–73 % at +0.4 ms per chunk Cost: — | post 2026-10-25, VLA chunk-boundary jumps; EXPERIMENTS.md: G13b in vla.cpp Post · Gate row |
| Lived at: loader (1) | |||||
| F-03 2026-10-18 | model A world-model result: G11-A | loader The benchmark loaded weights_final.pt with strict=False and a key remap that did not fit transformers 5.5.4: the ViT encoder was random | The Jetson strict-load check (192 missing / 192 unexpected encoder keys) | G11-A re-check: no latent result stands; strict load mandatory before any result Cost: one gate voided (the 12.78 ms timing stands) | EXPERIMENTS.md: G11-A re-check (2026-10-18) Gate row · Post: Two gates I voided |
| Lived at: data (4) | |||||
| F-02 2026-10-11 | model C1: a world-model probe that might predict failures | data Failures were concentrated in one policy: policy identity alone (AUROC 0.76) beat every probe (current 0.68, imagined 0.72) | Scoring policy identity as a baseline before reading the probe | Verdict not proven: the data is confounded with the policy Cost: one gate not proven | EXPERIMENTS.md: C1 probe (2026-10-11) Row |
| F-06 2026-10-23 | engine TurboVLA ignores its instruction because of our port | data The instruction is redundant on ~94 % of the bowls frames (the pose leaks the side); upstream's own trainer on the same data ignores it too (0.020°) | Path A: retrain with the authors' own code, full LIBERO init, against a pre-registered bar | Port claim withdrawn, data claim stands Cost: the cause in the TurboVLA write-up, corrected 15 days after publication | EXPERIMENTS.md: TurboVLA correction (2026-10-23) Gate row · Post |
| F-07 2026-10-23 | model REAP v1: a risk head that catches failures (AUROC 0.93) | data Elapsed time and episode length carried the outcome; with them removed and a common horizon, AUROC falls to 0.79 | Removing the time feature, truncating to a common horizon, adding shuffled- and zero-action controls (REAP v2) | Verdict moved from not proven to fail Cost: a monitor result that would not have survived the arm | EXPERIMENTS.md: REAP v2 (2026-10-23) Gate row · Post |
| F-08 2026-10-23 | evaluation Failure-onset labels for the 25 failed rollouts | data VLM-marked onsets: median offset +6 frames on a spot check, but +110 on one episode | A 5-episode spot check against the video | Episodes 18 and 36 sent for relabelling by hand; REAP v2 dropped onsets for a ramp target Cost: REAP's first verdict held as not proven | EXPERIMENTS.md: Failure onsets (2026-10-23) Row · Post: Audit your labels where they matter |
| Lived at: evaluation (2) | |||||
| F-04 2026-10-18 | model REKI's latency on the Orin: 318 ms per chunk (G12) | evaluation The benchmark timed a random-init Qwen2.5-1.5B without vision tower or Gated DeltaNet plus a hand-written expert; its parts did not sum (135.4 + 230.3 ≠ 318.2 ms) | Checking that the parts of a timing add up to the total | G12 void; the parts-must-sum rule Cost: one gate voided; REKI had been dropped from the architecture on it (D39) | EXPERIMENTS.md: G12 re-check (2026-10-18) Gate row · Post: Two gates I voided |
| F-05 2026-10-19 | model W1's live reflex thresholds 2.2 / 10.0, ready for the arm (G11-B) | evaluation The bar did not hold on real policy rollouts: on the 50 judge episodes the thresholds flag 42/42 successes | Screening the reflex on recorded rollouts before the robot day (W1-A) | Refit from episode maxima (G11-B-cal, fail); the reflex runs in shadow mode Cost: a robot-day gate that would have stopped every normal trial | EXPERIMENTS.md: W1-A (2026-10-19), G11-B-cal (2026-10-19) W1-A row · G11-B-cal row |
| Lived at: register (3) | |||||
| F-10 2026-10-26 | evaluation TurboVLA's LIBERO control "0.105°" on a card | register The value is in normalised LIBERO action units (≈ 5.5° of pan), not degrees | An outside audit of the public repo's L0 checks and units | Card and figure restated (0.105 ≈ 5.53° of pan) Cost: one wrong unit on the landing page | workstack WS-033 log (2026-10-26), site aa78fdd |
| F-11 2026-10-26 | evaluation 21 failed gates on the scoreboard | register Unescaped |Δ| pipes in the vla.cpp determinism row shifted its cells and faked a fail | Hand-counting the fails against the parser before publishing the count | Pipes escaped; the parser splits on unescaped pipes only (20 fails) Cost: one phantom failed gate | workstack WS-039 log (2026-10-26); EXPERIMENTS.md: vla.cpp determinism Row |
| F-12 2026-10-26 | evaluation G-flora-abl: one verdict for the ablation | register The public row merged three arms with three verdicts (27B pass, 9B fail, Gemini Flash narrow fail) | Syncing every gate of the journal into the public log | Row restated with the three verdicts Cost: a pass read as one result | workstack WS-039 log (2026-10-26); EXPERIMENTS.md: G-flora-abl (2026-10-20) Row |
Layers, bottom up: hardware (arm, servos, cables, cameras, mounts) · platform (Jetson image, drivers, builds, thermal) · engine (vla.cpp, ggml, the port) · loader (checkpoints, key maps, strict load) · data (datasets, features, splits, normalisation, leakage) · model (architecture, objective, training) · evaluation (benchmarks, judges, statistics, bars) · register (the log itself: parsing, units, cells). Source data: data/faults.yaml.