How I re-test robot-learning papers: the gate protocol as it ran¶

Every result on this site went through the same loop: write the pass rule down, commit it, run, apply the rule and nothing else, log the row, publish a folder that recomputes the number. That loop has produced 74 gates in the experiment log, and 20 failed their bar. This post is about the loop itself, and only about the parts of it that have actually run.

1. The bar goes in before the run¶

A gate starts as a protocol file with the pass rule in it, committed before anything is computed, and the row in the log cites that commit. Three real ones:

• G-flora (would a robot-trained reward model judge my rollouts?). The bar was committed at 1c50a5d: at the largest threshold that flags at most 2 of 42 successes, flag at least 6 of 8 failures. The image layout was amended at 179c1d6, also before the first call. Result: 8/8 failures at 2/42 successes, pass. Row.
• AW-0 Phase 2 (a per-frame planner for task slices). The settings were fixed at c8b8d96 before any fit; the bar was 85 % per-frame accuracy and p95 ≤ 30 ms. Result: 75.7 % and 81.7 ms, fail on both. Row.
• HWM (the hierarchical world-model crossover). Bars committed before training; the first run turned out to be fed red-only images, so a deviation note was added and the run redone. The bars stayed as they were. Post.

The point of the hash is that anyone can check the bar's date against the result's date. When I change something after seeing data, it goes in its own file, and anything decided after the number is labelled exploratory or post-hoc.

2. Four verdicts, and what separates fail from not proven¶

Each paper in the register gets one of: reproduces, fail, not proven, not tested (plus withdrawn, for my own mistakes). The count of each verdict is on the register's scoreboard, generated from the same file as the table.

The line between fail and not proven is whether the miss could be explained by something the run did not control. RoboJEPA is a clean fail: the 300M model got 11.8 % below copy-last against a 15 % bar, on the setup the bar was written for, and nothing in the run argued for a different reading. HWM is not proven: its high-level model never beat the low level, but it was trained on about 0.6 h of one task against the paper's ~130 h, and its error sat barely below predicting the average latent. An open confound like that makes it not proven rather than fail, even though two of its three bars were missed.

3. The judge at rest¶

Most gates need an outcome per trial. G9c-3 is the deterministic judge that gives it: top camera, arm at rest, a fixed rule. On the test split it scored 94.3 % on 87 judged episodes, with 0 false positives and 5 false negatives (row, post).

Its own bar was 95 %, so it missed by 0.7 points, and the row says so. It was still adopted, because the number that matters for anything that learns from rollouts is false positives: a failure judged as a success goes into the data as a demonstration. A false negative only costs a trial. Judged only at rest also means the verdict never depends on a frame where the arm is moving.

The same reasoning sets the rule for weaker judges (a VLM, a learned scorer): they never decide alone, they pair with a deterministic check, and they earn a place per task through a report card of false positives, false negatives and ambiguous calls against my labels.

4. What is recorded per trial¶

What has actually been recorded on robot days: per trial, the policy, the engine, latency (p50, p99, mean), control rate, VRAM, the condition, success and failure mode, plus video. Those CSVs sit in each folder's expected/ directory. The 69 rollouts behind most monitor gates are public with their outcome labels. One rule runs through all of it: nothing is trained on failed rollouts; failures only feed verifiers and the world model.

5. The work stack¶

Gates run through a queue: one item per task, each with what it needs (offline, vast, arm, rig), what it depends on, a prompt file that is the whole brief, and a stop point. When this was written (29 October) it held 55 items: 37 done, and 8 waiting on the arm or the rig.

Nothing that costs money or moves the arm runs on its own. Each needs its own explicit go: go vast WS-NNN with a cost cap per item, go arm WS-NNN for arm time, go rig for staging, go for publishing. They are separate because they fail differently: a wrong spend is recoverable, a wrong arm motion may not be, and a publish can't be taken back.

6. Cost per result, retries included¶

Every post ends with a compute block: external GPU yes or no, the dollar figure, and the retries. The register's tested papers add up to $45 of GPU and API. Across all published folders it is $46.3–47.3 on vast.ai and $4.9–5.9 of API. Those totals include the failed starts, the hosts that never connected and the runs I threw away, like HWM's red-only first run. That is the number a reader would pay to repeat the work, so it is the one I publish.

7. Three levels of replication¶

Every result has a folder in paper-vs-hardware:

• L0: recompute the post's numbers from the small files in the folder, in seconds, on a CPU. There are 23 folders and all 23 pass their L0 check.
• L1: rerun the analysis from the pinned inputs (checkpoints, datasets, scripts).
• L2: rerun on hardware. The protocol is in the post; the folder does not claim it is re-verified.

What it does not have yet¶

• The v1.5 harness has not run end to end on the arm. It is specified and partly built, and tested on simulated hardware only. Nothing on this site describes it as working.
• The adoption protocol (G24) has not run, and there is no autonomous loop in service.
• The data-engine fields (layout ids, executor per trial, a curated demos/ view) are proposed, not recorded. The LEGO verdict now comes from camera masks, and its verifier card has not run either.
• Gates from the v0.8–v1.1 era say so in their rows; the vocabulary and the judge above are the parts that carried through.

That is the same discipline as the rest: the register lists what ran, at the bar it was given.

The experiment log has every row; the register has every paper; the folders recompute every number.

Evaluation and judges, 13 of 14. ← Evaluation and judges 12: Audit your labels where they matter: a VLM labeller that is right on average · Evaluation and judges 14: VLA-JEPA's surprise as a failure warning: a robot day with nothing to warn about →