Findings¶

29 results the tests taught, in three groups after the disagreements: the cheap baselines to try first, what runs on the robot, and the rules the method now follows. Nothing on this page is typed by hand: every number is re-read from its source file in the replication repository when the site builds.

Where the papers and this rig disagree¶

5 comparisons where a paper states one relation and the same comparison, run on this rig, gives the other. Found by the build from the claims in the register (same two terms, same metric, relations that cannot both hold); the conditions that differ between the two sides are listed, so a disagreement under different conditions is visible, not hidden.

LeWorldModel: latent world-model ranking above action-only baseline on G1b top-1
Paper, §4 (planning by ranking imagined outcomes): Push-T planning 90 %
This rig: 0.68–0.90 vs 0.60 action-only baseline above latent world-model ranking
gate · folder · post · claims C-001-2 and C-001-3
DINO-WM: latent world-model ranking above action-only baseline on G1b top-1
Paper, §3.2 and §4.3, Table 1 (planning with DINO-WM): Push-T planning success 0.90
This rig: 0.68–0.90 vs 0.60 action-only baseline above latent world-model ranking Conditions that differ: model.
gate · folder · post · claims C-002-1 and C-001-3
ContactGuard: imagined-latent probe above current-latent probe on failure AUROC
Paper, Table 2 (ROC AUC, Cup .982 vs .840): AUROC 0.98–0.99 vs 0.84–0.92 over four grasp tasks (Cup 0.98 vs 0.84)
This rig: 0.82 vs 0.70 current-latent probe above imagined-latent probe
gate · folder · post · claims C-006-1 and C-006-2
Robometer: learned reward model above episode length on failure AUROC
Paper, not a paper comparison: the paper's premise that a learned judge beats hand-made signals: no episode-length baseline in the paper
This rig: 0.938 vs 0.818 episode length above learned reward model
gate · folder · post · claims C-010-2 and C-010-3
PACT-VLA: PACT cycle equals naive async cycle on control-step delay
Paper, ("no added control-step delay" on the evaluated hardware; Table 4: generation 35–49 ms, with verification 59–65 ms, both d = 2): no added delay
This rig: 37 ticks vs 8 PACT cycle above naive async cycle
gate · folder · post · claims C-020-2 and C-020-3

Same comparison, same relation, stated by two papers: LeWorldModel and DINO-WM (latent world-model ranking above action-only baseline on G1b top-1).

Cheap baselines that matched or beat the method¶

Across 93 gates, a cheap baseline matched or beat the published (or our more elaborate) method 8 times. Each entry says what was measured, on what, and where the code is; the order within a problem is the recommendation.

I need to tell success from failure in a rollout.¶

A deterministic judge at rest adopted CPU, free
0 false positives 94.3 % on 87 judged, 5 false negatives (its own bar was 95 %); Robometer-4B zero-shot on the same rollouts: AUROC 0.577
Wait until the arm is still (or a timeout), then judge from the top camera with fixed colour and geometry rules. Pick it for anything that learns from rollouts: a false positive puts a failure into your demonstrations.
gate · folder · post · checked against the source file at build: 2026-10-23-paper-vs-hardware-robometer/expected/g9c3_test_results.json
Episode length and policy identity, as the bar a learned judge must clear works here free, one column of the log
0.938 Robometer-4B pooled 0.818 on ArmnetBench (2,099 rollouts). On our 88: length 0.521, policy identity 0.878, Robometer 0.577
Score every rollout by its length, and by which policy ran it, before trying a reward model. Where failures run to the timeout, length alone is hard to beat; where they don't, it is near chance, and that tells you too.
gate · folder · post · checked against the source file at build: 2026-10-23-paper-vs-hardware-robometer/expected/g9a_ab_complete.json
A general VLM given the right input, never alone works here GPU dollars (~$4.3 for the comparison)
7/8 at 2/42 a general Qwen3.5-27B with the same input passed the same bar as the robot-trained Flora-Rewarder-27B (8/8 at 2/42); Gemini 3.8 Flash reached AUROC 0.964 but failed the flag rule
If you add a model judge, fix the input first: 64 frames, top and wrist side by side. Pair it with the deterministic check and give it a report card (false positives, false negatives, ambiguous) against your labels before it decides anything.
gate · folder · post · checked against the source file at build: 2026-10-20-flora-rewarder-input-format/expected/g_flora_abl_summary.json

Scope: Our SO-101, the two-task bowls rollouts (88 labelled, 26 failures); ArmnetBench's 2,099 rollouts (8 tasks, 7 policies) for the length baseline; the general-model comparison on 50 episodes (8 failures).

I need to rank candidate action chunks.¶

Rank by the commanded joints alone, with no model works here CPU, free
0.684–0.905 top-1 from the action-only baseline (horizon 5–30 steps) vs the world model W1's best 0.6046 (bar 0.8)
Before any latent planner, rank candidates by how close their commanded joints are to the target. If a world model can't beat that, its latents aren't what the ranking needs.
gate · folder · post · checked against the source file at build: 2026-10-23-paper-vs-hardware-leworldmodel/expected/g1_investigation.json

Scope: Our SO-101 bowls data, 25 validation episodes, 703 anchors, 8 candidates per anchor (chance 0.125).

I need a probe on a world model's latents.¶

Probe the current latent before the imagined one works here CPU, free
0.820 AUROC on the current latent vs 0.701 on the imagined one (ContactGuard's recipe); neither reached the 0.85 bar
Fit the same probe on the latent you already have at the anchor before imagining forward. An imagined-latent probe has to beat that by a margin to be worth its rollout.
gate · folder · post · checked against the source file at build: 2026-10-23-paper-vs-hardware-contactguard/expected/w1c_summary.json

Scope: W1 on our SO-101 rollouts: 65 scored episodes (44 successes, 21 failures), one grasp anchor each, 5 seeds × 5 folds.

I need an online failure monitor.¶

Judge at rest first; know what failed mid-episode here failed here CPU free; ~$2–3 of API for the VLM monitors
2/8 at 2/42 the best of the six (W1 L2 surprise, AUROC 0.86) against a bar of 5/8 at ≤ 1/20
Six mid-episode monitors, three on W1's latents and three VLM-based, missed the same bar in one day. Start from the judge at rest above and add a monitor only if it clears this bar on your data.
gate · folder · post · checked against the source file at build: 2026-10-19-six-monitors-w1/README.md
Run the shuffled-action and zero-action controls failed here CPU, free
0.708 the shuffled-action control vs 0.789 for the ReAP-style risk head with the true actions: the margin met the action-conditioning rule, but most of the signal did not need the actions, and no head reached the bar
Score the same monitor with the actions shuffled across episodes, and with them zeroed. Whatever the control reaches did not come from the action conditioning.
gate · folder · post · checked against the source file at build: 2026-10-23-paper-vs-hardware-reap/expected/monitors_v2.json

Scope: Our SO-101 bowls rollouts: 50 episodes (42 successes, 8 failures), bar ≥ 5/8 failures at ≤ 1/20 false interventions.

I need a long-horizon latent predictor.¶

Copy the current latent, and predict the average one, first works here CPU, free
0.759 vs 0.806 a hierarchical high level (HWM recipe) at 2 s vs predicting the train-mean latent; W1's own rollout had lower error than copying at every horizon (0.382–0.563 vs 0.613–0.954, 0.73–2 s)
Put both trivial predictors in the table before the model. A useful predictor beats copying at short horizons and the average at long ones; one that barely beats the average has learned little.
gate · folder · post · checked against the source file at build: 2026-10-28-paper-vs-hardware-hwm/expected/posthoc_baselines.json

Scope: W1 latents of our SO-101 bowls data: 157 training episodes (~0.6 h, one task), 700 validation anchors from 25 episodes; the two baselines were added after the result (post-hoc).

I need lower latency on a Jetson.¶

vla.cpp with flash attention adopted free
974 → 238 ms per chunk, PyTorch vs vla.cpp with flash attention
Serve the same checkpoint through vla.cpp with --flash-attn --mm-prec default before optimising anything else. The details, and what didn't fit on the device, are on the systems page.
gate · folder · post · checked against the source file at build: 2026-10-23-paper-vs-hardware-smolvla/expected/vlacpp_trials.csv

Scope: SmolVLA on a Jetson Orin NX 16 GB at MAXN_SUPER, one checkpoint, 10 trials per engine; the numbers and the rest are in the systems group of this page.

I need smoother chunk boundaries.¶

LeRobot's identity RTC mode, tuned, as the baseline to beat works here GPU minutes on the Orin, free
0.95 / 1.02 / 1.12 median jump ratio of identity RTC at β = 10 vs the prefix overwrite's 0.93 / 0.96 / 1.06 (naive async 3.42 / 3.69 / 3.77)
Before claiming a win at chunk boundaries, tune LeRobot's own guided mode (identity, β) on the same boundaries. The prefix overwrite's real edge is no β to tune, an exact prefix and no gradient, not a much smaller jump.
gate · folder · post · checked against the source file at build: 2026-10-25-vla-chunk-boundary-jumps/expected/g13b_summary.json

Scope: SmolVLA on recorded SO-101 episodes, offline: 20 episodes, 254 boundaries per delay (d = 8 / 12 / 20); task success with either method on the arm is not tested.

I need to encode an action chunk.¶

Concatenate the raw actions before learning an encoder works here GPU dollars (part of ~$0.4)
0.602 vs 0.608 cosine alignment, concatenated actions vs the learned macro-action encoder (± 0.006 / 0.007); net joint displacement 0.551
A linear map of the zero-padded primitive actions did as well as a learned transformer encoder here; both beat a handcrafted net-displacement summary. Report SEs and say when two are indistinguishable.
gate · folder · post · checked against the source file at build: 2026-10-28-paper-vs-hardware-hwm/expected/gate0.json

Scope: HWM's high level on W1 latents of our SO-101 bowls data, 700 validation anchors from 25 episodes, one training seed; SEs over anchors (correlated within an episode).

Not tested yet¶

Problems the register has no baseline for yet, so this page says nothing about them:

Choosing a policy for a new task: no baseline yet (every policy here is at 8–10/10 on bowls, a ceiling).
Weighting or filtering training data (curation scores against random or length): planned, no run.
Task success with smoother chunk boundaries on the arm: measured offline only.
Adaptive chunk length chosen by sample entropy (AAC): running offline now, no result yet.

What runs on the robot, and what did not¶

Every number here was measured on a ≈ €1,350 rig. The block that matters most is the second one: what did not fit.

8 results from the experiment log, each linked to its gate row, its replication folder and its post; the pull-request states are read from GitHub.

Shipped on the robot¶

238 ms

SmolVLA chunk inference on the Orin shipped Orin NX, MAXN_SUPER

974 ms (PyTorch) → 238 ms (vla.cpp, flash attention)

Same checkpoint, same 10-trial gate: PyTorch 974 ms → vla.cpp 390 ms → vla.cpp with flash attention 238 ms per chunk (median p99 over the trials).

post · folder · gate · checked against the source file at build: 2026-10-23-paper-vs-hardware-smolvla/expected/vlacpp_trials.csv

0 false positives

Deterministic top-camera judge at rest shipped CPU

Judges a rollout from the top camera once the arm is still: 94.3 % accuracy on 87 judged episodes, 0 false positives, 5 false negatives.

post · folder · gate · checked against the source file at build: 2026-10-23-paper-vs-hardware-robometer/expected/g9c3_test_results.json

Rejected on device¶

does not build

REKI 2.4B on the Orin rejected on device rented GPU

REKI needs flash-attn, FLA / Triton and causal-conv1d, which build for x86 CUDA only, so it ran from a rented RTX 3090 (19/20 on each task). The earlier 318 ms Orin figure (G12) was void: it timed a stand-in model.

post · folder · gate · checked against the source file at build: 2026-10-23-paper-vs-hardware-reki/README.md

526 ms

RoboJEPA's encoder on the Orin rejected on device Orin NX, MAXN_SUPER

The two-view V-JEPA 2.1-G encoder takes 526 ms per step on the Orin NX against a 200 ms bar, before the predictor runs. Usable off-board only.

post · folder · gate · checked against the source file at build: 2026-10-23-paper-vs-hardware-robojepa/expected/gateB_orin.json

238 → 304 ms

W1 as an online monitor next to the policy rejected on device Orin NX (power mode not recorded)

238 ms policy chunk alone → 304 ms with the monitor on the same GPU

W1's check is cheap on its own (17.6 ms per tick, 25-step imagination 153 ms), but sharing the GPU slows SmolVLA's chunk from 238 to 304 ms. W1 stays in shadow mode.

post · folder · gate · checked against the source file at build: 2026-10-23-paper-vs-hardware-reap/expected/latency.json

Upstreamed¶

5

LeRobot pull requests merged upstreamed

5 merged; 26 others reviewed or commented on. Counted from the GitHub search API at build time.

merged PRs

+0.4 ms

Prefix conditioning in vla.cpp upstreamed Orin NX, MAXN_SUPER

Optional prefix conditioning for SmolVLA (prev_actions, inference_delay): the chunk-boundary fix below, inside the engine, at +0.4 ms per chunk.

post · folder · gate · VinRobotics/vla.cpp #60 open · checked against the source file at build: 2026-10-25-vla-chunk-boundary-jumps/expected/vlacpp_overwrite_summary.json

ACT port

ACT in vla.cpp upstreamed Orin NX, MAXN_SUPER

LeRobot's ACT added to vla.cpp. The vla.cpp ACT path has not been through the 10-trial gate on the arm; the PyTorch checkpoint scored 9/10.

post · folder · gate · VinRobotics/vla.cpp #38 merged · checked against the source file at build: 2026-10-12-act-vlacpp-port/README.md

Measured¶

Result	Number	Device	Gate	Code
Prefix overwrite at chunk boundaries (offline): Median jump at chunk boundaries cut 71.0 / 73.3 / 71.5 % against naive async (d = 8 / 12 / 20, 254 boundaries), prefix error 0.0. Offline on recorded episodes: task success with it is not tested and the robot client does not send prev_actions yet.	−71 to −73 %	Orin NX, MAXN_SUPER	gate	folder · post

Findings from the method itself¶

Results that are not a baseline and not a systems number, but a rule the register now follows. Each is re-read from its source file at build, like the two groups above.

A deterministic judge at rest, never a false positive method CPU, free
0 false positives 94.3 % on 87 judged, 5 false negatives
Judge a rollout only once the arm is still, from the top camera, with fixed rules; tune for zero false positives, because a false positive puts a failure into the training data and a false negative only costs one demonstration.
gate · folder · post · checked against the source file at build: 2026-10-23-paper-vs-hardware-robometer/expected/g9c3_test_results.json
The input format did the work, not the robot fine-tune method GPU and API dollars (~$4.95)
7/8 at 2/42 Flora-Rewarder-27B 8/8 at 2/42 on the same 50 episodes
Given Flora-Rewarder's input (64 frames, top and wrist side by side), a general Qwen3.5-27B passes the same flag bar as the robot-trained Flora-Rewarder-27B. Fix the judge's input before choosing the model.
gate · folder · post · checked against the source file at build: 2026-10-20-flora-rewarder-input-format/expected/g_flora_abl_summary.json
Mid-episode monitors miss; verify at rest instead method Orin NX; Gemini API ~$2.1–3.1
2/8 at 2/42 the best of six monitors (W1 L2 surprise); bar ≥ 5/8 at ≤ 5 % false interventions
Six latent and VLM monitors were scored against one bar on the same rollouts in one day and all missed it; the best, W1's L2 surprise, caught 2 of 8 failures. Spend the effort on the check at rest.
gate · folder · post · checked against the source file at build: 2026-10-19-six-monitors-w1/expected/w1a_summary.json
Latent distance to the goal is not progress method Mac, free
ρ −0.73 bar −0.80; level-1 W1 −0.72, level-2 H-JEPA −0.73
Distance from the current latent to a goal latent does not fall monotonically as the task advances, at either level of the hierarchy; it cannot be used as a progress or stall signal.
gate · folder · post · checked against the source file at build: 2026-10-14-g6-progress-latents/expected/g6b_results.json
Two gates voided by a two-line check method Orin NX, free
2 gates void W1's encoder: 192 missing and 192 unexpected keys under strict load
G11-A timed an encoder that loaded with random weights and G12 timed a stand-in model; a strict load and a check that the parts sum to the whole would have caught both, and now open every protocol.
gate · folder · post · checked against the source file at build: 2026-10-22-voided-gates/expected/w1_strict_load.txt
An upstream correctness fix made a checkpoint 47 % worse method Orin NX, free
R = 1.473 loss ratio after / before the fix; worse on 25 of 25 episodes
LeRobot's rotary-embedding fix is right, and it raises a legacy VLA-JEPA checkpoint's world-model loss on every episode. Pin the version a checkpoint was trained with, and re-check after upgrades.
no gate row: measured as a compatibility check, not a gate · folder · post · checked against the source file at build: 2026-10-26-vla-jepa-rope-compat/expected/results.json
p99 is not a tail for chunked policies method Orin NX and arm, free
974 ms SmolVLA's p99 on the Orin NX = one chunk inference (median over 10 trials)
For a policy that predicts chunks, the slow calls are the chunk inferences themselves, on a fixed schedule; p99 equals one chunk inference. Report time per chunk inference and how often it runs.
gate · folder · post · checked against the source file at build: 2026-09-29-three-policies-so101/expected/trials_orin_nx.csv
Audit your labels where they matter method API cents; Mac
+6 frames median onset offset; +110 and +289 frames on the missed-grasp retries
A VLM failure-onset labeller is right on average and wrong exactly on the retries, where the label matters; spot-check the boundary cases before training on its labels.
gate · folder · post · checked against the source file at build: 2026-10-23-paper-vs-hardware-reap/expected/onsets_README.md
The $200 arm executes what it is told method Orin NX (CPU), free
18.9 % gripper gap, failures / successes 0.77 (CI 0.64–1.06): not seen; settle band ≥ 0.3° on 1 of 5 joints: not sub-servo
Measured from every recording's command and follower streams: the SO-101 misses its commanded joint targets by 18.9 % of the commanded motion, about half the 32.9–35.0 % reported for AgileX Piper arms, and failed rollouts do not track worse than successful ones. A policy's failure on this arm is a choice, not an execution error.
gate · folder · post · checked against the source file at build: 2026-11-01-execution-gap/expected/exec_gap.json
A one-timestamp indexing bug, found by a consistency check method Orin NX, free
0.624 → 0.565 W1's mean R² (latent → joints), red channel only → all three channels
Indexing a single-timestamp dataset item as if it had a time axis fed both encoders the red channel only. A consistency check on a later run found it; both affected extractions were redone and LeJEPA's verdict did not change.
gate · folder · post · checked against the source file at build: 2026-10-29-paper-vs-hardware-lejepa/expected/gate0.json

Scope, per entry: A deterministic judge at rest, never a false positive: Our SO-101, 88 bowls rollouts (26 failures), one top camera, judged at rest. · The input format did the work, not the robot fine-tune: 50 bowls episodes (8 failures, 42 successes); flag bar ≥ 6/8 failures at ≤ 2/42 successes. · Mid-episode monitors miss; verify at rest instead: 50 rollouts (8 failures, 42 successes) on our SO-101; W1, a 192-d LeWM trained on our teleop data. · Latent distance to the goal is not progress: Held-out teleop episodes of the bowls task, W1 and a two-level H-JEPA on its latents. · Two gates voided by a two-line check: Two Orin gates (G11-A, G12) of 2026-10-18; both re-checked and voided. · An upstream correctness fix made a checkpoint 47 % worse: One legacy VLA-JEPA checkpoint, 200 windows from 25 bowls episodes. · p99 is not a tail for chunked policies: ACT and SmolVLA on our SO-101, 10 trials per row, Orin NX and M1 Mac. · Audit your labels where they matter: A 5-episode spot check of Gemini onsets against a second labeller, on the 88 bowls rollouts. · The $200 arm executes what it is told: 69 SO-101 rollouts (44 success / 25 failure, 9 policies) and 182 teleop demos; the residual is our reading of arXiv 2609.37334's ‖η‖/‖a‖ in joint degrees, so the comparison with the Piper is indicative. · A one-timestamp indexing bug, found by a consistency check: Two extractions (HWM gate 0, LeJEPA gate 0) on 8,447 validation frames; the superseded file is kept beside the corrected one.

Built 2026-11-01 from data/findings.yaml.