Papers¶
One post per paper I have run against the SO-101 / Orin setup. Each post has the same shape: what the paper claims and how it measures it (protocol, n, baselines), what I ran with the gate ID, their number beside mine, what the gap means, a one-line verdict, and what would change it. The gates are in the experiment log; the long-form reading of every paper is in the private repo's journal/book/ and is being folded in here as each experiment runs.
Verdict words are the register's: reproduces (their claim holds here within the bar), fail (the pre-registered bar was not met and the confound checklist is clean), not proven (bar not met, a confound is open), not tested (no run that tests the paper's claim yet).
22 papers tested: 4 reproduce, 16 fail or are not proven; $45.10 of GPU and API for all of them, retries included. Click a column header to sort. Of the 10 papers demonstrated on rigs over $5k, tested on a ≈ €1,350 one: 0 transfer, 8 do not, 2 not proven, 0 untested or withdrawn. Their rig is the setup the tested claim was shown on; ~ is the list-price class of the named arm.
Their rig:
| # | Paper | Verdict | Their rig | Theirs | Ours | Bar (fixed before the run) | Cost | Code | Claims |
|---|---|---|---|---|---|---|---|---|---|
| 1 | LeWorldModel Train a small world model end to end from pixels, with no frozen encoder and no EMA… arXiv 2603.19312 · code | fail | sim Push-T, OGBench-Cube, Reacher, Two-Room (sim); one GPU (model not stated) (§4.1, App. D) | Push-T planning 90 % at matched FLOPs | dynamics 0.31 vs 0.89 copy-last (reproduces); planning top-1 0.60 vs 0.68–0.90 for an action-only baseline (fail) | Dynamics: beat copy-last, every corruption ≥ 1.3 at K = 30. Planning: rank the executed chunk first among 8 in ≥ 0.8 of anchors. | $6.60 | folder | 3 tested · 0 not |
| 2 | DINO-WM Plan for manipulation and navigation with a world model built on frozen DINOv2 patch… arXiv 2411.04983 · code | fail | sim PointMaze, Push-T, Wall, Reacher (sim), XArm rope / granular in NVIDIA Flex; A6000 GPU (§4.1, App. A.1, A.8) | Push-T planning success 0.90 | arm-pose R² 0.92 (vs 0.57 for W1); G1b top-1 0.55–0.61; no-model baseline 0.68–0.90 | G1b top-1 ≥ 0.8 at some target distance. | $1.70 | folder | 2 tested · 0 not |
| 3 | RoboJEPA Measure how an action-conditioned predictor on frozen V-JEPA 2.1 features scales with… arXiv 2610.10515 · code | fail | ~>$20k Franka Panda (DROID stack), CEM planning on a multi-GPU cluster; RoboCasa sim (App. G.1, §2.2) | rollout error frontier near E ≈ 0.22; latency not reported | 11.8 % below copy-last (300M; 1B 15.8 % on 30 clips); 9–10 % below shuffled; encoder 526 ms on the Orin NX | 9-step L1 ≥ 15 % below copy-last (300M) and ≥ 5 % below shuffled actions; 2-view encoder ≤ 200 ms on the Orin. | $1.20 | folder | 2 tested · 0 not |
| 4 | H-JEPA Stack JEPA world models into a hierarchy with a distinct latent space per level, so that… arXiv 2610.06805 · no code link found | fail | sim FourRoom, Visual AntMaze, Push-T, OGBench-Cube (sim); DROID video scored offline (§3, App. B) | subgoal-tracking Spearman 0.89–1.00 | level-2 ρ −0.73; false stalls 52 % | Progress monotonicity Spearman ρ ≤ −0.80; false stalls ≤ 10 %. | $0.00 | folder | 1 tested · 0 not |
| 5 | D-JEPA Close the gap between a world model's global predictive geometry and the decision among… arXiv 2609.24749 · code | not tested | ~<$5k Push-T, Reacher, Granular, RoboTwin (sim); AgileX PiPER arm with a V-JEPA 2-AC planner (§5, Table 2) | +4 to +17 points | regret 0.263 (W1 alone 0.263; action-only −0.007) | G1b regret ≤ 0.10 and ≤ the action-only baseline + 0.05. | $0.00 | folder | 1 tested · 0 not |
| 6 | ContactGuard Detect grasp failures before contact for a chunked policy, by rolling an… arXiv 2608.13438 · no code link found | fail | ~$5–20k AgileX Piper dual-arm (14-DoF), three RGB cameras; RTX 5090 (§4.1, Table 4) | AUROC 0.98–0.99 imagined vs 0.84–0.92 current over four grasp tasks (Cup 0.98 vs 0.84) | 0.70 imagined vs 0.82 current | AUROC ≥ 0.85 and ≥ 0.05 above the current latent. | $0.00 | folder | 2 tested · 0 not |
| 7 | FAIL-Detect and SAFE Detect policy failures at run time from scores built without failure data (FAIL-Detect)… arXiv 2503.08558, arXiv 2506.09937 · code · code | fail | ~>$20k bimanual Franka Panda station (FAIL-Detect); Franka Panda + WidowX (SAFE); A100 GPUs (FAIL-Detect §V; SAFE §5.1) | false alarms ≤ α on in-distribution successes (conformal coverage bound) | 22.7 % of successes flagged at α = 5 %; 46 % of failures caught | Successes flagged ≤ 1/20; failures flagged ≥ 50 %. | $0.00 | folder | 2 tested · 0 not |
| 8 | ReAP Warn of a VLA's failure before it happens, with a detector on a compact visual state… arXiv 2610.16309 · no code link found | fail | ~$5–20k LIBERO-10, RoboTwin (sim); two AgileX PIPER arms, π0.5 on a remote server, RTX 4070 detector (§5.1, App. C.4) | bTWA 0.65–0.78 in simulation (Table 1), 0.60–0.82 on the real robot (Table 2a) | v1 AUROC 0.93 came from time and length; v2 0.789 vs 0.708 shuffled, 3/8 at 5/42 | ≥ 5/8 failures caught at ≤ 1/20 false interventions; action-conditioned iff true ≥ shuffled + 0.05. | $0.10 | folder | 2 tested · 0 not |
| 9 | VLA-Corrector Keep a frozen π0.5 on task by detecting deviations with a learned verifier on observable… arXiv 2609.06508 · no code link found | fail | sim LIBERO and LIBERO-Plus (sim, Franka Panda); real robot not named (Experimental Setup) | 54-feature GRU verifier AUROC 0.90 | the paper's verifier family on our rollouts (a GRU on pooled RGB statistics and actions): 2/8 caught at 3/42 false interventions, AUROC 0.59; cosine drift of the W1 latent 0/8, AUROC 0.30 | ≥ 5/8 failures at ≤ 1/20 false interventions. | $0.00 | folder | 1 tested · 0 not |
| 10 | Robometer Train a general robot reward model on a vision-language backbone, from about a million… arXiv 2603.02115 · code | fail | ~>$20k 21 embodiments, training hardware not stated in the text; RL on a DROID Franka Panda (App. B.3, App. E.2) | VOC 0.94 on out-of-distribution evaluation sets (0.92 in-distribution) | pooled AUROC 0.818 on 2099 ArmnetBench rollouts; episode length alone 0.938 | Pooled AUROC ≥ 0.85 and within-policy ≥ 0.8 on ≥ 5 of 7 policies. | $9.55 | folder | 3 tested · 0 not |
| 11 | SmolVLA Build a small, open vision-language-action model trained on community LeRobot data that… arXiv 2506.01844 · code | reproduces | ~<$5k SO-100 / SO-101 arms; pretraining on 4 GPUs (~30k GPU-hours) (§4.1–4.3) | SO-101 LEGO 90 % in distribution; latency not reported for Jetson | 9/10 on bowls; 974 ms in PyTorch, 238 ms with vla.cpp on the Orin NX | 10-trial operator-judged gate (no numeric pass mark registered); chunk inference within the async budget (667 ms). | $0.00 | folder | 2 tested · 0 not |
| 12 | TurboVLA Make a lightweight VLA without a full VLM backbone (DINOv3 vision and a BERT text encoder… arXiv 2607.27205 · code | withdrawn | sim LIBERO and RoboTwin 2.0 (sim) on an RTX 4090; real: AgileX Piper (§4.1, Table 1) | LIBERO-Goal 97.4 → 11.6 without language (Table 3) | upstream's own trainer on our data: task swap 0.020°, per-layout 38.9 %; control, their released LIBERO checkpoint run here: 0.105 (≈ 5.5° of pan) | Task swap ≥ 5° and per-layout ≥ 75 % → our port was the bug; < 1° with the control ≥ 0.095 → the data. | $2.80 | folder | 1 tested · 0 not |
| 13 | VLA-JEPA Pretrain a VLA on human videos without actions through a latent world model, so that its… arXiv 2602.10098 · code | reproduces | sim LIBERO, SimplerEnv (sim); real: Franka Research 3; trained on 8× A100 (§4.1–4.4, App. B) | +1.1 over no-human-video on LIBERO (Table 1); no latency reported | 9/10 for ~$9.7; 355 ms per chunk on the Orin NX; reranking no-go (0.76, 1.84×) | 10-trial operator-judged gate (no numeric pass mark registered); for reranking (H0): oracle / mean error ≤ ~0.7 and N = 8 ≤ 1.5× the latency. | $9.70 | folder | 2 tested · 0 not |
| 14 | REKI Give a VLA a short memory of past frames with register tokens cached across steps, so it… arXiv 2610.22178 · code | reproduces | sim RoboTwin (sim, AgileX ALOHA); real: UR5 + WSG50; pretrained on 32× H100 (§5.2, §6.2, §7) | 18.6 % without memory, 4.1 % with | 20.3 % of all episodes, 24.0 % of failures; REKI trained on my data: 19/20 + 19/20 on the arm, oscillates in 1/40 | ≥ 10 % of failed episodes oscillate (their criterion). | $13.05 | folder | 2 tested · 0 not |
| 15 | Real-Time Chunking Let a flow-matching policy run asynchronously without jumps at chunk boundaries, by… arXiv 2506.07339 · code | not proven | unstated two 6-DoF arms with parallel grippers (arm model not named); π0.5, RTX 4090 inference; Kinetix sim (§4.2, App. A.7) | throughput flat across added latency (π0.5, real robot) | prefix overwrite cuts the jump 71–73 % against naive async at +0.4 ms; tuned LeRobot identity mode is close | Best gradient-free method cuts the median jump ≥ 50 % against naive async, drift ≤ +10 %, within 667 ms. | $0.00 | folder | 1 tested · 0 not |
| 16 | HWM (hierarchical latent world models) Plan long manipulation tasks with a two-level world model: a high level over learned… arXiv 2604.03208 · code | not proven | ~>$20k Franka Emika Panda + Robotiq; ~130 h DROID/RoboSet (§3.1 (Franka, Robotiq, data); §3.4 (Fig. 6, Table 4b)) | low level better up to 1 s, high level from 1.5 s (Fig. 6; ≈ 0.367 vs 0.385 at 2 s, read off the plot); cosine 0.88 ± 0.03 vs 0.80 ± 0.02 | no crossover: W1 0.382 ± 0.005 → 0.563 ± 0.007, high level 0.769 ± 0.009 → 0.759 ± 0.009 (0.73–2 s); cosine 0.608 ± 0.007 vs 0.551 ± 0.006 | Crossover in [1.0, 2.0] s; at 2 s high-level error ≤ 0.95 × low-level; learned macro-actions ≥ net joint displacement on cosine. | $0.40 | folder | 2 tested · 0 not |
| 17 | LeJEPA identifiability Derive the conditions under which a JEPA trained with SIGReg recovers the true latent… arXiv 2605.26379 · no code link found | reproduces | sim DMC Reacher (simulation) (§6.1–6.2 (synthetic 2D worlds, DMC Reacher)) | anisotropic recovery on trajectory data, identifiable under isotropic random walks (simulation) | W1 mean R² 0.565, anisotropy 0.639 (shoulder lift 0.91, wrist flex 0.27); frozen DINOv2-S 0.867 / 0.301 | W1's mean R² from latent to joints < 0.80; anisotropy (max − min per-joint R²) ≥ 0.20; the higher-R² model has the higher G1b top-1. | $0.00 | folder | 2 tested · 1 not |
| 18 | SURVEYOR Let a latent-world-model planner consume drafted subgoals speculatively, accepting each… paper · no code link found | fail | sim simulation only (PushT, Reacher, OGBench-Cube, Two-Room) (experiments section, Table 1) | τ = 0.20 in simulation (PushT, Reacher, OGBench-Cube, Two-Room) | τ = 1.249 on W1 latents of 69 real rollouts; as a monitor 3/8 at 6/42 (S = 30), AUROC 0.756 vs shuffled 0.560 | τ below the 10th percentile of non-equivalent pairs and ≤ 0.40; as a monitor ≥ 5/8 failures at ≤ 2/42; ≥ 0.05 AUROC over a shuffled drafter. | $0.00 | folder | 2 tested · 0 not |
| 19 | Adaptive Action Chunking (AAC) Choose each action chunk's length at inference from how much N sampled chunks disagree… arXiv 2604.04161 · no code link found | fail | ~>$20k Realman arm + Mycobot gripper; A800 (real-world experiments and Table 4 (efficiency)) | h* follows the semantic phases (Fig. 3–4); real world 67.0 → 82.0 %; N = 20 costs 106 ms vs 83 ms on an A800 ("about 20 ms inference delay"; negligible below N = 10) | h* 2 in transport and 2 at the grasp (86 % of decisions at the floor); jump 0.973 vs 0.940; N = 20 decision 4,751 ms (sequential) on the Orin | median h* in transport ≥ grasp window + 2; jump with AAC + overwrite ≤ fixed + overwrite at d = 8; an N = 20 decision ≤ 298 ms on the Orin. | $0.00 | folder | 3 tested · 0 not |
| 20 | PACT-VLA Under asynchronous execution, sample several candidate chunks and keep the one the frozen… arXiv 2610.28019 · no code link found | fail | ~$5–20k bimanual AgileX PiperX, three RealSense cameras; π0.5 on RTX 5090 / H100 / RTX Pro 6000 (§4.4, Fig. 4; Table 4 (latency)) | bimanual PiperX 82.0 % vs naive async 60.0 %; no added control-step delay on the evaluated GPUs (generation 35–49 ms, with verification 59–65 ms, both d = 2 at 30 Hz, Table 4) | offline −0.27° vs random and vs the stale residual (both CIs < 0); on the Orin 1,200 ms per cycle, 37 control ticks vs 8 | PACT beats a random pick and the stale-observation residual at d = 8 (CIs < 0); its delay on the Orin ≤ the naive delay; at that delay it still beats a single sample. | $0.00 | folder | 3 tested · 0 not |
| 21 | Horizon × depth (Euler steps vs execution horizon) Show that, for frozen flow-matching VLAs, the effect of cutting Euler steps depends on… arXiv 2610.32102 · no code link found | not proven | ~>$20k Franka Research 3 with a ZED 2i (real); π0 / π0.5 / GR00T on A100s (LIBERO, RoboTwin 2.0 sim) (§IV-C (real robot), Table I and Fig. 3 (sim)) | R(2)/R(10) 0.51, centroid shift 3.2 %; 14.3 → 2.6 ms per step (A100); "eliminating edge GPU thermal throttling" | R(2)/R(10) 0.78 (CI 0.73–0.81), centroid 0.018 and 2.35°; 196.1 vs 237.5 ms per chunk (0.83, vision 151 ms); no throttling at N = 10 on the Orin, at MAXN_SUPER or 25 W | median R(2)/R(10) ≤ 0.70 with the centroid within 0.05 relative and 3°; N = 2 latency ≤ 0.85 × N = 10 on the Orin, N = 10 within 5 % of 238 ms. | $0.00 | folder | 2 tested · 0 not |
| 22 | REMORA Track task progress through errors and recovery: a progress distribution from a… arXiv 2610.39125 · no code link found | fail | ~>$20k AgiBot G02, head + 2 wrist cameras, 30 Hz; 8 × H200 (§4.1, §5.1) | D2 (25 error episodes): peak selection vs the median readout MAE 6.04 → 3.31 (×10⁻²), Spearman 0.870 → 0.959 | premature 0.22 vs 0.25, missed equal (P1 pass); flips 0.70 × argmax, CI 0.48–0.94 (P2 fail); setback recall 2/7, false 0.52 (P3 fail); MAE 0.184 vs median 0.218 | Against per-frame argmax: premature not higher and missed ≤ +2 points (P1); flips halved, CI upper ≤ 0.7 (P2); a decrease inside ≥ 50 % of the reversal windows with ≤ 20 % false regressions (P3). | $0.00 | folder | 1 tested · 2 not |
Across all experiments: 93 gates run, 27 failed, 14 passed, 3 voided — see the experiment log.
Scheduled (12): the post follows the run
- SEVO (arXiv 2605.11114): §18 wrist-camera blanking, then top+side fine-tune
- Planning Limits of JEPA (arXiv 2609.39235): §19 VLA + WM selection; ranking test on the 69 rollouts
- PSG-JEPA + SA/IDM (arXiv 2608.06799, arXiv 2609.03565): §7 three-arm grounding
- CF-JEPA (arXiv 2610.00727): §8 participation ratio, then factorisation
- PEEK (arXiv 2509.18282): §21 mask overlay for ACT
- Mulligan (arXiv 2610.05882): §11 Sobol + failed-layouts-first
- Spotter + ReTrace (arXiv 2609.36808, arXiv 2610.15814): §4 repair vs retreat vs retry
- FARM (arXiv 2609.11445): readout on W1 / VLA-JEPA states, common-horizon truncation
- TD-JEPA (arXiv 2607.25337): §16 pre/post-grasp split
- Augmentation in post-training (arXiv 2610.05994): §20 noise floor
- Robo-COP (arXiv 2610.09228, arXiv 2610.18792): G24 adoption window; flywheel controls
- HyMeS (arXiv 2608.09410): harness stage verification (k-of-w vote)
Read, not testable here (12)
- V-JEPA 2.1: encoder too slow for the loop; used offline
- JEPA-WM design study: rules adopted in W1 training
- DeepJEPA: planner not the bottleneck
- Imagine-RL and the residual-RL pair: no torque sensing, no RL budget
- FailSafe: no simulator injection; leader-arm version noted
- STDR: RL from scratch
- FOREWARN: VLM on imagined latents; at-rest only
- VLAs-as-Tools: unit-level fine-tuning noted for G24
- RegenHarness and HexaShell: design papers; rules adopted
- No Free Checker: survey; report-card template adopted
- blind precise assembly: torque control; its Wilson-interval figure is the evaluation standard
- 2610.19375 one-step fidelity: the matched-epochs control in §7