Experiment log¶
Every gate, in date order. Bar is the pass rule written down before the run; Result is what was measured; Verdict applies the bar and nothing else. A row is never rewritten to fit a later result; corrections are added as new rows (see G9a-AB, G11-A, G12). Costs are vast.ai invoices where they exist.
Method
Pre-register the rule, compute the cheap baseline first (episode length, policy identity, commanded joints), run the experiment only if one of its outcomes changes the next decision, and label anything decided after the number as exploratory. This is what kept the negative results below trustworthy.
Policies on the arm¶
Task: "put the blue bowl in the pink bowl" (both directions), 10 trials, operator-judged, 85 trials recorded with video.
| Date | Policy / engine | Machine | Success | Chunk inference | Verdict | Code |
|---|---|---|---|---|---|---|
| 09-29 | ACT | M1 Mac | 10/10 | – | works | folder |
| 09-29 | SmolVLA (PyTorch) | M1 Mac | 9/10 | – | works | folder |
| 09-29 | ACT | Orin NX | 9/10 | 59 ms | works | folder |
| 09-29 | SmolVLA (PyTorch) | Orin NX | 9/10 | 974 ms | works, slow | folder |
| 10-01 | SmolVLA via vla.cpp | Orin NX | 9/10 | 390 ms | works | folder |
| 10-01 | SmolVLA via vla.cpp, flash attention | Orin NX | 8/10 | 238 ms | works | folder |
| 09-29 → 10-08 | TurboVLA, several variants | M1 / Orin NX | 0/3 last variant | – | closed: ignores the instruction ("copycat"); task-distinguishing action was ~6 % of frames | folder |
| 10-08 | VLA-JEPA (Qwen3-VL-2B + DiT-B, world model off at inference), fine-tuned ~$9.7 | Orin NX | 9/10 | 355 ms | works | folder |
| 10-27 | REKI 2.4B trained on the bowls data (WS-026), served from a rented RTX 3090 | RTX 3090 (off-board) | 19/20 + 19/20 (20 trials per instruction) | – | works; oscillates in 1/40 | folder |
Reading: ceiling. Bowls cannot separate the policies; the next comparison runs on the hardest primitive (flip, inside grasp, instruction switch).
World model¶
LeWM = stable-worldmodel JEPA, trained from pixels. W1: 192-d pooled latent, multi-view (ContactGuard recipe). W2: frozen DINOv2-S patches + learned dynamics.
| Date | ID | Question | Bar | Result | Verdict | Cost | Code |
|---|---|---|---|---|---|---|---|
| 10-08 | H0 | Do VLA-JEPA's 8 samples vary enough that picking one helps? | oracle/mean error ≤ 0.7; latency N=8 ≤ 1.5×; baseline < 85 % | 0.76 (null 0.92); 1.84×; baseline 90 %; averaging the 8 chunks gets 45 % of the oracle gain with no model | no-go | free | folder |
| 10-08 | LeWM env | Does LeWM train on our data? | losses fall, no collapse | pass on vast.ai and on the Jetson GPU; 11 setup problems fixed | pass | $0.28 | folder |
| 10-08 | H3 plan | Does predicted goal distance on logged rollouts predict success? | AUROC 1 s before failure ≥ 0.75, beats gripper aperture by 0.10 | never run | superseded by C1 (10-11) | – | – |
| 10-09 | MV 5-epoch | Multi-view LeWM (ContactGuard recipe), short test | none (diagnostic) | beats copy-last, but an action shuffle changes the error only ×1.04; effective rank 5.8/192 | weak (too short) | small | folder |
| 10-10 | Action corruption | Does the 5-epoch model react to wrong actions and to timing? | none (diagnostic) | reacts to gross errors (episode swap, frozen action), not to ±0.5 s timing shifts | weak at 5 epochs | free | folder |
| 10-11 | G0 | Faithful 100-epoch multi-view LeWM | beats copy-last; every corruption ratio ≥ 1.3 at K=30; no collapse | K=30 error 0.31 vs copy-last 0.89; timing shifts 1.99 / 1.58; effective rank 24–30; grasp-onset AUROC 0.94 | pass | $3.30 (+$2.96 failed runs) | folder |
| 10-11 | fault F-02C1 | Does the imagined latent predict failures better than simpler signals? (81 rollouts) | imagined beats current latent | policy identity alone 0.76 beats every probe (current 0.68, imagined 0.72) | not proven (confounded: TurboVLA made most failures) | free | folder |
| 10-12 | G1b W1 | Plannable range: rank the expert chunk first among 8 | top-1 ≥ 0.8 at some distance | best 0.60 (d = 20–30) | fail | free | folder |
| 10-12 | G1 probes | Does the latent encode the arm? | pose R² > 0.57 | W1 0.57; W2 0.92; gripper-open AUROC 0.999 | W2 pass | $0.06 | folder |
| 10-12 | G2 | One decision (K=10 rollouts, H=30) within a chunk? | ready before the next chunk | W1 108 ms; W2 1 814 ms (Mac MPS) | W2 too slow | free | folder |
| 10-12 | G1 W2 | W2 on G1b; decides whether W3 (bigger) runs | ≥ 0.8 at any d | 0.55–0.61; open-loop worse than W1; reacts less to wrong actions | fail, W3 not run | < $1 | folder |
| 10-12/13 | G1 diagnosis | Why does selection fail? | regret ≤ 0.2 | top-1 ≈ pairwise^7; action-only baseline (no model) top-1 0.68–0.90, regret ≈ 0; W2 regret 0.18, W1 0.26; no conditioning bug | G1b measured kinematics, not world-model value; regret becomes the metric | free | folder |
| 10-12 | BotFails | Action-conditioned surprise as an anomaly detector on public SO-100 data (vs FIDeL 70.5) | beat FIDeL and the baselines | data, split, scorer ready; no model trained or scored | parked until G1 picks a winner | ~$0 | – |
| 10-13 | RoboJEPA Gate B | Can the V-JEPA 2.1-G encoder run on the Orin NX? | 2-view encoder p50 ≤ 200 ms | 526 ms (1 view 398 ms); predictor step 252–487 ms | no-go on device; the world model goes off-board | free | folder |
| 10-13 | RoboJEPA Gate B' | Same on the Mac | same | encoder 1 025 ms (MPS); predictor attention fails on MPS | no-go; the off-board host must be CUDA | free | folder |
| 10-13 | RoboJEPA Gate A | Does pretrained RoboJEPA predict our latents and use the actions? | 9-step L1 ≥ 15 % below copy-last; ≥ 5 % below shuffled actions | 300M 11.8 %, 1B 15.8 % on 30 clips; shuffled-action gap 8.9–10.4 % | no-go for the primary; action conditioning works on our arm | $1.2 | folder |
| 10-14 | G1b-RJ | RoboJEPA 1B on G1b | – | cancelled: neither outcome changes the build | – | – | folder |
| 10-14 | G6a | Done / stall from latent distance, zero labels | monotonicity ρ ≤ −0.8; done AUROC ≥ 0.95 | ρ −0.32 to −0.50; done 0.97–1.00 (easy contrast); 32 % false stalls | stall by distance dropped | free | folder |
| 10-17 | G10 | Outcome readout at the judge's time, and anticipation 30 steps earlier | step 1 AUROC ≥ 0.90; step 2 beats policy identity by +0.10 | step 1: W1 1.000, W2 0.976; step 2: +0.179; surprise spikes 1.82× on misses, 2.58× on stalls | pass: pre-execution veto and surprise monitor | free | folder |
| 10-18 | G6b | H-JEPA level-2 slow-feature progress (stride 15, SIGReg) | ρ ≤ −0.80; false stalls ≤ 10 % | ρ −0.73; false stalls 52 % | fail; hierarchy dropped | free | folder |
| 10-18 | G11-A | Live dual-camera surprise check latency on the Orin NX | < 20 ms inside a 33 ms tick | 12.78 ms mean, 12.95 max | pass (timing) | free | folder |
| 10-18 | fault F-03G11-A re-check | Did it run the trained W1? | – | no: strict=False + key remap under a different transformers version left the encoder random; timing stands, latents void |
void for the latents (correction row; timing stands) | free | folder |
| 10-18 | W1 noise floor | W1 error on a still scene; strict checkpoint load | – (measured, not a gate) | loads strictly only with transformers 5.19.0 (the saving version); noise floor not run (no cameras attached); found G11-A's non-strict load | measured (load only) | free | folder |
| 10-18 | W1 timing next to vla.cpp | W1 ms per check while vla-server runs on the same GPU | – (measured) | not run: no cameras attached | not run | free | – |
| 10-19 | fault F-05W1-A | W1 as mid-rollout screener on 50 fresh episodes (L2 surprise, cosine drift, live reflex) | one signal ≥ 5/8 failures at ≤ 1/20 false interventions | L2 2/8 at 2/42 (AUROC 0.86); cosine drift 0/8; live reflex flags 42/42 successes | fail → shadow mode | free | folder |
| 10-19 | fault F-05G11-B-cal | Refit the reflex thresholds from episode maxima (69 rollouts) | recall ≥ 50 % on 25 failures | 3/25 at 1/44 false alarms; success and failure maxima overlap | fail → shadow mode | free | folder |
| 10-19 | W1-C1 | Event-anchored grasp check from the imagined latent (5-fold × 5 seeds) | AUROC ≥ 0.85 and ≥ 0.05 above the current latent | imagined 0.70, current 0.82 | fail | free | folder |
| 10-19 | W1-C2 | Time-varying conformal band (FAIL-Detect / SAFE) on W1 surprise | successes flagged ≤ 1/20; failures ≥ 50 % | successes 22.7 %, failures 46 %; band not conformal across time when μ, σ are fit on the same successes | fail | free | folder |
| 10-19 | W1-B | Latent stage tracker: distance to grasp / place prototypes + gripper opening (G15's split and scoring) | premature ≤ 2 %, missed ≤ 10 % | premature 22.1 %, missed 31.1 %; the early grasp fires during the approach | fail (better than G15's 48.5 / 46.7) | free | folder |
| 10-22 | W1-R | D-JEPA's relational ranking operator on the frozen W1: does it re-rank G1b's candidates? | mean regret ≤ 0.10 and ≤ action-only + 0.05 | W1-R 0.2631 vs W1 0.2630; action-only −0.0075 | fail; does not test D-JEPA's claim (no executed outcomes) | free | folder |
| 10-22 | REAP step 4 | Monitor latency on the Orin, alone and with vla-server serving SmolVLA | – (measured) | W1 check 17.6 / 19.1 ms; 25-step imagination 153 / 158 ms; risk head 17.9 / 23.9 ms; SmolVLA chunk 238 → 304 ms with the monitor running | measured | free | folder |
| 10-23 | fault F-08Failure onsets | Gemini marks the onset frame of the 25 failed published rollouts (deviation: the owner normally labels) | – (labels; 5-episode spot check) | 25/25 labelled; spot check median +6 frames, but +110 and +289 on two missed grasps | labels in use, bias disclosed | API, cents | folder |
| 10-23 | REAP monitors | W1 L2 surprise vs a ReAP-style risk head vs elapsed time, 5-fold × 5 seeds, 50 episodes (+ all 69) | ≥ 5/8 failures at ≤ 1/20 false interventions | surprise 2/8 at 2/42 (AUROC 0.86); risk head 6/8 at 7/42 (0.93); MLP 7/8 at 13/42; elapsed 0/8 | not proven pending the onset check; superseded by REAP v2 | free | folder |
| 10-23 | fault F-07REAP v2 | Ramp-target risk head (no onsets, no time feature), common horizon, true / shuffled / zero action controls, GRU floor | ≥ 5/8 at ≤ 1/20 false interventions; action-conditioned iff true AUROC ≥ shuffled + 0.05 | logistic 3/8 at 5/42, AUROC 0.789 (shuffled 0.708); MLP 6/8 at 13/42; GRU floor 2/8 at 3/42 | fail (no monitor at the bar); both heads action-conditioned | free | folder |
| 10-23 | W2′ | W2 re-trained (weights lost), same recipe, new seed | reproduce W2's recorded G1 values | arm-pose R² 0.9193 (= W2), gripper AUROC 0.9994, open loop / copy 0.633 (W2 0.623) | reproduced; replaces W2 | ~$0.65 | folder |
| 10-26 | §30 inverse dynamics | Do W1 latents carry the action beyond the arm's state? Probes on W1, random-init W1 and W2′ | latents beat the state probes by ≥ 0.10 (CI > 0), add ≥ 0.02 over joints, random-init gain < half of W1's | +0.748 over the state probes, but only +0.007 [0.003, 0.011] over joints; random-init gain 1.254 vs W1 1.180 | not proven (the random-init control blocks) | free | – |
| 10-28 | HWM gate 0 | Does a high-level world model over learned macro-actions (HWM, CoRL 2026) beat W1's rollout beyond ~1 s on SO-101 data? Trained on W1's frozen latents, 157 / 25 episodes | crossover in [1.0, 2.0] s; at 2 s high ≤ 0.95 × low; learned macro-actions ≥ net displacement on cosine | W1 0.382 → 0.563, high level 0.769 → 0.759 (0.73–2 s); cosine learned 0.608, concatenated 0.602, displacement 0.551. First run fed red-only frames: not scored, redone | H1 fail, H2 fail, H3 pass; verdict not proven: ~0.6 h of one task vs the paper's ~130 h | ~$0.4 | folder |
| 10-27 | LeJEPA identifiability gate 0 | Does W1 (SIGReg LeWM, teleop data) recover the logged joints only partially and anisotropically (arXiv 2605.26379)? Ridge latent → 6 joints, 8,447 validation frames | W1 mean R² < 0.80; anisotropy ≥ 0.20; higher R² ↔ higher G1b top-1 | W1 R² 0.565, anisotropy 0.639; DINOv2-S 0.867 / 0.301 (corrected extraction) | pass (reproduces) | $0 | folder |
| 10-27 | SURVEYOR gate 0 | Does the accept threshold τ separate equivalent from non-equivalent states on real W1 latents? 69 rollouts | τ < p10 of non-equivalent pairs and τ ≤ 0.40 | τ = 1.249 (p10 1.106) | fail | $0 | folder |
| 10-27 | SURVEYOR gate 1 | The accept rule as a failure monitor, 50 rollouts (42 / 8), S = 10 and 30; shuffled-drafter control | ≥ 5/8 at ≤ 2/42; AUROC ≥ shuffled + 0.05 | S = 30: 3/8 at 6/42, AUROC 0.756 vs 0.560 | fail | $0 | folder |
Judges and rewards¶
| Date | ID | Question | Bar | Result | Verdict | Cost | Code |
|---|---|---|---|---|---|---|---|
| 10-15/16 | G9a-AB | Robometer-4B zero-shot on ArmnetBench (2 099 rollouts, 7 policies) | pooled AUROC ≥ 0.85; within-policy ≥ 0.8 on ≥ 5/7 | partial set 0.861 (462 episodes lost to OOM) | qualified pass, superseded | $5 | folder |
| 10-16 | G9a-AB complete | same, all episodes re-scored | same | pooled 0.818; episode length alone 0.938 | fail: not a zero-shot judge | $0.78 | folder |
| 10-16 | G9a-own | Robometer-4B zero-shot on own rollouts (88) | AUROC ≥ 0.9 | 0.58–0.62; policy identity 0.88 | fail | $0.25 | folder |
| 10-16 | G9c | Deterministic top-camera judge (HSV + geometry) | accuracy ≥ 95 %, AUROC ≥ 0.95 | 86.4 % / 0.919; 0 false positives; all 12 errors are recordings ending after the operator reset | pre-registered fail | free | folder |
| 10-16 | G9c-2 | + automatic end rule (arm at rest) | same | 29/88 invalid: failing policies rarely return to rest | fail | free | folder |
| 10-16 | G9c-3 | + stillness-or-timeout end rule | same | 94.3 %, 0 false positives | narrow fail / adopted | free | folder |
| 10-16 | G9b | Robometer-4B + LoRA on ArmnetBench held-out-task folds (written 10-22) | median held-out AUROC ≥ 0.85, beats episode length on ≥ 3/4 tasks | no LoRA trained (the pilot stopped at preprocessing); zero-shot baseline pooled 0.736 vs episode length 0.923 | not run: no verdict | ~$2.3–3.3 | folder |
| 10-19 | Judge τ | Fit Gemini Flash sliding-window FAIL threshold (50 episodes, 319 windows) | ≤ 1/20 successes interrupted | τ 0.98, 1/42 false; caught 0/8; ~10 s per call | not usable | ~$1–2 | folder |
| 10-19 | G16-B | Two checkpoint VLM calls instead (after grasp, at rest) | ≤ 2/42 successes flagged, ≥ 6/8 failures | 41/42 successes flagged (nested bowls, pre-lift checkpoint) | fail → VLM judge dropped | $0.26 | folder |
| 10-20 | G-flora | Flora-Rewarder-27B, 64-frame side-by-side clip, terminal prompt | at the largest k with ≤ 2/42 successes flagged, ≥ 6/8 failures | 8/8, 2/42, AUROC 1.00, deterministic 5/5, 6 s/episode | pass (non-commercial licence; used as evidence only) | $0.7 | folder |
| 10-20 | fault F-12G-flora-abl | Same input on general models: is it the fine-tune or the input? | same, per model | Qwen3.5-27B 7/8, 2/42 (0.91) pass; Qwen3.5-9B 16/42 fail; Gemini Flash 8/8, 3/42 (0.96) narrow fail | 27B pass, 9B fail, Gemini Flash narrow fail: the input format did the work; size matters | $4.3 | folder |
| 10-23 | Verifier cards | False positives, false negatives, Wilson CIs and latency for G9c-3 and Flora, from existing data | – (template fixed before filling) | G9c-3: FP 0/25, FN 5/62, ~1.2 s per episode; Flora: FP 0/8, FN 2/42, 6 s per episode | cards filled | $0 | – |
| 10-29 | PRIME | Does a PRIME experience graph (90 teleop episodes, DINOv2-S visual gate, TCP + gripper weights, Dijkstra distance to success) catch policy failures on the REAP bar, and does the visual gate add anything over the same graph by configuration alone? | ≥ 5/8 at ≤ 2/42 AND ≥ 1 more than the kinematic-only graph (PRIME_PROTOCOL.md, 2026-10-23); held-out teleop monotonicity ≥ 0.8 required first |
parity: no reference code, 3-episode hand example passes; nodes 10 564 (= expected); ε_xy 7–8 mm, ε_z 2.8 mm; held-out teleop Spearman 0.51; off-graph 59 % of success queries | void (the graph check failed; no rollout number read, E1–E4 not run) | free (Orin) | repo 2026-10-23_prime/README.md |
Execution and engines¶
| Date | ID | Question | Bar | Result | Verdict | Code |
|---|---|---|---|---|---|---|
| 10-18 | G12 | REKI 2.4B on the Orin NX | ≤ 8 GB, ≤ 250 ms | 3.46 GB; 318 ms | fail (voided by the re-check below) | folder |
| 10-18 | G14 | Do our policies oscillate between poses? (REKI's metric, 69 rollouts) | ≥ 10 % of failed episodes | 24 % of failed, 18 % of succeeded; real re-grasp loops on inspection | pass: register tokens stay in the plan | folder |
| 10-18 | fault F-04G12 re-check | Was G12 a measurement of REKI? | – | no: the bench timed a random-init text-only stand-in plus a hand-written expert; its parts do not sum | void: REKI on the Orin is unmeasured | folder |
| 10-18 | G13a timing | Sync vs async blocked time with the real vla-server and cameras | – (measured; the bars need the arm) | not run: no cameras attached | not run | – |
| 10-19 | G13b | RTC guidance vs gradient-free prefix conditioning at chunk boundaries (20 episodes, 254 boundaries) | best gradient-free cuts median jump ≥ 50 %, drift ≤ +10 % | prefix overwrite 72–74 % at d = 8/12/20; exact RTC 21–41 % at 2× latency; every method misses the 667 ms budget, the policy itself is too slow | propose prefix overwrite | folder |
| 10-19 | fault F-09G13b in vla.cpp | Port of prefix overwrite into vla.cpp's SmolVLA | same | 71–73 %; prefix error 0.0; +0.4 ms per chunk; default path bit-identical; 14/14 tests | pass | folder |
| 10-19 | G15 | Stage tracker from proprio and masks: does it advance at the right moment? (69 rollouts, Gemini labels) | premature ≤ 2 %, missed ≤ 10 % | premature 48.5 %, missed 46.7 %; place never fires (the gripper opens less than fitted; nested bowls hide the pink blob) | fail → stage changes on the judge at rest only | folder |
| 10-19 | G15-v2 dev fit | New tracker (grasp = masks AND gripper band), thresholds fit on the 69 rollouts | (dev only; the gate is fresh rollouts) | premature 7.2 %, missed 86.7 %: while the bowl is held the gripper reads ~1.1°, so the AND never holds | design unusable on this gripper | folder |
| 10-19 | G15-v3 fit | Fit the contact-cue tracker's S and D on the 69 rollouts (contact proxied by the gripper command) | (fit only; the gate is a robot-day test) | S = 50, D = 20: premature 18.1 %, missed 37.3 % on the proxy | params committed, contact threshold pending | folder |
| 10-22 | TurboVLA faithful, step 1 | LIBERO-Goal control: how much does a task swap move the released TurboVLA checkpoint? | ≥ 0.095 (= 5° of SO-101 pan) makes the data reading available | 0.105 (5.5° pan-equivalent); four port divergences confirmed against upstream | control ≥ line (small margin) | folder |
| 10-22 | 21-task SmolVLA parity | Does the converted GGUF give the PyTorch chunk for the same frame and noise? (8 frames) | median max abs Δ ≤ 1° | 0.80° (PyTorch noise-to-noise spread 12.2°); one frame at 11.4° | pass (outlier reported) | folder |
| 10-23 | TurboVLA faithful, path A | Upstream's own trainer on the bowls data, full LIBERO init, 35k steps | task swap ≥ 5° → the port was the bug; < 1° with control ≥ 0.095 → the data | task swap 0.020°, per-layout 38.9 % | fact 7 is the full story: the paper's recipe ignores the instruction on this data too | folder |
| 10-23 | fault F-06TurboVLA correction | What stands of the TurboVLA write-up after path A? (no new run) | – (reading of the frozen bar) | data claim stands (the instruction is redundant on ~94 % of frames); port claim withdrawn | corrected | folder |
| 10-23 | fault F-11vla.cpp determinism | 21-task SmolVLA GGUF, 69 rollouts, same noise, two passes with the model reloaded | – (expected ~0) | max abs Δ 0.0° on all 1 534 anchors | deterministic | – |
| 10-23 | Wrist blank, proxies | Wrist input replaced by the dataset-mean image (offline, recordings of other policies) | – (measurement) | chunk error 2.28° → 3.48° (blank worse in 54/69); first action changes > 3° on 17 % of anchors | measured; the decision is a robot-day test | – |
| 10-25 | AW-0 Phase 1 | Can a coding agent annotate the 182 bowls episodes as program slices that match the gripper events? | ≥ 95 % of episodes matched; owner spot check ≥ 9/10 | 182/182; spot check 9/10 | pass | – |
| 10-26 | AW-0 Phase 2 | Does a per-frame slice planner predict the annotated slice on held-out episodes? | accuracy ≥ 85 %, instruction consistency ≥ 98 %, p95 ≤ 30 ms | accuracy 75.7 %, consistency 99.97 %, p95 81.7 ms | fail (accuracy, latency) | – |
| 10-29 | AAC gate 0 | Adaptive Action Chunking (CVPR 2026) on SmolVLA, 20 G13b episodes, 254 decisions, N = 20: does the entropy-chosen chunk length follow the task phases, and lower the boundary jump beyond the prefix overwrite? | A1 median h* transport ≥ grasp window + 2; A3 AAC + overwrite ≤ fixed + overwrite at d = 8 | h* 2 vs 2 (219 / 254 at the floor ξ = 2); jump 0.973 vs 0.940 | A1 fail, A3 fail | folder |
| 10-30 | AAC gate 1 | Cost of an N-sample decision in vla.cpp on the Orin NX (MAXN_SUPER + jetson_clocks), N = 1–40, sequential (no batched sampler) | A4 N = 20 decision ≤ 298 ms | N = 1 241 ms, N = 20 4,751 ms, N = 40 9,506 ms; peak RAM 2.9 GB | fail | folder |
| 10-29 | PACT gate B | PACT-VLA selection on SmolVLA, 20 episodes, 254 boundaries: does FlowRes under the pre-commit frame pick chunks closer to what was executed? | B-P1 PACT < random, B-P2 PACT < stale-frame FlowRes at d = 8, CIs < 0 | −0.27° (CI −0.44 to −0.09) and −0.27° (−0.48 to −0.09); stale FlowRes = random | B-P1 pass, B-P2 pass | folder |
| 10-30 | PACT gate A | PACT's cycle on the Orin NX (MAXN_SUPER + jetson_clocks): 4 candidates + pre-commit vision/prefill + verification (projected) | d_PACT ≤ d_naive; fail if ≥ d_naive + 2 | 1,200 ms vs 237 ms: 37 ticks vs 8; B-P3 not computable (13 of 25 executed steps left in the chunk) | fail | folder |
| 10-29 | Horizon × depth A1 | Does N = 2 contract SmolVLA's action cloud around an unchanged centroid (arXiv 2610.32102)? 20 frames × 32 draws, N = 1–10 | R(2)/R(10) ≤ 0.70, centroid ≤ 0.05 and ≤ 3°; fail R > 0.85 | R(2)/R(10) 0.78 (CI 0.73–0.81); centroid 0.018, 2.35° | not proven | folder |
| 10-30 | Horizon × depth A2 | vla.cpp chunk latency vs Euler steps on the Orin NX, MAXN_SUPER + jetson_clocks, 50 calls per N | N = 2 ≤ 0.85 × N = 10; N = 10 within ±5 % of 238 ms | 196.1 vs 237.5 ms (0.83); 5.3 ms per denoise step, vision 150.7 ms | pass | folder |
| 10-30 | Horizon × depth A4 | Thermal: 10 min at N = 10, 10 idle, 10 min at N = 2, one chunk per 1.67 s, in MAXN_SUPER and in 25 W | reading rule (descriptive) | no throttling in either mode (MAXN clock 1,173 MHz throughout, max 68.6 °C; 25 W max 63.2 °C) | not applicable here | folder |
| 10-31 | REMORA readout | REMORA's causal nearest-peak readout (arXiv 2610.39125) of a 3-atom progress distribution (logistic on G15's evidence, dev-half fit) vs per-frame argmax, median and the stage machine; 34 test episodes of the 69 rollouts, 7 G14 windows | P1 premature ≤ argmax, missed ≤ +2 pt; P2 flips ≤ 0.5 × argmax (CI ≤ 0.7); P3 setback recall ≥ 0.5 at ≤ 0.2 false | premature 0.22 vs 0.25, missed 0.13 = 0.13; flips 0.70 × (CI 0.48–0.94); recall 2/7, false 0.52 | P1 pass, P2 fail, P3 fail | folder |
| 10-31 | Smooth MP chunking A1 | Does projecting SmolVLA's chunk onto 10 ProMP bases cost accuracy? 20 G13b episodes, s = 25 and 50, offline | S2 − S0 error ≤ +0.30° (CI ≤ +0.50°) and within-chunk jerk ≤ 0.5 × S0 at both (projects/smooth-chunking/PROTOCOL.md) |
error −0.025° / −0.020° (CIs below 0); jerk × 0.057 / × 0.058 | pass | projects/smooth-chunking/README.md |
| 10-31 | Smooth MP chunking A2 | Does via-point conditioning remove the seam without training? same chunks | S3 boundary jerk ≤ 0.40 × S2 and S3 − S2 error ≤ +0.30° at both | × 0.174 / × 0.125; +0.184° / +0.191°; descriptive: blending leaves 2.74 × S3's seam at s = 25 | pass; gate B (arm, raw vs S3) next | projects/smooth-chunking/README.md |
| 11-01 | TOPReward vs elapsed time | Does a training-free video-language reward model (Qwen3-VL-8B, the paper's success score) separate failed from successful SO-101 rollouts better than episode length, within policy? ArmnetBench 2,047 (+ 69 own, descriptive) | paired Δ within policy (TOPReward − length) 95 % CI above 0 = beats, contains 0 = matches, below 0 = below; same at the 50 % cut | within policy 0.661 (CI 0.631–0.692) vs length 0.942, Δ −0.31 to −0.25; 50 % cut 0.667 vs 0.851; own 69: 0.825 vs length 0.56 | below elapsed time (full and 50 % cut) | folder |
| 11-01 | RACE diagnostic | Is SmolVLA's chunk error concentrated near sub-skill transitions (gripper crossings) and growing with chunk length? 20 demos, WS-064 chunks | R(50) = arm error near / far ≥ 2 with CI > 1, rising from H = 10 to 50 | R(10) 0.78, R(25) 0.66, R(50) 0.53 (CI 0.45–0.63); gripper channel 32× near transitions (descriptive) | not seen | folder |
| 11-01 | VLA-Corrector (ZJU) monitor | Does the paper's latent-vision monitor, trained on demos only, catch failures on the 50-episode set? | ≥ 5/8 failures at ≤ 2/42 successes | 8/8 at 40/42 (10 interrupts/min on successes); AUROC max E 0.72 vs shuffled actions 0.67 | fail | folder |
| 11-01 | Execution gap C1 + C2 | C1: does the follower settle inside a band smoothing corrections fall into? C2: do failures track their commands worse? 182 demos, 69 rollouts | C1: settle band CI ≥ 0.3° on ≥ 3 of 5 joints; C2: gripper gap ratio ≥ 2 with CI > 1 | C1: 1 of 5 joints (wrist_flex 1.13°); C2: 0.77 (CI 0.64–1.06); normalised residual 18.9 % (Piper: 32.9–35.0 %, 2609.37334) | C1 not sub-servo, C2 not seen | folder |
Data¶
| Date | What | Result | Code |
|---|---|---|---|
| 10-01 | ravediamond/so101_bowls_all_trimmed |
182 episodes, idle starts trimmed, 61 462 frames at 30 fps, public | – |
| 10-14 | Community SO-101 data catalogue (metadata only) | 203 usable datasets / 50 h; strict fit 71 / 14.4 h; ArmnetBench 2 099 labelled rollouts | – |
| 10-14 | ravediamond/so101-bowls-policy-rollouts |
69 policy rollouts, 46 success / 23 failure, labelled, public | – |
| 10-13 → 15 | Tidy-loop teleop (pink bowl + cubes: towers, transfers, flips, corrections) | ~180 episodes; merged into a 21-task set of 653 episodes | – |
| 10-16 | SmolVLA fine-tuned on the 21-task set, 30k steps | $2.94; first Orin gate: single objects 6/6, named cube among two 2/6 | folder |
Lessons the numbers paid for¶
- A world model that predicts well can still be useless for choosing actions. Prediction quality is not decision quality.
- On a position-controlled arm, the commanded joints already say where the arm will be. Test the world model on object state and contact, not on arm motion.
- Raw latent distance is not progress. A progress signal has to be trained on progress labels.
- Spatial features beat a single pooled vector (W1 pose R² 0.57 vs W2 0.92). Never PCA-compress the latent.
- Judging is easier than imagining, but zero-shot judges are not free either: compute episode length and policy identity before any model.
- Top-1 of N compounds errors (top-1 ≈ pairwise^7). Use regret.
- Missing data isn't random: 22 % of episodes lost to one OOM were two whole tasks, and re-scoring them flipped the verdict.
- Idle frames teach freezing; instruction-dependent behaviour needs data where the instruction matters.
- Freeze the rule before the number. It is what makes the failures above worth publishing.