Shisso Robotics

REMORA

Sat, 31 Oct 2026 00:00:00 +0000

REMORA¶

They said reading progress at the peak nearest the last estimate keeps it steady and lets it fall after setbacks. On 34 of our recorded runs we compared it with simpler readouts of the same evidence. It did not hold: it cut changes to 0.70 of the simplest readout, not half, and missed most setbacks.

Papers tested · 22 of 22

Goal (the paper's): Track task progress through errors and recovery: a progress distribution from a frozen-VLM backbone with a per-camera recurrent memory, trained against a target that falls inside error intervals, read out causally at the peak nearest the previous estimate; shown on ReSTEP (AgiBot G02, two logistics tasks).

Its hypotheses:

Picking the supported peak nearest the previous estimate tracks progress through errors better than a per-frame readout. (§3.3 (Eq. 3); Table 2 (A5)) fail
A recurrent memory beats an explicit window of past frames. (Table 2) not tested here
REMORA tracks progress better than SARM. (Table 2) not tested here

Verdict: fail · Claim tested: Reading a progress distribution out at the supported peak nearest the previous estimate steadies the estimate and lets it fall during real setbacks, with no worse detection of the stages. · Bar: Against per-frame argmax: premature not higher and missed ≤ +2 points (P1); flips halved, CI upper ≤ 0.7 (P2); a decrease inside ≥ 50 % of the reversal windows with ≤ 20 % false regressions (P3). · n: 34 test episodes of the 69 rollouts, 7 reversal windows

Theirs vs ours: D2 (25 error episodes): peak selection vs the median readout MAE 6.04 → 3.31 (×10⁻²), Spearman 0.870 → 0.959 vs premature 0.22 vs 0.25, missed equal (P1 pass); flips 0.70 × argmax, CI 0.48–0.94 (P2 fail); setback recall 2/7, false 0.52 (P3 fail); MAE 0.184 vs median 0.218 · Not proven: Three progress atoms from hand-written evidence, not a learned 201-bin distribution; the reversal windows are a mechanical criterion, not human error labels; the recurrent memory and the backbone are not tested.

Paper: arXiv 2610.39125 · Code: no code link found

Claims:

fail nearest-peak progress readout below per-frame progress readout on readout flips per episode — theirs MAE 6.04 → 3.31 (×10⁻²) vs the median readout · ours flips 0.70 × argmax (CI 0.48–0.94); setback recall 0.29 (2/7), false regressions 0.52 (paper §3.3 and Table 2 (peak selection vs median readout on the error task))
not tested recurrent progress memory below explicit frame window on progress MAE — theirs MAE −3.3 %, curvature error −14 % (paper Table 2 (MAE −3.3 %))
not tested REMORA progress model below SARM reward model on progress MAE — theirs lower MAE and smoother on both tasks (paper Table 2)

Compute: External GPU: no. Orin NX CPU (feature extraction in the lerobot-jetson container). Cost: $0. · Reproduce: 2026-10-31-paper-vs-hardware-remora-readout · First published: 2026-10-31 · Updated: 2026-10-31

Tracking Task Progress Through Errors and Recovery (Wang, Niu, Zhu et al., arXiv 2610.39125) argues that a progress estimate should be able to go backwards: when the robot drops the object, progress should fall, then recover. Its model needs eight H200s. Its readout rule does not, so I tested the rule on the 69 rollouts this rig already has.

What the paper claims, and how it measures it¶

REMORA predicts a distribution over progress at every frame, from a frozen VLM backbone with a recurrent memory per camera, and is trained against a target that falls inside labelled error intervals and climbs back after them. It reads the distribution out causally: among the peaks of the distribution, take the one closest to the previous estimate. On the task with real errors (25 error episodes in its ReSTEP test set, AgiBot G02), peak selection against the per-frame median cuts the error from 6.04 to 3.31 (×10⁻²) and lifts Spearman from 0.870 to 0.959. On the task with one error episode it gains nothing.

What I ran¶

The model is out of reach; the readout rule and the error-aware target are model-free. The rig has G15's per-frame grasp and place evidence on 69 rollouts, a monotone stage machine that reads it (premature on 48.5 % of events, missed on 46.7 %), and G14's reversal windows (re-grasp loops, pan dithering) as its own error intervals.

I turned the evidence into a distribution over three progress values (reach, holding, placed) with two logistic regressions fitted on G15's development half, and read the same distribution four ways on the 34 test episodes: the stage machine, the per-frame argmax, the per-frame median (the paper's ablation), and the nearest peak. Bars frozen before any number, against the argmax:

P1: the nearest peak calls no more events too early, and misses at most 2 points more.
P2: it halves the flips per episode (CI upper bound ≤ 0.7).
P3: it decreases inside at least half of the reversal windows, with at most 20 % of the other episodes showing a decrease.

Their number, my number¶

readout	premature	missed	flips / episode	setbacks caught (7)	false setbacks	error vs target
stage machine	0.485	0.467	1.00	0	0.00	0.248
per-frame argmax	0.250	0.133	5.97	3	0.56	0.183
per-frame median	0.397	0.133	6.35	3	0.74	0.218
nearest peak	0.221	0.133	4.21	2	0.52	0.184

P1 passes: premature −5.9 points (CI −17.6 to 0), missed unchanged. P2 fails: the nearest peak keeps 0.70 of the argmax's flips (CI 0.48–0.94), not half. P3 fails: it falls inside 2 of the 7 windows (0.29) and in 52 % of the episodes with none. Against the median, the paper's own comparison, it is closer to the target (0.184 vs 0.218) and steadier, the direction the paper reports, but that comparison had no bar here.

I checked the seven stalled or retried-grasp failures by eye: the nearest peak stays at "reach" for the whole episode, which is right, since none of them ever lifted a bowl. They have no progress to lose, so they cannot exercise the claim.

What the gap means¶

The readout does what it says in the direction it says. What it selects between here is three values from hand-written predicates, and those predicates are noisy in exactly the places a setback would show. The rule cannot separate a real setback from evidence noise when the distribution itself cannot.

Verdict: P1 passes, P2 and P3 fail: not adopted; the progress slot stays on the stage machine.

What would change it: a learned, denser progress distribution (the paper's 201 bins) on this rig, and human-labelled error intervals instead of a mechanical reversal criterion.

Compute. External GPU: no. Orin NX CPU (feature extraction in the lerobot-jetson container). Cost: $0.
Reproduce this: 2026-10-31-paper-vs-hardware-remora-readout

Papers tested, 22 of 22. ← Papers tested 21: Horizon × depth (Euler steps vs execution horizon)

Adaptive Action Chunking (AAC)

Fri, 30 Oct 2026 00:00:00 +0000

Adaptive Action Chunking (AAC)¶

They said a robot policy can pick each motion chunk's length from how much 20 samples disagree. On recorded runs and on the robot's computer we checked the chosen lengths and the cost. It did not hold: lengths mostly stayed at the minimum, and one decision took 4,751 ms.

Papers tested · 19 of 22

Goal (the paper's): Choose each action chunk's length at inference from how much N sampled chunks disagree, so a VLA commits to long chunks where it is confident and short ones where it is not; shown on RoboCasa, LIBERO and three real tasks.

Its hypotheses:

The entropy-chosen chunk length follows the task's phases: long in transport, short near the grasp. (Fig. 3–4) fail
Adaptive chunks raise success over fixed chunks in simulation and on a real arm. (Tables 1–3; Table 5 (real world 67.0 → 82.0 %)) not tested here
Sampling N chunks in a batch adds little cost: negligible below N = 10, about 20 ms at N = 20. (Table 4 (A800)) fail

Verdict: fail · Claim tested: Chunk length chosen from the entropy of N sampled chunks follows the task's phases (long in transport, short at the grasp), lowers the boundary jump, and costs little with batched sampling. · Bar: median h in transport ≥ grasp window + 2; jump with AAC + overwrite ≤ fixed + overwrite at d = 8; an N = 20 decision ≤ 298 ms on the Orin. · n:* 254 decisions on 20 recorded episodes; 10 timed decisions per N

Theirs vs ours: h follows the semantic phases (Fig. 3–4); real world 67.0 → 82.0 %; N = 20 costs 106 ms vs 83 ms on an A800 ("about 20 ms inference delay"; negligible below N = 10) vs h 2 in transport and 2 at the grasp (86 % of decisions at the floor); jump 0.973 vs 0.940; N = 20 decision 4,751 ms (sequential) on the Orin · Not proven: No arm run (the bowls task is at ceiling for SmolVLA); vla.cpp has no batched sampler, which would change the cost but not the entropy rule.

Paper: arXiv 2604.04161 · Code: no code link found

Claims:

fail entropy-chosen chunk length above grasp-window chunk length on median chunk length h in transport — theirs long in transport, short at the grasp · ours 2 vs 2 (paper Fig. 3–4 (h follows the semantic phases))
fail AAC with prefix overwrite at most fixed chunk with prefix overwrite on median boundary jump — theirs no boundary-jump measurement; real-world success 67.0 → 82.0 % (Table 5) · ours 0.973 vs 0.940 (paper §1 and Fig. 6 (qualitative; no boundary metric))
fail N-sample decision at most pass mark on ms per decision on the Orin NX — theirs 106 ms (A800, batched) · ours 4,751 ms (paper Table 4 (N = 20: 106 ms vs 83 ms on an A800, "negligible"))

Compute: External GPU: no. Gate 0 on the Orin NX at 25 W (PyTorch, ~3 h); gate 1 on the Orin at MAXN_SUPER + jetson_clocks (vla.cpp). Cost: $0. · Reproduce: 2026-10-30-paper-vs-hardware-adaptive-chunking · First published: 2026-10-30 · Updated: 2026-10-30

Adaptive Action Chunking at Inference-time for VLA Models (AAC; Liang et al., CVPR 2026, arXiv 2604.04161) lets a flow-matching policy choose each chunk's length from how much N sampled chunks disagree. SmolVLA is a flow-matching policy, so the rule applies to my SO-101 as written.

What the paper claims, and how it measures it¶

Draw N = 20 chunks from the same observation; the step where their entropy jumps most sets the chunk length h. Where the policy is confident (transport) chunks get long, where it is not (approach and grasp) they get short. The paper shows h following the task's phases (Fig. 3–4), gains on RoboCasa, LIBERO and three real tasks (67.0 → 82.0 % on a Realman arm with a Mycobot gripper), and calls the cost negligible: N = 20 takes 106 ms against 83 ms for one chunk, batched on an A800.

What I ran¶

SmolVLA (the checkpoint behind every vla.cpp number here) on the 20 recorded episodes of the bowls task I use for chunk-boundary tests: 254 decisions, N = 20 samples each, floor ξ = 2. Three bars, frozen before any number:

A1: median h in transport ≥ median h in the grasp window + 2 (the phase claim).
A3: with the prefix overwrite I already run, AAC's boundary jump ≤ the fixed chunk's, at a delay of 8 steps.
A4: on the Orin NX at full power, one N = 20 decision ≤ 298 ms (25 % over one 238 ms chunk).

Their number, my number¶

	Paper	This rig
h* in transport vs at the grasp	long vs short	2 vs 2
jump, AAC + overwrite vs fixed + overwrite (d = 8)	—	0.973 vs 0.940
N = 20 decision	106 ms (A800, batched)	4,751 ms (Orin NX, sequential)

All three bars fail. h* sits at the floor in 219 of 254 decisions: SmolVLA's samples spread fastest at the very first step, so the largest entropy increase is always at h = 1 and the rule returns the minimum. AAC's earlier switch alone does lower the naive jump (3.42 → 2.28), because the new chunk is sampled closer in time, but the prefix overwrite already brings the jump to about one normal step (0.94), and AAC adds nothing on top. On the Orin, vla.cpp has no batched sampler, so 20 samples are 20 chunk inferences: 4.75 s per decision.

What the gap means¶

The entropy rule needs a policy whose samples agree for a while and then diverge at hard moments. SmolVLA's samples diverge from the first step everywhere, so on this policy the rule degenerates to "near-minimal chunks". And the paper's "negligible" cost is a property of batched sampling on a data-centre GPU, not of the method.

Verdict: fail on this rig.

What would change it: a policy whose sample spread grows late in the chunk, and a batched sampler on the edge device.

Compute. External GPU: no. Gate 0 on the Orin NX at 25 W (PyTorch, ~3 h); gate 1 on the Orin at MAXN_SUPER + jetson_clocks (vla.cpp, ~5 min). Cost: $0.
Reproduce this: 2026-10-30-paper-vs-hardware-adaptive-chunking

Papers tested, 19 of 22. ← Papers tested 18: SURVEYOR · Papers tested 20: PACT-VLA →

Horizon × depth (Euler steps vs execution horizon)

Fri, 30 Oct 2026 00:00:00 +0000

Horizon × depth (Euler steps vs execution horizon)¶

They said a robot that commits to long motion chunks can use 2 sampling steps instead of 10. On our arm's computer we measured how tightly 2 steps group the planned motions, and the time saved. Not proven: on 20 frames, 2 steps kept 0.78 of the 10-step spread (paper: 0.51) and saved 41 ms.

Papers tested · 21 of 22

Goal (the paper's): Show that, for frozen flow-matching VLAs, the effect of cutting Euler steps depends on how long the robot commits to a chunk: fewer steps hurt under frequent replanning and help under long open-loop commitment; shown on LIBERO, RoboTwin 2.0 and a real Franka.

Its hypotheses:

Few Euler steps contract the sampled action cloud around an unchanged centroid. (Proposition 1; §III-D) not proven
Under long open-loop commitment, two steps match or beat ten while cutting inference time. (Table I; Fig. 3; latency figures) reproduces
Two steps avoid thermal throttling on edge GPUs. (§IV (end): 0.04 NFE per step; asserted, no thermal measurement) not tested here

Verdict: not proven · Claim tested: With long open-loop commitment, cutting a flow-matching VLA's Euler steps from 10 to 2 contracts the action cloud around an unchanged centroid, saves most of the inference time and avoids edge-GPU thermal throttling. · Bar: median R(2)/R(10) ≤ 0.70 with the centroid within 0.05 relative and 3°; N = 2 latency ≤ 0.85 × N = 10 on the Orin, N = 10 within 5 % of 238 ms. · n: 20 frames × 32 draws; 50 timed chunks per N; 2 × 10 min thermal per power mode

Theirs vs ours: R(2)/R(10) 0.51, centroid shift 3.2 %; 14.3 → 2.6 ms per step (A100); "eliminating edge GPU thermal throttling" vs R(2)/R(10) 0.78 (CI 0.73–0.81), centroid 0.018 and 2.35°; 196.1 vs 237.5 ms per chunk (0.83, vision 151 ms); no throttling at N = 10 on the Orin, at MAXN_SUPER or 25 W · Not proven: The contraction is between the bar and the fail line, so the arm gate (success at N = 2 vs 10) is not run; one policy, one task.

Paper: arXiv 2610.32102 · Code: no code link found

Claims:

not proven two Euler steps below ten Euler steps on action-cloud radius — theirs R(2)/R(10) 0.51 · ours R(2)/R(10) 0.78 (paper Proposition 1; §III-D (R(2)/R(10) = 0.51 on π0))
reproduces two Euler steps below ten Euler steps on ms per chunk on the Orin NX — theirs 14.3 → 2.6 ms per step · ours 196.1 vs 237.5 ms (paper Table I (14.3 → 2.6 ms per control step, A100))

Compute: External GPU: no. A1 on the Orin NX at 25 W (PyTorch); A2 and A4 at MAXN_SUPER + jetson_clocks and at 25 W (vla.cpp). Cost: $0; A2 and the MAXN half of A4 re-run once. · Reproduce: 2026-10-30-paper-vs-hardware-horizon-depth · First published: 2026-10-30 · Updated: 2026-10-30

Coupling Execution Horizon and Sampling Depth in Flow-Matching VLA Policies (Yang et al., arXiv 2610.32102) says that when a robot commits to a long chunk, two Euler steps are as good as ten and much cheaper. My SO-101 already runs SmolVLA with its whole 50-step chunk committed, so the claim lands on its default setting.

What the paper claims, and how it measures it¶

A flow-matching policy draws an action chunk by integrating from noise in N Euler steps. With few steps the samples cluster: the paper's Proposition 1 says the cloud contracts around an unchanged centroid, so under long open-loop commitment, where averaging out sample noise matters, N = 2 matches N = 10. On π0 the cloud's radius falls to 0.51 of N = 10's with a 3.2 % centroid shift, the per-step latency drops from 14.3 to 2.6 ms on an A100, and the authors add that N = 2 avoids thermal throttling on edge GPUs. Real-robot results are on a Franka Research 3.

What I ran¶

SmolVLA (10 Euler steps, H = 50, the checkpoint behind every vla.cpp number here) on 20 frames from the bowls episodes. Bars frozen before any number:

A1, mechanism: median R(2)/R(10) ≤ 0.70, with the centroid within 0.05 relative and 3°; fail above 0.85.
A2, latency on the Orin: N = 2 ≤ 0.85 × N = 10, with N = 10 within 5 % of the 238 ms reference.
A4, thermal: 10 minutes at N = 10, 10 idle, 10 at N = 2, at full power and at 25 W; throttling = the GPU clock 5 % below its first minute for 10 s.

The arm test (success at N = 2 vs 10) runs only if A1 and A2 pass.

Their number, my number¶

	Paper	This rig
cloud radius, N = 2 / N = 10	0.51	0.78 (CI 0.73–0.81)
centroid shift	3.2 %	1.8 %, 2.35°
inference saving with N = 2	14.3 → 2.6 ms per step (A100)	237.5 → 196.1 ms per chunk (Orin NX)
thermal throttling at N = 10	"eliminated" by N = 2	none at full power or at 25 W

A1 is not proven: two steps do contract the cloud around a nearly unchanged centroid, but by 22 %, not 49 %, and the confidence interval sits between the bar and the fail line. A2 passes, barely, and for a reason the paper's A100 hides: on the Orin a chunk is 151 ms of vision, 15 ms of prefill and 5.3 ms per denoise step, so cutting 8 of 10 steps saves 41 ms. A4 has nothing to act on: one chunk every 1.67 s never throttled the Orin, the GPU staying at 1,173 MHz and under 69 °C.

What the gap means¶

The mechanism is there but weaker on a 450M policy, and the cost it removes is a small slice of an edge chunk: the vision encoder dominates, so the denoising budget is not where the Orin's time goes. Because A1 did not pass, the arm test did not run and SmolVLA stays at ten steps.

Verdict: not proven.

What would change it: a policy whose denoising dominates the chunk (larger action expert, cheaper vision), or an arm test showing N = 2 keeps success, which the protocol only runs after A1 passes.

Compute. External GPU: no. A1 on the Orin NX at 25 W (PyTorch); A2 and A4 at MAXN_SUPER + jetson_clocks and at 25 W. Cost: $0. A2 and the full-power A4 were re-run once (a root-owned results folder; a tegrastats format without the GPU clock).
Reproduce this: 2026-10-30-paper-vs-hardware-horizon-depth

Papers tested, 21 of 22. ← Papers tested 20: PACT-VLA · Papers tested 22: REMORA →

PACT-VLA

Fri, 30 Oct 2026 00:00:00 +0000

PACT-VLA¶

They said choosing among several motion chunks with the newest camera frame adds no delay. On recorded runs we tested the choice, and on the robot's computer the delay. The choice helped, but a full cycle took 1,200 ms, 37 control ticks instead of 8.

Papers tested · 20 of 22

Goal (the paper's): Under asynchronous execution, sample several candidate chunks and keep the one the frozen policy finds most consistent with the newest observation before the switch, with no verifier to train and no added control-step delay; shown on Kinetix, LIBERO and a bimanual PiperX.

Its hypotheses:

Selecting by flow residual under the pre-commit observation picks better chunks than a random pick or the stale observation. (Appendix D.2; Fig. 4) reproduces
The selection adds no control-step delay, because the extra pass overlaps with execution. (§3 (pipelining), Table 4) fail

Verdict: fail · Claim tested: Picking among N sampled chunks by their flow residual under the pre-commit observation fits what happens next better than a random or stale pick, with no added control-step delay. · Bar: PACT beats a random pick and the stale-observation residual at d = 8 (CIs < 0); its delay on the Orin ≤ the naive delay; at that delay it still beats a single sample. · n: 20 episodes, 254 boundaries per delay

Theirs vs ours: bimanual PiperX 82.0 % vs naive async 60.0 %; no added control-step delay on the evaluated GPUs (generation 35–49 ms, with verification 59–65 ms, both d = 2 at 30 Hz, Table 4) vs offline −0.27° vs random and vs the stale residual (both CIs < 0); on the Orin 1,200 ms per cycle, 37 control ticks vs 8 · Not proven: The verification time is projected (vla.cpp has no velocity query); at the measured delay a 50-step chunk cannot fill the executed window, so the transfer clause is not computable; no arm run.

Paper: arXiv 2610.28019 · Code: no code link found

Claims:

reproduces PACT selection below random pick on executed-window error — theirs +0.44 points over random (Kinetix); 82.0 vs 60.0 % (PiperX) · ours −0.27° error vs a random pick (CI −0.44 to −0.09) (paper Appendix D and Fig. 4)
fail PACT cycle equals naive async cycle on control-step delay — theirs no added delay · ours 37 ticks vs 8 (paper ("no added control-step delay" on the evaluated hardware; Table 4: generation 35–49 ms, with verification 59–65 ms, both d = 2))
reproduces PACT cycle above naive async cycle on control-step delay · ours 37 ticks vs 8 (our measured comparison)

Compute: External GPU: no. Gate B on the Orin NX at 25 W (PyTorch); gate A on the Orin at MAXN_SUPER + jetson_clocks. Cost: $0. · Reproduce: 2026-10-30-paper-vs-hardware-pact · First published: 2026-10-30 · Updated: 2026-10-30

PACT-VLA: Speculative Action-Chunk Verification with Pre-Commit Observations (Zhu et al., arXiv 2610.28019) samples several chunks and keeps the one the policy itself finds most consistent with the latest frame before the switch, "with no added control-step delay". The selection works on my recorded episodes. The delay is where it breaks on a Jetson.

What the paper claims, and how it measures it¶

Under asynchronous execution the robot keeps moving while the next chunk is computed, so the chunk is planned from a stale frame. PACT generates N = 4 candidates, then scores each with the policy's own flow residual (FlowRes) under the pre-commit observation, the frame one control step before the switch, and keeps the lowest. On a bimanual AgileX PiperX with π0.5 it reaches 82.0 % against 60.0 % for naive async, at 35–49 ms per chunk on an RTX 5090 or H100, and the extra pass is hidden by overlap.

What I ran¶

SmolVLA on the same 20 recorded bowls episodes, 254 chunk boundaries per delay, four candidates per boundary. Bars frozen before any number:

B-P1 / B-P2: at a delay of 8 steps (238 ms at 30 Hz, this rig's real delay), PACT's chunk is closer to what was actually executed than a random pick, and than the same residual under the stale frame (episode bootstrap, CI entirely below 0).
A: on the Orin at full power, PACT's cycle adds no control step: d_PACT ≤ d_naive (fail if two or more).
B-P3: at the delay PACT actually costs on the Orin, it still beats a single sample.

Checks before reading any number: candidate 0 bit-identical to a single sample, the recorded future scored below the median candidate on 68 % of boundaries (bar 60 %), and a shuffled observation does not pass B-P1.

Their number, my number¶

	Paper	This rig
selection vs random pick	+0.44 points (Kinetix, one decision); 82.0 vs 60.0 % on the robot	−0.27° error (CI −0.44 to −0.09)
selection vs stale-frame residual	stale = random (Kinetix)	−0.27° (CI −0.48 to −0.09); stale = random here too
cycle time	35–49 ms per chunk, extra pass overlapped	1,200 ms vs 237 ms
control-step delay	none added	37 ticks vs 8

B-P1 and B-P2 pass: choosing with the fresher frame picks better chunks, and the residual under the stale frame is no better than chance, as the paper's own appendix found. The fresher observation does the work.

A fails by 29 ticks. vla.cpp has no batched sampler, so the four candidates are four chunk inferences (950 ms), plus a second vision and prefill pass on the pre-commit frame (166 ms) and the verification (85 ms, projected from a PyTorch ratio because vla.cpp has no velocity query). The Orin's GPU is already 94–98 % busy during a chunk, so nothing overlaps. B-P3 cannot be computed as written: at a 37-step delay, a 50-step chunk holds only 13 of the 25 steps PACT would have to execute.

What the gap means¶

The idea is sound and the selection signal is real on a 450M policy and a ≈ €1,350 rig. "No added delay" is a property of a desktop GPU with spare capacity: on an edge GPU running at capacity, every candidate and the second pass are paid in sequence.

Verdict: the selection replicates offline; the paper's no-delay claim fails on the Orin, so PACT does not transfer to this rig.

What would change it: batched sampling and a velocity query in vla.cpp, which would make the cycle closer to one chunk plus a pass.

Compute. External GPU: no. Gate B on the Orin NX at 25 W (PyTorch); gate A on the Orin at MAXN_SUPER + jetson_clocks. Cost: $0.
Reproduce this: 2026-10-30-paper-vs-hardware-pact

Papers tested, 20 of 22. ← Papers tested 19: Adaptive Action Chunking (AAC) · Papers tested 21: Horizon × depth (Euler steps vs execution horizon) →

LeJEPA identifiability

Thu, 29 Oct 2026 00:00:00 +0000

LeJEPA identifiability¶

They said a world model trained on goal-directed runs recovers the true robot state only partly and unevenly. On our arm we compared its internal numbers with the joint angles the arm logs. It held: it explained 0.565 of the joints' variance on average, 0.91 at the shoulder, 0.27 at the wrist.

Papers tested · 17 of 22

Goal (the paper's): Derive the conditions under which a JEPA trained with SIGReg recovers the true latent state of the system, and show where goal-directed data breaks that recovery.

Its hypotheses:

On goal-directed trajectory data, recovery of the true state is partial and anisotropic. (§6.2, Table 2 (DMC Reacher trajectories)) reproduces
Under an isotropic random walk the latent recovers the state up to a linear map. (§5.1, Theorem 1; §6.2, Table 2 (OU pairs R² = 0.95)) not tested here

Verdict: reproduces · Claim tested: A SIGReg JEPA trained on goal-directed data recovers the true latent state only partially and unevenly. · Bar: W1's mean R² from latent to joints < 0.80; anisotropy (max − min per-joint R²) ≥ 0.20; the higher-R² model has the higher G1b top-1. · n: 8 447 validation frames, 25 episodes; 2 models

Theirs vs ours: anisotropic recovery on trajectory data, identifiable under isotropic random walks (simulation) vs W1 mean R² 0.565, anisotropy 0.639 (shoulder lift 0.91, wrist flex 0.27); frozen DINOv2-S 0.867 / 0.301 · Not proven: The random-walk side (gate 1: random-walk data on the arm and a retrain) is not run; B3 compares two models.

Paper: arXiv 2605.26379 · Code: no code link found

Claims:

reproduces SIGReg JEPA latent below pass mark on latent-to-joints R² — theirs anisotropic recovery on trajectory data (simulation) · ours 0.565 (paper §6.2, Table 2)
reproduces SIGReg JEPA latent at least pass mark on per-joint R² anisotropy — theirs anisotropic (simulation) · ours 0.639 (paper §6.2, Table 2)
not tested SIGReg JEPA latent equals pass mark on latent-to-joints R² — theirs identifiable (simulation) (paper §5.1 Theorem 1; §7)

Compute: External GPU: no (Orin NX). Cost: $0. One extraction redone after a red-channel bug. · Reproduce: 2026-10-29-paper-vs-hardware-lejepa · First published: 2026-10-29 · Updated: 2026-10-29 · Findings: A one-timestamp indexing bug, found by a consistency check →

When Does LeJEPA Learn a World Model? (Klindt, LeCun, Balestriero; arXiv 2605.26379) asks when a SIGReg JEPA's latent recovers the true state of the system. A robot arm logs that state for free: the joint positions. So on my SO-101 I can measure directly what the paper could only measure in simulation.

What the paper claims, and how it measures it¶

The theory says a SIGReg-trained JEPA recovers the true latent up to a linear map when the data moves through state space like an isotropic random walk. When the data is goal-directed, as in demonstrations, recovery becomes anisotropic: the directions the task moves along a lot are recovered well, the rest poorly. The paper checks this in simulation (DMC Reacher), where the true state is known.

What I ran¶

W1 is my multi-view LeWM, trained with SIGReg on teleoperated bowls demonstrations: goal-directed data, the case the paper predicts goes anisotropic. I fitted a ridge regression from W1's latent to the six logged joints on 17,724 training frames and scored R² on 8,447 validation frames from 25 held-out episodes. The comparison is a frozen DINOv2-S encoder (W2′), which was never trained on this data.

Three bars, frozen before any latent was computed: B1 W1's mean R² stays below 0.80 (recovery is partial); B2 W1's anisotropy, the gap between its best and worst joint, is at least 0.20; B3 the model with the higher R² also ranks action chunks better (G1b top-1).

My first extraction fed both encoders the red channel only (a one-timestamp indexing bug). I found it on a later run, fixed it and re-extracted. The verdict didn't change; the numbers below are the corrected ones.

Their number, my number¶

	W1 (SIGReg LeWM, teleop)	W2′ (frozen DINOv2-S)
mean R², latent → joints	0.565	0.867
per joint: pan / lift / elbow / wrist flex / wrist roll / gripper	0.58 / 0.91 / 0.88 / 0.27 / 0.30 / 0.45	0.92 / 0.98 / 0.97 / 0.74 / 0.68 / 0.91
anisotropy (best − worst joint)	0.639	0.301
G1b ranking top-1	0.605	0.646

B1, B2 and B3 are all met. W1 recovers the joints the task moves most (shoulder lift 0.91, elbow 0.88) and loses the ones it barely varies (wrist flex 0.27, wrist roll 0.30): the anisotropic recovery the paper predicts for trajectory data, on a real arm, with the true state logged.

What the gap means¶

The direction of B3 holds, but it's two models, and both sit well below the action-only baseline on G1b (0.68–0.90). A 0.30 R² gap moved ranking by 0.04, so identifiability isn't what limits ranking here. The paper's other half, that an isotropic random walk would make W1's recovery even, needs new data on the arm and a retrain. That's gate 1, and it hasn't run.

Verdict: the anisotropic recovery reproduces on a real arm; the random-walk side is untested.

What would change it: a random-walk dataset of the same size on the arm, the same recipe retrained, and the same measurement.

Compute. External GPU: no (Orin NX). Cost: $0. One extraction redone after the red-channel bug.
Reproduce this: 2026-10-29-paper-vs-hardware-lejepa

Papers tested, 17 of 22. ← Papers tested 16: HWM (hierarchical latent world models) · Papers tested 18: SURVEYOR →

SURVEYOR

Thu, 29 Oct 2026 00:00:00 +0000

SURVEYOR¶

They said a fixed distance threshold between imagined and reached states accepts good plans and rejects bad ones. On 69 real runs we measured that threshold in our world model. It did not hold: the threshold here is 1.249, not their 0.20, and it caught 3/8 failures.

Papers tested · 18 of 22

Goal (the paper's): Let a latent-world-model planner consume drafted subgoals speculatively, accepting each step while the imagined and reached latents stay within a calibrated threshold; shown in simulation (PushT, Reacher, OGBench-Cube, Two-Room).

Its hypotheses:

A threshold calibrated on task-equivalent state pairs (τ = 0.20) separates acceptable from unacceptable plan steps. (method and experiments) fail
The accept rule keeps good drafted plans and rejects bad ones. (experiments) fail

Verdict: fail · Claim tested: A threshold on the imagined-vs-reached latent distance accepts good plans and rejects bad ones (τ = 0.20). · Bar: τ below the 10th percentile of non-equivalent pairs and ≤ 0.40; as a monitor ≥ 5/8 failures at ≤ 2/42; ≥ 0.05 AUROC over a shuffled drafter. · n: 69 rollouts (primary 50: 42 successes, 8 failures)

Theirs vs ours: τ = 0.20 in simulation (PushT, Reacher, OGBench-Cube, Two-Room) vs τ = 1.249 on W1 latents of 69 real rollouts; as a monitor 3/8 at 6/42 (S = 30), AUROC 0.756 vs shuffled 0.560 · Not proven: FF-JEPA's diffusion drafter is not available (W1's own imagination stands in); no closed-loop planning.

Paper: none (NeurIPS 2026 workshop) · Code: no code link found

Claims:

fail latent accept threshold at most pass mark on accept threshold τ — theirs τ = 0.20 · ours τ = 1.249 (paper, τ = 0.20 (simulation))
fail latent accept rule at least pass mark on failures caught — theirs simulation only · ours 3/8 at 6/42 (paper (accepts good plans, rejects bad ones))

Compute: External GPU: no (Orin NX). Cost: $0. · Reproduce: 2026-10-29-paper-vs-hardware-surveyor · First published: 2026-10-29 · Updated: 2026-10-29

SURVEYOR (Certified Speculative Plan Consumption for Latent World-Model Planning, a NeurIPS 2026 workshop paper) accepts a drafted plan step when the latent the world model imagined and the latent actually reached are close enough. Its encoder is the one released with LeWorldModel, the same recipe as my W1, so the accept rule can be tested on real rollouts.

What the paper claims, and how it measures it¶

A drafter proposes subgoals in latent space; the planner consumes them while the relative distance between imagined and reached latents stays below a threshold τ, calibrated on pairs of states that should count as equivalent. The paper's τ is 0.20. Everything is in simulation (PushT, Reacher, OGBench-Cube, Two-Room); there is no real robot.

What I ran¶

Gate 0, τ. On W1's latents of 69 real SO-101 rollouts, I calibrated τ the paper's way: the median relative distance between equivalent pairs (two at-rest success frames from different episodes). Bars: S1, τ must sit below the 10th percentile of non-equivalent pairs; S2, τ ≤ 0.40.

Gate 1, the accept rule as a failure monitor. FF-JEPA's diffusion drafter isn't available, so W1's own imagination stands in for it, and a shuffled-drafter control checks whether the rule uses the plan at all. Bars: S3, at least 5 of 8 failures caught with at most 2 false interventions out of 42; S4, at least 0.05 AUROC above the shuffled drafter.

Their number, my number¶

	Paper	This arm
τ	0.20	1.249 (non-equivalent pairs: p10 1.106)
equivalent vs non-equivalent	separable	AUROC 0.71
monitor, S = 10 frames	—	1/8 caught at 1/42, AUROC 0.619 (shuffled 0.628)
monitor, S = 30 frames	—	3/8 at 6/42, AUROC 0.756 (shuffled 0.560)

S1 and S2 fail: on real rollouts, two finished successes from different episodes are about as far apart in W1's latent as a success and a frame from mid-episode. The scene changes between episodes (bowl positions, lighting) dominate the distance. S3 fails at both step sizes. S4 is met at S = 30 only: over one second the discrepancy carries plan–execution signal beyond chance, but its warnings come after the failure onset.

What the gap means¶

In simulation, equivalent states look alike and τ can be small. On a real arm, the same latent puts "done, in a different scene" far from "done": a threshold on latent distance has no operating point, and the accept rule becomes the seventh monitor here to miss the standing bar. The positive bit is S4 at one second: the discrepancy isn't noise, it just arrives late.

Verdict: fail on this arm.

What would change it: a latent where at-rest successes cluster across scenes (an object-centric or goal-conditioned one), and the real drafter.

Compute. External GPU: no (Orin NX). Cost: $0.
Reproduce this: 2026-10-29-paper-vs-hardware-surveyor

Papers tested, 18 of 22. ← Papers tested 17: LeJEPA identifiability · Papers tested 19: Adaptive Action Chunking (AAC) →

How I re-test robot-learning papers: the gate protocol as it ran

Thu, 29 Oct 2026 00:00:00 +0000

How I re-test robot-learning papers: the gate protocol as it ran¶

We wanted to know how our gate loop actually ran, not how it was meant to run. We went back over every gate in the log: bar committed, run, verdict, row, recompute folder. Of 74 gates, 20 failed their bar.

Evaluation and judges · 13 of 14

Type: method · Result: Bars committed before the run, a four-word verdict vocabulary, a judge at rest with 0 false positives, separate gos for money and arm time, costs with retries; the v1.5 harness has not run end to end. · n: 74 gates, 16 papers, 55 work-stack items

Not proven: That the protocol catches every error; the v1.5 harness, G24 and the data-engine fields have not run.

Compute: External GPU: no; no new run. Every figure is recomputed from the experiment log, the register and the paper-vs-hardware folders. · Reproduce: – · First published: 2026-10-29 · Updated: 2026-10-29

Every result on this site went through the same loop: write the pass rule down, commit it, run, apply the rule and nothing else, log the row, publish a folder that recomputes the number. That loop has produced 74 gates in the experiment log, and 20 failed their bar. This post is about the loop itself, and only about the parts of it that have actually run.

1. The bar goes in before the run¶

A gate starts as a protocol file with the pass rule in it, committed before anything is computed, and the row in the log cites that commit. Three real ones:

G-flora (would a robot-trained reward model judge my rollouts?). The bar was committed at 1c50a5d: at the largest threshold that flags at most 2 of 42 successes, flag at least 6 of 8 failures. The image layout was amended at 179c1d6, also before the first call. Result: 8/8 failures at 2/42 successes, pass. Row.
AW-0 Phase 2 (a per-frame planner for task slices). The settings were fixed at c8b8d96 before any fit; the bar was 85 % per-frame accuracy and p95 ≤ 30 ms. Result: 75.7 % and 81.7 ms, fail on both. Row.
HWM (the hierarchical world-model crossover). Bars committed before training; the first run turned out to be fed red-only images, so a deviation note was added and the run redone. The bars stayed as they were. Post.

The point of the hash is that anyone can check the bar's date against the result's date. When I change something after seeing data, it goes in its own file, and anything decided after the number is labelled exploratory or post-hoc.

2. Four verdicts, and what separates fail from not proven¶

Each paper in the register gets one of: reproduces, fail, not proven, not tested (plus withdrawn, for my own mistakes). The count of each verdict is on the register's scoreboard, generated from the same file as the table.

The line between fail and not proven is whether the miss could be explained by something the run did not control. RoboJEPA is a clean fail: the 300M model got 11.8 % below copy-last against a 15 % bar, on the setup the bar was written for, and nothing in the run argued for a different reading. HWM is not proven: its high-level model never beat the low level, but it was trained on about 0.6 h of one task against the paper's ~130 h, and its error sat barely below predicting the average latent. An open confound like that makes it not proven rather than fail, even though two of its three bars were missed.

3. The judge at rest¶

Most gates need an outcome per trial. G9c-3 is the deterministic judge that gives it: top camera, arm at rest, a fixed rule. On the test split it scored 94.3 % on 87 judged episodes, with 0 false positives and 5 false negatives (row, post).

Its own bar was 95 %, so it missed by 0.7 points, and the row says so. It was still adopted, because the number that matters for anything that learns from rollouts is false positives: a failure judged as a success goes into the data as a demonstration. A false negative only costs a trial. Judged only at rest also means the verdict never depends on a frame where the arm is moving.

The same reasoning sets the rule for weaker judges (a VLM, a learned scorer): they never decide alone, they pair with a deterministic check, and they earn a place per task through a report card of false positives, false negatives and ambiguous calls against my labels.

4. What is recorded per trial¶

What has actually been recorded on robot days: per trial, the policy, the engine, latency (p50, p99, mean), control rate, VRAM, the condition, success and failure mode, plus video. Those CSVs sit in each folder's expected/ directory. The 69 rollouts behind most monitor gates are public with their outcome labels. One rule runs through all of it: nothing is trained on failed rollouts; failures only feed verifiers and the world model.

5. The work stack¶

Gates run through a queue: one item per task, each with what it needs (offline, vast, arm, rig), what it depends on, a prompt file that is the whole brief, and a stop point. When this was written (29 October) it held 55 items: 37 done, and 8 waiting on the arm or the rig.

Nothing that costs money or moves the arm runs on its own. Each needs its own explicit go: go vast WS-NNN with a cost cap per item, go arm WS-NNN for arm time, go rig for staging, go for publishing. They are separate because they fail differently: a wrong spend is recoverable, a wrong arm motion may not be, and a publish can't be taken back.

6. Cost per result, retries included¶

Every post ends with a compute block: external GPU yes or no, the dollar figure, and the retries. The register's tested papers add up to $45 of GPU and API. Across all published folders it is $46.3–47.3 on vast.ai and $4.9–5.9 of API. Those totals include the failed starts, the hosts that never connected and the runs I threw away, like HWM's red-only first run. That is the number a reader would pay to repeat the work, so it is the one I publish.

7. Three levels of replication¶

Every result has a folder in paper-vs-hardware:

L0: recompute the post's numbers from the small files in the folder, in seconds, on a CPU. There are 23 folders and all 23 pass their L0 check.
L1: rerun the analysis from the pinned inputs (checkpoints, datasets, scripts).
L2: rerun on hardware. The protocol is in the post; the folder does not claim it is re-verified.

What it does not have yet¶

The v1.5 harness has not run end to end on the arm. It is specified and partly built, and tested on simulated hardware only. Nothing on this site describes it as working.
The adoption protocol (G24) has not run, and there is no autonomous loop in service.
The data-engine fields (layout ids, executor per trial, a curated demos/ view) are proposed, not recorded. The LEGO verdict now comes from camera masks, and its verifier card has not run either.
Gates from the v0.8–v1.1 era say so in their rows; the vocabulary and the judge above are the parts that carried through.

That is the same discipline as the rest: the register lists what ran, at the bar it was given.

The experiment log has every row; the register has every paper; the folders recompute every number.

Evaluation and judges, 13 of 14. ← Evaluation and judges 12: Audit your labels where they matter: a VLM labeller that is right on average · Evaluation and judges 14: VLA-JEPA's surprise as a failure warning: a robot day with nothing to warn about →

VLA-JEPA's surprise as a failure warning: a robot day with nothing to warn about

Thu, 29 Oct 2026 00:00:00 +0000

VLA-JEPA's surprise as a failure warning: a robot day with nothing to warn about¶

We wanted to know if a policy's own world model is surprised before it fails on the arm. We ran it on sealed layouts, some with the bowls moved, with the bar fixed first. It succeeded on 20 of 20, so there was no failure to warn about.

Evaluation and judges · 14 of 14

Type: method · Result: Not proven: VLA-JEPA completed 20 of 20 sealed layouts (9 off-nominal), so there was no failure for its surprise to warn about. · n: 20 rollouts on the arm, 0 failures

Not proven: Whether VLA-JEPA's surprise warns before a failure; a harder layout set is next.

Compute: External GPU: no (Orin NX, arm). Cost: $0. · Reproduce: 2026-10-29-vla-jepa-surprise-robot-day · First published: 2026-10-29 · Updated: 2026-10-29

VLA-JEPA carries its own world model. If that model is surprised when a grasp goes wrong, the policy could warn about its own failures with no extra network. I set a robot day up to test it, and the policy didn't fail once.

The question¶

Does VLA-JEPA's own world-model surprise warn before a failure on the arm, earlier or more reliably than W1's surprise, a joint-stall rule, elapsed time and the REAP risk head? The bar was frozen on 2026-10-25, before the robot day, together with the scorer and a sealed list of 60 layouts across 30 strata, nominal (teleop positions) and off-nominal (each bowl shifted 3–6 cm). A warning only counts if it comes before the failure onset marked by the owner, and the bar needs at least 6 failures before any monitor is scored.

What happened¶

On 2026-10-27 VLA-JEPA (checkpoint c65c98b, Orin NX at MAXN_SUPER, sync loop) ran the sealed layouts in order, the owner placing the bowls and calling each outcome at rest.

	rollouts	success
teleop positions	11	11
bowls shifted 3–6 cm	9	9
all	20	20

The owner stopped at rollout 20 instead of 60: with no failure in 13 (and one in 10 on the earlier gate), reaching 6 failures in the remaining 47 was about 1–2 % likely, and the bar's branch for fewer than 6 failures doesn't depend on the rollouts not run.

Verdict¶

Not proven: too few failures (0 < 6). No monitor was scored, so this says nothing yet about VLA-JEPA's surprise.

What the day does show: this VLA-JEPA completes the bowls task on 20 of 20 sealed layouts, including 9 with both bowls off their teleop positions. The task is too easy to produce the failures a monitor needs. The next step is a harder layout set, with its own bar written before the run, and we'll come back to it then.

Compute. External GPU: no (Orin NX, arm). Cost: $0.
Reproduce this: 2026-10-29-vla-jepa-surprise-robot-day

Evaluation and judges, 14 of 14. ← Evaluation and judges 13: How I re-test robot-learning papers: the gate protocol as it ran

HWM (hierarchical latent world models)

Wed, 28 Oct 2026 00:00:00 +0000

HWM (hierarchical latent world models)¶

They said a second, coarser world model predicts further ahead than the fine one beyond about one second. On our data we trained both levels and compared their errors up to 2 seconds. Not proven: the coarse level never beat the fine one, with error 0.759 against 0.563 at 2 seconds.

Papers tested · 16 of 22

Goal (the paper's): Plan long manipulation tasks with a two-level world model: a high level over learned macro-actions proposes waypoints, a low level plans primitive actions toward them; shown on a real Franka trained on about 130 hours of DROID and RoboSet.

Its hypotheses:

Beyond about one second, the high-level model predicts the future latent better than the low-level model. (Fig. 6 (crossover near 1.25 s)) not proven
Learned macro-actions align with the true motion better than a handcrafted delta-pose summary. (Table 4b (cosine 0.88 vs 0.80)) reproduces
Hierarchical planning succeeds on real pick-and-place where flat planning does not. (§3.1, Table 1 (70 % vs 0 %)) not tested here

Verdict: not proven · Claim tested: A high-level world model over learned macro-actions beats the low-level model beyond about 1 s (their Fig. 6 crossover). · Bar: Crossover in [1.0, 2.0] s; at 2 s high-level error ≤ 0.95 × low-level; learned macro-actions ≥ net joint displacement on cosine. · n: 700 val anchors, 25 episodes (157 train, ~0.6 h)

Theirs vs ours: low level better up to 1 s, high level from 1.5 s (Fig. 6; ≈ 0.367 vs 0.385 at 2 s, read off the plot); cosine 0.88 ± 0.03 vs 0.80 ± 0.02 vs no crossover: W1 0.382 ± 0.005 → 0.563 ± 0.007, high level 0.769 ± 0.009 → 0.759 ± 0.009 (0.73–2 s); cosine 0.608 ± 0.007 vs 0.551 ± 0.006 · Not proven: Open confound: ~0.6 h of one task vs their ~130 h, so data scale is not separated from the method. Baselines and the overfitting reading are post-hoc.

Paper: arXiv 2604.03208 · Code: kevinghst/HWM_PLDM (minimal implementation on PLDM, linked from the project page)

Claims:

not proven high-level world model below low-level world model on latent error at 2 s — theirs 0.367 vs 0.385 at 2 s · ours 0.759 vs 0.563 at 2 s (ratio 1.35) (paper §3.4, Fig. 6 (crossover between 1 s and 1.5 s; ≈ 0.367 vs 0.385 at 2 s, read off the plot))
reproduces learned macro-actions at least net joint displacement on cosine to the true macro-action — theirs 0.88 vs 0.80 · ours 0.608 vs 0.551 (paper §3.4, Table 4b (0.88 ± 0.03 vs 0.80 ± 0.02))

Compute: External GPU: yes (vast.ai RTX 3090): ≈ $0.4, including the invalid first run. Latent extraction on the Orin NX: $0. · Reproduce: 2026-10-28-paper-vs-hardware-hwm · First published: 2026-10-28 · Updated: 2026-10-28 · Findings: I need a long-horizon latent predictor →, I need to encode an action chunk →

HWM (Zhang, Terver, Zholus et al., FAIR at Meta / NYU / Mila / Brown; CoRL 2026, arXiv 2604.03208) stacks two latent world models: a low level over primitive actions, and a high level over learned macro-actions that jumps several seconds at once. The figure I could test offline is the one that says when each level wins. On my data the high level never wins, and I can't tell yet whether that is the method or the amount of data.

What the paper claims, and how it measures it¶

A transformer action encoder compresses the actions between two waypoints into one macro-action (its CLS token). The high-level predictor F⁽²⁾ takes (macro-action, waypoint latent) and predicts the next waypoint latent, trained with a teacher-forced ℓ1 loss on three waypoints sampled up to 4 s apart. At planning time the high level picks a subgoal and the low level plans primitive actions toward it.

The real-robot results are on a Franka Panda with a Robotiq gripper, trained on about 130 h of DROID and RoboSet: pick-and-place 70 % against 0 % for the flat planner. Figure 6 is the offline part: on held-out observations, the low-level model has the lower latent error up to about 1 s, the high level from about 1.5 s, and the curves cross near 1.25 s. Table 4b adds that learned macro-actions align better with the true motion than a handcrafted delta-pose summary (cosine 0.88 vs 0.80).

What I ran¶

Low level: W1, my multi-view LeWM, unchanged. High level: the paper's §2.1 recipe trained on W1's frozen latents, on my bowls data (182 episodes, 157 train / 25 val, about 0.6 h). Three versions that differ only in the action input: the learned encoder, the concatenated raw actions, and the net joint displacement over the chunk (my stand-in for delta-pose). Error is mean |ẑ − z| over W1's 192 latent dimensions on 700 val anchors.

Three bars, frozen before training: H1 a crossover between 1.0 and 2.0 s; H2 at 2 s, high-level error ≤ 0.95 × W1's; H3 learned macro-actions ≥ net displacement on cosine alignment.

The first run was wrong: my extraction script fed W1 the red channel only, three times. A check against my earlier W1 numbers caught it before scoring (W1 looked four times worse than copying the last frame, which it never was). I fixed it, checked that the new latents match the earlier W1 numbers, retrained, and scored only the second run.

Their number, my number¶

horizon	0.73 s	1.0 s	1.5 s	2.0 s
low level (W1 rollout)	0.382 ± 0.005	0.438 ± 0.006	0.511 ± 0.006	0.563 ± 0.007
high level, learned macro-actions	0.769 ± 0.009	0.766 ± 0.009	0.764 ± 0.009	0.759 ± 0.009
high level, net joint displacement	0.790 ± 0.005	0.793 ± 0.006	0.793 ± 0.006	0.799 ± 0.007

	Paper	This arm
Crossover (H1)	~1.25 s	none: the high level is above W1 at every horizon
2 s ratio, high / low (H2)	≈ 0.95	1.35
Learned vs handcrafted cosine (H3)	0.88 ± 0.03 vs 0.80 ± 0.02	0.608 ± 0.007 vs 0.551 ± 0.006
Training data	~130 h, DROID + RoboSet	~0.6 h, one task

H1 fails, H2 fails, H3 holds. The SEs are over anchors, and anchors from the same episode are correlated, so they are on the small side.

Added after seeing the result¶

None of this was in the protocol. I computed it to understand the numbers, and it should be read that way.

Copying the current latent gives 0.613 at 0.73 s and 0.954 at 2 s. Predicting the average training latent gives 0.81 at every horizon. The learned high level sits at 0.76, flat: barely better than the average, and better than copying only from about 1.3 s on.
Its last training loss was 0.15. That loss is not the same measurement as the val error, so the gap to 0.76 points to overfitting on 157 episodes without measuring it.

What the gap means¶

My reading: the high level didn't learn anything that generalises from 0.6 h of one task, and W1, trained end to end on the same data, did. That fits "the high level needs far more data than the low level" just as well as "there is no crossover on this arm", and this run can't tell them apart. So the verdict is not proven, not fail. H3 is the part that holds even here: a learned summary of the actions beats the handcrafted one.

I had one reason to care beyond the paper. My G1b gate ranked action chunks at 0.17–1.0 s, which by Figure 6 is where the low level should win anyway. If the crossover had shown up here, G1b would have been measured in the wrong band for a hierarchy argument. It didn't show up, but not cleanly enough to close that question either.

What would change it: the same high level trained on a lot more data, or on top of a pretrained multi-task latent. Neither is planned.

Compute. External GPU: yes (vast.ai RTX 3090): ≈ $0.4, including the invalid first run. Latent extraction on the Orin NX: $0.
Reproduce this: 2026-10-28-paper-vs-hardware-hwm

Papers tested, 16 of 22. ← Papers tested 15: Real-Time Chunking · Papers tested 17: LeJEPA identifiability →

Audit your labels where they matter: a VLM labeller that is right on average

Mon, 26 Oct 2026 00:00:00 +0000

Audit your labels where they matter: a VLM labeller that is right on average¶

We wanted to know whether labels that look fine on average are right where a metric needs them. We checked a vision-language labeller's failure onsets, frame by frame, on 5 episodes against the video. The median error was 6 frames, but on retried grasps it was up to +289 frames late.

Evaluation and judges · 12 of 14

Type: method · Result: VLM onsets are +6 frames on median but +110 / +289 on retries; all judge errors were recordings past the reset. · n: 5-episode spot check; 88 rollouts

Not proven: The spot check is n = 5.

Compute: External GPU: no. External API: Gemini onset labels, cents; the judge ran on a Mac. · Reproduce: 2026-10-23-paper-vs-hardware-reap · 2026-10-23-paper-vs-hardware-robometer · First published: 2026-10-26 · Updated: 2026-10-26 · Findings: Audit your labels where they matter →

Two evaluation problems on my SO-101 rollouts had the same shape: the labels were fine on average and wrong exactly where the metric looked.

Failure onsets marked by a VLM. The median error was 6 frames, but the error was 110 and 289 frames on the episodes a failure monitor most needed.
A deterministic success judge. It missed its accuracy bar, and every one of its errors was a recording that ran past the end of the task.

Neither showed in an aggregate number. Both showed when I looked per failure type and per error.

Labels 1: failure onsets from Gemini¶

A failure monitor is scored on when it warns. For that it needs an onset frame per failed episode: the moment the attempt went wrong.

Setup. I had 25 failed rollouts. By owner decision, logged as a protocol deviation before labelling, Gemini 3.8 Flash marked the onsets instead of a person.
How it labelled. A coarse-to-fine contact-sheet search, told that the attempt failed and given the operator's failure note. 52 calls, temperature 0, 25 of 25 labelled.
The check. A spot check against the video, to the frame, on 5 episodes:

episode	failure	Gemini	by eye	offset
2	missed grasp, retried	180	174	+6
18	missed grasp	349	239	+110
30	wrong object (released beside)	201	200	+1
36	missed grasp	418	129	+289
66	wrong object (went to pink)	7	10	−3

The median offset is +6 frames (0.2 s), which reads as "good enough". It is not one error, though. Where the failure is a single event (release beside, wrong bowl from the start, one miss), Gemini is within 6 frames. Where the arm missed and tried again, Gemini marked a later retry, not the first missed closure: 3.7 s and 9.6 s late.

That matters because of what the labels were for. A monitor that warns between the first miss and the retry looks "early" against a late onset. The bias moves every positive later, in the direction that flatters the monitor.

The REAP monitor scored on these onsets (AUROC 0.93) went into the register as not proven, pending the onset check, with episodes 18 and 36 queued for hand relabelling.
The second version of that monitor removed onsets entirely: a ramp target, and every episode cut to a common horizon of 374 frames. It found no monitor at the bar either.

Labels 2: where does the recording end?¶

The second case is a deterministic judge for the bowls task. It uses colour masks on the top camera plus geometry, with no learning. On 88 labelled rollouts the pre-registered bar was accuracy ≥ 95 % and AUROC ≥ 0.95. It scored 86.4 % [78.4, 93.2], AUROC 0.919.

The breakdown was strange in a useful way. All 12 errors were successes called failure; there were no false positives. The error frames showed the scene being undone: bowls apart again, the operator's hands, the arm still moving.

The judge was looking at the last frame of the recording. The protocol meant the end of the task. For many successes, the operator had started resetting the scene before the recording stopped.

On 2026-10-14, before this judge existed, 11 Orin episodes had been marked with a reset_start by eye, all successes. 9 of the 12 errors were among them.
Judged at the marked reset (exploratory, same frozen thresholds), the same judge scores 95.5 % [90.9, 98.9], AUROC 0.967, still with 0 false positives.

The fix that went into the next pre-registered version (G9c-3) was a rule, not a label: judge at the first still moment after the last gripper release, or at the timeout. Coverage was 98.9 %, accuracy 94.3 % [88.5, 98.9], 0 false positives, and every failure type caught. That is a narrow fail, 0.7 points under the bar, reported as such. The remaining errors are unmarked resets or labels to recheck.

What is not proven¶

The onset spot check is n = 5. It shows the failure mode exists (retries) but not how often it happens across the 25. Episodes 18 and 36 are still waiting for hand labels.
The effect on monitor scores is not measured with corrected onsets. The monitor that replaced the onsets also failed its bar, so no verdict depends on them today.
The reset marks are by eye and exist for 11 Orin episodes only. M1 sessions have none.
The judge's last 0.7 points are attributed to unmarked resets or label errors, not verified.

The reusable check¶

Spot-check machine labels per failure type, not by the median. Pick at least one episode of each type, especially multi-attempt ones. A single median hides a bimodal error.
Ask which direction the label error moves your metric. Late onsets flatter early-warning monitors. If the bias favours your method, treat the result as not proven until the labels are fixed.
When a judge's errors are all one-sided, look at the frames it judged before tuning a threshold. Here they all came from recording past the end of the task.
Judge at a defined task event, not at the end of the file. The rule from the judge (g9c3_test.py) is a few lines:

opened = np.nonzero(st[:, 5] > rule["g_open"])[0] # gripper open frames t_rel = int(opened[-1]) if len(opened) else 0 # last release w = max(1, int(round(rule["still_s"] * fps))) # 0.5 s window jf, why = T - 1, "timeout" for t in range(t_rel + w, T): if np.abs(arm[t] - arm[t - w]).max() <= rule["still_tol_deg"]: # 1.5 deg jf, why = t, "still"; break

Links¶

Rollouts: ravediamond/so101-bowls-policy-rollouts (69 episodes, labelled).
Earlier posts:
the zero-false-positive judge;
ReAP.
Every test and verdict: the paper vs hardware page.

Compute. External GPU: no. External API: Gemini onset labels, cents; the judge ran on a Mac.
Reproduce this: 2026-10-23-paper-vs-hardware-reap · 2026-10-23-paper-vs-hardware-robometer

Evaluation and judges, 12 of 14. ← Evaluation and judges 11: VLA-JEPA on the Orin for $9.7, and why best-of-N reranking is a no-go · Evaluation and judges 13: How I re-test robot-learning papers: the gate protocol as it ran →

p99 is not a tail: latency numbers that lie about chunked policies

Mon, 26 Oct 2026 00:00:00 +0000

p99 is not a tail: latency numbers that lie about chunked policies¶

We wanted to know why our latency numbers showed a fast policy with occasional hiccups. We logged every call's latency and marked where each motion chunk began. Every slow call was a full chunk inference, and the SmolVLA loop was blocked 39 % of each trial.

Engineering on the Orin · 7 of 8

Type: method · Result: For chunked policies p99 is one chunk inference; SmolVLA's synchronous loop is blocked 39 % of each trial. · n: 10 trials per row

Not proven: Synchronous loop only.

Compute: External GPU: no (Orin NX, M1 Mac, arm). Cost: $0. · Reproduce: 2026-09-29-three-policies-so101 · First published: 2026-10-26 · Updated: 2026-10-26 · Findings: p99 is not a tail for chunked policies →

My first gate runs reported select_action latency the usual way: p50 and p99 per trial. For SmolVLA on a Jetson Orin NX, p50 was a few milliseconds and p99 was about a second. That reads like a fast policy with an occasional hiccup.

It is not. Every one of those slow calls was a full chunk inference, on a fixed schedule, and the control loop spent 39 % of every trial blocked on them. For chunked policies, p50 and p99 describe two different operations, not one distribution.

The symptom¶

A chunked policy predicts H actions at once (ACT: 100, SmolVLA: 50) and queues them. The next H − 1 calls to select_action only pop the queue. When the queue is empty, the next call runs the model.

In a synchronous loop the robot waits for that call. On the arm this showed as a visible pause at every chunk, and the control-rate table looked inconsistent: - ACT on the Orin was twice as fast per inference as on the M1 Mac; - SmolVLA ran at about 19 Hz with about 130 ms of jitter on both machines.

How it was caught¶

The harness logs one tick file per trial, with the latency of every select_action call and its timestamp. A short script (shared/gate/chunk_latency.py) does three things: - flags calls slower than 10 × the trial's median; - checks whether each flagged call falls on a chunk boundary (call 0, H, 2H, …); - plots the first 300 calls of one trial with the boundaries drawn.

The spikes are not a tail. They are the boundaries:

policy	machine	chunk	spikes on a chunk boundary	chunk inference	queue pop	p99	chunk inferences / s	sync loop blocked
ACT	M1	100	100 %	122 ms	5.86 ms	118 ms	0.32	4 %
ACT	Orin NX	100	100 %	59 ms	0.92 ms	58 ms	0.32	2 %
SmolVLA	M1	50	98 %	954 ms	2.42 ms	954 ms	0.41	39 %
SmolVLA	Orin NX	50	100 %	974 ms	4.29 ms	975 ms	0.40	39 %

10 trials of 25 s per row, 30 Hz loop, 2026-09-29. The two off-boundary SmolVLA spikes on the M1 (2 of 102) are isolated outliers.

What the right numbers say¶

p50 is the queue pop. p99 is one chunk inference.
The useful numbers are the time per chunk inference and how often it runs. Together they give how long the robot stalls and how often, and so the share of the trial the loop is blocked.
The control-rate puzzle resolves. SmolVLA is bound by its own inference (about 0.95–0.97 s on both machines), so faster hardware did not help it. ACT, at 59 ms per 100 actions, barely blocks at all.
The lever was not "more hardware". It was the engine, then asynchronous inference.

Moving SmolVLA from PyTorch to vla.cpp on the same Orin and checkpoint (2026-10-01, 10 trials each), the metrics that matter moved:

	SmolVLA PyTorch	vla.cpp defaults	vla.cpp `--flash-attn --mm-prec default`
chunk inference	974 ms	390 ms	238 ms
sync loop blocked	39 %	20 %	14 %
control Hz achieved	19.32	24.59	26.56
control jitter	130.7 ms	51.0 ms	29.8 ms
success (n = 10)	9/10	9/10	8/10

Spikes still sit on the boundaries (100 / 100 / 99 %). The stall is shorter, not gone. That is what led to running inference asynchronously, and to the chunk-boundary post.

What is not proven¶

Synchronous loop only. The asynchronous loop on the arm, where inference overlaps execution, has not been timed end to end. That timing run (G13a) has not happened.
n = 10 trials per row, one task, two machines. The percentages are stable across trials here, but this is not a benchmark suite.
Success at n = 10 says nothing either way about the engine change (9 vs 9 vs 8). The _fa flags change numerics, and its two missed grasps are flagged, not explained.

The reusable check¶

For any chunked policy, compute these from per-call logs instead of reporting p50 / p99:

def summarize(trials: list[dict], chunk: int) -> dict: spikes, rest, on_boundary, blocked, wall = [], [], 0, 0.0, 0.0 for t in trials: s = np.array(t["select_action_ms"]) idx = np.where(s > 10 * np.median(s))[0] on_boundary += int(np.sum(idx % chunk == 0)) spikes += s[idx].tolist() rest += np.delete(s, idx).tolist() ts = np.array(t["tick_ts"]) wall += ts[-1] - ts[0] blocked += s[idx].sum() / 1e3 return { "spikes": len(spikes), "on_boundary_pct": 100 * on_boundary / len(spikes), "chunk_ms": float(np.median(spikes)), "pop_ms": float(np.median(rest)), "p99_ms": float(np.percentile(spikes + rest, 99)), "chunks_per_s": len(spikes) / wall, "blocked_pct": 100 * blocked / wall, }

If on_boundary_pct is near 100, report chunk_ms, chunks_per_s and blocked_pct. Treat p99 as what it is: one inference.

Links¶

Earlier posts:
three policies on a €350 arm kit;
SmolVLA four times faster with vla.cpp.
Tick files and chunk_latency.py: in my research repo, public with the replication repo.
Every test and verdict: the paper vs hardware page.

Compute. External GPU: no (Orin NX, M1 Mac, arm). Cost: $0.
Reproduce this: 2026-09-29-three-policies-so101

Engineering on the Orin, 7 of 8. ← Engineering on the Orin 6: Why your VLA's actions jump at every chunk boundary on real hardware, and a 0.4 ms fix · Engineering on the Orin 8: A correctness fix upstream made my checkpoint 47 % worse →

A correctness fix upstream made my checkpoint 47 % worse

Mon, 26 Oct 2026 00:00:00 +0000

A correctness fix upstream made my checkpoint 47 % worse¶

We wanted to know if an upstream bug fix changed what our trained checkpoint computes. We ran its world predictor on its own validation data with the old and the fixed code, nothing else changed. Under the fix its loss was 47 % higher, worse on all 25 episodes.

Engineering on the Orin · 8 of 8

Type: engineering · Result: LeRobot's rotary fix raises a legacy checkpoint's world-model loss by 47 % (R = 1.473), worse on 25/25 episodes. · n: 200 windows, 25 episodes

Not proven: One checkpoint, in distribution.

Compute: External GPU: no (Orin NX, ~10 min). Cost: $0. · Reproduce: 2026-10-26-vla-jepa-rope-compat · First published: 2026-10-26 · Updated: 2026-10-26 · Findings: An upstream correctness fix made a checkpoint 47 % worse →

LeRobot fixed a real bug in VLA-JEPA's world predictor: its rotary position embedding was not a rotation. I had a checkpoint trained before the fix. Run under the fixed code, its world-model loss is 47 % higher on its own training distribution, worse on all 25 episodes. The fix is right. The checkpoint was trained around the bug, so the fix silently changes what the checkpoint computes. This post is about checking for that before you upgrade.

The symptom¶

VLA-JEPA (LeRobot's port) has a world predictor next to the action head. It predicts future V-JEPA 2 latents and is trained with an L1 loss. LeRobot #4835 changed how the predictor expands its rotary frequencies:

# before #4835 emb_sin = freqs.sin().squeeze(-1).repeat(1, 1, 1, 2) emb_cos = freqs.cos().squeeze(-1).repeat(1, 1, 1, 2) # after #4835: each adjacent coordinate pair rotated by one frequency emb_sin = freqs.sin().repeat_interleave(2, dim=-1) emb_cos = freqs.cos().repeat_interleave(2, dim=-1)

On random tensors the difference is easy to confirm. The fixed version rotates each pair by one angle: pair norms are preserved to 8.9e-16, and the angle equals position × frequency to 1.3e-15. The old version does not preserve pair norms (error 2.19): the two coordinates of a pair get the cos and sin of different frequencies. So it was never a rotation.

My checkpoint, ravediamond/vla_jepa_so101_bowls_trimmed (an SO-101 fine-tune), was trained with the world model on, on the old code. LeRobot's docs say pretrained world-model quality "needs separate evaluation when continuing training". So I evaluated it.

How it was caught: one change, everything else held fixed¶

The protocol was frozen and committed before any fixed-rotary number on real data existed:

Data: the checkpoint's own 25 validation episodes, 8 windows each, start frames from a seeded generator per episode: 200 windows.
Sharing: for each window the V-JEPA 2 encoder and Qwen run once. Only the predictor runs more than once, on the same tensors.
Three arms:
legacy: the old expansion patched in, checked bit-identical to the pre-fix commit 2e88044 (max abs diff 0.0);
fixed: LeRobot main d3793b7;
copy_last: predict each next position with the current one.
Loss: exactly _world_model_loss's L1.
Statistics: paired bootstrap over episodes, 2,000 resamples.

Before the real comparison, three checks: 1. The legacy patch reproduces the pre-fix code exactly. 2. Two runs give identical losses. 3. The instrument separates a trained predictor from a random-init one. On 6 windows the trained predictor scored 1.14–1.26, random init 1.97–2.05, copy-last 2.20–2.28.

The bar had five labels: instrument fails, COLLAPSES, DEGRADES, SAFE, NOT PROVEN. DEGRADES needed R = fixed / legacy ≥ 1.10 with the CI of fixed − legacy above 0, and the fixed predictor still beating copy-last.

The measurement¶

	world-model L1 (mean over episodes)
`legacy`	1.175
`fixed`	1.731
`copy_last`	2.235

statistic	value	95 % CI
R = fixed / legacy	1.473	[1.462, 1.484]
fixed − legacy	+0.556	[0.547, 0.564]
fixed − copy_last	−0.505	[−0.508, −0.501]
legacy − copy_last	−1.060	[−1.069, −1.051]

Verdict: DEGRADES.
Episodes: fixed is worse on 25 of 25.
Position: the gap does not grow with distance from the context. It costs 0.53–0.61 at each of the three predicted positions, and about the same at the start, middle and end of episodes.
Not a collapse: the fixed predictor still beats copy-last, so it has not forgotten everything.
What it means for fine-tuning: continuing from this checkpoint on current main starts the world-model loss about half way back to the copy-last baseline.

The decision¶

The protocol fixed the consequence in advance: - My own fine-tunes from legacy checkpoints pin the old expansion, and say so in the model card. - I drafted an opt-in flag for upstream (world_model_rope_legacy: bool = False). Old checkpoints set it in their config, new training keeps the fix, and nothing changes by default.

Upstream V-JEPA 2 made the same call for its released checkpoints (facebookresearch/vjepa2#75 kept the old expansion). The draft is not posted yet.

What is not proven¶

One checkpoint, one small fine-tune. The LIBERO checkpoint (lerobot/VLA-JEPA-LIBERO) was not run: its files and dataset did not fit on the Orin's disk. The effect on the large pretrained model is unmeasured.
In distribution only. The windows come from the checkpoint's own training data. This is a compatibility test (does the predictor still do what it learned), not a generalisation test.
No closed-loop effect. My arm runs VLA-JEPA with the world model off at inference, so the robot's success rate is not affected. Fine-tuning is.
Recovery speed unmeasured. I did not measure how many fine-tuning steps it takes to recover under the fixed rotary.

The reusable check¶

When an upstream fix touches model code your checkpoint was trained with, run this before upgrading. It is an afternoon.

Freeze the bar first. What ratio counts as harmless, and what counts as degraded.
Re-create the old behaviour as a patch, and prove it bit-identical to the old commit on a few inputs.
Share everything upstream of the change. Run the unchanged parts once per sample and feed the same tensors to old and new. Then the only difference is the change.
Add a trivial baseline (copy-last, mean prediction) and require the old code to beat it. Otherwise your instrument cannot see anything.
Pair by episode and bootstrap over episodes, not windows.

Steps 2 and 3 in the evaluation script (eval_rope_compat.py): the old expansion is swapped in at module level, and each window is encoded once and predicted under both (excerpt; the per-position fields are trimmed).

def set_mode(mode): import lerobot.policies.vla_jepa.world_model as W if not hasattr(W, "_rope_fixed"): W._rope_fixed = W.rotate_queries_or_keys W.rotate_queries_or_keys = rope_legacy if mode == "legacy" else W._rope_fixed for e, s in windows(spans, val, a.windows_per_episode, d[-1]): i, g, ac, tpf = encode(pol, pre, ds[spans[e][0] + s]) # encoders + Qwen once pl, pf = predict(pred, i, ac, "legacy"), predict(pred, i, ac, "fixed") rows.append({"episode": e, "legacy": l1(pl, g), "fixed": l1(pf, g), "copy_last": l1(i, g)})

Links¶

LeRobot #4835.
vjepa2#75.
Data: ravediamond/so101_bowls_all_trimmed.
Code and checkpoint: in my research repo, which goes public with the replication repo.
Every test and verdict: the paper vs hardware page.

Compute. External GPU: no (Orin NX, ~10 min). Cost: $0.
Reproduce this: 2026-10-26-vla-jepa-rope-compat

Engineering on the Orin, 8 of 8. ← Engineering on the Orin 7: p99 is not a tail: latency numbers that lie about chunked policies

Why your VLA's actions jump at every chunk boundary on real hardware, and a 0.4 ms fix

Sun, 25 Oct 2026 00:00:00 +0000

Why your VLA's actions jump at every chunk boundary on real hardware, and a 0.4 ms fix¶

We wanted to know why the arm's commands jump each time a new motion chunk takes over. We measured the jump offline on 254 chunk boundaries, then tried overwriting the new chunk's start. The jump fell to the size of a normal step, at 0.4 ms per chunk on the Jetson.

Engineering on the Orin · 6 of 8

Type: engineering · Result: A prefix overwrite cuts the chunk-boundary jump 71–73 % against naive async at +0.4 ms; tuned LeRobot identity is close. · n: 254 boundaries per delay, 20 episodes

Not proven: Task success on the arm is not measured.

Compute: External GPU: no (Orin NX). Cost: $0. · Reproduce: 2026-10-25-vla-chunk-boundary-jumps · First published: 2026-10-25 · Updated: 2026-10-26 (LeRobot RTC baselines added) · Findings: I need smoother chunk boundaries →, Prefix conditioning in vla.cpp →, Prefix overwrite at chunk boundaries (offline) →

Run a chunked VLA asynchronously on a real arm and the command jumps every time a new chunk takes over. On my SO-101 with SmolVLA, the step at the switch was a median 3.4 times a normal step, and on a test boundary served by the real engine the jump was 58.7°. Against naive async, a gradient-free prefix overwrite inside the denoising loop brings the jump down to the size of a normal step, adds 0.4 ms per chunk on a Jetson Orin NX, and leaves the default path bit-identical. LeRobot's own gradient-free RTC mode, tuned, gets close on the jump; the overwrite's edge over it is no parameter to tune and an exact prefix. It has not been tested for task success on the arm yet. This post covers what I measured, and what I did not.

The symptom¶

SmolVLA predicts a chunk of 50 actions at a time. Run synchronously, the arm waits for every chunk. Run asynchronously, the next chunk is computed while the current one is still executing, and the arm switches to it when it arrives.

I measured the switch offline on recorded episodes: 20 episodes of ravediamond/so101_bowls_all, 254 boundaries per setting, inference delays of d = 8, 12 and 20 control steps. The metric is the jump ratio: the size of the joint step at the switch divided by the median step inside the chunk. 1.0 means the switch looks like any other step.

Naive asynchronous execution gave a median jump ratio of 3.39 / 3.38 / 3.68 (d = 8 / 12 / 20) through vla.cpp, and 3.42 / 3.69 / 3.77 in PyTorch. On a test boundary served by the real vla-server (episode 2, d = 8), the jump at the switch was 58.7°. With the fix it was 1.4°.

Why it happens¶

The new chunk is sampled from the observation at request time. By the time it arrives, d steps have passed and the arm has executed d more actions of the old chunk. The new chunk also starts from fresh noise. Nothing in the sampling ties its first rows to the actions that are still executing, so its row d can pick a different trajectory from where the old chunk left the arm.

chunk k |== executed ==|-- d steps of inference --|-- leftover (not executed) --| ^ request (state at t) ^ chunk k+1 arrives chunk k+1 |-- rows 0..d-1 (skipped) -|-- row d onward executes ---> ^ switch: row d of k+1 vs last executed row of k

The leftover is the part of chunk k that would have run next. Naive execution throws it away and jumps to row d of chunk k+1.

Budget first¶

None of this matters if the chunk does not arrive in time. With 50-action chunks and a new chunk started after 30 actions, the next one has to arrive within H − s = 20 steps, about 667 ms at 30 Hz.

SmolVLA in PyTorch on the Orin NX took 974 ms per chunk, so it does not fit with any correction. In the offline study, naive PyTorch was 1028 ms, the gradient-free variants about 1032–1039 ms, and RTC's exact guidance 1962 ms. Served by vla.cpp with --flash-attn --mm-prec default, the same checkpoint takes 238 ms per chunk, inside the budget with room to spare. So the fix had to live in the engine.

The fix¶

Inside SmolVLA's Euler denoising loop, I correct the predicted clean chunk towards the leftover at every step. With x_t the current sample, v the velocity and t the flow time:

x1 = x_t − t·v v ← v − W ⊙ (Y − x1) / t

Y is the leftover, normalised with the checkpoint's action stats.
W is Real-Time Chunking's soft prefix weight (Black, Galliker, Levine; arXiv 2506.07339). It is 1 on the first d rows and decays as c·expm1(c)/(e − 1) to 0 at the end of the leftover, over the real action dimensions.
Without a leftover, W = 0 and the update does nothing. With fixed inputs and noise, the default path is bit-identical to the existing build (max |Δ| = 0.0), and vla.cpp's 14 tests pass on the Orin.

There are no gradients. prev_actions (the not-yet-executed rest of the previous chunk) and inference_delay are new fields in the engine inputs, the C API, the Python bindings and the server's request. The change is on my fork, branch feat/smolvla-prefix-overwrite, commit 805570d, on top of vla.cpp.

Results¶

Medians over 254 boundaries per cell, 20 episodes, degrees, five arm joints. The vla.cpp rows go through the engine; the PyTorch rows are the same study in LeRobot, including its RTC modes: identity (LeRobot's gradient-free guided mode) and exact (vector-Jacobian product, RTC as published), at guidance weight β. Prefix error is how far the first d rows of the new chunk are from the leftover. Drift is the distance to the recorded actions, a check that the correction does not drag the chunk away from the demonstrations.

	d	jump ratio	accel ratio	prefix error	drift
vla.cpp naive	8 / 12 / 20	3.39 / 3.38 / 3.68	3.63 / 3.91 / 4.63	1.60 / 1.69 / 1.88	1.51 / 1.55 / 1.51
vla.cpp overwrite	8 / 12 / 20	0.98 / 0.90 / 1.05	1.21 / 1.19 / 1.29	0 / 0 / 0	1.50 / 1.50 / 1.45
PyTorch naive	8 / 12 / 20	3.42 / 3.69 / 3.77	3.81 / 4.13 / 4.27	1.65 / 1.69 / 1.91	1.48 / 1.53 / 1.53
PyTorch overwrite	8 / 12 / 20	0.93 / 0.96 / 1.06	1.23 / 1.26 / 1.34	0 / 0 / 0	1.49 / 1.55 / 1.49
PyTorch identity, β = 5	8 / 12 / 20	1.26 / 1.19 / 1.34	1.50 / 1.54 / 1.67	0.48 / 0.45 / 0.42	1.47 / 1.50 / 1.57
PyTorch identity, β = 10	8 / 12 / 20	0.95 / 1.02 / 1.12	1.26 / 1.28 / 1.33	0.08 / 0.08 / 0.08	1.51 / 1.52 / 1.62
PyTorch exact RTC, β = 5	8 / 12 / 20	2.43 / 2.55 / 2.99	2.83 / 3.04 / 3.77	0.99 / 0.98 / 0.97	1.57 / 1.61 / 1.60

Against naive async, the jump drops by 71.0 / 73.3 / 71.5 % through vla.cpp (72.8 / 74.0 / 71.7 % in PyTorch). The switch now looks like an ordinary step.
Drift is unchanged.
The prefix matches the leftover exactly: max prefix error 0.0 over 762 boundaries.
Cost: +0.4 ms per chunk, measured as 371.0 → 371.4 ms wall per call from Python. That includes copying two 512² images; the engine's own inference is the 238 ms above.

Against the real server, the first 8 rows of the corrected chunk match the leftover to 1e-6°. The naive chunk was off by up to 47° on the same rows.

Against LeRobot's RTC modes¶

The 72 % is against naive async. LeRobot already ships RTC, and its gradient-free identity mode, tuned to β = 10, reaches jump ratios of 0.95 / 1.02 / 1.12 against the overwrite's 0.93 / 0.96 / 1.06 in PyTorch. On the jump alone the gain over a tuned identity mode is small. What the overwrite adds is no β to tune (at β = 5 identity is clearly worse: 1.26 / 1.19 / 1.34), an exact prefix (0 against 0.08), and no gradient, which is what made it portable into vla.cpp. Exact RTC cost 1.9x per chunk in PyTorch (1962 against 1028 ms) and did worse than both gradient-free modes here. That is the opposite of the RTC paper's ordering, and I have not ruled out a problem in my VJP path, so I do not read it as a verdict on RTC.

How it differs from RTC¶

RTC solves the same problem with guidance: one vector-Jacobian product through the policy per denoising step. On π0.5 it costs 14 → 35 ms for 5 denoising steps, 76 → 97 ms per chunk (RTC, App. A.3, Table 3). In vla.cpp that path is not available today, because ggml lacks backward rules for flash attention and concat.

RTC does test an inpainting variant in its appendix (A.4, Fig. 8). That variant overwrites the noisy prefix with forward-noised committed actions under a hard mask on the first d rows. It ran in simulation only (Kinetix), gave "some benefit" over naive async, and was beaten by guided RTC on solve rate. RTC's real-robot runs compare guided RTC, synchronous, naive async and temporal ensembling only; no overwrite variant ran on hardware.

Mine is a different variant. It corrects the predicted clean chunk with RTC's soft mask over the whole overlap, which amounts to guidance with an identity Jacobian at weight 1/t. So the Kinetix ranking does not transfer as is, in either direction.

What is not proven¶

This is smoothness, measured offline on recorded episodes through the real engine. Task success on the arm has not been measured for any of these modes: not for the overwrite, not for LeRobot's identity or exact RTC. That test is registered as G13a and has not run. The comparison with identity and exact RTC is offline and in PyTorch only; neither runs in vla.cpp.

Forcing the new chunk to agree with the old one could hurt success: if the old chunk was heading the wrong way, the new one now follows it for d steps. If G13a shows that, the fallback is training-time action conditioning (arXiv 2512.05964), which fine-tunes the policy to take a clean prefix instead of correcting it at inference.

Try it¶

The model is the SmolVLA checkpoint ravediamond/smolvla_2026-09-24_14-34-40 (step 20000), converted to GGUF with vla.cpp's scripts/convert_smolvla_to_gguf.py. The data is ravediamond/so101_bowls_all. Build the branch on the Orin with -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=87 -DCMAKE_BUILD_TYPE=Release -DVLA_SPM=OFF. The boundary study ran as:

VLA_LIBRARY=~/workspace/git/vla.cpp/build-orin/libvla.so PYTHONPATH=~/workspace/git/vla.cpp/bindings/python \  python scripts/check_vlacpp_prefix_overwrite.py \  --gguf ~/vla-models/smolvla-so101-bowls-020000.gguf \  --episodes-from shared/gate/data/2026-10-19_g13b_offline/g13b_summary.json --out outputs/vlacpp_overwrite

The check script and the per-boundary rows live in my research repo, which is not public yet.

Every paper I test this way, with its bar and verdict, is on the paper vs hardware page.

Updated 2026-10-26: added LeRobot's identity and exact RTC modes as baselines.

Compute. External GPU: no (Orin NX). Cost: $0.
Reproduce this: 2026-10-25-vla-chunk-boundary-jumps

Engineering on the Orin, 6 of 8. ← Engineering on the Orin 5: Porting ACT to vla.cpp, so a Jetson can run it without PyTorch · Engineering on the Orin 7: p99 is not a tail: latency numbers that lie about chunked policies →

ContactGuard

Fri, 23 Oct 2026 00:00:00 +0000

ContactGuard¶

They said a world model can predict a failed grasp before contact by imagining the next moments. On our arm we built the same monitor and compared it with the current camera view. It did not hold: imagining scored 0.70, the plain current view 0.82.

Papers tested · 6 of 22

Goal (the paper's): Detect grasp failures before contact for a chunked policy, by rolling an action-conditioned latent world model forward under the chunk the policy already emitted and reading the imagined latent with a small probe; shown on a dual-arm AgileX Piper with ACT.

Its hypotheses:

A probe on the imagined latent after the grasp predicts grasp failure better than a probe on the current latent. (Table 2 (0.98 vs 0.84)) fail
The monitor can abort a grasp before contact, without access to the policy. (§3.5 (online pre-contact monitor), §1) not tested here

Verdict: fail · Claim tested: An event-anchored probe on imagined latents predicts grasp failure better than the current latent. · Bar: AUROC ≥ 0.85 and ≥ 0.05 above the current latent. · n: 65 scored rollouts, 21 failures

Theirs vs ours: AUROC 0.98–0.99 imagined vs 0.84–0.92 current over four grasp tasks (Cup 0.98 vs 0.84) vs 0.70 imagined vs 0.82 current · Not proven: Mixed policies and two views against their single policy and three views; corruption control pending.

Paper: arXiv 2608.13438 · Code: no code link found

Claims:

fail imagined-latent probe above current-latent probe on failure AUROC — theirs AUROC 0.98–0.99 vs 0.84–0.92 over four grasp tasks (Cup 0.98 vs 0.84) · ours 0.70 vs 0.82 (paper Table 2 (ROC AUC, Cup .982 vs .840))
reproduces current-latent probe above imagined-latent probe on failure AUROC · ours 0.82 vs 0.70 (our measured comparison)

Compute: External GPU: no (Orin NX). Cost: $0. · Reproduce: 2026-10-23-paper-vs-hardware-contactguard · First published: 2026-10-23 · Updated: 2026-10-23 · Findings: I need a probe on a world model's latents →

ContactGuard (Zheng, Johnson-Roberson, Zhi; arXiv 2608.13438) is the closest paper to what I run: a LeWM-style world model used as a pre-contact failure monitor on a real arm. Their monitor works. Mine, built to the same protocol, does not, with the opposite sign.

What the paper claims, and how it measures it¶

Shared ViT-tiny per camera, mean-pool three views then a linear projection → one latent; predictor of four AdaLN-zero blocks; LeWM loss trained teacher-forced on unlabelled rollouts and teleop; then frozen. The probe: anchor 15 frames (0.5 s) before the first gripper-closure command, roll out K = 30 steps under the policy's actual chunk, standardise the imagined latent, fit a class-balanced ℓ2 logistic regression on ~250 labelled grasp attempts per task, freeze one threshold per task on validation. Online: trigger on the first open→close in the upcoming chunk, abort before contact if P(fail) > τ.

Evaluation on an AgileX Piper with an ACT policy, four grasp settings, test episodes excluded from everything. Closed loop: 50 monitored live rollouts per setting with a 5-s pause-and-resume so the would-be outcome is recorded; AUC 0.99 / 0.95 / 0.90 / 0.92 vs a single-view LeWM and a no-rollout probe. Offline pools (n = 101–144, five seeded 5-fold CV): ContactGuard 0.98 vs current latent 0.84 — same encoder, labels, probe, pools, only the imagined future replaced by the present one — with paired bootstrap CIs excluding zero on every task. Action corruption (Table 3): shuffled chunks from other episodes collapse the full model from 0.965 to 0.535; zero and mean actions to ~0.84. So the signal is the specific action's imagined consequence. A counterfactual action swap raises P(fail) by +0.25 to +0.65. Proprioception added to the probe hurts on all four (small-data shortcut). Runtime 19 ms on a 5090.

What I ran¶

W1-C1 (2026-10-19). The protocol as written: anchor 15 frames before the first close, 30-step imagination under the executed chunk, logistic probe, current latent as the matched baseline, 5-fold × 5 seeds on the 69 published rollouts (65 scored; two stalls and two others never close). Bar: AUROC ≥ 0.85 and ≥ 0.05 above the current latent. Result: imagined 0.70 (folds 0.72 ± 0.13), current 0.82 (0.83 ± 0.11). Imagination is 0.12 worse. The fold threshold gives 27 % held-out false aborts.

W1 itself follows the multi-view ContactGuard recipe (mean-pooled views), which is why the comparison is as direct as it gets.

Their number, my number¶

	Paper (Piper, ACT)	This arm (SO-101, mixed policies)
Imagined-latent probe	0.98	0.70
Current-latent probe	0.84	0.82
Difference	+0.14, CI excludes 0	−0.12
Labelled attempts	~250 per task	65
Views	3	2
World-model training data	rollouts + teleop	teleop only
Arm	torque-controlled	position-controlled servos

What the gap means¶

The current-latent probes agree (0.84 vs 0.82): the encoder reads "this state looks risky" about as well on both arms. The imagined-latent probes do not: theirs adds information, mine subtracts it. The candidates, none proven: their world model saw the deployed policy's own rollouts in training and mine saw only teleop, so my imagination is off-distribution exactly at the states the monitor is for; 65 labels vs 250; two views vs three; and a torque-controlled approach that carries contact information a position-controlled one does not. The paper gives the control that would say which: Table 3's action corruption. If shuffled chunks do not push my imagined-latent AUROC below 0.70, the rollout carries no action information on my data and the encoder, not the probe, is the gap. That test is offline and free and is now in the amendments.

Verdict: does not reproduce; the matched baseline wins on my arm; mechanism unknown.

What would change it: the corruption test, then W1 retrained with policy rollouts in the mix.

Compute. External GPU: no (Orin NX). Cost: $0.
Reproduce this: 2026-10-23-paper-vs-hardware-contactguard

Papers tested, 6 of 22. ← Papers tested 5: D-JEPA · Papers tested 7: FAIL-Detect and SAFE →

D-JEPA

Fri, 23 Oct 2026 00:00:00 +0000

D-JEPA¶

They said a small ranking add-on, trained on the outcome of every candidate, picks better motions than the world model. We trained it on our recorded runs with stand-in labels, not with real outcomes. Not tested: with those labels the regret stayed at 0.263, the same as the world model alone.

Papers tested · 5 of 22

Goal (the paper's): Close the gap between a world model's global predictive geometry and the decision among the few candidates that compete for execution, with a set operator that corrects each candidate's score, trained on counterfactual execution of every candidate.

Its hypotheses:

Among the top few candidates, latent goal distance misorders successes and failures. (§3, Fig. 2c–d) not tested here
A permutation-equivariant ranking operator on frozen world models raises planning success over the base models. (§4 (operator), §6, Tables 1–2) not tested

Verdict: not tested · Claim tested: A decision-aligned ranking operator on a frozen world model picks better action chunks. · Bar: G1b regret ≤ 0.10 and ≤ the action-only baseline + 0.05. · n: 703 val anchors

Theirs vs ours: +4 to +17 points vs regret 0.263 (W1 alone 0.263; action-only −0.007) · Not proven: Proxy labels (executed chunk = success); the executed-candidate version is not run.

Paper: arXiv 2609.24749 · Code: NEBULIS-Lab/D-JEPA

Claims:

not tested decision-aligned ranking operator below latent world-model ranking on G1b regret — theirs +4 to +17 points · ours regret 0.2631 (W1 alone 0.2630) (paper §6, Tables 1, 2 and 4 (+4 to +17 points))

Compute: External GPU: no (Orin NX). Cost: $0. · Reproduce: 2026-10-23-paper-vs-hardware-d-jepa · First published: 2026-10-23 · Updated: 2026-10-23

D-JEPA (Liu, Wu, Wen, You, Zhang, Hao, Lin, Su; arXiv 2609.24749) names my strongest negative result — latent distance misorders the few candidates that compete for execution — and proposes a fix. I ran the fix with the wrong labels. This post says what the paper's test needs and why mine was not it.

What the paper claims, and how it measures it¶

The "decision-local prediction gap": a world model's latent geometry can be globally informative (Spearman 0.90 between predicted and realised cost over all 63 candidates on a Push-T start) and still misorder the top four (0.11). The fix is not a training loss. It is a permutation-equivariant set operator over the whole candidate set: per candidate, a descriptor from the terminal latent minus the goal and its native rank; a shared encoder, two Set-Transformer layers, and a bounded correction δ = ε·tanh(·) with ε = 0.2, so the operator can only reorder within 2ε of the native minimum and can never overturn a confident choice (Prop. 2). Loss: success mass under a softmax over scores, a local ordering term near the boundary, and a trust penalty on unnecessary correction.

The supervision is the expensive part: every candidate in the set is executed from the restored start and labelled — counterfactual execution. Evaluation: Push-T confirmation cohort of 256 starts (Reacher 128, Granular 64) with fitting, calibration and evaluation starts disjoint, Wilson intervals in the figures, paired counts; Push-T 83.6 → 87.9, Reacher 86.7 → 93.8. Real PiPER with a V-JEPA 2-AC planner, 50 paired starts per task: Push-T 56 → 74, two-cube stack 72 → 88 (gains/losses 12/3 and 10/2).

What I ran¶

W1-R (2026-10-22). The operator exactly as App. A.2 describes it, on frozen W1's G1b candidates, five seeds, τ calibrated on 20 % of training episodes. Labels: proxy — the executed expert chunk marked success, the seven decoys marked failure. Bar: mean regret ≤ 0.10 and ≤ action-only + 0.05. Result: W1-R regret 0.2631 vs W1 0.2630; the gate chose "never replace" in 24 of 25 operators because W1 ranks its own training episodes at 0.67–0.91 and validation at 0.45–0.61, so calibration found nothing to fix. An exploratory perfect rank-2 swap would reach 0.115–0.154: the bar is out of reach for a rank-local operator with K = 8 even with perfect labels.

An earlier draft of the angle (NEXT_ANGLES §1) described D-JEPA as a loss term on the world model. Claude Code caught the error when the protocol had to cite the section; the implementation is the operator, the write-up says plainly that it does not test the paper's claim.

Their number, my number¶

	Paper	This arm
Labels	executed outcomes of every candidate	proxy: expert = success, decoys = failure
Candidate set	63 (sim), planner's pool (real)	8
Fit / calibration / eval	disjoint starts, Wilson CIs	train / 20 % cal / held-out episodes
Gain	+4 to +17 points	0.000 regret

What the gap means¶

Nothing about D-JEPA was tested. Proxy labels are a tautology for the operator (the "success" is whichever candidate the expert took), so the operator has nothing to learn that the native rank does not already express. The paper's claim needs the labels the paper pays for: the outcome of actually executing each candidate from the same start. On my arm that is affordable only in one place — the G7 cube session, where the undo primitive restores the layout and G9c-3 labels at rest; four to eight executed candidates per start for a subset of starts is two hours of a robot day. The bounded-correction design, independently of the result, is the right safety property for any re-ranker on a physical arm.

Verdict: not tested; the proxy version fails and is renamed. The paper's protocol is written into the amendments for G7.

What would change it: executed outcomes on ≥ 25 held-out starts.

Compute. External GPU: no (Orin NX). Cost: $0.
Reproduce this: 2026-10-23-paper-vs-hardware-d-jepa

Papers tested, 5 of 22. ← Papers tested 4: H-JEPA · Papers tested 6: ContactGuard →

DINO-WM

Fri, 23 Oct 2026 00:00:00 +0000

DINO-WM¶

They said a planner built on a large frozen image model plans to a goal image better than learned ones. On our arm we measured how well its features track the arm and rank candidate motions. Partly: it reads the arm's pose well (0.92 of the variance) but ranks motions worse than ignoring the camera.

Papers tested · 2 of 22

Goal (the paper's): Plan for manipulation and navigation with a world model built on frozen DINOv2 patch features, so that no task-specific visual representation has to be learned; a transition model predicts the next patch features and CEM plans toward a goal image, in six simulated environments.

Its hypotheses:

A transition model on frozen DINOv2 patches plans to goal images better than world models that learn their own representation. (Table 1 (Push-T 0.90 vs DreamerV3 0.30)) fail
Patch features, not a pooled global vector, carry the spatial detail manipulation needs. (Table 2 (encoder ablation)) not tested here

Verdict: fail · Claim tested: A predictor on frozen DINOv2 patches plans better than one on a learned latent. · Bar: G1b top-1 ≥ 0.8 at some target distance. · n: 25 val episodes, 703 anchors

Theirs vs ours: Push-T planning success 0.90 vs arm-pose R² 0.92 (vs 0.57 for W1); G1b top-1 0.55–0.61; no-model baseline 0.68–0.90 · Not proven: One training each (W2 lost, W2′ retrained); planning not run in closed loop.

Paper: arXiv 2411.04983 · Code: gaoyuezhou/dino_wm

Claims:

fail latent world-model ranking above action-only baseline on G1b top-1 — theirs Push-T planning success 0.90 · ours 0.61 vs 0.68–0.90 action-only (paper §3.2 and §4.3, Table 1 (planning with DINO-WM))
reproduces frozen-DINOv2 world model above JEPA world model on arm-pose R² — theirs not reported for an arm pose · ours 0.92 vs 0.57 (paper §4.4, Table 2 (frozen pretrained features encode the scene))

Compute: External GPU: yes (vast.ai): W2 < $1, W2′ ~$0.65, probes ~$0.06; ~$1.7. · Reproduce: 2026-10-23-paper-vs-hardware-dino-wm · First published: 2026-10-23 · Updated: 2026-10-23

DINO-WM (Zhou, Pan, LeCun, Pinto; arXiv 2411.04983) is the recipe behind W2: frozen DINOv2 patches plus a learned dynamics model. It had the better representation and lost the same planning gate as W1, more slowly.

What the paper claims, and how it measures it¶

Three parts, one trained: a frozen DINOv2 encoder gives N patch embeddings per 224×224 frame; a ViT transition model takes H frames of patches with the action MLP-embedded and concatenated to every patch, attends causally at the frame level, and predicts the whole next frame of patches at once; a transposed-conv decoder trained separately is for visualisation only (backpropagating its loss into the predictor hurt, their A.4.3). Loss: ‖p(enc(o), φ(a)) − enc(o_{t+1})‖², teacher-forced. Because the encoder is frozen, collapse is impossible by construction; the price is a representation that cannot adapt to the task. Planning: CEM in MPC to an image goal, cost ‖ẑ_T − z_g‖²; gradient descent through the model was tried and was worse (A.5.3).

Six simulated environments. Metric: success on 50 sampled (start, goal) pairs (Chamfer distance on 10 instances for the deformable ones). Table 1: Push-T 0.90 vs DreamerV3 0.30, IRIS 0.32, TD-MPC2 0.00. Table 2, the number that matters most for anyone picking a representation: Push-T with DINO CLS 0.44, ResNet 0.20, R3M 0.42, DINO patches 0.90. No standard errors, no real robot, no non-learned baseline.

What I ran¶

W2: frozen DINOv2-S patches, learned dynamics, trained on the same 182 episodes. Lost to a disk failure; W2′ retrained on 2026-10-23 (seed 3073, $0.65) reproduces its probes exactly.

G1 probes. Arm-pose R² 0.92 (W1: 0.57), gripper-open AUROC 0.999, grasp onset 0.995. The frozen patch representation encodes the arm far better than W1's end-to-end CLS latent, which is Table 2's CLS-vs-patch result on real frames.

G1b, ranking. Bar: top-1 ≥ 0.8. Result: 0.55–0.61 at every horizon. Fail. Open-loop error 0.62 of copy-last (W1: 0.36) — the better representation came with worse dynamics; action-corruption ratio only 1.13 (reacts less to wrong actions than W1).

G2, latency. A single decision (10 rollouts × 30 steps) on the Mac: W1 108 ms, W2 1 814 ms. The DINOv2 encoder alone is hundreds of ms per frame on an Orin. W3 (bigger) was not run because the decision rule said it would not change the build.

G10 outcome readout. AUROC 0.976 (W1 1.000).

Their number, my number¶

	Paper (sim)	This arm
Patch > CLS for state	0.90 vs 0.44 (Push-T success)	R² 0.92 vs 0.57 (arm pose)
Planning success	0.90 Push-T	top-1 0.55–0.61; no-model baseline 0.68–0.90
Planning time	47 s/episode (LeWM's measurement)	1.8 s per decision on a Mac; off the Orin
Pairwise accuracy	not reported	0.92–0.93; top-1 ≈ pairwise⁷

What the gap means¶

The representation result transfers and the planning result does not, for the same reason as W1: nothing in the paper is compared to a planner with no model. The one new thing W2 taught: a better frozen encoder does not give better dynamics for free; the learned predictor on top of DINOv2 patches reacted less to wrong actions than the small end-to-end model. And DINO-WM-style online planning is off the table on an Orin on latency alone; the role is offline ranking or verification, where W2′'s patch tokens are the ones to use for anything object-level (G7).

Verdict: representation reproduces; planning fails the baseline; the encoder cost rules out the loop.

What would change it: an object-state test (G7) where the patch representation's spatial detail is the thing being measured, scored offline.

Compute. External GPU: yes (vast.ai): W2 < $1, W2′ ~$0.65, probes ~$0.06; ~$1.7.
Reproduce this: 2026-10-23-paper-vs-hardware-dino-wm

Papers tested, 2 of 22. ← Papers tested 1: LeWorldModel · Papers tested 3: RoboJEPA →

FAIL-Detect and SAFE

Fri, 23 Oct 2026 00:00:00 +0000

FAIL-Detect and SAFE¶

They said a statistical band fitted on successful runs keeps false alarms at a chosen rate. On our arm we fitted it to a world model's surprise and set that rate to 5 %. It did not hold: it flagged 22.7 % of successful runs and caught 46 % of failures.

Papers tested · 7 of 22

Goal (the paper's): Detect policy failures at run time from scores built without failure data (FAIL-Detect) or from a probe on a VLA's own features trained on both outcomes (SAFE), with a conformal band that fixes the false-alarm rate.

Its hypotheses:

A conformal band calibrated on successful rollouts keeps false alarms at or below the chosen rate α on in-distribution successes. (FAIL-Detect §IV-B; SAFE §4.3) fail
Learned scores flag failures early enough to matter, on simulated and real robots. (FAIL-Detect Fig. 4–5; SAFE Table 1, Fig. 4 and 6) fail

Verdict: fail · Claim tested: A conformal band on a learned failure score keeps false alarms at α while catching failures. · Bar: Successes flagged ≤ 1/20; failures flagged ≥ 50 %. · n: 22 + 22 successes, 25 failures, 20 splits

Theirs vs ours: false alarms ≤ α on in-distribution successes (conformal coverage bound) vs 22.7 % of successes flagged at α = 5 %; 46 % of failures caught · Not proven: One world model's surprise as the score; their own scores were not reimplemented.

Paper: arXiv 2503.08558, arXiv 2506.09937 · Code: CXU-TRI/FAIL-Detect, vla-safe/SAFE

Claims:

fail conformal failure band equals conformal level α on successes flagged — theirs false alarms ≤ α · ours 22.7 % at α = 5 % (FAIL-Detect §IV-B; SAFE §4.3 (coverage ≥ 1 − α))
fail conformal failure band at least pass mark on failures caught — theirs high recall at the target α · ours 46 % (FAIL-Detect Fig. 4–5; SAFE Table 1)

Compute: External GPU: no (Orin NX). Cost: $0. · Reproduce: 2026-10-23-paper-vs-hardware-contactguard · First published: 2026-10-23 · Updated: 2026-10-23

Two papers, one experiment: both calibrate a time-varying conformal band on successful rollouts and flag a run that leaves it. FAIL-Detect (Xu et al., TRI; arXiv 2503.08558) does it on learned OOD scores with no failure data; SAFE (Gu et al., UofT/TRI; arXiv 2506.09937) on a probe of the VLA's own features trained with failures. I ran the band on W1's surprise (W1-C2). It failed, and the way it failed is the way FAIL-Detect's own hardware section describes.

What the papers claim, and how they measure it¶

FAIL-Detect: a scalar score per step from the policy's observation embedding — logpZO (a flow from the embedding distribution to N(0, I); the score is the log-density), NatPN, RND — plus post-hoc baselines (STAC, which needs 256 policy samples per step; PCA-kmeans; smoothness). A band over time is fitted to the success distribution; a flag is raised when the score exceeds it. Sixteen simulated settings (four robomimic tasks × two policies × ID/OOD); summary is top-1 count across settings: logpZO 10/16. Hardware: Franka bimanual, two tasks, fewer rollouts; logpZO top-1 on 8/12. The caveat is in §VI: with ID-only bands, OOD-but-successful rollouts are flagged, "at times yielding low or close to zero TNR"; the fix needs successes from every new setting. Detection time for STAC exceeds the average success duration.

SAFE: the VLA's last-layer features already separate success from failure across tasks (a shared "failure zone" in t-SNE). A tiny MLP or LSTM on them, trained on seen tasks with trajectory-level labels, generalises to unseen tasks. Threshold by functional conformal prediction on successful calibration rollouts, FPR ≤ α at any time step. Three random seen/unseen splits; ROC-AUC on the max-so-far score (a flag anywhere on a success is a false positive); 14 adapted baselines. Sim unseen AUROC ~0.78 vs best baseline ~0.74; real Franka 30 + 30 per task and WidowX 532 rollouts; < 1 ms overhead.

What I ran¶

W1-C2 (2026-10-19). A time-varying functional CP band on W1's long surprise, fitted on 22 successes, tested on the other 22 successes and the 8 failures, 20 seeds. Bar (median): successes flagged ≤ 1/20, failures ≥ 50 %. Result: successes 22.7 % (IQR 13.6–27.3), failures 46 %, stalls 34 %. Fail. The band as registered fits μ, σ and the radius on the same successes: on synthetic exchangeable data its false-alarm rate is 0.14 at 20 anchors — about α only with separate μ/σ data. So the band was not conformal across time as written, and even a correct band sits on a surprise distribution that widens on fresh data (W1-A: the live reflex fitted on one session flagged 42/42 successes on the next).

C1 probe (2026-10-11), the SAFE-shaped question — does the imagined latent predict failure better than simpler signals? — found policy identity alone at 0.76 beating every probe (current 0.68, imagined 0.72), because the failure set was dominated by one policy.

Their number, my number¶

	Papers	This arm
False alarms at the registered α	≈ α by construction	22.7 % (α = 5 %)
Failures caught	logpZO top-1 in 10/16 settings; SAFE unseen AUROC 0.78	46 %
Transfer across sessions	"close to zero TNR" without new-setting successes (FAIL-Detect §VI)	42/42 successes flagged on the next session
Policy-identity baseline	not run	0.76, beats every probe

What the gap means¶

The unsupervised band fails on hardware the same way in Palo Alto and in Paris: the success distribution shifts with the setting and the band does not know. The supervised probe (SAFE's design) is the one that works on hardware in every paper I read, and the C1 result says my version of it is blocked on a confound the papers do not check — the failure labels came mostly from one policy. The register now requires a policy-identity and episode-length baseline under every monitor result, which neither paper runs.

Verdict: FAIL-Detect's band fails here as its authors warn it can; SAFE's probe is not yet tested cleanly.

What would change it: SAFE-MLP on SmolVLA's action-expert input features exported from vla.cpp, three random task splits, with episode length beside it.

Compute. External GPU: no (Orin NX). Cost: $0.
Reproduce this: 2026-10-23-paper-vs-hardware-contactguard

Papers tested, 7 of 22. ← Papers tested 6: ContactGuard · Papers tested 8: ReAP →

H-JEPA

Fri, 23 Oct 2026 00:00:00 +0000

H-JEPA¶

They said a slow upper layer in a stacked world model tracks progress toward the goal. On our arm we checked whether its distance to the goal falls steadily along recorded runs. It did not hold: it signalled a stall that was not there 52 % of the time.

Papers tested · 4 of 22

Goal (the paper's): Stack JEPA world models into a hierarchy with a distinct latent space per level, so that upper levels plan in abstract states over longer horizons and hand subgoals down; shown in simulation, with an open-loop fidelity measure on DROID.

Its hypotheses:

A hierarchy of distinct latent spaces plans long-horizon tasks with less planner compute than a flat model. (Fig. 6 (AntMaze 18 % → 73 %)) not tested here
The lower level's cost to the subgoals handed down falls steadily along expert trajectories (Spearman 0.89–1.00), more consistently than its cost to the final goal (0.48–0.75). (§4.2, Fig. 8; Table 10 (Spearman 0.89–1.00)) fail
Upper levels drop fast-varying entities when the data has an entity-frequency gap above about 2×. (Fig. 4–5) not tested here

Verdict: fail · Claim tested: A slow second level makes latent distance track task progress. · Bar: Progress monotonicity Spearman ρ ≤ −0.80; false stalls ≤ 10 %. · n: teleop val episodes (monotonicity, stalls); 88 rollouts (sanity check)

Theirs vs ours: subgoal-tracking Spearman 0.89–1.00 vs level-2 ρ −0.73; false stalls 52 % · Not proven: Outside the paper's regime (no entity-frequency gap on this task).

Paper: arXiv 2610.06805 · Code: no code link found

Claims:

fail second-level latent distance at most pass mark on progress Spearman ρ — theirs subgoal-tracking (level-1 cost to level-2 subgoals) Spearman 0.89–1.00; no level-2 distance-to-goal reported · ours ρ −0.73; false stalls 52 % (paper, subgoal-tracking Spearman 0.89–1.00)

Compute: External GPU: no (Mac). Cost: $0. · Reproduce: 2026-10-14-g6-progress-latents · First published: 2026-10-23 · Updated: 2026-10-23

H-JEPA (Zhang, Terver, Rabbat, LeCun, Balestriero; arXiv 2610.06805) stacks LeWorldModels across timescales. I tested its level-2 slow feature as a progress signal (G6b) and it failed. Reading the paper afterwards, the paper's own Fig. 5 predicts that failure.

What the paper claims, and how it measures it¶

Level 1 is LeWM. Each level ℓ > 1 has its own state encoder (a pointwise MLP over the level below), its own action encoder pooling s_ℓ lower actions, and its own causal predictor, stride 2 per level, all trained end-to-end with a LeWM loss per level. On real data (DROID) an inverse-dynamics term is required or the model collapses to "encode the predictable background, drop the arm" (their App. L). Planning is top-down by gradient descent: the top level plans macro-actions; its predicted states become subgoals for the level below.

Evaluation: four simulated environments; success rate; three paired model + planner seeds with SE; planner compute swept and reported as TFLOPs per episode; depth 1–4 at a fixed 100-TFLOP budget on 50 tasks per environment. Baseline HWM (same hierarchy, shared latent) separates "temporal decomposition" from "distinct abstract spaces". AntMaze 18 → 73 % at three levels; two levels match LeWM on Push-T, three levels fail there because episodes are short. The mechanism figure: probe NMSE per entity vs level shows body pose becoming unrecoverable while position stays; the entity-frequency gap (ratio of spectral centroids of fastest and slowest state components) predicts when this selective abstraction appears — above ~2× it does; Push-T, Cube and DROID have gaps below 2× and show none. On DROID the metric is Fréchet fidelity of the planned end-effector path against the expert's (15 clips, three seeds). No closed-loop real robot.

What I ran¶

G6b. A level-2 slow feature (stride 15, SIGReg loss) on W1, tested as a progress signal on the bowls task. Bars: T1 Spearman between latent distance and time ≤ −0.80; false-stall rate ≤ 10 %; sanity AUROC ≥ 0.95. Result: T1 −0.73 (fail), false stalls 52 % (fail), sanity 0.998 (pass). The hierarchy was dropped.

Their number, my number¶

	Paper	This arm
Where hierarchy helps	AntMaze (frequency gap > 2×): 18 → 73 %	bowls task: one arm, one object, ~1 Hz everything
Where it does not	Push-T, Cube, DROID (gap < 2×)	same regime
Progress monotonicity	subgoal-tracking Spearman 0.89–1.00 (Fig. 8)	level-2 distance vs time −0.73

What the gap means¶

I tested the hierarchy on data the paper says it should not change. There is no frequency gap on a single-arm pick-and-nest task; the paper's own prediction for that regime is "no selective abstraction", and that is what G6b found. So the negative result and the paper's claim do not conflict; the gate was outside the claim. Two things do transfer: the inverse-dynamics term, which W1 does not need on single-scene data but W2′ on mixed lighting might; and Fréchet fidelity as an offline, open-loop plan-quality metric against expert paths — it costs nothing on my 69 rollouts and would have told me before G1b that the plans do not follow the demos.

Verdict: fail, outside the paper's regime; the paper's mechanism test (Fig. 5) is the better reading than my gate.

What would change it: a task with a real timescale gap (a tidy loop with navigation-scale structure), which my bench does not have.

Compute. External GPU: no (Mac). Cost: $0.
Reproduce this: 2026-10-14-g6-progress-latents

Papers tested, 4 of 22. ← Papers tested 3: RoboJEPA · Papers tested 5: D-JEPA →

LeWorldModel

Fri, 23 Oct 2026 00:00:00 +0000

LeWorldModel¶

They said a small world model, trained from camera images, learns a robot's motion well enough to plan with. On our arm we checked whether it ranks the motion actually taken first among 8 candidates. It did not hold: it was right 0.60 of the time, below a check that ignores the camera.

Papers tested · 1 of 22

Goal (the paper's): Train a small world model end to end from pixels, with no frozen encoder and no EMA target, kept from collapse by SIGReg alone, and plan with it by sampling actions in its latent space; shown in simulation (Push-T, Two-Room, Reacher, OGBench Cube).

Its hypotheses:

A 15M-parameter JEPA world model trained end to end with SIGReg learns dynamics stable enough to plan with. (§3.1, Eq. (3); Fig. 6) reproduces
At a matched planning-compute budget it plans more successfully than DINO-WM and about 48× faster; without that match it leads on Push-T and Reacher only. (Fig. 3 (Push-T at fixed FLOPs 90 % vs 13 %; 0.98 s vs 47 s per plan)) fail
A linear readout of its latent recovers object state. (Table 1 (Push-T probes)) not tested here

Verdict: fail · Claim tested: A small JEPA world model learns the arm's dynamics well enough to plan with it. · Bar: Dynamics: beat copy-last, every corruption ≥ 1.3 at K = 30. Planning: rank the executed chunk first among 8 in ≥ 0.8 of anchors. · n: 25 val episodes, 703 anchors

Theirs vs ours: Push-T planning 90 % at matched FLOPs vs dynamics 0.31 vs 0.89 copy-last (reproduces); planning top-1 0.60 vs 0.68–0.90 for an action-only baseline (fail) · Not proven: Closed-loop planning on the arm was never run; one training seed.

Paper: arXiv 2603.19312 · Code: lucas-maes/le-wm

Claims:

reproduces JEPA world model below copy-last baseline on open-loop latent error at K = 30 — theirs Push-T planning 90 % at matched FLOPs · ours 0.31 vs 0.89 copy-last (not stated: the premise of §3–4 (planning needs a usable dynamics model))
fail latent world-model ranking above action-only baseline on G1b top-1 — theirs Push-T planning 90 % · ours 0.60 vs 0.68–0.90 action-only (paper §4 (planning by ranking imagined outcomes))
reproduces action-only baseline above latent world-model ranking on G1b top-1 · ours 0.68–0.90 vs 0.60 (our measured comparison)

Compute: External GPU: yes (vast.ai): ~$6.60 incl. $2.96 of failed attempts. · Reproduce: 2026-10-23-paper-vs-hardware-leworldmodel · First published: 2026-10-23 · Updated: 2026-10-23 · Findings: I need to rank candidate action chunks →

One paper, one arm. This series takes each paper I actually tested on a SO-101 + Jetson Orin NX, states what the paper claims and how its authors measured it, puts my number next to theirs, and says what the gap means. Numbers are from journal/EXPERIMENTS.md; gate IDs are the register's.

LeWorldModel (Maes, Le Lidec, Scieur, LeCun, Balestriero; arXiv 2603.19312) is the recipe behind W1, the world model that runs on my arm. Four gates tested it: dynamics (G0), planning (G1b), progress (G6a), outcome readout (G10).

What the paper claims, and how it measures it¶

A 15M-parameter JEPA trained end-to-end from pixels: a ViT-tiny encoder (patch 14, 12 layers, width 192, ~5M) whose CLS token becomes one 192-d latent per frame; a 6-layer predictor with AdaLN zero-init action conditioning (~10M); loss ‖ẑ_{t+1} − z_{t+1}‖² + 0.1 · SIGReg, where SIGReg pushes the batch of latents toward an isotropic Gaussian by testing 1024 random 1-d projections for normality. No EMA target, no stop-gradient; SIGReg alone prevents collapse. Planning is CEM with a terminal cost ‖ẑ_H − z_g‖² inside MPC.

Evaluation is simulation only: Push-T, Two-Room, Reacher, OGBench Cube. The headline is success at matched training FLOPs (Fig. 6): Push-T 90 % vs DINO-WM's 13 % at the same compute, planning 48× faster (0.98 s vs 47 s). Table 1 is a linear-probe table: a linear readout of the latent predicts Push-T block location with r = 0.986 and block angle with r = 0.902. There is no real robot, no non-learned planner baseline, and the authors list "short planning horizons" as the limitation.

What I ran¶

W1 is LeWM's recipe on 182 teleop episodes from two webcams (multi-view variant, ContactGuard-style mean-pool), 100 epochs, $3.30 on vast.ai.

G0, dynamics. Bar: beat copy-last; every action-corruption ratio ≥ 1.3 at K = 30; no collapse. Result: open-loop error at K = 30 0.31 vs copy-last 0.89; timing-shift ratios 1.99 / 1.58; effective rank 24–30 of 192 (the 5-epoch run was 5.8). Pass. The model learns real action-conditioned dynamics.

G1b, planning as ranking. Bar: rank the expert chunk first among 8 candidates at top-1 ≥ 0.8 at some horizon. Result: best 0.60 at d = 20–30. Fail. The diagnosis (G1 diagnosis row) found why: an action-only baseline with no model at all, ranking by commanded-joint distance, scores 0.68–0.90. On a position-controlled arm the commanded joints already determine the outcome of a 7-step chunk; the latent adds noise.

G6a, progress from latent distance. Bar: Spearman ≤ −0.8 between latent distance to goal and time. Result: −0.32 to −0.50. Fail. Latent distance is not progress here (the TD-JEPA paper, 2607.25337, finds the same gap in simulation: Spearman 0.65 for LeWM's Euclidean distance vs step count).

G10, outcome readout at rest. Bar: AUROC ≥ 0.90 on success/failure from the final latent. Result: 1.000. Pass. Table 1's probe result transfers: the latent contains the task outcome.

Their number, my number¶

	Paper (sim)	This arm
Dynamics learned	yes (VoE test, qualitative)	yes: 0.31 vs 0.89 copy-last
Linear probe of state	r 0.986 (block location)	arm-pose R² 0.57 (W1), outcome AUROC 1.00
Planning success	Push-T 90 % at matched FLOPs	top-1 0.60 vs no-model 0.68–0.90
Baseline without a world model	not run	run; it wins

What the gap means¶

Not that the paper is wrong. The paper's planner is compared to other learned planners at equal compute; it is never compared to "do nothing clever". On a cheap arm the clever-nothing baseline is strong because the commanded trajectory is the plan, and the latent planner has to beat it, not DINO-WM. The honest role for the model here is the one G10 passed: a verifier at rest. The planning-limits paper (2609.39235) later showed with a real simulator as a perfect predictor that terminal-latent-distance scoring against a distant goal falls from 92 % to 41 %; G1b scored against a goal many chunks away, so it sat in exactly that regime. A subgoal inside the rollout is the fix the paper proves; it has not been tried here yet.

Verdict: dynamics and readout reproduce; planning does not survive the baseline the paper did not run.

What would change it: G7 (object-state ranking where the joints are blind), with a near subgoal. That gate is frozen and waiting for a cube-rollout session.

Compute. External GPU: yes (vast.ai): ~$6.60 incl. $2.96 of failed attempts.
Reproduce this: 2026-10-23-paper-vs-hardware-leworldmodel

Papers tested, 1 of 22. Papers tested 2: DINO-WM →

Real-Time Chunking

Fri, 23 Oct 2026 00:00:00 +0000

Real-Time Chunking¶

They said building each new motion chunk to match the one already running removes jumps at the joins. On recorded runs we compared a simple overwrite of the shared part with their method. Not proven on the arm: the overwrite removed most of the jump, adding 0.4 ms.

Papers tested · 15 of 22

Goal (the paper's): Let a flow-matching policy run asynchronously without jumps at chunk boundaries, by generating each new chunk as an inpainting of the actions already committed; shown on dynamic simulated tasks and a bimanual robot with added latency.

Its hypotheses:

Conditioning the new chunk on the committed actions removes the discontinuity at chunk boundaries. (§3.1–3.2 (ΠGDM guidance, Eq. 2–3); Fig. 3–4) not proven
With it, task throughput stays flat as inference latency grows, where synchronous execution falls. (Fig. 6) not tested here

Verdict: not proven · Claim tested: Conditioning a new chunk on the executing one removes the jump at chunk boundaries. · Bar: Best gradient-free method cuts the median jump ≥ 50 % against naive async, drift ≤ +10 %, within 667 ms. · n: 254 boundaries per delay, 20 episodes

Theirs vs ours: throughput flat across added latency (π0.5, real robot) vs prefix overwrite cuts the jump 71–73 % against naive async at +0.4 ms; tuned LeRobot identity mode is close · Not proven: Offline smoothness only; task success on the arm not measured for any mode.

Paper: arXiv 2506.07339 · Code: Physical-Intelligence/real-time-chunking-kinetix

Claims:

not proven prefix overwrite at least pass mark on median boundary-jump cut — theirs throughput flat across added latency (π0.5, real robot) · ours 71–73 % at +0.4 ms (paper §3.1–3.2, Fig. 4 (qualitative; no jump metric reported))

Compute: External GPU: no (Orin NX). Cost: $0. · Reproduce: 2026-10-25-vla-chunk-boundary-jumps · First published: 2026-10-23 · Updated: 2026-10-26 (soft-mask correction)

Real-Time Chunking (Black, Galliker, Levine, Physical Intelligence; arXiv 2506.07339) treats the next action chunk as an inpainting problem. My gradient-free prefix overwrite uses the paper's soft mask without its gradient, and the paper says where gradient-free inpainting should lose.

Corrected 2026-10-26: an earlier version called the overwrite RTC's hard mask. It uses RTC's soft prefix weights, without the gradient (see the chunk-boundary post).

What the paper claims, and how it measures it¶

With chunk length H, control period Δt and inference time δ, the delay is d = ⌊δ/Δt⌋ steps. Asynchronous inference must start the next chunk d steps early, and the junction is arbitrary: a new chunk from fresh noise may pick a different strategy (their bifurcation figure). Temporal ensembling averages across chunks and is not guaranteed valid when the distribution is multimodal. RTC: freeze the first d actions (they will be executed before the new chunk arrives), attend to the remaining overlap with an exponentially decaying weight, leave the rest free, and enforce it through ΠGDM guidance on the flow's velocity field with a clipped guidance weight — one vector-Jacobian product through the policy per Euler step. The soft mask (eq. 5): weight 1 for i < d, decaying over the overlap, 0 after.

Evaluation: twelve dynamic Kinetix tasks chosen because quasi-static benchmarks are saturated by open-loop chunking; 2 048 rollouts per point with Wilson 95 % intervals; delays 0–4; baselines naive async, BID, temporal ensembling. RTC is above every baseline at every delay; hard masking (freeze only d steps) is below soft masking at small d; temporal ensembling is worst even at d = 0. Real: π0.5, bimanual, H = 50, Δt = 20 ms, 5 denoising steps, d ≈ 6, plus +100 / +200 ms injected latency; six tasks, 10 trials per task per method per delay, metric throughput (task progress per minute). RTC is flat across added latency; synchronous falls linearly; both temporal-ensembling variants trip the protective stop at +100 ms and score zero. Cost: 97 vs 76 ms per chunk on their stack.

What I ran¶

G13b (2026-10-19, offline, Orin). SmolVLA on 20 bowls episodes, 254 chunk boundaries, d = 8 / 12 / 20. Methods: naive, exact RTC guidance, and gradient-free prefix overwrite. Bars: propose the gradient-free method if it cuts the median jump ≥ 50 % vs naive and stays within 20 % of exact. Result: prefix overwrite cuts the median jump 72–74 % at every d, drift unchanged; exact RTC cut only 21–41 % in this implementation (weaker than LeRobot's identity mode — flagged to check). Timing: naive 1 028 ms, overwrite 1 035 ms, exact 1 962 ms per chunk in PyTorch on the Orin; every method misses the 667 ms line because the policy itself is too slow in PyTorch.

In vla.cpp (same day). Ported as prev_actions + inference_delay; jump cut 71.0 / 73.3 / 71.5 % through the engine, prefix error 0.0, +0.4 ms per chunk, default path bit-identical. Exact RTC is not possible in vla.cpp today: ggml lacks backward rules for flash attention and concat.

Their number, my number¶

	Paper	This arm
Hard vs soft mask	soft > hard at small d (Fig. 5)	soft only (RTC's prefix weights over the whole leftover); hard not run
Cost	+21 ms (VJP per Euler step)	+0.4 ms (overwrite); exact +930 ms in PyTorch
Metric	throughput, task progress/min	median boundary jump (instrument)
n	2 048 rollouts/point; real 10/task/delay	254 boundaries, 20 episodes

What the gap means¶

The overwrite already uses the paper's soft mask (weight 1 on the first d steps, decaying to 0 at the end of the leftover). What it drops is the gradient: it corrects the predicted clean chunk directly, which is guidance with an identity Jacobian at weight 1/t, instead of RTC's vector-Jacobian product. So the paper's hard-vs-soft result (Fig. 5) does not decide between the two; the paper's own gradient-free inpainting variant (App. A.4, simulation only) was beaten by guided RTC on solve rate. Only if guided RTC beats the overwrite by more than the pairing noise on the same 254 boundaries are ggml backward kernels worth writing. And the result metric should be theirs: a smoother junction is for throughput; the jump is the instrument check, not the claim.

Verdict: the gradient-free point reproduces in spirit; the paper predicts a better point I have not measured.

What would change it: the soft-mask comparison offline, then throughput on the robot day with sync / async / async + prefix.

Compute. External GPU: no (Orin NX). Cost: $0.
Reproduce this: 2026-10-25-vla-chunk-boundary-jumps

Papers tested, 15 of 22. ← Papers tested 14: REKI · Papers tested 16: HWM (hierarchical latent world models) →

ReAP

Fri, 23 Oct 2026 00:00:00 +0000

ReAP¶

They said a risk score that looks at the next planned motions warns before a robot fails. On our arm we built that risk score and asked it to warn before 8 recorded failures. It did not hold: it caught 3/8 failures while flagging 5/42 good runs.

Papers tested · 8 of 22

Goal (the paper's): Warn of a VLA's failure before it happens, with a detector on a compact visual state trained on a ramp target (no onset labels) and a predictor that rolls that state forward under the policy's next action chunk.

Its hypotheses:

Conditioning the risk prediction on the policy's next action chunk improves warning over the same model with shuffled or zero actions. (Table 3b) reproduces
The monitor warns earlier and more accurately than action-uncertainty, OOD and classifier baselines. (Table 1, Table 2a) fail

Verdict: fail · Claim tested: A risk head on an action-conditioned latent warns before failures. · Bar: ≥ 5/8 failures caught at ≤ 1/20 false interventions; action-conditioned iff true ≥ shuffled + 0.05. · n: 50 episodes, 8 failures (and all 69)

Theirs vs ours: bTWA 0.65–0.78 in simulation (Table 1), 0.60–0.82 on the real robot (Table 2a) vs v1 AUROC 0.93 came from time and length; v2 0.789 vs 0.708 shuffled, 3/8 at 5/42 · Not proven: 8 failures; v1 onsets VLM-labelled.

Paper: arXiv 2610.16309 · Code: no code link found

Claims:

reproduces action-conditioned risk head above shuffled-action risk head on failure AUROC — theirs bTWA 0.683 vs 0.618 shuffled (no AUROC reported) · ours 0.789 vs 0.708 shuffled (paper Table 3b (action-conditioned 0.683 vs shuffled 0.618 bTWA; the next planned chunk))
fail action-conditioned risk head at least pass mark on failures caught — theirs bTWA 0.70–0.78 · ours 3/8 at 5/42 (paper Table 1 (detection time), Table 2a)

Compute: External GPU: no (Orin NX). External API: Gemini onset labels, cents. · Reproduce: 2026-10-23-paper-vs-hardware-reap · First published: 2026-10-23 · Updated: 2026-10-23 · Findings: I need an online failure monitor →, W1 as an online monitor next to the policy →

ReAP (Liu, Qing, Wei, Hou; arXiv 2610.16309) trains a failure-supervised visual space and an action-conditioned risk predictor on top of it. My "ReAP-style risk head" is the closest any monitor got to my bar — and it is a stripped version with two confounds the paper's own design removes.

What the paper claims, and how it measures it¶

Stage 1, a detector: frozen DINOv2 patches from each camera → Perceiver-style cross-attention from 32 learned queries → a compact state z_t → a risk head, trained with class-weighted BCE against a ramp target: 0 on successes, t/(T−1) on failures. No onset labels. Stage 2, a predictor: the policy's action chunk cross-attends the frozen latent, a residual MLP predicts the latent H steps ahead, and the frozen head reads risk from it; loss is latent-transition error plus risk BCE. Alarm = detector OR predictor, each with its own conformal threshold on held-out successes.

Metric bTWA: half earliness-weighted recall (a failure detected at t_i of T_i scores 1 − t_i/T_i) plus half specificity; averaged over five α; mean ± std over three seeds. Offline: LIBERO-10 and RoboTwin-8 with four policies; baselines from three families (action uncertainty, OOD, classifiers). ReAP 0.70–0.78 vs best baseline 0.58–0.76. Real: three π0.5 tasks, monitor live, alarms logged without acting. The ablations are the valuable part: a random, untrained detector scores 0.650 vs the learned 0.683 and mean-pooled DINOv2 beats every baseline alone (Table 3a); shuffled chunks from other episodes of the same task drop 0.683 → 0.618, zero actions → 0.52 (Table 3b). Detector latency 9.7 ms, no policy inference needed.

What I ran¶

REAP monitors (2026-10-23). On W1-A's 50 episodes (42 ok / 8 fail), 5-fold × 5 seeds: W1 L2 surprise, a logistic or MLP risk head on [W1 latent, 30 commanded actions, step], elapsed time. Bar: ≥ 5/8 failures at ≤ 1/20 false interventions. Result: surprise 2/8 at 2/42 (AUROC 0.86); logistic head 6/8 at 7/42 (AUROC 0.93); MLP 7/8 at 13/42; elapsed 0/8. On all 69: logistic 23/25 at 4/44 (0.97), but 17 of the extra failures are TurboVLA's. Verdict: not proven. Two confounds: the secondary set's failures are mostly one policy, and the onsets were labelled by Gemini with two misses of 110 and 289 frames. Latency on the Orin: W1 tick 17.6 ms, 25-step imagination 153 ms off the control thread, risk head 18 ms; the policy's chunk slows 238 → 304 ms with the monitor sharing the GPU.

Their number, my number¶

	Paper	This arm
Representation	failure-supervised Perceiver over DINOv2	W1's prediction-trained latent
Target	ramp, no onsets	Gemini onsets + step feature
Action control	shuffled chunk: 0.683 → 0.618	not run
Best result, v1 (time feature, Gemini onsets)	bTWA 0.70–0.78	6/8 at 7/42, AUROC 0.93
v2 (ramp target, no time feature, common-horizon cut)	—	logistic 3/8 at 5/42, AUROC 0.789, bTWA 0.55; MLP 6/8 at 13/42
True vs shuffled chunk	Table 3b: 0.683 vs 0.618	0.789 vs 0.708 (+0.08)
Latency	9.7 ms detector	17.6 ms W1 + 18 ms head

What the gap means¶

Table 3a says the encoder is not where my gap is: a random projection of DINOv2 is within 0.03 of their learned one, so W1's latent is probably fine. The two things their design has and mine lacks are exactly my two confounds. The ramp target removes the onset labels from the verdict — the 25 hand labels I owe become a validation set instead of a prerequisite. The shuffled-chunk control decides whether "action-conditioned" belongs in the name: train once, evaluate with other episodes' chunks; if the number does not drop, the head is a state monitor. Both are offline and free and are in the amendments; bTWA goes beside my bar so the comparison is on their metric too.

REAP v2 (2026-10-23, evening). The paper's own fixes applied: ramp target (no onset labels), no time feature, every episode cut to the task's common horizon (374 frames), shuffled-chunk and zero-action controls, bTWA reported, VLA-Corrector's parameter-free GRU as the floor. Protocol committed before the run. Result on the primary 50, medians over 5 seeds: logistic head 3/8 at 5/42, AUROC 0.789 (shuffled 0.708, zero 0.693); MLP 6/8 at 13/42; GRU floor 2/8 at 3/42, 0.589. So v1's 0.93 was the time feature and the episode length. The one thing that survives is the action-conditioning test: both W1 heads gain 0.07–0.08 over shuffled chunks, above the +0.05 line, so the head does read the commanded action — it just does not read enough.

Verdict: fail. No monitor meets the bar once the confounds are removed; the W1 risk head is action-conditioned but weak; the floor baseline is a state monitor.

What would change it: a world model trained on the deployed policy's own rollouts (ContactGuard's recipe), not teleop only; then the same v2 protocol.

Compute. External GPU: no (Orin NX). External API: Gemini onset labels, cents.
Reproduce this: 2026-10-23-paper-vs-hardware-reap

Papers tested, 8 of 22. ← Papers tested 7: FAIL-Detect and SAFE · Papers tested 9: VLA-Corrector →

REKI

Fri, 23 Oct 2026 00:00:00 +0000

REKI¶

They said robot policies without memory get stuck going back and forth between poses. We measured that back-and-forth in our 69 recorded runs, then trained the memory version. It held: 24.0 % of failed runs oscillated; with memory, 1 run in 40 did.

Papers tested · 14 of 22

Goal (the paper's): Give a VLA a short memory of past frames with register tokens cached across steps, so it stops oscillating between visually aliased poses; trained on public data only and evaluated on LIBERO-Plus, RoboTwin, RoboCasa and real robots.

Its hypotheses:

Policies without memory oscillate between poses in a measurable share of episodes; register memory cuts it. (Fig. 5b (18.6 % without, 4.1 % with)) reproduces
Most of the gain comes from reading the first past frame at inference. (Fig. 5a) not tested here
A 2.4B VLA pretrained only on public data matches larger in-house models. (Fig. 4) not tested here

Verdict: reproduces · Claim tested: Policies without memory oscillate between poses at a measurable rate. · Bar: ≥ 10 % of failed episodes oscillate (their criterion). · n: 69 published rollouts

Theirs vs ours: 18.6 % without memory, 4.1 % with vs 20.3 % of all episodes, 24.0 % of failures; REKI trained on my data: 19/20 + 19/20 on the arm, oscillates in 1/40 · Not proven: That memory causes the drop to 1/40: the K = 1 (memory off) control is not run. REKI does not build for the Orin (served from an RTX 3090).

Paper: arXiv 2610.22178 · Code: kyutai-labs/reki

Claims:

reproduces policy without memory at least pass mark on oscillating share of failures — theirs 18.6 % of all episodes and about half of the failed ones oscillate without memory (RoboTwin Clean, §6.2, Fig. 5b) · ours 24.0 % of failures, 20.3 % of all episodes (paper, 18.6 % without memory, 4.1 % with)
reproduces REKI at least pass mark on success on the arm — theirs UR5 + WSG50 real-robot success · ours 19/20 + 19/20 (paper §7, Fig. 8)

Compute: External GPU: yes for REKI (vast.ai): training $11.25 + smoke $0.60, serving $1.20, total $13.05. Oscillation audit on the Orin NX: $0. · Reproduce: 2026-10-23-paper-vs-hardware-reki · First published: 2026-10-23 · Updated: 2026-10-28 (REKI trained and run on the arm) · Findings: REKI 2.4B on the Orin →

REKI (Marrie, Pacaud, Ramanana Rahary, Chen, Böhle; Kyutai / Inria; arXiv 2610.22178) gives a VLA a memory through register tokens and measures the oscillation it removes. I voided my own latency measurement of it and then found the same oscillation in my policies with their metric.

What the paper claims, and how it measures it¶

A 2.4B VLA (Qwen3.5-2B + a flow-matching expert) with 32 register tokens per frame in the expert: they cross-attend the frame's context, self-attend the registers of the K − 1 previous frames (K = 3), never read the action tokens, and are cached as keys/values across chunks — so memory costs +0.1 ms. Pretrained on public data only; 21.8 ms per chunk on an H100. The augmentation detail that matters for a webcam rig: random camera dropout and shuffled camera order, worth > 20 points under viewpoint change.

Evaluation: LIBERO-Plus 90.4; RoboTwin Clean2Rand 65.8; RoboCasa365 38.4; baselines quoted from their papers. The memory ablation (Fig. 5a): a K = 3 model evaluated at K = 1 scores like a K = 1 model — the gain comes from reading past frames at inference, mostly the first one. The oscillation result (Fig. 5b): an episode oscillates if an arm reverses direction at three or more consecutive chunks; 4.1 % with memory, 14.8 % when the same model is evaluated without it, 18.6 % trained without; two-arm tasks 24.5 → 3.1; RoboTwin, 50 tasks, 20 matched seeds each. Mechanism offered: perceptual aliasing — two poses each look like demonstration states heading toward the other.

What I ran¶

G12 (2026-10-18), voided the same day. The bar was memory ≤ 8 GB and chunk ≤ 250 ms on the Orin; the script reported 3.46 GB and 318 ms. The re-check found it timed a random-init Qwen2.5-1.5B with no vision tower and a hand-written random expert; the parts did not sum (135 + 230 ≠ 318). REKI's Orin latency is unmeasured.

G14 (2026-10-18). REKI's oscillation criterion on my 69 published rollouts: reversal at ≥ 3 consecutive chunks, cosine < −0.5, > 3° per chunk. Failed episodes 6/25 = 24.0 % (CI 11.5–43.4 %); succeeded 8/44; all episodes 14/69 = 20.3 %, which is the number comparable with REKI's (theirs is over all episodes). Trajectory check: the flagged failures are real re-grasp loops and pan dithering; two unflagged loops were missed. All my policies are memoryless.

Update 2026-10-28: REKI trained and run on my arm¶

Training. REKI's frozen-VLM recipe (their repo at cc16b94), 3,500 steps, batch 30 × 4, on my bowls data. One rented RTX PRO 5000, 16.1 GPU hours, $11.25 plus a $0.60 smoke test. Training loss 0.315 → 0.008. One seed.

Serving. REKI doesn't run on the Orin at all: it needs flash-attn, FLA / Triton and causal-conv1d, which only build for x86 CUDA. So the policy ran on a rented RTX 3090 (~100 ms per chunk) and the Orin drove the arm as a client over an SSH tunnel. $1.20 over two sessions. Every chunk is a network round trip, so the arm pauses about 10 s of every 25 s trial.

Protocol. Same as my SmolVLA gate: 25 s per trial, judged at rest by me, 20 trials per instruction. Bar, fixed before the first counted trial: not worse than SmolVLA (9/10) by more than 2 in 20.

instruction	REKI	failures
blue bowl in pink	19/20	one set down short (the pink bowl started outside the teleop area)
pink bowl in blue	19/20	one missed grasp

Four more trials were thrown out and replaced: after a network stall of 5 s or more the server cleared REKI's memory mid-trial. I wrote that rule before seeing the last affected trial's outcome, and applied it to all four (three were successes).

Oscillation. With the metric above, 1 of 40 REKI trials oscillates (2.5 %), against 20.3 % on my earlier policies. That's the direction the paper predicts, but it isn't their ablation: different policies, and I computed it on REKI's commanded trajectory rather than recorded states. The clean test is the same REKI model with memory switched off (K = 1) on the same setup. I haven't run it.

Two successes and the missed grasp, 47 s. Network pauses are cut, so it plays faster than real time.

Their number, my number¶

	Paper	This arm
Oscillation, no memory	18.6 % (trained without), 14.8 % (evaluated without)	20.3 % of all episodes (24.0 % of failed)
With memory	4.1 %	1/40 = 2.5 % (REKI on my arm; no memory-off control)
Task success	—	19/20 and 19/20 (SmolVLA 9/10, same protocol)
Criterion	reversal ≥ 3 consecutive chunks, 20 seeds/task	same criterion, 69 episodes
Latency on Orin	—	void; REKI doesn't build for the Orin (served from an RTX 3090)

What the gap means¶

G14 is the cleanest external replication of a mechanism in my register: a memoryless policy on a SO-101 dithers at about the rate a memoryless 2.4B VLA dithers on RoboTwin, under the same definition. What I cannot yet say is whether memory fixes it here, because adding registers to SmolVLA means retraining. The paper's Fig. 5a points at the cheap half: for any policy that has a past-frame cache, the same checkpoint evaluated with and without it is a free experiment. SmolVLA has none, so the port is the only route, and it costs $5–10 plus a robot session; the amendments put the free comparison and the real latency measurement before it.

Verdict: the oscillation mechanism reproduces. REKI itself works on my arm (19/20 on both instructions) and oscillates in 1 of 40 trials, which points the paper's way but doesn't isolate the memory. It still can't run on the robot's own computer.

What would change it: the same REKI checkpoint at K = 1 (memory off) on the same 40-trial protocol.

Compute. External GPU: yes for REKI (vast.ai): training $11.25 + smoke $0.60, serving $1.20, total $13.05. Oscillation audit on the Orin NX: $0.
Reproduce this: 2026-10-23-paper-vs-hardware-reki

Papers tested, 14 of 22. ← Papers tested 13: VLA-JEPA · Papers tested 15: Real-Time Chunking →

RoboJEPA

Fri, 23 Oct 2026 00:00:00 +0000

RoboJEPA¶

They said a world model pretrained on 15 000 hours of robot video predicts what comes next from the actions. On our arm we tested its predictions on our clips and timed it on the robot's computer. It used our actions, but its image encoder alone took 526 ms, too slow to run on the robot.

Papers tested · 3 of 22

Goal (the paper's): Measure how an action-conditioned predictor on frozen V-JEPA 2.1 features scales with compute across 12 embodiments, fit a law that extrapolates to held-out model sizes, and plan with the large models on a real Franka.

Its hypotheses:

Latent-rollout error on held-out scenes follows a compute law that extrapolates to 4B and 8B models. (§3.1, Table 2) fail
Planning capabilities emerge in order with compute: end-effector control, holding, scene geometry, object dynamics. (Fig. 6–9) not tested here
The larger models plan grasp, lift and pick-and-place on a real Franka from a single image goal. (Table 3) not tested here

Verdict: fail · Claim tested: Pretrained RoboJEPA predicts our latents, uses the actions, and runs on the robot's computer. · Bar: 9-step L1 ≥ 15 % below copy-last (300M) and ≥ 5 % below shuffled actions; 2-view encoder ≤ 200 ms on the Orin. · n: 60 clips (30 for 1B), 25 episodes

Theirs vs ours: rollout error frontier near E ≈ 0.22; latency not reported vs 11.8 % below copy-last (300M; 1B 15.8 % on 30 clips); 9–10 % below shuffled; encoder 526 ms on the Orin NX · Not proven: 1B on a partial sample; TensorRT not tried.

Paper: arXiv 2610.10515 · Code: facebookresearch/robo_jepa

Claims:

fail pretrained RoboJEPA below copy-last baseline on 9-step latent L1 — theirs rollout error frontier near E ≈ 0.22 · ours 11.8 % below copy-last (paper §3.1, Table 2 (DROID holdout, E = 0.218))
fail RoboJEPA encoder at most pass mark on ms per encoder call on the Orin NX — theirs latency not reported · ours 526 ms (our measured comparison)

Compute: External GPU: yes (vast.ai, Gate A): ~$1.2. Gate B on the Orin NX: $0. · Reproduce: 2026-10-23-paper-vs-hardware-robojepa · First published: 2026-10-23 · Updated: 2026-10-23 · Findings: RoboJEPA's encoder on the Orin →

RoboJEPA (Zholus, Assran et al., Meta FAIR; arXiv 2610.10515) is a scaling law for latent world models trained on 15 000 hours of robot video, including 21.5k LeRobot community episodes from arms like mine. Two gates: can the encoder run on the Orin, and does the pretrained predictor use my actions.

What the paper claims, and how it measures it¶

A deterministic next-feature predictor on a frozen V-JEPA 2.1-G encoder (width 1664, patch 16). Per timestep the sequence is all views' patches plus an action token and a proprio token; attention is joint space-time-view, timestep-causal, local to 8 timesteps so it rolls out past its training length; QK-norm for stability. Loss is half teacher-forced L1, half autoregressive L1 from a prefix (the exposure-bias fix). 23 datasets, 12 embodiments, 26 action spaces; AdamW with a shared flat learning-rate stage and per-budget cool-downs.

The scaling law is of per-token L1 error over a 9-step latent rollout on held-out scenes (DROID holdout and RoboCasa), compute counted in FLOPs, compute-optimal frontier across 22M–2B. The functional form was chosen by extrapolation error on 4B and 8B held out of the fit (Table 2): a second-order law wins (0.6 / 1.4 ×10⁻³) over a plain power law (2.0 / 3.7). Downstream: closed-loop CEM planning in RoboCasa with capability emerging in order (end-effector control → holding → scene geometry → object dynamics, the last only at ~10²² FLOPs); a real Franka table (Table 3: 8B grasp 67 %, lift 50 %, pick-and-place 27 % success; π0.5 as a reference, not a baseline); "over 50 000 evaluation episodes" across both platforms, per-task n in an appendix. Planning is "several seconds per step for 8B, not real-time".

What I ran¶

Gate B, encoder on device. Bar: 2-view encoder p50 ≤ 200 ms on the Orin NX. Result: 526 ms (one view 398 ms); the 100M predictor step 252 ms, 300M 487 ms. No-go on device. On the Mac (MPS) the encoder is 1 025 ms and the predictor's attention does not run. Any use is off-board on CUDA.

Gate A, does the pretrained predictor use my actions. Two bars on 9-step rollouts of my bowls latents: (1) L1 ≥ 15 % below copy-last for the 300M model; (2) ≥ 5 % below shuffled actions. Result: (1) 300M 11.8 %, 100M 10.5 %, 1B 15.8 % [14.6, 17.0] on only 30 clips; (2) 8.9–10.4 % for every size. The primary bar fails; the 1B is borderline on a partial sample; action conditioning works on my arm at every size.

G1b-RJ (the 1B on the ranking gate) was cancelled as a gate on 2026-10-15: neither outcome would change the build, since the encoder cannot run in the loop.

Their number, my number¶

	Paper	This arm
Action use	implicit in the rollout error	9–10 % below shuffled, all sizes
Rollout error vs model size	clean frontier, near irreducible E ≈ 0.22	100M 10.5 → 300M 11.8 → 1B 15.8 % below copy-last (30 clips)
Encoder latency	not reported (multi-GPU planning)	526 ms / 2 views, Orin NX
Real-robot n	appendix	30 clips, no robot

What the gap means¶

My arm is in their training mix, and the pretrained model does read its actions — the Gate A shuffled-action margin is the far-left point of their scaling curve, at zero fine-tuning cost. What it cannot do is run where the arm is. The paper is explicit that planning is not real-time even on a GPU cluster; on an Orin the encoder alone eats sixteen control ticks. So the honest use is offline: feature extraction for a judge or a dataset-level probe. The method lesson is the one to keep: a scaling claim is an extrapolation test — fit on some sizes, predict the held-out ones, report the error. My own scaling angle (50 / 100 / 182 / 362 episodes) is a plot until it does that.

Verdict: action conditioning reproduces at the small end; the encoder is off the robot.

What would change it: a distilled encoder at ≤ 100 ms, which is not my experiment to run.

Compute. External GPU: yes (vast.ai, Gate A): ~$1.2. Gate B on the Orin NX: $0.
Reproduce this: 2026-10-23-paper-vs-hardware-robojepa

Papers tested, 3 of 22. ← Papers tested 2: DINO-WM · Papers tested 4: H-JEPA →

Robometer

Fri, 23 Oct 2026 00:00:00 +0000

Robometer¶

They said a large reward model, trained on a million robot videos, can judge runs on robots it never saw. We asked it, untrained, to tell failed from successful runs on 2 099 public SO-101 arm runs. It did not hold: 0.818 out of 1, under our 0.85 bar; on our runs a robot-trained judge, Flora, passed.

Papers tested · 10 of 22

Goal (the paper's): Train a general robot reward model on a vision-language backbone, from about a million trajectories across 21 embodiments with progress, preference and success heads, and use it zero-shot to score progress, rank trajectories and detect failures.

Its hypotheses:

The reward model tracks task progress on embodiments it was not trained on. (Table I (VOC 0.92 / 0.94)) fail
It detects failures zero-shot better than other reward models and token-uncertainty baselines. (Table IV (F1 0.81)) fail
As a reward it improves online and offline RL, including on an SO-101. (Q3, Fig. 7) not tested here

Verdict: fail · Claim tested: A 4B reward model judges SO-101 rollouts zero-shot. · Bar: Pooled AUROC ≥ 0.85 and within-policy ≥ 0.8 on ≥ 5 of 7 policies. · n: 2099 + 88 rollouts

Theirs vs ours: VOC 0.94 on out-of-distribution evaluation sets (0.92 in-distribution) vs pooled AUROC 0.818 on 2099 ArmnetBench rollouts; episode length alone 0.938 · Not proven: Zero-shot only; the LoRA run never trained.

Paper: arXiv 2603.02115 · Code: robometer/robometer

Claims:

fail learned reward model at least pass mark on failure AUROC — theirs failure-detection F1 0.81 (no AUROC reported); VOC 0.94 OOD · ours 0.818 pooled (paper Table IV (zero-shot failure detection))
fail learned reward model above episode length on failure AUROC — theirs no episode-length baseline in the paper · ours 0.818 vs episode length 0.938 (not a paper comparison: the paper's premise that a learned judge beats hand-made signals)
reproduces episode length above learned reward model on failure AUROC · ours 0.938 vs 0.818 (our measured comparison)

Compute: External GPU: yes (vast.ai): ~$8.55–9.55 incl. a LoRA run that never trained and an OOM re-score. · Reproduce: 2026-10-23-paper-vs-hardware-robometer · First published: 2026-10-23 · Updated: 2026-10-23 · Findings: I need to tell success from failure in a rollout →

Robometer (Liang, Korkmaz, Zhang, Hwang et al.; arXiv 2603.02115) is a 4B reward model trained on a million trajectories. I tested it zero-shot as a judge on my own rollouts and on a 2 099-rollout public SO-101 benchmark. Episode length beat it on every held-out task. The paper never computes episode length.

What the paper claims, and how it measures it¶

Qwen3-VL-4B with learned tokens: a progress token after each frame (causal; progress in 10 bins, success at the last frame) and a preference token after two videos. Three losses: preference, progress, success. Pairs are built from expertise (expert vs failure), instruction negatives (same frames, wrong instruction → progress 0), and video rewind. Data: RBM-1M, 21 embodiments; SO-101 is not in it.

Evaluation on RBM-EVAL (976 OOD trajectories): VOC (Pearson r between predicted reward and frame index on successes) 0.92 and Kendall τ on the failed < suboptimal < successful ordering 0.64, vs RoboReward-8B 0.82 / 0.47. Zero-shot on RoboFAC, VOC 0.65. Ablations on LIBERO-90: adding failure data lifts τ from 0.74 to 0.92. Downstream: online RL on a DROID π0 base 20 → 85 % (20 trials); offline IQL on SO-101 "2.4× the best baseline" (0.85 vs 0.50 on an oven task; 0.15 vs 0.05 on a plate task — 3 vs 1 successes in 20); failure detection on 100 DROID trajectories, F1 0.81 vs GPT-5-mini 0.33. The authors' own limitation: no access to contact or grasp stability.

What I ran¶

G9a-AB (2026-10-15/16). Zero-shot on ArmnetBench, 2 099 rollouts, 7 policies, top camera. Bar: pooled AUROC ≥ 0.85 and within-policy ≥ 0.8 on ≥ 5/7. Complete set: pooled 0.818 [0.790, 0.843]; within-policy 5/7; episode length alone 0.938; beyond length (quintile median 0.687, CV gain +0.016) both bars missed. Fail: not a zero-shot judge.

G9a-own (2026-10-16). On my 88 labelled bowls rollouts: AUROC 0.577 [0.44, 0.70]; TPR 0.25 at the calibrated threshold; policy identity 0.878; wrong-object failures scored highest. Fail.

G9b (written 2026-10-22). The LoRA fine-tune on held-out-task folds never trained (the job died at preprocessing). The zero-shot baseline on policy rollouts only: stack 0.564, tool 0.616, cable 0.761, ring 0.933; length 0.903 / 0.855 / 0.946 / 0.965, better on 4/4; CV gain over length +0.008 [−0.001, 0.017]. No verdict.

Their number, my number¶

	Paper	This arm
Ranking quality	VOC 0.92, τ 0.64 (OOD embodiments)	AUROC 0.82 on ArmnetBench, 0.58 on own bowls
Episode-length baseline	not run	0.938 / 4-of-4 held-out tasks
Wrong-object failures	not analysed	scored highest
Downstream n	20 trials	—

What the gap means¶

Robometer and my result agree once the question is phrased right. The model answers "how far along does this look" and does that well (VOC 0.92 is real). My failure mode is identity — the wrong bowl, nested — and a trajectory-comparison objective with instruction negatives at the dataset level has no reason to catch it at the frame level on an embodiment it never saw. The paper's own "pull tissue" and "fold towel" rows (F1 0.58, 0.76) are its version of the same gap. The number that should be in every reward-model paper and is in none of the ten I read: episode length. On my tasks it is the judge every learned judge must beat first.

Verdict: fails as a zero-shot judge on this arm; the gap is the task's, not the model's.

What would change it: an instruction-negative test on my own frames (swap the instruction, fix the frames, does the score move?) before any LoRA; and labels for suboptimal-but-successful episodes, which τ needs and I do not have.

Compute. External GPU: yes (vast.ai): ~$8.55–9.55 incl. a LoRA run that never trained and an OOM re-score.
Reproduce this: 2026-10-23-paper-vs-hardware-robometer

Papers tested, 10 of 22. ← Papers tested 9: VLA-Corrector · Papers tested 11: SmolVLA →

SmolVLA

Fri, 23 Oct 2026 00:00:00 +0000

SmolVLA¶

They said a small robot policy works on a low-cost arm and runs on cheap hardware. On our arm we ran it 10 times and timed it on the robot's computer. It held: 9 of 10 runs succeeded, and our engine cut each step to 238 ms.

Papers tested · 11 of 22

Goal (the paper's): Build a small, open vision-language-action model trained on community LeRobot data that matches much larger VLAs, runs on consumer hardware, and can run asynchronously; shown in simulation and on SO-100 / SO-101 arms.

Its hypotheses:

A 450M VLA pretrained on community SO-100 data matches or beats much larger VLAs on real low-cost arms. (Table 3–4 (SO-101 LEGO 90 %)) reproduces
Asynchronous inference keeps the arm moving while the next chunk is computed. (Algorithm 1; Fig. 5) not tested here
Pretraining on community data raises real-arm success (51.7 → 78.3 % on SO-100, Table 5). (Table 5) not tested here

Verdict: reproduces · Claim tested: A 450M VLA succeeds on an SO-101 task and can run on low-cost hardware. · Bar: 10-trial operator-judged gate (no numeric pass mark registered); chunk inference within the async budget (667 ms). · n: 10 trials per engine

Theirs vs ours: SO-101 LEGO 90 % in distribution; latency not reported for Jetson vs 9/10 on bowls; 974 ms in PyTorch, 238 ms with vla.cpp on the Orin NX · Not proven: The bowls task is at ceiling for every policy; asynchronous execution not timed on the arm.

Paper: arXiv 2506.01844 · Code: huggingface/lerobot

Claims:

reproduces SmolVLA at least pass mark on success on the arm — theirs 90 % (SO-101 LEGO) · ours 9/10 (paper Table 4 (SO-101 Pick-Place-Lego, 90 % in distribution))
reproduces SmolVLA chunk inference below async budget on ms per action chunk on the Orin NX — theirs latency not reported for Jetson · ours 238 ms (vla.cpp + FA), 974 ms PyTorch (our measured comparison)

Compute: External GPU: no (Orin NX, arm). Cost: $0; SmolVLA training cost not recorded. · Reproduce: 2026-10-23-paper-vs-hardware-smolvla · First published: 2026-10-23 · Updated: 2026-10-23

SmolVLA (Shukor, Aubakirova, Capuano et al., Hugging Face; arXiv 2506.01844) is the policy behind almost every number on this site. Its paper's ablation tables are the manual for running it on a Jetson; three of them predicted things I later measured.

What the paper claims, and how it measures it¶

A 450M VLA: SmolVLM-2 (SigLIP + SmolLM2) with the image at 512×512, no tiling, pixel-shuffled to 64 visual tokens per frame; the state projected to one token into the VLM prefix; only the first 16 of 32 LLM layers run; a ~100M action expert alternating cross-attention (to the VLM's keys/values) and causal self-attention over the chunk, trained by conditional flow matching with 10 Euler steps, chunk 50. Only the expert is trained. Pretraining: 481 community SO-100 datasets, 22.9k episodes, with task strings rewritten by a VLM and cameras manually mapped to a consistent order.

Evaluation: LIBERO and Meta-World at 10 trials per task; real SO-100 and SO-101 tasks with 50 demos each (10 per 5 start positions), scored by sub-steps; baselines π0 fine-tuned on the same data and ACT from scratch. Headline: LIBERO 87.3 for 0.45B vs π0 86.0; SO-100 real 78.3 vs π0 61.7 vs ACT 48.3; SO-101 LEGO 90 / 50 in-/out-of-distribution. The ablations: community pretraining is worth +27 points on SO-100 (Table 5); cross+self attention 85.5 vs cross-only 79.0; flow matching 80.3 vs L1 75.3; state in the VLM prefix 80.3 vs in the expert 73.3 (Table 11); chunk 50 fine; and executing 1 / 10 / 30 / 50 actions before re-observing: 80.3 / 82.8 / 70.8 / 51.8 (Table 13). Asynchronous inference keeps success (73.3 vs 78.3) and doubles throughput (19 vs 9 cubes in fixed time). No seeds, no intervals.

What I ran¶

The bowls gate (10 trials, operator-judged). SmolVLA PyTorch on the Orin: 9/10 at 974 ms per chunk. Through vla.cpp: 9/10 at 390 ms; with flash attention 8/10 at 238 ms. Same weights, 4× faster, same task success. The task is at ceiling (ACT 9–10/10, VLA-JEPA 9/10), so it separates engines, not policies.

Parity. The GGUF against PyTorch on fixed frames and shared noise: median max |Δ| 0.80° against a 1° line (noise-to-noise spread in PyTorch itself 12.2°).

Chunk boundaries (G13b, G13b-in-vla.cpp). With a 238 ms chunk at 30 Hz the inference delay is d ≈ 7 steps; a fresh chunk from fresh noise jumps at the junction. Gradient-free prefix overwrite cuts the median jump 72–74 % offline and 71–73 % through the engine at +0.4 ms per chunk.

Oscillation (G14). 24 % of failed episodes reverse direction at ≥ 3 consecutive chunks.

Their number, my number¶

	Paper	This arm
Real-arm success	SO-101 LEGO 90 % ID	bowls 8–9/10 (ceiling)
Latency	not reported for Jetson	974 ms → 238 ms (vla.cpp + FA)
Re-observe interval	Table 13: full chunk costs 29 pts	d ≈ 7, within the "free" band; boundary handled by prefix overwrite
Pretraining value	+27 pts	not re-measured; every fine-tune here starts from it

What the gap means¶

Nothing here contradicts the paper; the paper predicts the work. Table 13 is my boundary problem in the authors' own numbers: executing a whole chunk before re-observing is ruinous, re-observing every ≤ 10 steps is nearly free — if the junction is handled, which is what prefix overwrite and RTC are for. Table 11 is a checklist item for the plugin (state in the prefix, not the expert). Table 5 is the honest footnote under every fine-tune: the community pretraining is doing more than any architecture choice I could make. The one thing I measured that the paper does not is the engine: on an Orin the first 4× is engineering, and the control-loop research cannot start until it is done.

Verdict: reproduces; the ablation tables are the operating manual.

What would change it: nothing needs changing; the open question from this paper on my arm is the async stack's refill threshold g, which I run through vla.cpp and have not pre-registered.

Compute. External GPU: no (Orin NX, arm). Cost: $0; SmolVLA training cost not recorded.
Reproduce this: 2026-10-23-paper-vs-hardware-smolvla

Papers tested, 11 of 22. ← Papers tested 10: Robometer · Papers tested 12: TurboVLA →

TurboVLA

Fri, 23 Oct 2026 00:00:00 +0000

TurboVLA¶

They said a light policy without a large language model still follows its instruction. On our arm we trained it on our two tasks and swapped the instruction. Our claim that it ignored the instruction is withdrawn: the data caused the 0.020° change, not the model.

Papers tested · 12 of 22

Goal (the paper's): Make a lightweight VLA without a full VLM backbone (DINOv3 vision and a BERT text encoder fused for an action decoder) that is fast enough for real-time control; shown on LIBERO and RoboTwin 2.0 in simulation and on an AgileX Piper.

Its hypotheses:

A policy without a large VLM backbone still follows the language instruction. (§4 ablations, Table 3 (LIBERO-Goal 97.4 → 11.6 without language)) withdrawn

Verdict: withdrawn · Claim tested: Our claim that the architecture made the policy ignore its instruction. · Bar: Task swap ≥ 5° and per-layout ≥ 75 % → our port was the bug; < 1° with the control ≥ 0.095 → the data. · n: 25 val episodes, 8 frames

Theirs vs ours: LIBERO-Goal 97.4 → 11.6 without language (Table 3) vs upstream's own trainer on our data: task swap 0.020°, per-layout 38.9 %; control, their released LIBERO checkpoint run here: 0.105 (≈ 5.5° of pan) · Not proven: One seed; two tasks only, instruction redundant on 94 % of frames.

Paper: arXiv 2607.27205 · Code: H-EmbodVis/TurboVLA

Claims:

withdrawn TurboVLA on our data at least pass mark on task-swap action change — theirs LIBERO control 5.5° of pan · ours 0.020° on our data; LIBERO control 5.5° (our measured comparison)

Compute: External GPU: yes (vast.ai): path A ≈ $2.0 + ~$0.8 of failed starts. · Reproduce: 2026-10-23-paper-vs-hardware-turbovla · First published: 2026-10-23 · Updated: 2026-10-23

TurboVLA (Xie, Yao, Wu et al.; arXiv 2607.27205) drops the language model from the VLA and binds vision to text with Grounding-DINO-style cross-attention. My port failed on a two-task dataset, I concluded it ignored the instruction, then I found my port differed from the paper in four places, then I ran the paper's own trainer on my data. The faithful model ignores the instruction too. The data was the whole story.

What the paper claims, and how it measures it¶

V + L → A instead of V → L → A: DINOv3 patch features and BERT token features, each projected to 256-d, fused by six bidirectional cross-attention layers initialised from Grounding DINO (--pretrained_init_ckpt groundingdino_swint_ogc.pth), then an ACT-style decoder with 12 learnable action queries; the robot state enters the decoder memory only (eq. 6/9), because "it is required for translating task-conditioned scene features into executable actions but not necessary for visual-language correspondence". Plain ℓ1 on the chunk. 0.2B parameters, 31 ms per chunk on a 4090.

Evaluation: LIBERO with one mixed-suite model, 50 rollouts per task, 97.7 average (π0.5 96.9 at 3× the latency, SmolVLA 88.8); RoboTwin 2.0 multi-task 60.2; real Piper, four tasks, 40 trials each, against π0.5 on the same data. The ablation that matters here, Table 3: remove language, LIBERO average 97.7 → 70.8, LIBERO-Goal 97.4 → 11.6. On data where the instruction is the only thing that distinguishes targets, this architecture uses it. One-way cross-attention costs 1.2–1.6 points (Table 5). The Grounding DINO initialisation is stated once and never ablated.

What I ran¶

Several variants, Sept 29 – Oct 8. 0/3 on the last; a two-week diagnosis with six probes concluded the model ignored the instruction on the two-task bowls data. The data finding survived: on 94 % of frames the instruction is redundant with the arm state ("fact 7": the side decision is visible in ~6 % of frames and the pose leaks it), so a copycat policy fits the data.

The port review (2026-10-22). Against upstream turbovla/models/: (1) the six fusion layers were random-init in every variant — the checkpoint converter transferred DINOv3 only; (2) the fusion block is a different module — two independent cross-attentions instead of BiAttentionBlock (shared affinity, layer-scale 1e-4, drop-path) plus a text self-attention layer; (3) the state token was prepended to vision before fusion instead of entering the decoder memory; (4) decoder depth, head and normalisation differ. So "TurboVLA ignores the instruction" was never established; a port that ignores it was.

Faithful reproduction, step 1 (2026-10-22). The free control: upstream code, released LIBERO checkpoint, task swap over the ten LIBERO-Goal instructions on 40 frames. Threshold 0.095 normalised (= 5° of SO-101 pan). Result 0.105 — the real model moves its actions when the instruction changes; my port moved 0.1–0.25°, 20–55× less. Path A was frozen on that number: task swap ≥ 5° with per-layout ≥ 75 % → the port was the bug; task swap < 1° with the control ≥ 0.095 → fact 7.

Faithful reproduction, path A (2026-10-23). Upstream's own trainer on the bowls data, full LIBERO init (666 of 672 tensors transferred; the 6 width-dependent ones re-initialised), 35k steps, batch 64 (128 OOM on 24 GB), ≈ $2.0 plus ~$0.8 of failed starts. Result: task swap 0.020° (frames 0.006–0.053°), per-layout 38.9 %, start reach 26.7°; the control still 0.105. The paper's recipe, with the paper's initialisation and the paper's fusion block, moves its actions by two hundredths of a degree when the instruction changes on this data. Fact 7 is the full story.

Their number, my number¶

	Paper	This arm
Instruction sensitivity	LIBERO-Goal 97.4 → 11.6 without language	LIBERO control 0.105 (reads it); port 0.1–0.25° and faithful 0.020° on bowls (neither does)
Fusion init	Grounding DINO, never ablated	port random vs faithful Grounding DINO: no effect on instruction use on this data
Real-arm success	80–92.5 %, 40 trials	0/3 on the last port
Data	multi-task, instruction decisive	two tasks, instruction redundant on 94 % of frames

What the gap means¶

Two claims, now both answered. The data claim stands and is now the whole result: on frames where the instruction is redundant with the arm state 94 % of the time, the paper's architecture with the paper's weights learns the copycat too (0.020° vs 0.105 for the same model on LIBERO-Goal, where the instruction is decisive). The model claim — that my port was the reason — is false: the four divergences are real, documented, and made no difference to instruction use. Table 3 of the paper and path A together say the same thing from two sides: this architecture reads language exactly when the data makes language necessary, and not otherwise. A by-product: the Grounding DINO initialisation, never ablated in the paper, is ablated here on one dataset (random vs faithful) with no effect on the quantity that mattered.

The lesson I am keeping is about order, not architecture: parity against upstream should have come before the two-week diagnosis, and the diagnosis should have ended with a data-side control (train the paper's model on data where the instruction is decisive) before any sentence about the model. Both are now rules in the register.

Verdict: the data claim (fact 7) is the full story; the port claim is withdrawn; correction row written.

What would change it: nothing about this dataset. The next test is the paper's model on data where the instruction decides early and often (the tidy-loop sessions), which is the only way to learn whether it helps on this arm.

Compute. External GPU: yes (vast.ai): path A ≈ $2.0 + ~$0.8 of failed starts.
Reproduce this: 2026-10-23-paper-vs-hardware-turbovla

Papers tested, 12 of 22. ← Papers tested 11: SmolVLA · Papers tested 13: VLA-JEPA →

VLA-Corrector

Fri, 23 Oct 2026 00:00:00 +0000

VLA-Corrector¶

They said a small model watching camera statistics and actions flags a failing run early enough to fix it. On our arm we trained the same kind of watcher on 50 recorded runs. It did not hold: it caught 2/8 failures, below our bar.

Papers tested · 9 of 22

Goal (the paper's): Keep a frozen π0.5 on task by detecting deviations with a learned verifier on observable history (cheap visual statistics, state and action, a GRU) and prompting recovery when it fires; shown in LIBERO and LIBERO-Plus and on a physical arm.

Its hypotheses:

A verifier on observable history detects execution risk (AUROC 0.902) close to a privileged rule-based verifier. (§ Failure Detection and Recovery Analysis (AUROC 0.902); Fig. 4) fail
Prompting recovery when the verifier fires raises success under perturbations. (Fig. 3, Table 3; real-robot results) not tested here

Verdict: fail · Claim tested: A verifier on observable history flags a failing rollout early enough to correct it. · Bar: ≥ 5/8 failures at ≤ 1/20 false interventions. · n: 50 episodes, 8 failures

Theirs vs ours: 54-feature GRU verifier AUROC 0.90 vs the paper's verifier family on our rollouts (a GRU on pooled RGB statistics and actions): 2/8 caught at 3/42 false interventions, AUROC 0.59; cosine drift of the W1 latent 0/8, AUROC 0.30 · Not proven: Their exact 54-feature verifier (with proprioceptive state) and its recovery prompt were not run here; our GRU uses pooled RGB statistics and actions.

Paper: arXiv 2609.06508 · Code: no code link found

Claims:

fail observable-history verifier at least pass mark on failures caught — theirs AUROC 0.90 · ours 2/8 caught at 3/42, AUROC 0.59 (§ Failure Detection and Recovery Analysis (AUROC 0.902); Fig. 4)

Compute: External GPU: no (Orin NX). Cost: $0. · Reproduce: 2026-10-19-six-monitors-w1 · First published: 2026-10-23 · Updated: 2026-10-23

VLA-Corrector (Song, Qian, Feng, Song; arXiv 2609.06508) wraps a frozen π0.5 in a tiny learned verifier that routes a recovery prompt. I borrowed one of its signals — cosine drift of the latent — as a monitor. It caught nothing. The part of the paper I should have borrowed is the verifier's design.

What the paper claims, and how it measures it¶

The policy is frozen (OpenPI pi05_libero). The verifier reads only observable history: two 112×112 views → parameter-free spatial statistics (3×4×4 pooled RGB plus mean and std, 54 features per view) → Linear 64; state and action → MLP 32; a GRU over an 8-step window; two heads, binary risk and a five-way stage. Teacher labels come from a rule-based state machine on simulator state (f = 1 within 10 steps of a failure event, 0 beyond 20). Gate: τ = 0.40, four consecutive positive windows, 10-step cooldown; the stage selects a recovery prompt fragment appended to the instruction; up to 100 recovery actions, then the original instruction resumes. Verifier training: 120 episodes, 30 epochs, CPU; 0.46 ms inference.

Evaluation: LIBERO and LIBERO-Plus difficulty 4–5 under action noise, target shift, goal shift and all three, 1 000 episodes per condition; three configurations — no assistance, a privileged rule verifier (the efficacy upper bound), the learned verifier. Combined perturbation 70.7 → 82.6 (rule) / 80.2 (learned): the learned verifier is within 2.4 points of the oracle. Verifier quality: accuracy 93.1 %, precision 61.1 %, recall 94.9 %, AUROC 0.90; recovery success 75 %. Ablations: single frame −15 F1, state-action only −15.8. Real arm (unnamed), three tasks, 40 trials each, target moved after execution starts: 82.5 → 88.3 average, zero-shot from sim-trained features.

What I ran¶

W1-A (2026-10-19). Cosine drift of W1's latent as a mid-rollout screener on 50 episodes, beside L2 surprise and the live reflex. Bar: ≥ 5/8 failures at ≤ 1/20 false interventions. Result: cosine drift 0/8, AUROC 0.30 — below chance; L2 surprise 2/8 at 2/42; the live reflex flagged 42/42 successes. Fail.

I never built the paper's verifier itself.

Their number, my number¶

	Paper	This arm
Signal tested	54-feature GRU verifier, AUROC 0.90	cosine drift of W1 latent, AUROC 0.30
Gate design	recall-tuned (95 %), 4-window confirm, cooldown	single threshold
Cost of a false trigger	a recovery prompt, ~100 steps	a wasted trial (my bar: ≤ 1/20)
Upper bound	privileged verifier, within 2.4 pts	none

What the gap means¶

Cosine drift was a wrong borrowing — the paper's verifier is not a drift threshold. Two things from the paper matter more than the result. First, the precision/recall split: a 61 % / 95 % verifier is a usable gate when a false trigger costs a re-prompt rather than a trial. My monitor bar (≤ 1/20 false interventions) assumes the expensive kind; if the intervention is "re-observe and re-prompt", the bar can relax and the 6/8-at-7/42 risk head from the ReAP post may already be a useful gate. Second, the parameter-free front end plus a GRU reaches 0.90 on LIBERO at zero cost on the Orin; it is the right floor baseline to put under any W1-based monitor, and it is now in the amendments as one.

Verdict: the borrowed signal fails; the paper's verifier is untested here and is the next floor baseline.

What would change it: the 54-feature GRU on my 69 rollouts, beside the risk head, with the recovery cost stated in the bar.

Update 2026-10-31 (claims audit, WS-068). The register now files this paper's claim under its own method, not under latent drift. The closest test this rig has run is the GRU "floor" of the REAP v2 gate: a GRU on pooled RGB statistics and the commanded actions, the verifier's family without the proprioceptive features, on the same 50 episodes and bar. It caught 2/8 failures at 3/42 false interventions (AUROC 0.59), below the bar; the verdict stays fail. Their exact 54-feature verifier is still not run here.

Compute. External GPU: no (Orin NX). Cost: $0.
Reproduce this: 2026-10-19-six-monitors-w1

Papers tested, 9 of 22. ← Papers tested 8: ReAP · Papers tested 10: Robometer →

VLA-JEPA

Fri, 23 Oct 2026 00:00:00 +0000

VLA-JEPA¶

They said a robot policy pretrained on human videos without actions trains cheaply and runs well. We trained it on our data for about $10 and ran it on the arm 10 times. It held: 9 of 10 runs succeeded, at 355 ms per chunk on the robot's computer.

Papers tested · 13 of 22

Goal (the paper's): Pretrain a VLA on human videos without actions through a latent world model, so that its latent action tokens carry state-transition structure without leaking future frames; evaluated on LIBERO, SimplerEnv, LIBERO-Plus and a real Franka.

Its hypotheses:

Pretraining on action-free human video through a leakage-free latent world model improves robustness to perturbations. (Table 3 (LIBERO-Plus 79.5 vs 62.9 without human video)) not tested here
The model reaches state-of-the-art success on LIBERO and transfers to a real arm with 100 demonstrations. (Table 1; Fig. 4) reproduces

Verdict: reproduces · Claim tested: VLA-JEPA trains cheaply on SO-101 data and runs on the robot. · Bar: 10-trial operator-judged gate (no numeric pass mark registered); for reranking (H0): oracle / mean error ≤ ~0.7 and N = 8 ≤ 1.5× the latency. · n: 10 trials; H0 on the teleop val split

Theirs vs ours: +1.1 over no-human-video on LIBERO (Table 1); no latency reported vs 9/10 for ~$9.7; 355 ms per chunk on the Orin NX; reranking no-go (0.76, 1.84×) · Not proven: Perturbation robustness (their main claim) not tested.

Paper: arXiv 2602.10098 · Code: ginwind/VLA-JEPA

Claims:

reproduces VLA-JEPA at least pass mark on success on the arm — theirs real Franka: ahead of π0 and π0.5 in-distribution and with object-layout changes, second on unseen tasks (Fig. 4) · ours 9/10 (paper §4.4, Fig. 4 (real Franka))
fail best-of-N reranking at most pass mark on oracle-to-mean error ratio · ours 0.76, 1.84× latency (our measured comparison)

Compute: External GPU: yes (vast.ai): ~$9.7 for the fine-tune. Everything else on the Orin NX, $0. · Reproduce: 2026-10-23-paper-vs-hardware-vla-jepa · First published: 2026-10-23 · Updated: 2026-10-23

VLA-JEPA (Sun, Zhang, Qi et al.; arXiv 2602.10098) pretrains a VLA on human video through a latent world model. I trained it for $10, ran it on the arm at 9/10, and tested whether its eight samples were worth choosing between. The paper's honest ablation row says what to expect: robustness, not accuracy.

What the paper claims, and how it measures it¶

A Qwen3-VL backbone with a flow-matching head. Latent-action tokens are read out of the VLM given the initial frame only plus the instruction — no future frame ever enters the VLM, which is the leakage-free principle — and an autoregressive latent world model predicts frozen V-JEPA 2 state chunks from them; L_WM is the teacher-forced prediction error. Robot data adds action tokens after the latent ones and a flow-matching loss; the joint objective is L_FM + β L_WM. Pretrained on Something-Something-v2 (220k human videos) and DROID; fine-tuned per benchmark.

Evaluation: LIBERO at 500 episodes per suite, 97.2 vs π0.5 96.9 — and 96.1 without human video (+1.1). SimplerEnv: Google Robot 65.2 with human video, 78.4 without (the authors attribute the reversal to the real-to-sim gap). LIBERO-Plus, seven perturbation axes: 79.5 vs OpenVLA-OFT 69.6, and 62.9 without human video (+16.6, the number the paper is about). Real Franka, 100 demos, three tasks: 0.70 vs π0 0.57 in distribution. No intervals; ±1 on LIBERO is inside binomial noise at this n.

What I ran¶

Training. Qwen3-VL-2B + DiT-B, world model off at inference, ~$9.7 on vast.ai over two days.

The bowls gate. 9/10 on the Orin NX at 355 ms per chunk (PyTorch); in the twelve-VLA latency benchmark through vla.cpp, 177 ms minimum.

H0 (2026-10-08). The question the policy's design invites: do its 8 flow samples vary enough that picking one with a world model would help? Bars: oracle/mean error ≤ 0.7, latency at N = 8 ≤ 1.5×, baseline success < 85 %. Result: 0.76 (null 0.92); 1.84× (643 vs 349 ms); baseline 90 %. No-go. Averaging the eight chunks got 45 % of the oracle's gain with no model at all. Reranking was dropped the same day, and with it the first architecture version.

Their number, my number¶

	Paper	This arm
In-distribution success	+1.1 over no-human-video (LIBERO)	9/10, same as ACT and SmolVLA (ceiling)
Perturbation robustness	+16.6 (LIBERO-Plus)	not tested
Sample diversity useful?	not asked	oracle/mean 0.76; mean-of-8 gets 45 % of it free
Latency	108.7 ms (4090)	355 ms PyTorch / 177 ms vla.cpp on Orin NX

What the gap means¶

The paper's own ablation rows say the human-video pretraining buys robustness to perturbation, not nominal accuracy, and sometimes costs accuracy (SimplerEnv). A nominal gate at ceiling, which is all I ran, cannot see that and did not. If I ever run a perturbation gate — lighting, background, camera — this paper predicts which policy survives and why; a nominal gate showing no difference would confirm it, not refute it. The H0 result is mine, not theirs: the samples do not differ enough to be worth a chooser, and the cheapest chooser (the mean) is half as good as an oracle for free. That one afternoon ended a design.

Verdict: trains and runs as advertised; the robustness claim is untested here; sample reranking is a no-go.

What would change it: a perturbation gate against SmolVLA on the same layouts, after the camera question (SEVO) is settled.

Compute. External GPU: yes (vast.ai): ~$9.7 for the fine-tune. Everything else on the Orin NX, $0.
Reproduce this: 2026-10-23-paper-vs-hardware-vla-jepa

Papers tested, 13 of 22. ← Papers tested 12: TurboVLA · Papers tested 14: REKI →

v1.5: the station as a harness for a coding agent

Thu, 22 Oct 2026 00:00:00 +0000

v1.5: the station as a harness for a coding agent¶

We wanted to know what a one-person robot lab needs to run trials unattended and keep data clean. We specified a harness from nine recent papers on agent harnesses and self-improvement. It is a design: the agent stays outside the 33 ms control tick, and nothing has run end to end.

Evaluation and judges · 7 of 14

Type: design · Result: A station design where the coding agent dispatches unit actions and batches and never sits inside the 33 ms tick. · n: —

Not proven: Not run end to end on the arm.

Compute: None; reads results from the folders above. · Reproduce: no run behind this post · First published: 2026-10-22 · Updated: 2026-10-22

Three weeks of gates reduced the world-model stack to a verifier and a shadow signal. What remained was the actual bottleneck of a one-person robotics lab: deciding what to record next, running trials unattended, and keeping the training data clean. v1.5 is that loop, specified from nine recent papers on agent harnesses and self-improvement, and partly built. It has not run end to end on the arm yet; this post is a design note, not a result.

The shape¶

Claude Code (operator) ── skills: use-case brief, station-session, failure-review │ MCP Run layer journal, budgets per failure class, resume, full trace Harness tools observe · do_action · run_batch · run_policy · undo · tickets · estop Executors scripted_pick_place | run_policy (SmolVLA 450M via vla.cpp) Task plugin catalogue · planner · perception · VERIFIER (tiered) · gates Robot SO-101 · cameras · scale · Jetson Orin NX · W1 in shadow mode

The agent dispatches unit actions and batches; it never sits inside the 33 ms tick (HexaShell's 739 s of agent time for 26 s of robot time is the number to avoid). A new task is a plugin: catalogue, planner, perception config, verifier, gates. The harness and the logging schema do not change.

Three contracts¶

Two-field results. executor_status (done / timeout / clamped / error) and verdict (satisfied / violated / ambiguous / stale) are separate; ok is their conjunction. The executor never writes the verdict, the verifier never reads the agent's text. ambiguous can only lead to a budgeted re-observe.
Idempotent dispatch. Every call carries (run, action_id, attempt); a repeat returns the existing trial. This closes "the agent retried after a lost tool response and the arm picked twice".
Verifier tiers. 1 physical measurement (a scale, a count), 2 deterministic perception at rest (the G9c-3 judge), 3 learned or VLM judge, 4 owner label. Tiers 1–2 can pass a gate and admit trials to demos/ alone. Tier 3 only pairs with tier 2, and only after a report card on that task. The gate on tier 3 is the 10-19 and 10-20 results written as policy.

The flywheel¶

Every trial is a LeRobot v3 record with task string, layout id, executor and version, verdict and evidence. demos/ is a view over satisfied trials from tier-1/2 verifiers. The next teleop session is planned from the log: failed layouts first, at most a third of the budget, then farthest-point coverage of the workspace. A policy replaces a scripted executor only through an adoption protocol (G24): enough verified demos on the failing skill, a verification window with scripted fallback, per-call finish rate not below the incumbent, revert otherwise.

First task¶

LEGO kitting: pick bricks from bulk into a carousel, verify by weight. Chosen because it has a tier-1 verifier for free, and because a slow robot only caps volume on a product, where it would lose outright on a service. First teleop session: 60 episodes; harness code: 284 tests. The gates for it (H1, G22, K1, G24) are frozen and waiting for the arm.

Compute. None; reads results from the folders above.
No run behind this post; it reads results from: 2026-10-23-paper-vs-hardware-robometer (the judge at rest).

Evaluation and judges, 7 of 14. ← Evaluation and judges 6: Two gates I voided, and the two-line check that would have caught them · Evaluation and judges 8: Training a JEPA world model on SO-101 data for $3.30 →

Two gates I voided, and the two-line check that would have caught them

Thu, 22 Oct 2026 00:00:00 +0000

Two gates I voided, and the two-line check that would have caught them¶

We wanted to know whether two benchmark results on the Jetson measured what they claimed. We reloaded each script's model the strict way and checked whether its timing parts added up. One timed a random encoder, with 192 missing keys; the other timed the wrong model.

Evaluation and judges · 6 of 14

Type: correction · Result: G11-A timed a random encoder and G12 a stand-in model; both gates are void. · n: two gates

Not proven: The voided numbers support nothing.

Compute: External GPU: no (Orin NX). Cost: $0. · Reproduce: 2026-10-22-voided-gates · First published: 2026-10-22 · Updated: 2026-10-22 · Findings: Two gates voided by a two-line check →

On 2026-10-18 two Jetson gates went into the register with bars frozen beforehand: G11-A passed, G12 failed. Both were voided the same day. G11-A timed the right model with the wrong weights; G12 timed the wrong model. Neither was a training or hardware problem. Both were caught by checking what the script had actually loaded.

G11-A: 12.78 ms of a random encoder¶

G11 Phase A asked whether W1, the multi-view JEPA world model, can run its surprise check inside the 33.3 ms control tick on the Orin NX, motors off. Bar: under 20 ms per check. Result: mean 12.78 ms, p95 12.87, max 12.95. Pass, 20.5 ms of headroom.

The next Jetson run was meant to measure W1's error on a still scene. No camera was attached, so only the checkpoint load could run, through the live loader, which refuses strict=False. It failed: 192 missing keys, 192 unexpected, all in the ViT encoder.

The checkpoint was saved under transformers 5.19.0, which names ViT layers encoder.layers.N.attention.q_proj.*. The Docker image ships 5.5.4, which names the same tensor encoder.encoder.layer.N.attention.attention.query.*. A key remap renamed layers → layer and q_proj → attention.query, but did not add the inner encoder. level, so the remapped keys still missed. Under 5.19.0 all 279 checkpoint keys load verbatim, no remap.

G11-A's benchmark used the same remap with strict=False. All 192 encoder tensors stayed at random initialisation. The timing stands, since the architecture and compute are the same, but no latent and no surprise value from that script came from the trained encoder. The unit tests passed throughout: they only exercised the layers → layer rename on a stand-in model.

G12: parts that don't sum¶

G12 asked whether REKI, a 2.4B VLA with register-token memory, fits the Orin: memory ≤ 8 GB, chunk ≤ 250 ms. Result: 3.46 GB, 318.2 ms. Fail, and a decision to port the registers into SmolVLA instead.

Re-reading the report: 135.4 + 230.3 is not 318.2. The expert time came from a separate loop. Then the benchmark itself: a randomly initialised Qwen2.5-1.5B, text only, 256 random tokens, no vision tower, none of Qwen3.5's Gated DeltaNet layers, followed by a hand-written random expert, in eager PyTorch. And 3.46 GB matches that 1.64B-parameter proxy, not REKI's 2.4B. So 318 ms is neither REKI's latency on the Orin nor a bound on it.

The two-line check¶

m = W1LiveMonitor.from_checkpoint(ckpt) # load_state_dict(strict=True), never strict=False assert set(m.model.state_dict()) == set(torch.load(ckpt, map_location="cpu", weights_only=False)["state_dict"])

The first line raises on any missing or unexpected key. The second confirms the checkpoint's keys were used verbatim, not through a remap that happened to fit. On G11-A's setup the first line stops at 192 keys before a single timing is taken.

Parts must sum¶

G12's check is arithmetic. If a report splits a latency into parts, the parts add up to the total, measured in the same loop. And the thing measured must be the thing named: parameter count, vision tower, layer types. A 2.4B model that peaks at the memory of a 1.64B one is a second sum that does not hold.

What changed¶

Both gates had frozen bars, and both results were real measurements of something. Pre-registration stops the bar from moving after the number; it says nothing about what produced the number. So the checks now come before the number: a strict load with a key-set comparison for any checkpoint, and a sum over the parts for any latency breakdown.

G11-A's 12.78 ms survives. G12 is open again: the REKI decision now rests on the G14 oscillation audit and on a real measurement, with the vision tower and the real expert.

Compute. External GPU: no (Orin NX). Cost: $0.
Reproduce this: 2026-10-22-voided-gates

Evaluation and judges, 6 of 14. ← Evaluation and judges 5: What the world model is actually for · Evaluation and judges 7: v1.5: the station as a harness for a coding agent →

Judging, part 2: the input format did the work

Tue, 20 Oct 2026 00:00:00 +0000

Judging, part 2: the input format did the work¶

We wanted to know which judge could read a run's outcome from video, beyond our two-bowl rule. We gave a robot-trained judge and three general models the same clip and question, with a bar fixed first. The robot judge caught every failure at 2/42 false alarms, and its untrained base model nearly matched it.

Evaluation and judges · 3 of 14

Type: method · Result: With Flora-Rewarder's input, a general Qwen3.5-27B passes too (7/8 at 2/42): the input format did the work. · n: 50 episodes, 8 failures

Not proven: One prompt per model; Flora's licence is noncommercial.

Compute: External GPU: yes (vast.ai, ≥ 80 GB): ~$2.2. External API: Gemini ~$2.75. Total ~$4.95. · Reproduce: 2026-10-20-flora-rewarder-input-format · First published: 2026-10-20 · Updated: 2026-10-20 · Findings: I need to tell success from failure in a rollout →, The input format did the work, not the robot fine-tune →

The deterministic judge is right but narrow: it knows about two coloured bowls. Any new task needs a judge that reads video. NVIDIA's Flora-Rewarder-27B, a robot-trained outcome scorer, passed where Gemini had failed the day before. The ablation says why, and it is not the robot training.

G-flora¶

Input exactly as NVIDIA's terminal prompt expects: 64 frames, top and wrist camera side by side, no telemetry, greedy decoding. 50 episodes, 42 successes and 8 failures, bar frozen before any call: at the largest score threshold flagging ≤ 2/42 successes, ≥ 6/8 failures must be flagged.

8/8 failures, 2/42 successes, AUROC 1.00. Successes score 4–5, failures 1–3, including both wrong-object cases and both nested-bowl successes that fooled Gemini.
Deterministic (5/5 identical), 0 unparseable, 6 s per episode on an A100, $0.70 total.

Flora's licence is non-commercial, so it is evidence, not a component.

G-flora-abl: fine-tune or format?¶

Same clip, same prompt, three general models.

Model	Successes flagged	Failures flagged	AUROC	Verdict
Qwen3.5-27B (Flora's base, no robot fine-tune)	2/42	7/8	0.91	pass
Qwen3.5-9B	16/42	–	0.75	fail
Gemini 3.8 Flash, 64 JPEGs	3/42	8/8	0.96	narrow fail

Flora's base model, untouched, passes the same bar. The 9B does not. Gemini, which caught 0/8 with sliding windows the day before, catches 8/8 with the whole episode and a terminal question. The fine-tune added a little; the input format (whole episode, both views, one outcome question at the end) and model size did the work.

What goes into the stack¶

A commercial rewarder fine-tuned from Qwen3.5 on own labels is the planned tier-3 verifier. It pairs with the deterministic judge, never replaces it, and it gets a report card (false positives and negatives against owner labels) on every new task before any of its verdicts enter a training set. That rule is written into v1.5.

Compute. External GPU: yes (vast.ai, ≥ 80 GB): ~$2.2. External API: Gemini ~$2.75. Total ~$4.95.
Reproduce this: 2026-10-20-flora-rewarder-input-format

Evaluation and judges, 3 of 14. ← Evaluation and judges 2: Verify at rest: six mid-episode monitors fail in one day · Evaluation and judges 4: Latent distance is not progress →

Verify at rest: six mid-episode monitors fail in one day

Mon, 19 Oct 2026 00:00:00 +0000

Verify at rest: six mid-episode monitors fail in one day¶

We wanted to know if any monitor could catch a failing run before it ended. We ran every mid-run checker in the design on 50 fresh runs, with the bars fixed the night before. Every monitor failed; the world model's live reflex fired on 42/42 successful runs.

Evaluation and judges · 2 of 14

Type: method · Result: Six mid-episode monitors miss their bars; the best, L2 surprise, catches 2/8 at 2/42. · n: 50 episodes, 8 failures

Not proven: One world model; one prompt per VLM.

Compute: External GPU: no (Orin NX). External API: Gemini 3.8 Flash, ~$2.1–3.1 (one figure estimated: the API did not report it). · Reproduce: 2026-10-19-six-monitors-w1 · First published: 2026-10-19 · Updated: 2026-10-19 · Findings: I need an online failure monitor →, Mid-episode monitors miss; verify at rest instead →

One Jetson day, no arm motion, every mid-episode verifier in the v1.0 design run against 50 fresh rollouts with the bars frozen the night before. Stage trackers, VLM judges and the world-model reflex all failed. One offline engine experiment passed. The architecture that came out of it (v1.1) verifies only at rest, and that rule has not moved since.

The world model, on fresh data¶

Test	Bar	Result
W1-A: L2 surprise, cosine drift, live reflex (2.2 / 10.0) as screeners	one signal catches ≥ 5/8 failures at ≤ 1/20 false interventions	L2 2/8 (AUROC 0.86); drift 0/8; the live reflex fires on 42/42 successes
G11-B-cal: refit the reflex from episode maxima	recall ≥ 50 % on 25 failures	3/25 at 1/44 false alarms; success and failure maxima overlap
W1-C1: event-anchored grasp check from the imagined latent	AUROC ≥ 0.85, ≥ 0.05 above the current latent	imagined 0.70, current 0.82: imagination is worse
W1-C2: conformal band on the surprise trace (FAIL-Detect / SAFE)	successes flagged ≤ 1/20, failures ≥ 50 %	22.7 % / 46 %; the band is not conformal across time when μ and σ are fit on the same successes

G10's thresholds did not transfer. The world model went to shadow mode: scored and logged on every trial, never allowed to stop the arm.

Stage trackers and VLM judges¶

G15 (proprio + masks): 48 % premature, 47 % missed. The gripper reads 1.1° when holding a bowl rim, below every "closed on nothing" threshold, so a grasp detector built on aperture cannot fire.
G15-v2 and the latent-prototype tracker W1-B: unusable on this gripper; 22 % / 31 %.
Gemini Flash with sliding-window FAIL probability: threshold fitted at 0.98, 0/8 failures caught, 10 s per call.
Gemini Flash with two checkpoint calls (after grasp, at rest): 41/42 successes flagged. Nested bowls read as "blue alone on the mat"; the post-grasp checkpoint comes before the lift.

The one that passed: G13b¶

Chunk boundaries are where SmolVLA jerks. Real-time chunking (RTC) does inference-time guidance; the gradient-free alternative overwrites the new chunk's prefix with the tail of the old one. On 20 episodes, 254 boundaries: prefix overwrite cuts the median boundary jump 72–74 %; exact RTC 21–41 % at twice the latency. Ported into vla.cpp the same day: 71–73 %, prefix error 0.0, +0.4 ms per chunk, default path bit-identical, 14/14 tests.

v1.1¶

Verification and repair at rest with the deterministic judge; prefix overwrite in the engine; world model in shadow mode; every trial recorded. The simplest design of the series, and the first one forced entirely by its own negative results.

Compute. External GPU: no (Orin NX). External API: Gemini 3.8 Flash, ~$2.1–3.1 (one figure estimated: the API did not report it).
Reproduce this: 2026-10-19-six-monitors-w1

Evaluation and judges, 2 of 14. ← Evaluation and judges 1: Judging robot rollouts automatically: a success detector with zero false positives · Evaluation and judges 3: Judging, part 2: the input format did the work →

What the world model is actually for

Sat, 17 Oct 2026 00:00:00 +0000

What the world model is actually for¶

We wanted to know if a world model could read a run's outcome, and see it coming. We tested its latent at the moment of judgement and 30 steps earlier, against which policy ran. The outcome readout scored 1.000, but on runs it had seen in training.

Evaluation and judges · 5 of 14

Type: method · Result: The latent reads outcomes at rest (AUROC 1.00 vs the judge) and anticipates failures before execution (+0.18 over policy identity). · n: 88 rollouts

Not proven: Step 2 is compared against policy identity only.

Compute: External GPU: no (Mac). Cost: $0. · Reproduce: 2026-10-17-g10-outcome-readout · First published: 2026-10-17 · Updated: 2026-10-17

With a judge that is right at rest, the world model finally had a fair question: can it read the judge's verdict from the latent, and can it see it coming? G10 ran on the Mac for free. It is the one world-model gate that passed cleanly, and the next post is about how much of it survived contact with fresh data.

G10¶

Bars: outcome readout at the judge's timestamp AUROC ≥ 0.90; anticipation 30 steps earlier must beat policy identity by +0.10.

Step 1, readout at t_j: W1 AUROC 1.000, W2 0.976. The latent at rest contains the outcome.
Step 2, anticipation at t_j − 30: the imagined latent beats policy identity by +0.179 (0.993 against ground truth).
Mid-chunk surprise (prediction error of the next latent): 1.82× higher on missed grasps, 2.58× on stalls, from the first step.

Pass. The architecture of record changed the same day: no planning, no progress tracking. The world model is a pre-execution veto (imagine the chunk, refuse it if the imagined outcome reads as failure) and a mid-chunk surprise monitor.

G11-A: does it fit in the tick?¶

Dual-camera surprise check on the Orin NX, FP16 CUDA: 12.78 ms mean, 12.95 ms max, inside a 33 ms control tick with 20 ms to spare. Pass.

A correction row was added the next day: the benchmark had loaded the checkpoint with strict=False and a key remap that did not fit the image's transformers version, so its encoder was random. The timing stands (same architecture); the latent values from that run are void. A strict-load check now runs before every Jetson gate.

Honest reading, written before the next gates¶

G10 is retrospective: W1 was scored on the same 69 rollouts it had seen as unlabeled training video. The real test is a fresh rollout and a fixed threshold. That test came two days later.

Compute. External GPU: no (Mac). Cost: $0.
Reproduce this: 2026-10-17-g10-outcome-readout

Evaluation and judges, 5 of 14. ← Evaluation and judges 4: Latent distance is not progress · Evaluation and judges 6: Two gates I voided, and the two-line check that would have caught them →

Judging robot rollouts automatically: a success detector with zero false positives

Fri, 16 Oct 2026 00:00:00 +0000

Judging robot rollouts automatically: a success detector with zero false positives¶

We wanted to know if a robot run could be judged automatically without ever calling a failure a success. We wrote a fixed camera rule, with no learning, that checks the two bowls once the arm is still. On 88 runs it was right 94.3 % of the time and never called a failure a success.

Evaluation and judges · 1 of 14

Type: method · Result: A deterministic top-camera judge at rest: 0 false positives, 94.3 % on 88 rollouts. · n: 88 rollouts, 26 failures

Not proven: A narrow fail of its 95 % bar; reset marks set by eye.

Compute: External GPU: yes (vast.ai) for Robometer: ~$8.55–9.55 incl. a LoRA run that never trained and an OOM re-score. The deterministic judge: Mac, $0. · Reproduce: 2026-10-23-paper-vs-hardware-robometer · First published: 2026-10-16 · Updated: 2026-10-16 · Findings: I need to tell success from failure in a rollout →, Deterministic top-camera judge at rest →, A deterministic judge at rest, never a false positive →

Ten operator-judged trials are fine. An autonomous loop that records hundreds of rollouts and fine-tunes on the good ones is not, because every success it mislabels trains the policy on a failure. This is the day the judge became the most important component in the stack, and the day a published 4B reward model lost to 40 lines of OpenCV.

Robometer-4B, zero-shot¶

On ArmnetBench (2 099 SO-101 rollouts, 7 policies), a first run on the partial set gave a qualified pass at AUROC 0.861. Re-scoring the 462 episodes lost to one CUDA OOM (two whole tasks, the longest episodes) brought the complete set to 0.818: fail. Episode length alone scores 0.938. Missing data is not random, and a partial result is provisional until it covers every task.

On our own 88 rollouts: AUROC 0.58–0.62. The main failure mode (grasping the wrong bowl) is an object-identity question that a trajectory-comparison reward model has no reason to answer.

G9c: a deterministic judge¶

Top camera, HSV masks for the two bowls, centroid geometry: is the blue blob inside the pink one at the end? No learning.

Version	End rule	Accuracy
G9c	last frame	86.4 %
G9c-2	arm back at rest	29/88 invalid (failing policies rarely return to rest)
G9c-3	stillness or timeout	94.3 %

All twelve G9c errors are successes whose recording continues past the operator's reset, so the final frame shows the bowls separated again. Judged at the marked reset instead: 95.5 %. G9c-3 misses its pre-registered 95 % bar by 0.7 points and is adopted anyway, because the one number that matters for a training loop is the false-positive count, and it is zero.

The rule¶

A judge that never calls a failure a success can be slow, narrow and dumb. One that does, however clever, poisons the data. Everything later (the verifier tiers in v1.5) comes from this table.

Compute. External GPU: yes (vast.ai) for Robometer: ~$8.55–9.55 incl. a LoRA run that never trained and an OOM re-score. The deterministic judge: Mac, $0.
Reproduce this: 2026-10-23-paper-vs-hardware-robometer

Evaluation and judges, 1 of 14. Evaluation and judges 2: Verify at rest: six mid-episode monitors fail in one day →

Latent distance is not progress

Wed, 14 Oct 2026 00:00:00 +0000

Latent distance is not progress¶

We wanted to know if distance in a world model's latent space could measure progress toward the goal. We checked whether that distance falls steadily over successful runs, then tried a hierarchical version. It did not: the arm's path loops, and the hierarchical version still had 52 % false stalls.

Evaluation and judges · 4 of 14

Type: method · Result: Distance to a goal latent does not track progress (ρ −0.32 to −0.50; a second level reaches −0.73 against a −0.80 bar). · n: teleop val episodes

Not proven: Monotonicity measured on teleop episodes only.

Compute: External GPU: no (Mac). Cost: $0. · Reproduce: 2026-10-14-g6-progress-latents · First published: 2026-10-14 · Updated: 2026-10-14 · Findings: Latent distance to the goal is not progress →

If a world model cannot rank actions, maybe it can tell how far along a task is: distance in latent space to the goal frame should fall as the arm gets closer, and a stall is when it stops falling. Zero labels needed. G6a tested that; G6b tested a hierarchical version four days later. Both failed the same way.

G6a¶

Bars: Spearman correlation between step and distance-to-goal ≤ −0.8 over successful demos; done-detector AUROC ≥ 0.95.

Monotonicity: −0.32 to −0.50 for every variant. The arm's trajectory loops in physical space (approach, lift, swing, place), so distance to the final frame rises and falls on the way there.
Stall rule derived from it: 32 % false stalls on held-out successes.
Done detector: AUROC 0.97–1.00, but its negatives were at-rest frames before any grasp. An easy contrast proves little.

G6b: the hierarchy¶

H-JEPA's level-2 slow features (coarse stride 15, SIGReg loss) were supposed to smooth the loops out. Monotonicity −0.73 against a −0.80 bar; 52 % false stalls. The hierarchy was dropped, and with it the last planning-shaped role in the stack.

What the papers already said¶

TD-JEPA reports ~0.65 for raw latent distance against 0.95 for a learned cost; the Planning-Limits paper shows rollouts shorter than the target distance rank below chance even with a perfect simulator. I had read both. Running the gate anyway cost nothing on the Mac and turned "the papers say" into "on this arm, this number". A progress signal has to be trained on progress labels; nothing here produced one for free.

Compute. External GPU: no (Mac). Cost: $0.
Reproduce this: 2026-10-14-g6-progress-latents

Evaluation and judges, 4 of 14. ← Evaluation and judges 3: Judging, part 2: the input format did the work · Evaluation and judges 5: What the world model is actually for →

World-model planning on an SO-101: the commanded joints win (0.90 vs 0.60)

Tue, 13 Oct 2026 00:00:00 +0000

World-model planning on an SO-101: the commanded joints win (0.90 vs 0.60)¶

We wanted to know if a world model could plan on our arm by ranking candidate motions. We asked it to pick the real motion among 8, against a check that reads only the commanded joints. The joints won, scoring up to 0.90 vs 0.60 for the best world model.

Evaluation and judges · 9 of 14

Type: method · Result: Ranking action chunks, the world model's top-1 is 0.60 against 0.68–0.90 for an action-only baseline. · n: 703 val anchors

Not proven: Closed-loop planning was not run.

Compute: External GPU: no for these gates (Mac). The models: W1 ~$6.54 (see the $3.30 post), W2 < $1, probes ~$0.06. · Reproduce: 2026-10-23-paper-vs-hardware-leworldmodel · First published: 2026-10-13 · Updated: 2026-10-13

G1b asked the question that decides whether a latent world model can plan on this arm: given the expert chunk and seven decoys, does imagining each one rank the expert first? Two world models, one bigger pretrained one, and a baseline with no model at all. The baseline won.

G1b: plannable range¶

Bar: top-1 ≥ 0.8 at some target distance.

W1 (192-d, trained here): best 0.60 at d = 20–30. Fail.
W2 (frozen DINOv2-S patches + learned dynamics): arm-pose R² 0.92 vs W1's 0.57, so the representation is far better; on G1b, 0.55–0.61. Fail. A decision with W2 takes 1.8 s on the Mac anyway (G2), so a third, bigger model W3 was not run.
RoboJEPA (pretrained, V-JEPA 2.1 features): the encoder takes 526 ms for two views on the Orin NX and 1 025 ms on the Mac. Off the robot. Zero-shot on our latents, the 1B model is borderline on a partial sample and action conditioning does work on our arm (9–10 % below shuffled actions). Not enough to earn a role.

The diagnosis¶

Top-1 among 8 compounds pairwise errors: W2's pairwise accuracy is 0.92–0.93, and 0.92^7 ≈ 0.56, which is exactly its top-1. A 0.8 top-1 bar needs ~97 % per comparison. Regret replaced top-1 as the metric.

Then the control: score chunks with the commanded joint trajectory alone, no images, no model. Top-1 0.68–0.90, regret ≈ 0. On a position-controlled arm, the actions already say where the arm will be. G1b was measuring kinematics, and the joints are a better kinematics model than any latent trained on 182 episodes.

What it means¶

A world model only has something to add where the actions are blind: object state, contact, failures. Prediction quality (G0 passed) is not decision quality (G1b failed). The next gates test exactly that boundary, and the planning role is gone for good.

Compute. External GPU: no for these gates (Mac). The models: W1 ~$6.54 (see the $3.30 post), W2 < $1, probes ~$0.06.
Reproduce this: 2026-10-23-paper-vs-hardware-leworldmodel

Evaluation and judges, 9 of 14. ← Evaluation and judges 8: Training a JEPA world model on SO-101 data for $3.30 · Evaluation and judges 10: TurboVLA: a policy that ignored its instruction →

Porting ACT to vla.cpp, so a Jetson can run it without PyTorch

Mon, 12 Oct 2026 00:00:00 +0000

Porting ACT to vla.cpp, so a Jetson can run it without PyTorch¶

We wanted to know if the most common policy on this arm could run on a Jetson without PyTorch. We ported it into the vla.cpp engine and checked it against PyTorch on fixed inputs and on the arm. The port merged upstream; speed was never the point, since PyTorch already ran it at 59 ms per chunk.

Engineering on the Orin · 5 of 8

Type: engineering · Result: ACT added to vla.cpp (merged upstream as #38); the PyTorch checkpoint scores 9/10 on the Orin NX gate. · n: 10 trials

Not proven: The vla.cpp ACT path was not gated on the arm.

Compute: External GPU: no (Orin NX + Mac). Cost: $0; ACT training cost not recorded. · Reproduce: 2026-10-12-act-vlacpp-port · First published: 2026-10-12 · Updated: 2026-10-12 · Findings: ACT in vla.cpp →

ACT is the most common SO-101 policy and vla.cpp did not have it. Merged as vla.cpp #38 after a scoping issue (#37) the maintainer approved in a day. The pitch was never speed: PyTorch ACT already runs at 59 ms per 100-action chunk on the Orin. The pitch is no PyTorch at inference: no 5.5-hour source build, no 12–24 GB image, and CPU-only boards can run it.

What a port involves¶

CONTRIBUTING.md lists the seven mechanical sites for a new architecture. Most of ACT's pieces already existed in-tree:

An ACT-style transformer decoder with learned queries, from the TurboVLA implementation.
Conv + group norm via im2col, from Octo. ACT's ResNet-18 uses frozen batch norm, which folds into the conv weights at conversion time.
No flow-matching steps: ACT goes on the num_steps refusal list. The VAE encoder is training-only, so the latent is zeros at inference.

The converter and act.cpp read state and action dims, camera count and resolution from the checkpoint config rather than hard-coding the SO-101, because the maintainer asked for a parity check on lerobot/act_aloha_sim_transfer_cube_human as well: bimanual, 14-dim, one 480×640 camera.

How it was tested¶

vla_predict_check against PyTorch on fixed inputs for both checkpoints (ours and the ALOHA one).
The same gate harness as every other policy here, with an --engine vlacpp path for ACT, on the real arm. The PyTorch checkpoint scores 9/10 on the Orin NX.
An --arch act client preset in the LeRobot-side client fork, so the same client drives ACT and SmolVLA (no tokenizer, ImageNet normalization).

Why a hobby lab should do ports¶

A port is the one contribution nobody can casually redo without your hardware, and it forces you to read the model as code rather than as a paper. The same reading is what later turned up a silently wrong objective in another policy's training code (LeRobot #4381) and the chunk-boundary fix that went into this engine a week after (G13b).

Compute. External GPU: no (Orin NX + Mac). Cost: $0; ACT training cost not recorded.
Reproduce this: 2026-10-12-act-vlacpp-port

Engineering on the Orin, 5 of 8. ← Engineering on the Orin 4: Twelve VLAs on a Jetson Orin NX: latency and memory with vla.cpp · Engineering on the Orin 6: Why your VLA's actions jump at every chunk boundary on real hardware, and a 0.4 ms fix →

Training a JEPA world model on SO-101 data for $3.30

Sun, 11 Oct 2026 00:00:00 +0000

Training a JEPA world model on SO-101 data for $3.30¶

We wanted to know if a small world model trained on our arm's videos was worth building on. We trained it for $3.30 and tested it against fixed bars before any use. Its error 30 steps ahead was 0.31 vs 0.89 for copying the last frame, so it passed.

Evaluation and judges · 8 of 14

Type: engineering · Result: A multi-view LeWM (W1) trains for $3.30 and beats copy-last at every horizon (0.31 vs 0.89 at K = 30). · n: 25 val episodes

Not proven: One training seed; planning not shown here.

Compute: External GPU: yes (vast.ai): $3.30 for the run + $2.96 of failed attempts + ~$0.28 environment bring-up = ~$6.54. · Reproduce: 2026-10-23-paper-vs-hardware-leworldmodel · First published: 2026-10-11 · Updated: 2026-10-11

LeWM (the stable-worldmodel recipe: a JEPA trained end to end from pixels, no reconstruction) on the 182 bowls episodes, two camera views, 100 epochs. Gate G0 asked whether the model is real before anything is built on it.

Getting it to train¶

Eleven setup problems between the paper's code and this data and hardware, on vast.ai x86 and on the Jetson GPU. $0.28 of smoke runs. A 5-epoch model reacted to gross action errors (episode swap, frozen action) but not to ±0.5 s timing shifts, and its latent had an effective rank of 5.8 out of 192: too short to judge.

G0, the faithful run¶

Pre-registered bars: beat copy-last; every corruption ratio ≥ 1.3 at K = 30; no collapse.

Open-loop error at K = 30: 0.31 vs 0.89 for copy-last.
Timing shifts: ratios 1.99 and 1.58. The model uses the actions.
Effective rank 24–30. No collapse.
Linear probes on the latent: grasp onset AUROC 0.94, holding 0.995.
Cost: $3.30, plus $2.96 of failed attempts before it.

Pass. This model, called W1 from here on, is the one every later gate tests.

C1, the first disappointment¶

Does the imagined latent predict rollout failures better than the current one? On the 81 gate rollouts: current latent 0.68, imagined 0.72, and policy identity alone 0.76. Most failures came from TurboVLA, so "which policy ran" explains more than any latent. Not proven, confounded. The lesson that stuck: compute the cheap baseline before the model, every time.

Compute. External GPU: yes (vast.ai): $3.30 for the run + $2.96 of failed attempts + ~$0.28 environment bring-up = ~$6.54.
Reproduce this: 2026-10-23-paper-vs-hardware-leworldmodel

Evaluation and judges, 8 of 14. ← Evaluation and judges 7: v1.5: the station as a harness for a coding agent · Evaluation and judges 9: World-model planning on an SO-101: the commanded joints win (0.90 vs 0.60) →

TurboVLA: a policy that ignored its instruction

Thu, 08 Oct 2026 00:00:00 +0000

TurboVLA: a policy that ignored its instruction¶

We wanted to know why a policy trained on two bowl tasks did the same motion for both instructions. We traced it to the data, then reran the original authors' trainer on the same data. Only about 6 % of frames differ between the two tasks, so ignoring the instruction still fits the rest.

Evaluation and judges · 10 of 14

Type: correction · Result: The policy ignored its instruction on the two-task bowls data (0/3 on the last port); the claim that our port caused it was withdrawn on 2026-10-23. · n: 3 arm trials on the last port; 25 val episodes

Not proven: Only two tasks; the instruction is redundant on 94 % of frames.

Compute: External GPU: yes (vast.ai). The port-era trainings behind this post: cost not recorded; the later path-A rerun: ~$2.8 incl. ~$0.8 of failed starts. · Reproduce: 2026-10-23-paper-vs-hardware-turbovla · First published: 2026-10-08 · Updated: 2026-10-23 (port claim withdrawn)

Ten days of work, several variants, a published PyPI package, and a closed project. TurboVLA fine-tuned on the two-task bowls data learned to perform whichever task it had seen last, whatever the instruction said. The failure taught me two things about data that the later datasets were built around.

Corrected 2026-10-23

The conclusion below about the model was wrong. My port differed from the paper in four places, and the paper's own trainer, run on the same data with the paper's initialisation, also ignores the instruction (task swap 0.020° vs 0.105 for the same model on LIBERO-Goal). The data finding (fact 7) is the whole story. Full account: TurboVLA.

What happened¶

Checkpoint conversion from the LIBERO warm start worked (verified 2026-09-08).
Trained on bowls, it rolled out at 0/3 on the last variant. On inspection it did the same motion for "blue in pink" and "pink in blue": a copycat.
An upstream-port spec was written for the maintainers, and the honest estimate was about one chance in three of passing a 50-episode benchmark. I stopped.

Why¶

Instruction-dependent behaviour needs data where the instruction matters. In the bowls set, the frames where the two tasks differ (which bowl to grasp first) are about 6 % of all frames. The rest is identical reach, lift, place. A policy can fit 94 % of the data while ignoring the instruction entirely.
Idle frames teach freezing. The earlier recordings started with the arm at rest for a second or two. TurboVLA learned "stay" as a valid action. Every dataset since has its idle starts trimmed with the same script.

What survived¶

lerobot-policy-turbovla-so101 stays on PyPI as a working plugin policy; the bowls dataset was re-cut; the two lessons went into the recording guide. Not every project needs to end in a merge to be worth the time.

Compute. External GPU: yes (vast.ai). The port-era trainings behind this post: cost not recorded; the later path-A rerun: ~$2.8 incl. ~$0.8 of failed starts.
Reproduce this: 2026-10-23-paper-vs-hardware-turbovla

Evaluation and judges, 10 of 14. ← Evaluation and judges 9: World-model planning on an SO-101: the commanded joints win (0.90 vs 0.60) · Evaluation and judges 11: VLA-JEPA on the Orin for $9.7, and why best-of-N reranking is a no-go →

VLA-JEPA on the Orin for $9.7, and why best-of-N reranking is a no-go

Thu, 08 Oct 2026 00:00:00 +0000

VLA-JEPA on the Orin for $9.7, and why best-of-N reranking is a no-go¶

We wanted to know if picking the best of several planned motions could improve a robot policy. We sampled 8 motion chunks per step and measured how much they differed and what they cost. They barely differed, took 1.84× the time, and their plain average captured 45 % of the possible gain.

Evaluation and judges · 11 of 14

Type: engineering · Result: VLA-JEPA fine-tuned for ~$9.7 scores 9/10 on the arm; best-of-8 reranking is a no-go (0.76, 1.84×). · n: 10 trials; H0 on the teleop val split

Not proven: The paper's robustness claim is not tested.

Compute: External GPU: yes (vast.ai): ~$9.7 for the fine-tune (retries not itemised). H0 and the arm gate on the Orin NX: $0. · Reproduce: 2026-10-23-paper-vs-hardware-vla-jepa · First published: 2026-10-08 · Updated: 2026-10-08

The first world-model idea was the obvious one: let the policy propose several action chunks, let a latent world model imagine each one, keep the best. Before building any of it, I checked whether the premise held. It did not.

VLA-JEPA on the arm¶

Fine-tuned on vast.ai for about $9.7 (over the $3.5–4.7 estimate: interruptible bids cost more than they save on long runs). On the Orin NX, world model off at inference: 9/10 on bowls, 355 ms per chunk. Same score as SmolVLA at three times the training cost. The task is at its ceiling and cannot tell them apart; that comparison waits for a harder primitive.

H0: do the samples vary enough to pick from?¶

Pre-registered bars: oracle-to-mean error ratio ≤ 0.7, latency for 8 samples ≤ 1.5× one sample, baseline success below 85 %.

The 8 samples are bundled: 1.08° spread, oracle/mean ratio 0.76 (null 0.92).
Latency 1.84× (643 vs 349 ms).
Baseline success 90 %, already above the bar where reranking could pay.
Averaging the 8 chunks, with no model at all, captures 45 % of the oracle gain.

Verdict: no-go. A reranker cannot fix a bias the samples share, and the cheapest competitor (the mean) takes half the available gain for free.

What changed¶

Reranking left the architecture the same day. The question became whether a world model can see a failure, not whether it can choose an action. That turned out to be a much better question, with a much longer answer.

Compute. External GPU: yes (vast.ai): ~$9.7 for the fine-tune (retries not itemised). H0 and the arm gate on the Orin NX: $0.
Reproduce this: 2026-10-23-paper-vs-hardware-vla-jepa

Evaluation and judges, 11 of 14. ← Evaluation and judges 10: TurboVLA: a policy that ignored its instruction · Evaluation and judges 12: Audit your labels where they matter: a VLM labeller that is right on average →

Twelve VLAs on a Jetson Orin NX: latency and memory with vla.cpp

Tue, 06 Oct 2026 00:00:00 +0000

Twelve VLAs on a Jetson Orin NX: latency and memory with vla.cpp¶

We wanted to know how fast twelve robot policies run on the Jetson board most hobby arms use. We ran the engine's own benchmark template on each, one evening, with every model file checked. SmolVLA took 411.2 ms per inference at the defaults; the fastest model took 27.4 ms.

Engineering on the Orin · 4 of 8

Type: engineering · Result: Minimum ms per inference for twelve VLAs on an Orin NX, from Octo 27.4 to Evo-1 893.2. · n: 20 repetitions, best of 3 processes per model

Not proven: Engine latency on fixed inputs, not the robot loop.

Compute: External GPU: no (Orin NX only). Cost: $0. · Reproduce: 2026-10-06-twelve-vlas-orin · First published: 2026-10-06 · Updated: 2026-10-06

vla.cpp (the llama.cpp-style engine for vision-language-action models) had benchmark reports for the Orin Nano Super and the AGX Orin, and nothing in between. The Orin NX 16 GB is the board most hobby arms end up on. This report fills the gap and is merged upstream as vla.cpp #36.

Protocol¶

The project's own template: 3 warm-ups, 20 repetitions, best of 3 processes, a flag screen, then a full re-run of the fastest flag set. JetPack 6.2, CUDA 12.6, MAXN_SUPER, built on the board with -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=87. Twelve of thirteen models; OpenVLA-OFT does not fit in memory. Every GGUF's sha256 checked against the Hub. One evening, 18:10 to 20:57, no errors.

Headline, minimum ms per inference at the defaults¶

	Octo	TurboVLA	VLA-JEPA	GR00T N1.7	BitVLA	SmolVLA	π0	Evo-1
Orin Nano Super	31.9	89.3	193.9	271.7	335.0	462.8	845.2	1011.5
Orin NX 16 GB	27.4	79.7	176.7	251.4	269.1	411.2	739.6	893.2
AGX Orin	17.5	36.0	94.6	132.7	133.0	218.3	350.3	434.6

Every model lands between the two existing boards, as it should. The fastest flags match the Nano report except GR00T N1.6, where --weight-dtype f16 passes the 3 % rule on this board.

Two things worth knowing¶

On this L4T release, getrusage peak RSS does not include CUDA buffers (π0 reports 483 MiB here against 5 961 MiB on the Nano). I kept the template column as measured and added a whole-board tegrastats column with the idle baseline; the maintainer chose to drop the memory columns.
Throttling check: GPU clock stayed at 1 162–1 172 MHz, junction temperature ≤ 83 °C under the full sweep, so the numbers are not thermally capped.

Why it matters here¶

These are engine latencies on fixed inputs. The same board running SmolVLA on a real SO-101 through the full client measured 238–390 ms per chunk, which is the number the control loop actually sees. Both numbers are on the issue tracker (#31) so the next person with this board does not have to guess.

Compute. External GPU: no (Orin NX only). Cost: $0.
Reproduce this: 2026-10-06-twelve-vlas-orin

Engineering on the Orin, 4 of 8. ← Engineering on the Orin 3: SmolVLA on a Jetson Orin NX: 974 → 238 ms per chunk with vla.cpp · Engineering on the Orin 5: Porting ACT to vla.cpp, so a Jetson can run it without PyTorch →

SmolVLA on a Jetson Orin NX: 974 → 238 ms per chunk with vla.cpp

Thu, 01 Oct 2026 00:00:00 +0000

SmolVLA on a Jetson Orin NX: 974 → 238 ms per chunk with vla.cpp¶

We wanted to know if SmolVLA could run fast enough on the Jetson to stop stuttering. We served the same trained policy through a lighter engine, vla.cpp, and ran 10 trials per engine. One motion chunk took 238 ms instead of 974 ms, and 8 or 9 of 10 trials succeeded.

Engineering on the Orin · 3 of 8

Type: engineering · Result: SmolVLA 974 → 238 ms per chunk on the Orin NX with vla.cpp; 9/10 and 8/10 on the arm. · n: 10 trials per engine

Not proven: Success differences are within noise at n = 10.

Compute: External GPU: no (Orin NX, 20 trials on the arm). Cost: $0; SmolVLA training cost not recorded. · Reproduce: 2026-10-23-paper-vs-hardware-smolvla · First published: 2026-10-01 · Updated: 2026-10-01 · Findings: I need lower latency on a Jetson →, SmolVLA chunk inference on the Orin →

974 ms per chunk made SmolVLA stutter on the Orin NX. Serving the same checkpoint through vla.cpp brought it to 238 ms with no loss in task success.

Numbers, same bowls task, 10 trials¶

Engine	Success	Chunk inference
SmolVLA, PyTorch	9/10	974 ms
SmolVLA, vla.cpp	9/10	390 ms
SmolVLA, vla.cpp + flash attention	8/10	238 ms

The 8/10 on the flash-attention build is one failed trial out of ten; the gate is too small to say whether that is a regression or noise. Inference went from blocking the control loop most of the time to 14 % of the time at the 26.6 Hz tick.

What it took¶

A cppzmq workaround for Ubuntu 22.04 / JetPack 6, now documented upstream (vla.cpp #47).
Real SO-101 latency numbers posted on the project's benchmark issue (#31), which were the first on this hardware.
An --arch act client preset so the same client drives ACT and SmolVLA.

Why it matters¶

On GPU-poor hardware the engine is the lever, not the model. Everything that follows runs the policy through vla.cpp, and two weeks later the same engine is where the chunk-boundary fix lands at a cost of 0.4 ms.

Compute. External GPU: no (Orin NX, 20 trials on the arm). Cost: $0; SmolVLA training cost not recorded.
Reproduce this: 2026-10-23-paper-vs-hardware-smolvla

Engineering on the Orin, 3 of 8. ← Engineering on the Orin 2: ACT and SmolVLA on an SO-101: a 10-trial evaluation harness on a €350 arm kit · Engineering on the Orin 4: Twelve VLAs on a Jetson Orin NX: latency and memory with vla.cpp →

ACT and SmolVLA on an SO-101: a 10-trial evaluation harness on a €350 arm kit

Tue, 29 Sep 2026 00:00:00 +0000

ACT and SmolVLA on an SO-101: a 10-trial evaluation harness on a €350 arm kit¶

We wanted to know if a learned policy could run reliably on our arm, measured the same way each time. We built a harness that runs fixed trials and logs every motion, then tested two policies. Both worked, 9/10 or better, but one took 974 ms per motion chunk on the Jetson.

Engineering on the Orin · 2 of 8

Type: engineering · Result: ACT 10/10 (M1) and 9/10 (Orin NX), SmolVLA 9/10 on both; SmolVLA takes 974 ms per chunk on the Orin. · n: 10 trials per policy and machine

Not proven: The task is at ceiling for every policy; success is operator-judged.

Compute: External GPU: none for the gates (Orin NX, M1 Mac, 40 trials of 25 s on the arm). Checkpoint training: cost not recorded. · Reproduce: 2026-09-29-three-policies-so101 · First published: 2026-09-29 · Updated: 2026-10-27 · Findings: The $200 arm executes what it is told →

Correction, 2026-10-29

This post first said "$300 arm", then (2026-10-27) "$230 arm", the list price of the follower and leader. What I actually paid: €350 for the SO-101 pack (follower, leader and the wrist camera). The title now says €350 arm kit and the URL is unchanged. Every part with its price source is on the rig page.

Before any research question, the arm had to run a learned policy reliably and I had to be able to measure it the same way every time. This post is the baseline everything else is compared against.

Setup¶

SO-101 follower arm (6 DoF, STS3215 servos), SO-101 leader for teleop, Logitech C920 top camera, a wrist camera.
Jetson Orin NX 16 GB as the robot computer, headless, in a Docker image I published (the Dockerfile is merged upstream in LeRobot, #4599).
A MacBook M1 for a second machine, and vast.ai for anything that needs a real GPU.
Task: "put the blue bowl in the pink bowl", and the reverse. 182 teleop episodes, idle starts trimmed, public on the Hub as ravediamond/so101_bowls_all_trimmed.

The harness¶

monitor_rollout.py runs a fixed number of trials, logs a tick file per trial (joint commands, chunk latency, timestamps), records video from both cameras, and takes one key press from the operator at the end of each trial: success or failure. Every trial since this day sits in one catalogue, 85 of them with video by mid-October. The rule that mattered later: freeze the pass bar before the run.

Results, 10 trials each¶

Policy	Machine	Success	Chunk inference
ACT	M1 Mac	10/10	–
SmolVLA 450M (PyTorch)	M1 Mac	9/10	–
ACT	Orin NX	9/10	59 ms
SmolVLA 450M (PyTorch)	Orin NX	9/10	974 ms

Both policies work. SmolVLA at 974 ms per 50-action chunk is a problem on the Orin: the arm visibly pauses at every chunk boundary. That number is the reason the next post exists.

What I learned¶

The first thing a hobby setup needs is not a better model, it is a harness that produces the same number twice. Operator judgment is fine for 10 trials; it will not be fine for 100, which is a judging problem I only understood two weeks later.

Compute. External GPU: none for the gates (Orin NX, M1 Mac, 40 trials of 25 s on the arm). Checkpoint training: cost not recorded.
Reproduce this: 2026-09-29-three-policies-so101

Engineering on the Orin, 2 of 8. ← Engineering on the Orin 1: LeRobot on a Jetson: why there is no pip install torch, and what it took · Engineering on the Orin 3: SmolVLA on a Jetson Orin NX: 974 → 238 ms per chunk with vla.cpp →

LeRobot on a Jetson: why there is no pip install torch, and what it took

Wed, 09 Sep 2026 00:00:00 +0000

LeRobot on a Jetson: why there is no `pip install torch`, and what it took¶

We wanted to know why LeRobot will not install on a Jetson with one command, and what fixes it. We built the whole software stack from source in a container image, now merged into LeRobot. The full build took 5 h 33 min, and the PyTorch compile alone was 92 % of it.

Engineering on the Orin · 1 of 8

Type: engineering · Result: LeRobot runs on the Orin NX from a container with a JetPack-built torch and a source-built torchcodec. · n: one board

Not proven: One JetPack release.

Compute: External GPU: no (Orin NX only). Cost: $0. · Reproduce: 2026-09-09-lerobot-jetson-container · First published: 2026-09-09 · Updated: 2026-09-09

pip install lerobot fails on a Jetson Orin with an error that says almost nothing. The cause: LeRobot needs Python ≥ 3.12 and torchcodec ≥ 0.11 on aarch64 with CUDA, and no channel (NVIDIA's redist, the community index) ships a cp312 CUDA torch wheel for Jetson. The fix is a Docker image that builds the stack from source; it is merged upstream as LeRobot #4599 and published on GHCR. This post is about the traps underneath it.

Why not jetson-containers¶

The usual answer is dusty-nv/jetson-containers. Its last human-merged PR was months old and its own issue thread says it is no longer maintained; its lerobot package lacks torchcodec, references a renamed extra and a CLI that no longer exists. Depending on it would mean waiting on a lane with zero throughput.

Five things built from source¶

PyTorch 2.11 for cp312, TORCH_CUDA_ARCH_LIST=8.7.
torchcodec 0.11 with CUDA, against a local NVDEC ffmpeg.
torchvision 0.26.
FFmpeg with --enable-cuda-nvcc --enable-nvdec --enable-cuvid.
dav1d, because ffmpeg 8.1 wants a newer one than Ubuntu 22.04 ships.

The traps¶

PyTorch auto-enables NVSHMEM once USE_DISTRIBUTED=1 finds the nvidia-nvshmem pip package, and the link fails after five hours of compiling. USE_NVSHMEM=0, explicitly.
USE_FLASH_ATTENTION=0 is ignored if build/CMakeCache.txt exists from an earlier attempt; the parallel template-heavy kernels then OOM-kill a 16 GB board. Force it with cmake -DUSE_FLASH_ATTENTION=OFF ..
USE_DISTRIBUTED=0, the sensible choice for a single board, crashes lerobot-train: its is_main_process() calls torch.distributed.is_initialized() unconditionally, and the stub module has no such method. Build with distributed on and Gloo as the backend.
The PyPI torchvision installs and imports fine against a from-source torch and only fails on the first compiled op: operator torchvision::nms does not exist. Build it from source, matched.
ffmpeg's vendored NVENC code disagrees with the Video Codec SDK headers on a struct field. Decode only is needed, so drop NVENC; note that ./configure auto-enables it when headers are present and only --disable-nvenc is the off switch.
A dependency's install script swallows its own failure (|| echo FAILED without exit 1), so the Docker layer builds and the missing binary surfaces two stages later.

The number to plan around¶

Full build: 5 h 33 min, of which PyTorch alone is 92 %. Everything else is under 15 minutes.

Result¶

Verified on an Orin NX 16 GB: torch.cuda.is_available(), a real .cuda() tensor out of torchcodec, torch.distributed.is_gloo_available(), import lerobot, a SmolVLA import. docker pull ghcr.io/ravediamond/lerobot-jetson:latest. Source and full build log: ravediamond/lerobot-jetson. The write-up reached about 49k people on LinkedIn, with a public thumbs-up from LeRobot's tech lead; it is also the image every Jetson gate on this site runs in.

Compute. External GPU: no (Orin NX only). Cost: $0.
Reproduce this: 2026-09-09-lerobot-jetson-container

Engineering on the Orin, 1 of 8. Engineering on the Orin 2: ACT and SmolVLA on an SO-101: a 10-trial evaluation harness on a €350 arm kit →