VLA-Corrector¶
They said a small model watching camera statistics and actions flags a failing run early enough to fix it. On our arm we trained the same kind of watcher on 50 recorded runs. It did not hold: it caught 2/8 failures, below our bar.
Papers tested · 9 of 22
Goal (the paper's): Keep a frozen π0.5 on task by detecting deviations with a learned verifier on observable history (cheap visual statistics, state and action, a GRU) and prompting recovery when it fires; shown in LIBERO and LIBERO-Plus and on a physical arm.
Its hypotheses:
- A verifier on observable history detects execution risk (AUROC 0.902) close to a privileged rule-based verifier. (§ Failure Detection and Recovery Analysis (AUROC 0.902); Fig. 4) fail
- Prompting recovery when the verifier fires raises success under perturbations. (Fig. 3, Table 3; real-robot results) not tested here
Verdict: fail · Claim tested: A verifier on observable history flags a failing rollout early enough to correct it. · Bar: ≥ 5/8 failures at ≤ 1/20 false interventions. · n: 50 episodes, 8 failures
Theirs vs ours: 54-feature GRU verifier AUROC 0.90 vs the paper's verifier family on our rollouts (a GRU on pooled RGB statistics and actions): 2/8 caught at 3/42 false interventions, AUROC 0.59; cosine drift of the W1 latent 0/8, AUROC 0.30 · Not proven: Their exact 54-feature verifier (with proprioceptive state) and its recovery prompt were not run here; our GRU uses pooled RGB statistics and actions.
Paper: arXiv 2609.06508 · Code: no code link found
Claims:
- fail observable-history verifier at least pass mark on failures caught — theirs AUROC 0.90 · ours 2/8 caught at 3/42, AUROC 0.59 (§ Failure Detection and Recovery Analysis (AUROC 0.902); Fig. 4)
Compute: External GPU: no (Orin NX). Cost: $0. · Reproduce: 2026-10-19-six-monitors-w1 · First published: 2026-10-23 · Updated: 2026-10-23
VLA-Corrector (Song, Qian, Feng, Song; arXiv 2609.06508) wraps a frozen π0.5 in a tiny learned verifier that routes a recovery prompt. I borrowed one of its signals — cosine drift of the latent — as a monitor. It caught nothing. The part of the paper I should have borrowed is the verifier's design.
What the paper claims, and how it measures it¶
The policy is frozen (OpenPI pi05_libero). The verifier reads only observable history: two 112×112 views → parameter-free spatial statistics (3×4×4 pooled RGB plus mean and std, 54 features per view) → Linear 64; state and action → MLP 32; a GRU over an 8-step window; two heads, binary risk and a five-way stage. Teacher labels come from a rule-based state machine on simulator state (f = 1 within 10 steps of a failure event, 0 beyond 20). Gate: τ = 0.40, four consecutive positive windows, 10-step cooldown; the stage selects a recovery prompt fragment appended to the instruction; up to 100 recovery actions, then the original instruction resumes. Verifier training: 120 episodes, 30 epochs, CPU; 0.46 ms inference.
Evaluation: LIBERO and LIBERO-Plus difficulty 4–5 under action noise, target shift, goal shift and all three, 1 000 episodes per condition; three configurations — no assistance, a privileged rule verifier (the efficacy upper bound), the learned verifier. Combined perturbation 70.7 → 82.6 (rule) / 80.2 (learned): the learned verifier is within 2.4 points of the oracle. Verifier quality: accuracy 93.1 %, precision 61.1 %, recall 94.9 %, AUROC 0.90; recovery success 75 %. Ablations: single frame −15 F1, state-action only −15.8. Real arm (unnamed), three tasks, 40 trials each, target moved after execution starts: 82.5 → 88.3 average, zero-shot from sim-trained features.
What I ran¶
W1-A (2026-10-19). Cosine drift of W1's latent as a mid-rollout screener on 50 episodes, beside L2 surprise and the live reflex. Bar: ≥ 5/8 failures at ≤ 1/20 false interventions. Result: cosine drift 0/8, AUROC 0.30 — below chance; L2 surprise 2/8 at 2/42; the live reflex flagged 42/42 successes. Fail.
I never built the paper's verifier itself.
Their number, my number¶
| Paper | This arm | |
|---|---|---|
| Signal tested | 54-feature GRU verifier, AUROC 0.90 | cosine drift of W1 latent, AUROC 0.30 |
| Gate design | recall-tuned (95 %), 4-window confirm, cooldown | single threshold |
| Cost of a false trigger | a recovery prompt, ~100 steps | a wasted trial (my bar: ≤ 1/20) |
| Upper bound | privileged verifier, within 2.4 pts | none |
What the gap means¶
Cosine drift was a wrong borrowing — the paper's verifier is not a drift threshold. Two things from the paper matter more than the result. First, the precision/recall split: a 61 % / 95 % verifier is a usable gate when a false trigger costs a re-prompt rather than a trial. My monitor bar (≤ 1/20 false interventions) assumes the expensive kind; if the intervention is "re-observe and re-prompt", the bar can relax and the 6/8-at-7/42 risk head from the ReAP post may already be a useful gate. Second, the parameter-free front end plus a GRU reaches 0.90 on LIBERO at zero cost on the Orin; it is the right floor baseline to put under any W1-based monitor, and it is now in the amendments as one.
Verdict: the borrowed signal fails; the paper's verifier is untested here and is the next floor baseline.
What would change it: the 54-feature GRU on my 69 rollouts, beside the risk head, with the recovery cost stated in the bar.
Update 2026-10-31 (claims audit, WS-068). The register now files this paper's claim under its own method, not under latent drift. The closest test this rig has run is the GRU "floor" of the REAP v2 gate: a GRU on pooled RGB statistics and the commanded actions, the verifier's family without the proprioceptive features, on the same 50 episodes and bar. It caught 2/8 failures at 3/42 false interventions (AUROC 0.59), below the bar; the verdict stays fail. Their exact 54-feature verifier is still not run here.
Compute. External GPU: no (Orin NX). Cost: $0.
Reproduce this: 2026-10-19-six-monitors-w1
Papers tested, 9 of 22. ← Papers tested 8: ReAP · Papers tested 10: Robometer →