Robometer¶
They said a large reward model, trained on a million robot videos, can judge runs on robots it never saw. We asked it, untrained, to tell failed from successful runs on 2 099 public SO-101 arm runs. It did not hold: 0.818 out of 1, under our 0.85 bar; on our runs a robot-trained judge, Flora, passed.
Papers tested · 10 of 22
Goal (the paper's): Train a general robot reward model on a vision-language backbone, from about a million trajectories across 21 embodiments with progress, preference and success heads, and use it zero-shot to score progress, rank trajectories and detect failures.
Its hypotheses:
- The reward model tracks task progress on embodiments it was not trained on. (Table I (VOC 0.92 / 0.94)) fail
- It detects failures zero-shot better than other reward models and token-uncertainty baselines. (Table IV (F1 0.81)) fail
- As a reward it improves online and offline RL, including on an SO-101. (Q3, Fig. 7) not tested here
Verdict: fail · Claim tested: A 4B reward model judges SO-101 rollouts zero-shot. · Bar: Pooled AUROC ≥ 0.85 and within-policy ≥ 0.8 on ≥ 5 of 7 policies. · n: 2099 + 88 rollouts
Theirs vs ours: VOC 0.94 on out-of-distribution evaluation sets (0.92 in-distribution) vs pooled AUROC 0.818 on 2099 ArmnetBench rollouts; episode length alone 0.938 · Not proven: Zero-shot only; the LoRA run never trained.
Paper: arXiv 2603.02115 · Code: robometer/robometer
Claims:
- fail learned reward model at least pass mark on failure AUROC — theirs failure-detection F1 0.81 (no AUROC reported); VOC 0.94 OOD · ours 0.818 pooled (paper Table IV (zero-shot failure detection))
- fail learned reward model above episode length on failure AUROC — theirs no episode-length baseline in the paper · ours 0.818 vs episode length 0.938 (not a paper comparison: the paper's premise that a learned judge beats hand-made signals)
- reproduces episode length above learned reward model on failure AUROC · ours 0.938 vs 0.818 (our measured comparison)
Compute: External GPU: yes (vast.ai): ~$8.55–9.55 incl. a LoRA run that never trained and an OOM re-score. · Reproduce: 2026-10-23-paper-vs-hardware-robometer · First published: 2026-10-23 · Updated: 2026-10-23 · Findings: I need to tell success from failure in a rollout →
Robometer (Liang, Korkmaz, Zhang, Hwang et al.; arXiv 2603.02115) is a 4B reward model trained on a million trajectories. I tested it zero-shot as a judge on my own rollouts and on a 2 099-rollout public SO-101 benchmark. Episode length beat it on every held-out task. The paper never computes episode length.
What the paper claims, and how it measures it¶
Qwen3-VL-4B with learned tokens: a progress token after each frame (causal; progress in 10 bins, success at the last frame) and a preference token after two videos. Three losses: preference, progress, success. Pairs are built from expertise (expert vs failure), instruction negatives (same frames, wrong instruction → progress 0), and video rewind. Data: RBM-1M, 21 embodiments; SO-101 is not in it.
Evaluation on RBM-EVAL (976 OOD trajectories): VOC (Pearson r between predicted reward and frame index on successes) 0.92 and Kendall τ on the failed < suboptimal < successful ordering 0.64, vs RoboReward-8B 0.82 / 0.47. Zero-shot on RoboFAC, VOC 0.65. Ablations on LIBERO-90: adding failure data lifts τ from 0.74 to 0.92. Downstream: online RL on a DROID π0 base 20 → 85 % (20 trials); offline IQL on SO-101 "2.4× the best baseline" (0.85 vs 0.50 on an oven task; 0.15 vs 0.05 on a plate task — 3 vs 1 successes in 20); failure detection on 100 DROID trajectories, F1 0.81 vs GPT-5-mini 0.33. The authors' own limitation: no access to contact or grasp stability.
What I ran¶
G9a-AB (2026-10-15/16). Zero-shot on ArmnetBench, 2 099 rollouts, 7 policies, top camera. Bar: pooled AUROC ≥ 0.85 and within-policy ≥ 0.8 on ≥ 5/7. Complete set: pooled 0.818 [0.790, 0.843]; within-policy 5/7; episode length alone 0.938; beyond length (quintile median 0.687, CV gain +0.016) both bars missed. Fail: not a zero-shot judge.
G9a-own (2026-10-16). On my 88 labelled bowls rollouts: AUROC 0.577 [0.44, 0.70]; TPR 0.25 at the calibrated threshold; policy identity 0.878; wrong-object failures scored highest. Fail.
G9b (written 2026-10-22). The LoRA fine-tune on held-out-task folds never trained (the job died at preprocessing). The zero-shot baseline on policy rollouts only: stack 0.564, tool 0.616, cable 0.761, ring 0.933; length 0.903 / 0.855 / 0.946 / 0.965, better on 4/4; CV gain over length +0.008 [−0.001, 0.017]. No verdict.
Their number, my number¶
| Paper | This arm | |
|---|---|---|
| Ranking quality | VOC 0.92, τ 0.64 (OOD embodiments) | AUROC 0.82 on ArmnetBench, 0.58 on own bowls |
| Episode-length baseline | not run | 0.938 / 4-of-4 held-out tasks |
| Wrong-object failures | not analysed | scored highest |
| Downstream n | 20 trials | — |
What the gap means¶
Robometer and my result agree once the question is phrased right. The model answers "how far along does this look" and does that well (VOC 0.92 is real). My failure mode is identity — the wrong bowl, nested — and a trajectory-comparison objective with instruction negatives at the dataset level has no reason to catch it at the frame level on an embodiment it never saw. The paper's own "pull tissue" and "fold towel" rows (F1 0.58, 0.76) are its version of the same gap. The number that should be in every reward-model paper and is in none of the ten I read: episode length. On my tasks it is the judge every learned judge must beat first.
Verdict: fails as a zero-shot judge on this arm; the gap is the task's, not the model's.
What would change it: an instruction-negative test on my own frames (swap the instruction, fix the frames, does the score move?) before any LoRA; and labels for suboptimal-but-successful episodes, which τ needs and I do not have.
Compute. External GPU: yes (vast.ai): ~$8.55–9.55 incl. a LoRA run that never trained and an OOM re-score.
Reproduce this: 2026-10-23-paper-vs-hardware-robometer
Papers tested, 10 of 22. ← Papers tested 9: VLA-Corrector · Papers tested 11: SmolVLA →