REMORA¶
They said reading progress at the peak nearest the last estimate keeps it steady and lets it fall after setbacks. On 34 of our recorded runs we compared it with simpler readouts of the same evidence. It did not hold: it cut changes to 0.70 of the simplest readout, not half, and missed most setbacks.
Papers tested · 22 of 22
Goal (the paper's): Track task progress through errors and recovery: a progress distribution from a frozen-VLM backbone with a per-camera recurrent memory, trained against a target that falls inside error intervals, read out causally at the peak nearest the previous estimate; shown on ReSTEP (AgiBot G02, two logistics tasks).
Its hypotheses:
- Picking the supported peak nearest the previous estimate tracks progress through errors better than a per-frame readout. (§3.3 (Eq. 3); Table 2 (A5)) fail
- A recurrent memory beats an explicit window of past frames. (Table 2) not tested here
- REMORA tracks progress better than SARM. (Table 2) not tested here
Verdict: fail · Claim tested: Reading a progress distribution out at the supported peak nearest the previous estimate steadies the estimate and lets it fall during real setbacks, with no worse detection of the stages. · Bar: Against per-frame argmax: premature not higher and missed ≤ +2 points (P1); flips halved, CI upper ≤ 0.7 (P2); a decrease inside ≥ 50 % of the reversal windows with ≤ 20 % false regressions (P3). · n: 34 test episodes of the 69 rollouts, 7 reversal windows
Theirs vs ours: D2 (25 error episodes): peak selection vs the median readout MAE 6.04 → 3.31 (×10⁻²), Spearman 0.870 → 0.959 vs premature 0.22 vs 0.25, missed equal (P1 pass); flips 0.70 × argmax, CI 0.48–0.94 (P2 fail); setback recall 2/7, false 0.52 (P3 fail); MAE 0.184 vs median 0.218 · Not proven: Three progress atoms from hand-written evidence, not a learned 201-bin distribution; the reversal windows are a mechanical criterion, not human error labels; the recurrent memory and the backbone are not tested.
Paper: arXiv 2610.39125 · Code: no code link found
Claims:
- fail nearest-peak progress readout below per-frame progress readout on readout flips per episode — theirs MAE 6.04 → 3.31 (×10⁻²) vs the median readout · ours flips 0.70 × argmax (CI 0.48–0.94); setback recall 0.29 (2/7), false regressions 0.52 (paper §3.3 and Table 2 (peak selection vs median readout on the error task))
- not tested recurrent progress memory below explicit frame window on progress MAE — theirs MAE −3.3 %, curvature error −14 % (paper Table 2 (MAE −3.3 %))
- not tested REMORA progress model below SARM reward model on progress MAE — theirs lower MAE and smoother on both tasks (paper Table 2)
Compute: External GPU: no. Orin NX CPU (feature extraction in the lerobot-jetson container). Cost: $0. · Reproduce: 2026-10-31-paper-vs-hardware-remora-readout · First published: 2026-10-31 · Updated: 2026-10-31
Tracking Task Progress Through Errors and Recovery (Wang, Niu, Zhu et al., arXiv 2610.39125) argues that a progress estimate should be able to go backwards: when the robot drops the object, progress should fall, then recover. Its model needs eight H200s. Its readout rule does not, so I tested the rule on the 69 rollouts this rig already has.
What the paper claims, and how it measures it¶
REMORA predicts a distribution over progress at every frame, from a frozen VLM backbone with a recurrent memory per camera, and is trained against a target that falls inside labelled error intervals and climbs back after them. It reads the distribution out causally: among the peaks of the distribution, take the one closest to the previous estimate. On the task with real errors (25 error episodes in its ReSTEP test set, AgiBot G02), peak selection against the per-frame median cuts the error from 6.04 to 3.31 (×10⁻²) and lifts Spearman from 0.870 to 0.959. On the task with one error episode it gains nothing.
What I ran¶
The model is out of reach; the readout rule and the error-aware target are model-free. The rig has G15's per-frame grasp and place evidence on 69 rollouts, a monotone stage machine that reads it (premature on 48.5 % of events, missed on 46.7 %), and G14's reversal windows (re-grasp loops, pan dithering) as its own error intervals.
I turned the evidence into a distribution over three progress values (reach, holding, placed) with two logistic regressions fitted on G15's development half, and read the same distribution four ways on the 34 test episodes: the stage machine, the per-frame argmax, the per-frame median (the paper's ablation), and the nearest peak. Bars frozen before any number, against the argmax:
- P1: the nearest peak calls no more events too early, and misses at most 2 points more.
- P2: it halves the flips per episode (CI upper bound ≤ 0.7).
- P3: it decreases inside at least half of the reversal windows, with at most 20 % of the other episodes showing a decrease.
Their number, my number¶
| readout | premature | missed | flips / episode | setbacks caught (7) | false setbacks | error vs target |
|---|---|---|---|---|---|---|
| stage machine | 0.485 | 0.467 | 1.00 | 0 | 0.00 | 0.248 |
| per-frame argmax | 0.250 | 0.133 | 5.97 | 3 | 0.56 | 0.183 |
| per-frame median | 0.397 | 0.133 | 6.35 | 3 | 0.74 | 0.218 |
| nearest peak | 0.221 | 0.133 | 4.21 | 2 | 0.52 | 0.184 |
P1 passes: premature −5.9 points (CI −17.6 to 0), missed unchanged. P2 fails: the nearest peak keeps 0.70 of the argmax's flips (CI 0.48–0.94), not half. P3 fails: it falls inside 2 of the 7 windows (0.29) and in 52 % of the episodes with none. Against the median, the paper's own comparison, it is closer to the target (0.184 vs 0.218) and steadier, the direction the paper reports, but that comparison had no bar here.
I checked the seven stalled or retried-grasp failures by eye: the nearest peak stays at "reach" for the whole episode, which is right, since none of them ever lifted a bowl. They have no progress to lose, so they cannot exercise the claim.
What the gap means¶
The readout does what it says in the direction it says. What it selects between here is three values from hand-written predicates, and those predicates are noisy in exactly the places a setback would show. The rule cannot separate a real setback from evidence noise when the distribution itself cannot.
Verdict: P1 passes, P2 and P3 fail: not adopted; the progress slot stays on the stage machine.
What would change it: a learned, denser progress distribution (the paper's 201 bins) on this rig, and human-labelled error intervals instead of a mechanical reversal criterion.
Compute. External GPU: no. Orin NX CPU (feature extraction in the lerobot-jetson container). Cost: $0.
Reproduce this: 2026-10-31-paper-vs-hardware-remora-readout
Papers tested, 22 of 22. ← Papers tested 21: Horizon × depth (Euler steps vs execution horizon)