REMORA
They said reading progress at the peak nearest the last estimate keeps it steady and lets it fall after setbacks. On 34 of our recorded runs we compared it with simpler readouts of the same evidence. It did not hold: it cut changes to 0.70 of the simplest readout, not half, and missed most setbacks.
Papers tested · 22 of 22
Goal (the paper's): Track task progress through errors and recovery: a progress distribution from a frozen-VLM backbone with a per-camera recurrent memory, trained against a target that falls inside error intervals, read out causally at the peak nearest the previous estimate; shown on ReSTEP (AgiBot G02, two logistics tasks).
Its hypotheses:
- Picking the supported peak nearest the previous estimate tracks progress through errors better than a per-frame readout. (§3.3 (Eq. 3); Table 2 (A5)) fail
- A recurrent memory beats an explicit window of past frames. (Table 2) not tested here
- REMORA tracks progress better than SARM. (Table 2) not tested here
Verdict: fail · Claim tested: Reading a progress distribution out at the supported peak nearest the previous estimate steadies the estimate and lets it fall during real setbacks, with no worse detection of the stages. · Bar: Against per-frame argmax: premature not higher and missed ≤ +2 points (P1); flips halved, CI upper ≤ 0.7 (P2); a decrease inside ≥ 50 % of the reversal windows with ≤ 20 % false regressions (P3). · n: 34 test episodes of the 69 rollouts, 7 reversal windows
Theirs vs ours: D2 (25 error episodes): peak selection vs the median readout MAE 6.04 → 3.31 (×10⁻²), Spearman 0.870 → 0.959 vs premature 0.22 vs 0.25, missed equal (P1 pass); flips 0.70 × argmax, CI 0.48–0.94 (P2 fail); setback recall 2/7, false 0.52 (P3 fail); MAE 0.184 vs median 0.218 · Not proven: Three progress atoms from hand-written evidence, not a learned 201-bin distribution; the reversal windows are a mechanical criterion, not human error labels; the recurrent memory and the backbone are not tested.
Paper: arXiv 2610.39125 · Code: no code link found
Claims:
- fail nearest-peak progress readout below per-frame progress readout on readout flips per episode — theirs MAE 6.04 → 3.31 (×10⁻²) vs the median readout · ours flips 0.70 × argmax (CI 0.48–0.94); setback recall 0.29 (2/7), false regressions 0.52 (paper §3.3 and Table 2 (peak selection vs median readout on the error task))
- not tested recurrent progress memory below explicit frame window on progress MAE — theirs MAE −3.3 %, curvature error −14 % (paper Table 2 (MAE −3.3 %))
- not tested REMORA progress model below SARM reward model on progress MAE — theirs lower MAE and smoother on both tasks (paper Table 2)
Compute: External GPU: no. Orin NX CPU (feature extraction in the lerobot-jetson container). Cost: $0. · Reproduce: 2026-10-31-paper-vs-hardware-remora-readout · First published: 2026-10-31 · Updated: 2026-10-31
Tracking Task Progress Through Errors and Recovery (Wang, Niu, Zhu et al., arXiv 2610.39125) argues that a progress estimate should be able to go backwards: when the robot drops the object, progress should fall, then recover. Its model needs eight H200s. Its readout rule does not, so I tested the rule on the 69 rollouts this rig already has.