SURVEYOR¶
They said a fixed distance threshold between imagined and reached states accepts good plans and rejects bad ones. On 69 real runs we measured that threshold in our world model. It did not hold: the threshold here is 1.249, not their 0.20, and it caught 3/8 failures.
Papers tested · 18 of 22
Goal (the paper's): Let a latent-world-model planner consume drafted subgoals speculatively, accepting each step while the imagined and reached latents stay within a calibrated threshold; shown in simulation (PushT, Reacher, OGBench-Cube, Two-Room).
Its hypotheses:
- A threshold calibrated on task-equivalent state pairs (τ = 0.20) separates acceptable from unacceptable plan steps. (method and experiments) fail
- The accept rule keeps good drafted plans and rejects bad ones. (experiments) fail
Verdict: fail · Claim tested: A threshold on the imagined-vs-reached latent distance accepts good plans and rejects bad ones (τ = 0.20). · Bar: τ below the 10th percentile of non-equivalent pairs and ≤ 0.40; as a monitor ≥ 5/8 failures at ≤ 2/42; ≥ 0.05 AUROC over a shuffled drafter. · n: 69 rollouts (primary 50: 42 successes, 8 failures)
Theirs vs ours: τ = 0.20 in simulation (PushT, Reacher, OGBench-Cube, Two-Room) vs τ = 1.249 on W1 latents of 69 real rollouts; as a monitor 3/8 at 6/42 (S = 30), AUROC 0.756 vs shuffled 0.560 · Not proven: FF-JEPA's diffusion drafter is not available (W1's own imagination stands in); no closed-loop planning.
Paper: none (NeurIPS 2026 workshop) · Code: no code link found
Claims:
- fail latent accept threshold at most pass mark on accept threshold τ — theirs τ = 0.20 · ours τ = 1.249 (paper, τ = 0.20 (simulation))
- fail latent accept rule at least pass mark on failures caught — theirs simulation only · ours 3/8 at 6/42 (paper (accepts good plans, rejects bad ones))
Compute: External GPU: no (Orin NX). Cost: $0. · Reproduce: 2026-10-29-paper-vs-hardware-surveyor · First published: 2026-10-29 · Updated: 2026-10-29
SURVEYOR (Certified Speculative Plan Consumption for Latent World-Model Planning, a NeurIPS 2026 workshop paper) accepts a drafted plan step when the latent the world model imagined and the latent actually reached are close enough. Its encoder is the one released with LeWorldModel, the same recipe as my W1, so the accept rule can be tested on real rollouts.
What the paper claims, and how it measures it¶
A drafter proposes subgoals in latent space; the planner consumes them while the relative distance between imagined and reached latents stays below a threshold τ, calibrated on pairs of states that should count as equivalent. The paper's τ is 0.20. Everything is in simulation (PushT, Reacher, OGBench-Cube, Two-Room); there is no real robot.
What I ran¶
Gate 0, τ. On W1's latents of 69 real SO-101 rollouts, I calibrated τ the paper's way: the median relative distance between equivalent pairs (two at-rest success frames from different episodes). Bars: S1, τ must sit below the 10th percentile of non-equivalent pairs; S2, τ ≤ 0.40.
Gate 1, the accept rule as a failure monitor. FF-JEPA's diffusion drafter isn't available, so W1's own imagination stands in for it, and a shuffled-drafter control checks whether the rule uses the plan at all. Bars: S3, at least 5 of 8 failures caught with at most 2 false interventions out of 42; S4, at least 0.05 AUROC above the shuffled drafter.
Their number, my number¶
| Paper | This arm | |
|---|---|---|
| τ | 0.20 | 1.249 (non-equivalent pairs: p10 1.106) |
| equivalent vs non-equivalent | separable | AUROC 0.71 |
| monitor, S = 10 frames | — | 1/8 caught at 1/42, AUROC 0.619 (shuffled 0.628) |
| monitor, S = 30 frames | — | 3/8 at 6/42, AUROC 0.756 (shuffled 0.560) |
S1 and S2 fail: on real rollouts, two finished successes from different episodes are about as far apart in W1's latent as a success and a frame from mid-episode. The scene changes between episodes (bowl positions, lighting) dominate the distance. S3 fails at both step sizes. S4 is met at S = 30 only: over one second the discrepancy carries plan–execution signal beyond chance, but its warnings come after the failure onset.
What the gap means¶
In simulation, equivalent states look alike and τ can be small. On a real arm, the same latent puts "done, in a different scene" far from "done": a threshold on latent distance has no operating point, and the accept rule becomes the seventh monitor here to miss the standing bar. The positive bit is S4 at one second: the discrepancy isn't noise, it just arrives late.
Verdict: fail on this arm.
What would change it: a latent where at-rest successes cluster across scenes (an object-centric or goal-conditioned one), and the real drafter.
Compute. External GPU: no (Orin NX). Cost: $0.
Reproduce this: 2026-10-29-paper-vs-hardware-surveyor
Papers tested, 18 of 22. ← Papers tested 17: LeJEPA identifiability · Papers tested 19: Adaptive Action Chunking (AAC) →