H-JEPA¶
They said a slow upper layer in a stacked world model tracks progress toward the goal. On our arm we checked whether its distance to the goal falls steadily along recorded runs. It did not hold: it signalled a stall that was not there 52 % of the time.
Papers tested · 4 of 22
Goal (the paper's): Stack JEPA world models into a hierarchy with a distinct latent space per level, so that upper levels plan in abstract states over longer horizons and hand subgoals down; shown in simulation, with an open-loop fidelity measure on DROID.
Its hypotheses:
- A hierarchy of distinct latent spaces plans long-horizon tasks with less planner compute than a flat model. (Fig. 6 (AntMaze 18 % → 73 %)) not tested here
- The lower level's cost to the subgoals handed down falls steadily along expert trajectories (Spearman 0.89–1.00), more consistently than its cost to the final goal (0.48–0.75). (§4.2, Fig. 8; Table 10 (Spearman 0.89–1.00)) fail
- Upper levels drop fast-varying entities when the data has an entity-frequency gap above about 2×. (Fig. 4–5) not tested here
Verdict: fail · Claim tested: A slow second level makes latent distance track task progress. · Bar: Progress monotonicity Spearman ρ ≤ −0.80; false stalls ≤ 10 %. · n: teleop val episodes (monotonicity, stalls); 88 rollouts (sanity check)
Theirs vs ours: subgoal-tracking Spearman 0.89–1.00 vs level-2 ρ −0.73; false stalls 52 % · Not proven: Outside the paper's regime (no entity-frequency gap on this task).
Paper: arXiv 2610.06805 · Code: no code link found
Claims:
- fail second-level latent distance at most pass mark on progress Spearman ρ — theirs subgoal-tracking (level-1 cost to level-2 subgoals) Spearman 0.89–1.00; no level-2 distance-to-goal reported · ours ρ −0.73; false stalls 52 % (paper, subgoal-tracking Spearman 0.89–1.00)
Compute: External GPU: no (Mac). Cost: $0. · Reproduce: 2026-10-14-g6-progress-latents · First published: 2026-10-23 · Updated: 2026-10-23
H-JEPA (Zhang, Terver, Rabbat, LeCun, Balestriero; arXiv 2610.06805) stacks LeWorldModels across timescales. I tested its level-2 slow feature as a progress signal (G6b) and it failed. Reading the paper afterwards, the paper's own Fig. 5 predicts that failure.
What the paper claims, and how it measures it¶
Level 1 is LeWM. Each level ℓ > 1 has its own state encoder (a pointwise MLP over the level below), its own action encoder pooling s_ℓ lower actions, and its own causal predictor, stride 2 per level, all trained end-to-end with a LeWM loss per level. On real data (DROID) an inverse-dynamics term is required or the model collapses to "encode the predictable background, drop the arm" (their App. L). Planning is top-down by gradient descent: the top level plans macro-actions; its predicted states become subgoals for the level below.
Evaluation: four simulated environments; success rate; three paired model + planner seeds with SE; planner compute swept and reported as TFLOPs per episode; depth 1–4 at a fixed 100-TFLOP budget on 50 tasks per environment. Baseline HWM (same hierarchy, shared latent) separates "temporal decomposition" from "distinct abstract spaces". AntMaze 18 → 73 % at three levels; two levels match LeWM on Push-T, three levels fail there because episodes are short. The mechanism figure: probe NMSE per entity vs level shows body pose becoming unrecoverable while position stays; the entity-frequency gap (ratio of spectral centroids of fastest and slowest state components) predicts when this selective abstraction appears — above ~2× it does; Push-T, Cube and DROID have gaps below 2× and show none. On DROID the metric is Fréchet fidelity of the planned end-effector path against the expert's (15 clips, three seeds). No closed-loop real robot.
What I ran¶
G6b. A level-2 slow feature (stride 15, SIGReg loss) on W1, tested as a progress signal on the bowls task. Bars: T1 Spearman between latent distance and time ≤ −0.80; false-stall rate ≤ 10 %; sanity AUROC ≥ 0.95. Result: T1 −0.73 (fail), false stalls 52 % (fail), sanity 0.998 (pass). The hierarchy was dropped.
Their number, my number¶
| Paper | This arm | |
|---|---|---|
| Where hierarchy helps | AntMaze (frequency gap > 2×): 18 → 73 % | bowls task: one arm, one object, ~1 Hz everything |
| Where it does not | Push-T, Cube, DROID (gap < 2×) | same regime |
| Progress monotonicity | subgoal-tracking Spearman 0.89–1.00 (Fig. 8) | level-2 distance vs time −0.73 |
What the gap means¶
I tested the hierarchy on data the paper says it should not change. There is no frequency gap on a single-arm pick-and-nest task; the paper's own prediction for that regime is "no selective abstraction", and that is what G6b found. So the negative result and the paper's claim do not conflict; the gate was outside the claim. Two things do transfer: the inverse-dynamics term, which W1 does not need on single-scene data but W2′ on mixed lighting might; and Fréchet fidelity as an offline, open-loop plan-quality metric against expert paths — it costs nothing on my 69 rollouts and would have told me before G1b that the plans do not follow the demos.
Verdict: fail, outside the paper's regime; the paper's mechanism test (Fig. 5) is the better reading than my gate.
What would change it: a task with a real timescale gap (a tidy loop with navigation-scale structure), which my bench does not have.
Compute. External GPU: no (Mac). Cost: $0.
Reproduce this: 2026-10-14-g6-progress-latents
Papers tested, 4 of 22. ← Papers tested 3: RoboJEPA · Papers tested 5: D-JEPA →