HWM (hierarchical latent world models)¶
They said a second, coarser world model predicts further ahead than the fine one beyond about one second. On our data we trained both levels and compared their errors up to 2 seconds. Not proven: the coarse level never beat the fine one, with error 0.759 against 0.563 at 2 seconds.
Papers tested · 16 of 22
Goal (the paper's): Plan long manipulation tasks with a two-level world model: a high level over learned macro-actions proposes waypoints, a low level plans primitive actions toward them; shown on a real Franka trained on about 130 hours of DROID and RoboSet.
Its hypotheses:
- Beyond about one second, the high-level model predicts the future latent better than the low-level model. (Fig. 6 (crossover near 1.25 s)) not proven
- Learned macro-actions align with the true motion better than a handcrafted delta-pose summary. (Table 4b (cosine 0.88 vs 0.80)) reproduces
- Hierarchical planning succeeds on real pick-and-place where flat planning does not. (§3.1, Table 1 (70 % vs 0 %)) not tested here
Verdict: not proven · Claim tested: A high-level world model over learned macro-actions beats the low-level model beyond about 1 s (their Fig. 6 crossover). · Bar: Crossover in [1.0, 2.0] s; at 2 s high-level error ≤ 0.95 × low-level; learned macro-actions ≥ net joint displacement on cosine. · n: 700 val anchors, 25 episodes (157 train, ~0.6 h)
Theirs vs ours: low level better up to 1 s, high level from 1.5 s (Fig. 6; ≈ 0.367 vs 0.385 at 2 s, read off the plot); cosine 0.88 ± 0.03 vs 0.80 ± 0.02 vs no crossover: W1 0.382 ± 0.005 → 0.563 ± 0.007, high level 0.769 ± 0.009 → 0.759 ± 0.009 (0.73–2 s); cosine 0.608 ± 0.007 vs 0.551 ± 0.006 · Not proven: Open confound: ~0.6 h of one task vs their ~130 h, so data scale is not separated from the method. Baselines and the overfitting reading are post-hoc.
Paper: arXiv 2604.03208 · Code: kevinghst/HWM_PLDM (minimal implementation on PLDM, linked from the project page)
Claims:
- not proven high-level world model below low-level world model on latent error at 2 s — theirs 0.367 vs 0.385 at 2 s · ours 0.759 vs 0.563 at 2 s (ratio 1.35) (paper §3.4, Fig. 6 (crossover between 1 s and 1.5 s; ≈ 0.367 vs 0.385 at 2 s, read off the plot))
- reproduces learned macro-actions at least net joint displacement on cosine to the true macro-action — theirs 0.88 vs 0.80 · ours 0.608 vs 0.551 (paper §3.4, Table 4b (0.88 ± 0.03 vs 0.80 ± 0.02))
Compute: External GPU: yes (vast.ai RTX 3090): ≈ $0.4, including the invalid first run. Latent extraction on the Orin NX: $0. · Reproduce: 2026-10-28-paper-vs-hardware-hwm · First published: 2026-10-28 · Updated: 2026-10-28 · Findings: I need a long-horizon latent predictor →, I need to encode an action chunk →
HWM (Zhang, Terver, Zholus et al., FAIR at Meta / NYU / Mila / Brown; CoRL 2026, arXiv 2604.03208) stacks two latent world models: a low level over primitive actions, and a high level over learned macro-actions that jumps several seconds at once. The figure I could test offline is the one that says when each level wins. On my data the high level never wins, and I can't tell yet whether that is the method or the amount of data.
What the paper claims, and how it measures it¶
A transformer action encoder compresses the actions between two waypoints into one macro-action (its CLS token). The high-level predictor F⁽²⁾ takes (macro-action, waypoint latent) and predicts the next waypoint latent, trained with a teacher-forced ℓ1 loss on three waypoints sampled up to 4 s apart. At planning time the high level picks a subgoal and the low level plans primitive actions toward it.
The real-robot results are on a Franka Panda with a Robotiq gripper, trained on about 130 h of DROID and RoboSet: pick-and-place 70 % against 0 % for the flat planner. Figure 6 is the offline part: on held-out observations, the low-level model has the lower latent error up to about 1 s, the high level from about 1.5 s, and the curves cross near 1.25 s. Table 4b adds that learned macro-actions align better with the true motion than a handcrafted delta-pose summary (cosine 0.88 vs 0.80).
What I ran¶
Low level: W1, my multi-view LeWM, unchanged. High level: the paper's §2.1 recipe trained on W1's frozen latents, on my bowls data (182 episodes, 157 train / 25 val, about 0.6 h). Three versions that differ only in the action input: the learned encoder, the concatenated raw actions, and the net joint displacement over the chunk (my stand-in for delta-pose). Error is mean |ẑ − z| over W1's 192 latent dimensions on 700 val anchors.
Three bars, frozen before training: H1 a crossover between 1.0 and 2.0 s; H2 at 2 s, high-level error ≤ 0.95 × W1's; H3 learned macro-actions ≥ net displacement on cosine alignment.
The first run was wrong: my extraction script fed W1 the red channel only, three times. A check against my earlier W1 numbers caught it before scoring (W1 looked four times worse than copying the last frame, which it never was). I fixed it, checked that the new latents match the earlier W1 numbers, retrained, and scored only the second run.
Their number, my number¶
| horizon | 0.73 s | 1.0 s | 1.5 s | 2.0 s |
|---|---|---|---|---|
| low level (W1 rollout) | 0.382 ± 0.005 | 0.438 ± 0.006 | 0.511 ± 0.006 | 0.563 ± 0.007 |
| high level, learned macro-actions | 0.769 ± 0.009 | 0.766 ± 0.009 | 0.764 ± 0.009 | 0.759 ± 0.009 |
| high level, net joint displacement | 0.790 ± 0.005 | 0.793 ± 0.006 | 0.793 ± 0.006 | 0.799 ± 0.007 |
| Paper | This arm | |
|---|---|---|
| Crossover (H1) | ~1.25 s | none: the high level is above W1 at every horizon |
| 2 s ratio, high / low (H2) | ≈ 0.95 | 1.35 |
| Learned vs handcrafted cosine (H3) | 0.88 ± 0.03 vs 0.80 ± 0.02 | 0.608 ± 0.007 vs 0.551 ± 0.006 |
| Training data | ~130 h, DROID + RoboSet | ~0.6 h, one task |
H1 fails, H2 fails, H3 holds. The SEs are over anchors, and anchors from the same episode are correlated, so they are on the small side.
Added after seeing the result¶
None of this was in the protocol. I computed it to understand the numbers, and it should be read that way.
- Copying the current latent gives 0.613 at 0.73 s and 0.954 at 2 s. Predicting the average training latent gives 0.81 at every horizon. The learned high level sits at 0.76, flat: barely better than the average, and better than copying only from about 1.3 s on.
- Its last training loss was 0.15. That loss is not the same measurement as the val error, so the gap to 0.76 points to overfitting on 157 episodes without measuring it.
What the gap means¶
My reading: the high level didn't learn anything that generalises from 0.6 h of one task, and W1, trained end to end on the same data, did. That fits "the high level needs far more data than the low level" just as well as "there is no crossover on this arm", and this run can't tell them apart. So the verdict is not proven, not fail. H3 is the part that holds even here: a learned summary of the actions beats the handcrafted one.
I had one reason to care beyond the paper. My G1b gate ranked action chunks at 0.17–1.0 s, which by Figure 6 is where the low level should win anyway. If the crossover had shown up here, G1b would have been measured in the wrong band for a hierarchy argument. It didn't show up, but not cleanly enough to close that question either.
What would change it: the same high level trained on a lot more data, or on top of a pretrained multi-task latent. Neither is planned.
Compute. External GPU: yes (vast.ai RTX 3090): ≈ $0.4, including the invalid first run. Latent extraction on the Orin NX: $0.
Reproduce this: 2026-10-28-paper-vs-hardware-hwm
Papers tested, 16 of 22. ← Papers tested 15: Real-Time Chunking · Papers tested 17: LeJEPA identifiability →