RoboJEPA¶
They said a world model pretrained on 15 000 hours of robot video predicts what comes next from the actions. On our arm we tested its predictions on our clips and timed it on the robot's computer. It used our actions, but its image encoder alone took 526 ms, too slow to run on the robot.
Papers tested · 3 of 22
Goal (the paper's): Measure how an action-conditioned predictor on frozen V-JEPA 2.1 features scales with compute across 12 embodiments, fit a law that extrapolates to held-out model sizes, and plan with the large models on a real Franka.
Its hypotheses:
- Latent-rollout error on held-out scenes follows a compute law that extrapolates to 4B and 8B models. (§3.1, Table 2) fail
- Planning capabilities emerge in order with compute: end-effector control, holding, scene geometry, object dynamics. (Fig. 6–9) not tested here
- The larger models plan grasp, lift and pick-and-place on a real Franka from a single image goal. (Table 3) not tested here
Verdict: fail · Claim tested: Pretrained RoboJEPA predicts our latents, uses the actions, and runs on the robot's computer. · Bar: 9-step L1 ≥ 15 % below copy-last (300M) and ≥ 5 % below shuffled actions; 2-view encoder ≤ 200 ms on the Orin. · n: 60 clips (30 for 1B), 25 episodes
Theirs vs ours: rollout error frontier near E ≈ 0.22; latency not reported vs 11.8 % below copy-last (300M; 1B 15.8 % on 30 clips); 9–10 % below shuffled; encoder 526 ms on the Orin NX · Not proven: 1B on a partial sample; TensorRT not tried.
Paper: arXiv 2610.10515 · Code: facebookresearch/robo_jepa
Claims:
- fail pretrained RoboJEPA below copy-last baseline on 9-step latent L1 — theirs rollout error frontier near E ≈ 0.22 · ours 11.8 % below copy-last (paper §3.1, Table 2 (DROID holdout, E = 0.218))
- fail RoboJEPA encoder at most pass mark on ms per encoder call on the Orin NX — theirs latency not reported · ours 526 ms (our measured comparison)
Compute: External GPU: yes (vast.ai, Gate A): ~$1.2. Gate B on the Orin NX: $0. · Reproduce: 2026-10-23-paper-vs-hardware-robojepa · First published: 2026-10-23 · Updated: 2026-10-23 · Findings: RoboJEPA's encoder on the Orin →
RoboJEPA (Zholus, Assran et al., Meta FAIR; arXiv 2610.10515) is a scaling law for latent world models trained on 15 000 hours of robot video, including 21.5k LeRobot community episodes from arms like mine. Two gates: can the encoder run on the Orin, and does the pretrained predictor use my actions.
What the paper claims, and how it measures it¶
A deterministic next-feature predictor on a frozen V-JEPA 2.1-G encoder (width 1664, patch 16). Per timestep the sequence is all views' patches plus an action token and a proprio token; attention is joint space-time-view, timestep-causal, local to 8 timesteps so it rolls out past its training length; QK-norm for stability. Loss is half teacher-forced L1, half autoregressive L1 from a prefix (the exposure-bias fix). 23 datasets, 12 embodiments, 26 action spaces; AdamW with a shared flat learning-rate stage and per-budget cool-downs.
The scaling law is of per-token L1 error over a 9-step latent rollout on held-out scenes (DROID holdout and RoboCasa), compute counted in FLOPs, compute-optimal frontier across 22M–2B. The functional form was chosen by extrapolation error on 4B and 8B held out of the fit (Table 2): a second-order law wins (0.6 / 1.4 ×10⁻³) over a plain power law (2.0 / 3.7). Downstream: closed-loop CEM planning in RoboCasa with capability emerging in order (end-effector control → holding → scene geometry → object dynamics, the last only at ~10²² FLOPs); a real Franka table (Table 3: 8B grasp 67 %, lift 50 %, pick-and-place 27 % success; π0.5 as a reference, not a baseline); "over 50 000 evaluation episodes" across both platforms, per-task n in an appendix. Planning is "several seconds per step for 8B, not real-time".
What I ran¶
Gate B, encoder on device. Bar: 2-view encoder p50 ≤ 200 ms on the Orin NX. Result: 526 ms (one view 398 ms); the 100M predictor step 252 ms, 300M 487 ms. No-go on device. On the Mac (MPS) the encoder is 1 025 ms and the predictor's attention does not run. Any use is off-board on CUDA.
Gate A, does the pretrained predictor use my actions. Two bars on 9-step rollouts of my bowls latents: (1) L1 ≥ 15 % below copy-last for the 300M model; (2) ≥ 5 % below shuffled actions. Result: (1) 300M 11.8 %, 100M 10.5 %, 1B 15.8 % [14.6, 17.0] on only 30 clips; (2) 8.9–10.4 % for every size. The primary bar fails; the 1B is borderline on a partial sample; action conditioning works on my arm at every size.
G1b-RJ (the 1B on the ranking gate) was cancelled as a gate on 2026-10-15: neither outcome would change the build, since the encoder cannot run in the loop.
Their number, my number¶
| Paper | This arm | |
|---|---|---|
| Action use | implicit in the rollout error | 9–10 % below shuffled, all sizes |
| Rollout error vs model size | clean frontier, near irreducible E ≈ 0.22 | 100M 10.5 → 300M 11.8 → 1B 15.8 % below copy-last (30 clips) |
| Encoder latency | not reported (multi-GPU planning) | 526 ms / 2 views, Orin NX |
| Real-robot n | appendix | 30 clips, no robot |
What the gap means¶
My arm is in their training mix, and the pretrained model does read its actions — the Gate A shuffled-action margin is the far-left point of their scaling curve, at zero fine-tuning cost. What it cannot do is run where the arm is. The paper is explicit that planning is not real-time even on a GPU cluster; on an Orin the encoder alone eats sixteen control ticks. So the honest use is offline: feature extraction for a judge or a dataset-level probe. The method lesson is the one to keep: a scaling claim is an extrapolation test — fit on some sizes, predict the held-out ones, report the error. My own scaling angle (50 / 100 / 182 / 362 episodes) is a plot until it does that.
Verdict: action conditioning reproduces at the small end; the encoder is off the robot.
What would change it: a distilled encoder at ≤ 100 ms, which is not my experiment to run.
Compute. External GPU: yes (vast.ai, Gate A): ~$1.2. Gate B on the Orin NX: $0.
Reproduce this: 2026-10-23-paper-vs-hardware-robojepa
Papers tested, 3 of 22. ← Papers tested 2: DINO-WM · Papers tested 4: H-JEPA →