Skip to content

Real-Time Chunking

They said building each new motion chunk to match the one already running removes jumps at the joins. On recorded runs we compared a simple overwrite of the shared part with their method. Not proven on the arm: the overwrite removed most of the jump, adding 0.4 ms.

Jump at the chunk seam cut vs naive async, %: prefix overwrite 71–73 %; bar 50 %prefix overwrite71–73 %bar 50 %

Papers tested · 15 of 22

Goal (the paper's): Let a flow-matching policy run asynchronously without jumps at chunk boundaries, by generating each new chunk as an inpainting of the actions already committed; shown on dynamic simulated tasks and a bimanual robot with added latency.

Its hypotheses:

  • Conditioning the new chunk on the committed actions removes the discontinuity at chunk boundaries. (§3.1–3.2 (ΠGDM guidance, Eq. 2–3); Fig. 3–4) not proven
  • With it, task throughput stays flat as inference latency grows, where synchronous execution falls. (Fig. 6) not tested here

Verdict: not proven · Claim tested: Conditioning a new chunk on the executing one removes the jump at chunk boundaries. · Bar: Best gradient-free method cuts the median jump ≥ 50 % against naive async, drift ≤ +10 %, within 667 ms. · n: 254 boundaries per delay, 20 episodes

Theirs vs ours: throughput flat across added latency (π0.5, real robot) vs prefix overwrite cuts the jump 71–73 % against naive async at +0.4 ms; tuned LeRobot identity mode is close · Not proven: Offline smoothness only; task success on the arm not measured for any mode.

Paper: arXiv 2506.07339 · Code: Physical-Intelligence/real-time-chunking-kinetix

Claims:

  • not proven prefix overwrite at least pass mark on median boundary-jump cut — theirs throughput flat across added latency (π0.5, real robot) · ours 71–73 % at +0.4 ms (paper §3.1–3.2, Fig. 4 (qualitative; no jump metric reported))

Compute: External GPU: no (Orin NX). Cost: $0. · Reproduce: 2026-10-25-vla-chunk-boundary-jumps · First published: 2026-10-23 · Updated: 2026-10-26 (soft-mask correction)

Real-Time Chunking (Black, Galliker, Levine, Physical Intelligence; arXiv 2506.07339) treats the next action chunk as an inpainting problem. My gradient-free prefix overwrite uses the paper's soft mask without its gradient, and the paper says where gradient-free inpainting should lose.

Corrected 2026-10-26: an earlier version called the overwrite RTC's hard mask. It uses RTC's soft prefix weights, without the gradient (see the chunk-boundary post).

What the paper claims, and how it measures it

With chunk length H, control period Δt and inference time δ, the delay is d = ⌊δ/Δt⌋ steps. Asynchronous inference must start the next chunk d steps early, and the junction is arbitrary: a new chunk from fresh noise may pick a different strategy (their bifurcation figure). Temporal ensembling averages across chunks and is not guaranteed valid when the distribution is multimodal. RTC: freeze the first d actions (they will be executed before the new chunk arrives), attend to the remaining overlap with an exponentially decaying weight, leave the rest free, and enforce it through ΠGDM guidance on the flow's velocity field with a clipped guidance weight — one vector-Jacobian product through the policy per Euler step. The soft mask (eq. 5): weight 1 for i < d, decaying over the overlap, 0 after.

Evaluation: twelve dynamic Kinetix tasks chosen because quasi-static benchmarks are saturated by open-loop chunking; 2 048 rollouts per point with Wilson 95 % intervals; delays 0–4; baselines naive async, BID, temporal ensembling. RTC is above every baseline at every delay; hard masking (freeze only d steps) is below soft masking at small d; temporal ensembling is worst even at d = 0. Real: π0.5, bimanual, H = 50, Δt = 20 ms, 5 denoising steps, d ≈ 6, plus +100 / +200 ms injected latency; six tasks, 10 trials per task per method per delay, metric throughput (task progress per minute). RTC is flat across added latency; synchronous falls linearly; both temporal-ensembling variants trip the protective stop at +100 ms and score zero. Cost: 97 vs 76 ms per chunk on their stack.

What I ran

G13b (2026-10-19, offline, Orin). SmolVLA on 20 bowls episodes, 254 chunk boundaries, d = 8 / 12 / 20. Methods: naive, exact RTC guidance, and gradient-free prefix overwrite. Bars: propose the gradient-free method if it cuts the median jump ≥ 50 % vs naive and stays within 20 % of exact. Result: prefix overwrite cuts the median jump 72–74 % at every d, drift unchanged; exact RTC cut only 21–41 % in this implementation (weaker than LeRobot's identity mode — flagged to check). Timing: naive 1 028 ms, overwrite 1 035 ms, exact 1 962 ms per chunk in PyTorch on the Orin; every method misses the 667 ms line because the policy itself is too slow in PyTorch.

In vla.cpp (same day). Ported as prev_actions + inference_delay; jump cut 71.0 / 73.3 / 71.5 % through the engine, prefix error 0.0, +0.4 ms per chunk, default path bit-identical. Exact RTC is not possible in vla.cpp today: ggml lacks backward rules for flash attention and concat.

Their number, my number

Paper This arm
Hard vs soft mask soft > hard at small d (Fig. 5) soft only (RTC's prefix weights over the whole leftover); hard not run
Cost +21 ms (VJP per Euler step) +0.4 ms (overwrite); exact +930 ms in PyTorch
Metric throughput, task progress/min median boundary jump (instrument)
n 2 048 rollouts/point; real 10/task/delay 254 boundaries, 20 episodes

What the gap means

The overwrite already uses the paper's soft mask (weight 1 on the first d steps, decaying to 0 at the end of the leftover). What it drops is the gradient: it corrects the predicted clean chunk directly, which is guidance with an identity Jacobian at weight 1/t, instead of RTC's vector-Jacobian product. So the paper's hard-vs-soft result (Fig. 5) does not decide between the two; the paper's own gradient-free inpainting variant (App. A.4, simulation only) was beaten by guided RTC on solve rate. Only if guided RTC beats the overwrite by more than the pairing noise on the same 254 boundaries are ggml backward kernels worth writing. And the result metric should be theirs: a smoother junction is for throughput; the jump is the instrument check, not the claim.

Verdict: the gradient-free point reproduces in spirit; the paper predicts a better point I have not measured.

What would change it: the soft-mask comparison offline, then throughput on the robot day with sync / async / async + prefix.

Compute. External GPU: no (Orin NX). Cost: $0.
Reproduce this: 2026-10-25-vla-chunk-boundary-jumps

Papers tested, 15 of 22. ← Papers tested 14: REKI · Papers tested 16: HWM (hierarchical latent world models) →