質素 Shisso — plain, frugal, without pretence.
What survives when a paper leaves the lab?¶
Robot-learning papers are demonstrated on Franka-class arms, data-centre GPUs or in simulation, against the baselines the authors chose, with the pass mark read off after the result. Shisso re-runs their claims under three conditions the papers rarely meet.
1. The hardware people can buy. An SO-101 arm and a Jetson Orin NX, ≈ €1,350. Their rig and ours side by side on every paper; the discount we expect from the gap is written down before the run, so a fail is attributed to the gap or to the claim, not left ambiguous.
2. The cheapest baseline first. Episode length, the commanded joints, copy-the-last-frame, the current latent, a shuffled control. Measured before the method, every time. Across 93 gates so far, 8 times the baseline matched or beat the published method.
3. The bar before the run. The pass mark is committed to a git hash before any number exists. Verdicts are four words — reproduces, fail, not proven, withdrawn — and every correction is logged with its date.
What it covers. Not one family. World models, vision-language-action policies, failure monitors, reward models and judges, action chunking and real-time execution, memory, latency on the edge. Any claim that can be tested on one arm and one task family. Each paper is recorded as what its authors proposed, what this rig tested, and what it did not.
What it is not. Not a benchmark, not a leaderboard, not a review. One rig, one task family, small n, with the n printed. Every write-up says what the rig cannot test.
Why. Hugging Face democratised the models. The papers still assume the lab. Shisso is the register of what transfers, and of the cheap thing that would have done as well.
Checked against the papers. Every paper-side field (goal, hypotheses, the claims attributed to the paper) was last re-read from the PDFs on 2026-10-31: 107 of 112 fields (96 %), 21 of 22 papers.
How a paper gets tested¶
-
Read the protocol, not the abstract.
Main table, setup, the baselines that were run and the ones that were not.
-
State the hardware gap.
Their rig and ours, side by side; the discount we expect from it is written down before the run, so a fail can be attributed to the gap or to the claim.
-
Baseline first, bar before the run.
The cheapest explanation (commanded joints, episode length, copy-last, a random-init model) is measured first; the pass mark is committed before any result exists.
-
Publish the verdict.
Pass, fail or not proven, with the number, its n, the controls, the code, the data and the checkpoint.
Everything is open¶
34
Reproduce
replication folders, 34/34 pass their L0 check: one folder per result, numbers recomputed in seconds.
28
Data and checkpoints
13 datasets and 15 checkpoints on the Hub, including 69 policy rollouts labelled with their outcome.
5
Upstream
LeRobot pull requests merged; 26 others reviewed or commented on; vla.cpp PR #60.
29
Findings
results the tests taught: 8 times a cheap baseline matched or beat the method, 3 systems rejected on device with the number that rejected them, and the method's own rules.
44
Write-ups
posts, each with its bar, verdict, compute and code.
12
Faults traced
12 faults traced across 7 layers of the stack, each with where it showed up, where it was and what found it.
Who¶
Ravindu Somawansa — Staff Engineer, evaluation and agent harness, Paris. 10+ years in data and ML. Research engineer in robot learning on his own hardware: every result here is a pre-registered test, bar committed before the run, fails published. When a number does not make sense the fault is traced down the stack before the model is touched — the ledger is at /faults/.
LeRobot reviewer · vla.cpp contributor (PR #60) · 2nd place, Hugging Face × NVIDIA GTC hackathon
Get involved¶
Request
A paper or claim you want tested on low-cost hardware: the form below, contact@shissorobotics.com, or a GitHub issue.
Subscribe
New results and corrections by RSS (or with updates).
Reproduce
ravediamond/paper-vs-hardware: start with any folder's
l0_check.py.