Quick reference · 5 minutes
A smaller error is
one piece of evidence.
Ask what was measured, on which inputs, and what remains untested.
| Evidence | What it establishes | What it leaves open |
|---|---|---|
| Training loss decreases | The optimizer reduced its chosen fitting objective on sampled training examples. | Stable grasping, contact timing and unseen scenes. |
| Recorded-input evaluation | Predictions can be compared with saved target actions for the evaluated frames. | What happens after the policy changes the simulated scene. |
| Physical skill evaluation | Measured contact and stability checks passed or failed for that attempted skill. | Other skills, the continuous dinner task and generalization. |
| Frozen held-out task suite | Observed performance across the declared, untouched test cases. | Behavior outside the tested operating envelope. |
Read the units before the number
A normalized training loss is not automatically an angle in radians, a distance in millimetres, or a success probability. In this project, inspect the training configuration and loss weights, then use the evaluator's raw joint-angle errors for an interpretable action comparison. Do not compare two loss numbers if their weighting or normalization changed.
Averages can hide the important instant
Illustrative example: nine joint targets have 0.001 rad error, but one has 0.100 rad error. The mean is 0.0109 rad. That average hides the much larger error at one instant. If that instant is when a gripper releases an object, its timing matters. This example is invented; it is not a result from our robot.
Our September 11 experiment
The first full hand-off checkpoint trained for 20,000 updates on 630 observations from one authored scene. Its recorded-input evaluation and physical tests are now complete. These are development results, not held-out validation.
| Check | Observed result |
|---|---|
| All 630 recorded inputs | Finite, bounded forecasts. First-action mean absolute error: 0.00335 rad; largest individual error: 0.18855 rad. |
| Execute one action per forecast | Failed after 900 actions without completing the hand-off. |
| Execute five actions per forecast | Stopped when the forecast expired. The freshness limit was retained. |
| Execute two actions per forecast | Approached the pre-grasp pose, but failed after 900 actions without completing the hand-off. |
“Two actions per forecast” means executing the first two predictions from a ten-action sequence, checking each before applying it. It does not mean skipping observations or disabling contact checks. Our two-action experiment improved approach progress but did not establish grasping.
Check your interpretation: do valid, low-error predictions prove the robot can grasp?
No. They describe predictions on recorded inputs. Once the policy moves the arms, its new observations can differ from the teacher's. The physical run must independently demonstrate a grasp, transfer and stable release. Our failed runs are evidence of that gap.
The corrective follow-up
The later batch-four corrective checkpoint completed 20,000 updates using all 630 nominal frames plus 188 approach corrections. On the same 630 recorded inputs, its first-action mean absolute error was about 0.00116 rad and its maximum was 0.08593 rad. Those numbers improved, but both physical runs still failed.
| Physical attempt | Observed result |
|---|---|
| Two actions per forecast | Reached donor hold, then lost all grip at action 567. |
| Five actions per forecast | Completed shared hold, then lost all grip at action 676. |
Replaying the sealed contact logs localized the final fault. The receiver-gripper commands were 0.0707 rad and 0.1323 rad; the nearest nominal left-release target was -0.1 rad. The respective errors of +0.1707 rad and +0.2323 rad opened the receiving gripper. This supports a corrective-data decision about the release transition. It does not promote the checkpoint.
See the current engineering status, continuity analysis, and skill-data contract for the exact evidence. The original ACT paper discusses how policy errors can compound over time. The LeRobot ACT guide describes its camera/joint inputs and action-sequence outputs.