Lesson 04 · 20 minutes · Connected to local training

A policy learns from
aligned examples.

Your next win: tell the difference between one pretty trajectory and a training demonstration that can teach a policy.

Local ACT fitting and checkpoint inference now run. A corrective hand-off checkpoint matched recorded targets closely, yet both frozen physical attempts dropped the bar. This lesson explains why training evidence and physical success are different.

One time step, one aligned record

Camera images
declared order+Joint observation
documented units→Action target
same time base→Outcome
kept or failed

A demonstration is useful only if the images, joint observations, actions, timestamps, and instruction agree about the same moment. The scripted teacher may use simulator truth to generate safe examples, but the deployed policy must receive only its declared images and robot observations. That separation prevents hidden ground-truth leakage.

What makes a demonstration usable for imitation learning?

ACT predicts a short future sequence

Action Chunking with Transformers (ACT) predicts a chunk of future actions from the current observation. A chunk can smooth control and make the policy less reactive to tiny visual changes, but it also creates a safety obligation: actions must be bounded, checked, and cleared when the task changes or a stop is requested.

What is an ACT action chunk?

Falling loss does not finish the table

Training adjusts the model to reduce an error on recorded examples. During a physical run, an imperfect action changes the next image and robot state. The model may then encounter a situation absent from those examples. The ACT paper identifies this accumulation of policy errors as a challenge.

Our next checks ask different questions: compare predictions on all 630 recorded hand-off inputs, then execute the learned skill under contact and stability checks. The recorded inputs come from training, so their results cannot establish generalization. Keep the training-evidence reference beside a run report.

A report says only that training loss fell. Which conclusion is supported?

Explain in your own words why a model can match recorded actions yet drop an object when it runs. Ask the agent for feedback; mastery is recorded only after you demonstrate it.

A real failure at one critical joint

Our corrective checkpoint produced valid forecasts on all 630 recorded hand-off inputs, with a first-action mean absolute error of about 0.00116 rad. In the live simulation, the five-action-prefix run still failed after completing the shared two-arm hold. At the failure, the policy commanded the receiving right gripper to 0.1323 rad while the nearest nominal release target kept it closed at -0.1 rad. That 0.2323 rad error released the bar.

The average recorded-input error did not reveal the closed-loop failure. Once a small error changed contact, the next image and robot state differed from the demonstration. The next correction therefore targets receiver close, shared hold, and donor release. Read the sealed continuity analysis for the exact evidence.

Why did this checkpoint drop the bar despite a small average recorded-input error?

Practice a split decision

Suppose demonstrations use scene seeds 1–80. Choose a validation range and a frozen test range without looking at their outcomes. Explain why changing the test set after selecting a checkpoint would distort the evidence. Share your answer with the agent.

Primary reading: the LeRobot ACT documentation. Return to the project’s architecture to see how the implemented data and control boundaries fit together.