Lesson 06 · 15 minutes · Preparation concept
Evidence starts before
the test runs.
Your next win: recognize evaluation leakage before it turns a promising demo into an unreliable result.
Keep three scene sets apart
| Set | Purpose | May guide checkpoint choice? |
|---|---|---|
| Training | Teach the policy from demonstrations. | Yes |
| Validation | Compare choices while developing. | Yes |
| Frozen test | Estimate final behavior on held-out scenes. | No |
Randomization changes placements, mass, friction, shape, lighting, and background. It does not make a result reliable by itself. The seed list and conditions must be frozen before selecting the final checkpoint, and every attempted episode—including failures and interventions—must remain in the report.
Define success before the episode
For the full dinner-table task, success requires every requested step, stable placement after release, and the required physical hand-off. A nice final image, a partial task, or an operator correction does not meet that definition. Paired comparisons keep the scene and seed fixed when comparing a high-precision model with a quantized version.
Uncertainty is part of the result
A result such as 9 successes from 10 seeds estimates behavior for a limited sample; it is not evidence of 90% reliability in every future scene. We will report sample counts, confidence intervals, intervention rates, and failure classes alongside percentages.
Practice: explain why replacing one hard test seed after seeing a failure changes the meaning of a reported score. Ask the agent to check your explanation.
Primary reading: the project’s evidence rules and requirements matrix.