-
Eval-driven development with layered graders
Build a small, real-failure-derived test set early (Anthropic suggests 20-50 tasks) and grade agent output with a mix of code-based graders (fast, deterministic, for checkable outcomes), model-based graders (rubric-driven LLM judges, calibrated against humans), and human graders (gold standard, used sparingly) -- then keep iterating the eval set as the agent matures.
Why this verdict?
Anthropic's own 'demystifying evals' guidance lays this out as field-tested methodology: early on, effect sizes are large so small samples suffice, and the bar shifts to harder evals and tighter calibration only once an agent starts saturating existing tests. The discipline matches current LLM-as-judge practitioner writing too -- judges carry position/length/self-preference bias and need the same calibration rigor as any other measurement tool.