Anthropic's Head of Product for Research/Labs: Evals Are the Primary Artifact of Product Work, Not the PRD
Summary
Dianne Penn, Anthropic's first technical PM, describes how her team generates 30-40 representative examples encoded as prompts with golden answers for every major feature — that eval set, not the PRD, is now the primary artifact of product work. She also details the eval-driven development loop, reading AI transcripts as the new 'sweating the pixels,' and why alignment (reducing sycophancy) makes Claude more useful as a thinking partner.
Why this signal
Directly and concretely addresses Neo's #2 focus area (evals as product guidance) from inside Anthropic itself — a fully worked example of the eval-as-PRD practice Neo wants to bring to Thomas Engineering, plus a reusable framing ('sweat the tokens as much as you sweat the pixels') for reviewing AI transcripts.
Original email
Lennys Newsletter · 7/26/2026
Anthropic's first technical PM on token maxing, the jagged edge, and living in the future | Dianne Penn