Evals as Product Guidance
Non-Determinism at Temperature=0: A Hidden Eval-Design Trap
Deep technical walkthrough of everything between pressing Enter and the first token (context assembly, safety classification, tokenization, batching, prefill/decode). Notes that identical prompts can …
View details →A 'Design Critic' Subagent Pattern for Turning AI Output From Bland to Distinctive
Ex-Apple AI design lead Anshu Chimala lays out a Discover/Define/Deliver framework for getting genuinely distinctive design output from AI agents, centered on a 'design critic' subagent — shown only a…
View details →LLMs as a Judge: The Full Evaluation Stack Reference
Comprehensive explainer on why deterministic tests fail for LLMs and lays out the full evaluation stack: golden datasets, automated metrics, LLM-as-judge (point/pass-fail/pairwise/error-ID), human cal…
View details →Advanced Evals Sequel: 'Criteria Drift' and a Free Open-Source Error-Discovery Skill
Hamel Husain and Shreya Shankar publish a sequel focused on the step most teams skip before writing eval metrics: systematically discovering failure modes from real traces. They give a concrete 3-step…
View details →Anthropic's Head of Product for Research/Labs: Evals Are the Primary Artifact of Product Work, Not the PRD
Dianne Penn, Anthropic's first technical PM, describes how her team generates 30-40 representative examples encoded as prompts with golden answers for every major feature — that eval set, not the PRD,…
View details →Braintrust CEO: Evals Are the Modern PRD — Build the Feedback Loop First
Ankur Goyal (Braintrust CEO) lays out a concrete eval-driven product process: treat evals as the modern PRD, build a feedback loop that turns real production data into new evals continuously, and run …
View details →How DoorDash Built a Simulation + Eval Flywheel for LLM-Based Support
DoorDash built an offline simulator that generates multi-turn support conversations, then an eval framework grades them against quality criteria. The flywheel: simulation generates eval cases, eval fa…
View details →