Advanced Evals Sequel: 'Criteria Drift' and a Free Open-Source Error-Discovery Skill
Summary
Hamel Husain and Shreya Shankar publish a sequel focused on the step most teams skip before writing eval metrics: systematically discovering failure modes from real traces. They give a concrete 3-step workflow (instrument traces, human+agent hybrid annotation via a custom review UI, cluster failures into prioritized modes) and ship a free open-source 'evals-skills' Claude/Codex plugin, with cited results from Shopify, Cursor, and Ramp.
Why this signal
The single most directly actionable piece this cycle for Neo's #2 focus area — 'criteria drift' (you only learn your real quality bar by reading traces, not by writing metrics upfront) is a reusable mental model, and the open-source skill plugin is something Thomas Engineering could adopt directly this week.
Original email
Lennys Newsletter · 9/22/2026
Advanced evals: How to find (and fix) hidden AI failures in your product