How DoorDash Built a Simulation + Eval Flywheel for LLM-Based Support
Summary
DoorDash built an offline simulator that generates multi-turn support conversations, then an eval framework grades them against quality criteria. The flywheel: simulation generates eval cases, eval failures surface model gaps, model improvements feed back into simulation. This decouples evaluation from live production traffic and enables rapid iteration without user-facing risk.
Why this signal
Most concrete implementation of an eval flywheel from a named company at scale (not a lab — a real ops product). The simulation-first approach is a pattern Thomas Engineering could apply to any AI-assisted workflow that handles complex multi-step interactions (e.g., manufacturing support, quoting, supplier evaluation).
Original email
ByteByteGo · 5/30/2026
How DoorDash Built a Testing System to Evaluate LLMs