★MediumAct: MediumFull content
Non-Determinism at Temperature=0: A Hidden Eval-Design Trap
ByteByteGo·August 31, 2026
New
Summary
Deep technical walkthrough of everything between pressing Enter and the first token (context assembly, safety classification, tokenization, batching, prefill/decode). Notes that identical prompts can produce different outputs even at temperature=0 due to batching-dependent floating-point behavior.
Why this signal
The temperature=0 non-reproducibility point is a subtle, easy-to-miss source of eval flakiness worth flagging to anyone building automated eval pipelines for Thomas Engineering.
Original email
ByteByteGo · 8/31/2026
What Happens Inside an AI Chatbot Between Enter and the First Word?