AI quality can drift while uptime and latency remain perfect.
Prompts change, models update, documents evolve, user traffic shifts, and tools return new shapes. Detecting drift begins with versioned traces and stable evaluation signals.
Track quality by prompt, model, retrieval configuration, task, language, and user cohort. Watch distributions rather than only averages. A stable overall score can hide a sharp decline in one important segment.
Maintain a fixed sentinel dataset for comparability, then evaluate a rolling sample of production traffic for new behavior. Alert on sustained changes with enough volume, not every noisy score. Send examples with the alert so investigation begins with evidence.
Combine automated evaluation with user corrections, retries, abandonment, and escalation. When a new failure appears, label it and add it to the regression suite.
Drift monitoring is a loop: detect, inspect, reproduce, fix, and verify. The goal is not a perfectly flat chart; it is shortening the time between quality changing and the team understanding why.