Single-turn evals miss failures that appear only after a conversation develops.
A multi-turn assistant must preserve facts, respect corrections, avoid repeating questions, and complete a task across changing context. Evaluate scripted sessions with checkpoints: what should the system know, ask, or do after each turn?
Include user corrections, topic changes, ambiguous references, stale memories, tool failures, and long sessions. Check both the final outcome and the path. An assistant that eventually succeeds after losing information three times still creates a poor experience.
Measure consistency, instruction retention, recovery, unnecessary turns, tokens, latency, and cost. Use session-level human review for nuance, then extract deterministic checks where patterns emerge.
Production sessions are especially valuable. Preserve privacy-safe examples of abandonment, repeated questions, and escalation. Multi-turn quality is not the average quality of isolated messages; it is whether the conversation moves coherently toward the user’s goal.