Developers understand unit tests because the contract is crisp.
Given this input, return this output. Throw this error. Do not mutate that value. The test is either green or red, and a failure usually points toward a narrow piece of code.
Then a team adds an LLM and tries to apply the same mental model:
input: "Explain our refund policy"
expected: one exact paragraph
The test fails when the model writes an equally correct paragraph with different wording. So the team loosens the assertion until almost anything passes—or gives up and tests only that a string was returned.
Neither approach is useful.
AI evaluations are not replacements for unit tests. They measure a different layer of the system. The key is to preserve software-testing discipline while changing what counts as evidence.
Deterministic Code Still Needs Deterministic Tests
An AI application contains plenty of ordinary software:
- request validation;
- prompt rendering;
- permissions;
- tool handlers;
- parsers and schemas;
- retrieval filters;
- caching;
- rate limits;
- and billing logic.
Test those pieces normally. If a tool is given invalid arguments, assert the exact error. If an output must conform to a schema, parse it. If a user cannot access another tenant’s documents, write a hard authorization test.
Do not use an LLM judge to check something code can determine exactly.
This gives us the first rule of evaluation design: use the least subjective evaluator that can correctly measure the requirement.
AI Behavior Has Graded Contracts
Many model outputs are not exact, but they are not arbitrary either.
A support answer may need to be grounded in policy, complete, concise, empathetic, and free from forbidden promises. Several wordings could satisfy those requirements. Some outputs may be mostly correct but omit an important condition. Others may sound excellent while inventing the central fact.
The contract is graded across dimensions.
Instead of one expected string, define a rubric:
- Groundedness: every policy claim is supported by the provided documents.
- Completeness: the answer covers the time window, eligibility, and next step.
- Relevance: it answers the question without unrelated policy text.
- Tone: it is respectful and does not blame the customer.
- Safety: it does not request sensitive information in an insecure channel.
Now different valid phrasings can pass, while a fluent hallucination can fail.
An Eval Case Needs Context
A useful evaluation example is more than an input and output.
Depending on the system, it may include:
- user request;
- conversation history;
- retrieved documents;
- available tools and their results;
- model and prompt version;
- expected facts or reference answer;
- category and risk level;
- rubric;
- and known failure notes.
For an agent, the final answer may be insufficient. If the agent reached a correct result by calling a destructive tool unnecessarily, the run should fail. If it guessed after a tool error, the trace matters more than the polished prose.
Think of the evaluation unit as the behavior the user experienced. For a single-generation feature, that may be one input and output. For an agent, it is often the complete trace.
Four Evaluator Types
Most practical suites combine four kinds of evaluation.
1. Deterministic checks
Use code for schema validity, exact facts, numeric calculations, required citations, latency limits, tool-call rules, and other objective properties.
These checks are fast, cheap, explainable, and stable. Use them aggressively where they fit.
2. Reference-based metrics
When you have a verified answer or set of required facts, compare the result with that reference. Exact matching is rarely enough for prose, but fact coverage, semantic similarity, or task-specific extraction can help.
References work best when the task genuinely has a known answer. They are weaker for open-ended writing and situations with several equally good strategies.
3. LLM-as-a-judge
A capable model can apply a detailed rubric to an input and output, often with supporting context. This scales subjective review across many cases.
Judges need calibration. They can prefer verbosity, be influenced by answer order, miss domain nuance, or reward text that sounds authoritative. Treat a judge as a measurement instrument, not ground truth.
4. Human and user signals
Domain experts are essential for high-risk or ambiguous cases and for checking whether automated evaluators match actual judgment. User signals reveal product outcomes: acceptance, correction, retry, escalation, completion, or explicit rating.
These signals are costly and sometimes noisy, but they anchor the suite in reality.
Build the Dataset Around Risk
Random examples are not automatically representative.
Start with the product’s main jobs and the failures that matter most. A coding assistant may need tests for correct edits, repository conventions, test execution, and avoidance of destructive actions. A retrieval assistant needs groundedness, source selection, and appropriate abstention. A health-adjacent product needs conservative language and clear boundaries.
Include:
- common happy paths;
- edge cases;
- ambiguous requests;
- missing or conflicting context;
- adversarial inputs;
- multilingual or domain-specific cases where relevant;
- and historical production failures.
Tag the examples by category. An overall score can remain flat while one critical category collapses. Category-level results show where behavior changed.
The dataset should evolve. Every meaningful production failure is a candidate regression case.
Calibrate Before You Automate
Before trusting an evaluator, test it on examples humans have already labeled.
Create clear passes, clear failures, and difficult borderline cases. Ask multiple reviewers to apply the rubric. Resolve ambiguous wording. Then run the automated evaluator and measure agreement—not only overall accuracy, but the kinds of mistakes it makes.
If the judge regularly accepts unsupported claims, it cannot gate a groundedness release. If it disagrees mainly on stylistic borderline cases, it may still be useful as a triage signal.
Calibration also reveals bad rubrics. “Is this a good answer?” invites inconsistent judgment. “Are all factual claims supported by the supplied context? Return pass only when every material claim can be traced to a passage” is more testable.
Scores Need Uncertainty Too
An evaluation result is not automatically precise because it is numeric.
A score of 0.84 may represent 84 passes out of 100 deterministic cases. It may also be the average of subjective ratings from one model judge. Those numbers have very different meanings.
Track sample size, category distribution, evaluator version, and repeated-run variance when relevant. For stochastic systems, one run per case can hide instability. Run difficult cases multiple times if consistency matters.
Do not celebrate a two-point improvement without asking whether the dataset and evaluator can distinguish it from noise.
Offline and Online Evals Answer Different Questions
Offline evaluation runs a controlled dataset against a candidate. It helps compare prompt, model, retrieval, and agent changes before release.
Online evaluation scores or samples production traffic. It reveals distribution shift, new user behavior, tool failures, changing documents, and cases the offline dataset never anticipated.
The two should connect:
- Production exposes a failure.
- The trace is reviewed and labeled.
- A privacy-safe version becomes an offline case.
- Candidate fixes run against the full suite.
- The chosen change ships.
- Online evaluation verifies the behavior on new traffic.
Offline-only teams overfit to known examples. Online-only teams discover problems after users do and struggle to reproduce them. The loop needs both.
Use Thresholds That Reflect the Product
Not every metric deserves the same release rule.
A critical safety or authorization check may require zero known regressions. A tone score might tolerate small variation. Latency and cost may have budgets rather than pass/fail rules. A high-risk category can be weighted separately instead of disappearing inside the average.
A release policy might say:
- no regression on deterministic safety checks;
- no new failures in high-risk cases;
- groundedness improves by a meaningful margin;
- completeness stays within the baseline range;
- and p95 latency remains below the product budget.
The policy turns evaluation into a decision mechanism rather than a reporting exercise.
Evals Are Closer to a Measurement System
Unit tests verify deterministic contracts. AI evals measure probabilistic, contextual behavior. They need datasets, calibrated instruments, category analysis, repeated observation, and a connection to production.
That does not make them less rigorous. It means the rigor lives in different places:
- defining the behavior precisely;
- choosing the right evaluator;
- curating representative cases;
- measuring evaluator reliability;
- tracking versions;
- and setting release thresholds before seeing the result.
Keep your unit tests. Add integration tests around tools and data flow. Then build evaluations for the semantic layer those tests cannot see.
The question is not “Did the model return the exact sentence I expected?”
It is “Do we have credible evidence that this system behaves well enough for the job we are asking it to do?”