Changing a prompt is easy. Knowing whether the new prompt is better is one of the hardest parts of building an AI product.
You can read the text and agree with the intent. You can run five examples in a playground and prefer the new outputs. You can ask another model which version it likes. None of those steps tells you how the change behaves across the cases your product actually encounters.
Prompts are production logic. They deserve a release process.
With Currai, I think about that process as a path from a versioned change to evidence: define the intended improvement, build the right dataset, evaluate both versions, inspect tradeoffs, release deliberately, and verify the result on production traces.
Begin With a Hypothesis
“Make the prompt better” is not testable.
A useful prompt change begins with a named behavior:
- reduce unsupported claims when retrieval has weak evidence;
- produce a valid schema more consistently;
- ask a clarifying question when required context is missing;
- avoid unnecessary tool calls;
- or make answers shorter without reducing completeness.
The hypothesis determines the dataset and evaluator. If the goal is groundedness, you need examples with source context and a rubric for support. If the goal is valid JSON, use a parser and schema validator. If the goal is tool selection, inspect the action sequence rather than judging only the final prose.
Write the release claim before changing the prompt:
Version B should reduce unsupported refund-policy claims without lowering answer completeness or increasing median latency by more than our accepted limit.
Now the team knows what evidence would justify shipping.
Version More Than the Text
A prompt does not run alone. Its behavior depends on the model, parameters, system instructions, retrieved context, tools, output schema, and application logic around it.
If an experiment changes several of those at once, a better result will not reveal which change helped. A worse result will be equally hard to debug.
Keep the candidate narrow where possible. Record enough configuration to reproduce the run:
- prompt name and version;
- model and relevant parameters;
- dataset version;
- evaluator and rubric version;
- retrieval or tool configuration when material;
- and application release or environment.
This is not bureaucracy. It is the minimum context required to compare results honestly.
Build a Dataset From Three Sources
A useful evaluation dataset combines different kinds of evidence.
Designed cases
These represent the product contract: common requests, required formats, policy boundaries, and important edge cases the team already understands.
Historical failures
These come from production traces, support reports, user corrections, and previous incidents. They make the suite specific to the ways the real system has failed.
Fresh or adversarial cases
These prevent the team from optimizing only for yesterday’s mistakes. Include ambiguous inputs, missing context, conflicting sources, unusual tool results, and cases designed to expose the target failure.
The dataset does not need to be enormous. It needs coverage. Fifty well-labeled cases across meaningful categories can guide a small product better than thousands of synthetic inputs with unclear expectations.
Match Evaluators to Failure Modes
No evaluator should be asked to judge everything.
Use deterministic code for objective properties:
- schema validity;
- required fields;
- forbidden phrases;
- exact calculations;
- citation presence;
- tool-call constraints;
- and latency or token thresholds.
Use rubric-based evaluation for qualities that need interpretation:
- groundedness;
- completeness;
- relevance;
- tone;
- clarity;
- and policy adherence.
Use human review for high-impact ambiguity, evaluator calibration, and cases where reasonable people may disagree. User outcomes add another layer: did the answer resolve the request, get accepted, or lead to an immediate retry?
Currai keeps evaluation results with the trace inputs and outputs. That makes a low score inspectable. You can see whether the evaluator caught a real issue or misunderstood the case.
Establish the Baseline First
Run the current production version against the dataset before looking at the candidate.
This baseline tells you three things:
- whether the dataset reproduces the problem;
- whether the evaluators separate good and bad behavior;
- and how much room for improvement actually exists.
If the old prompt passes every groundedness case despite known hallucinations, the evaluation setup is not ready. Fix the cases or rubric before comparing candidates.
This step prevents a surprisingly common mistake: trusting an evaluation number that was never validated against examples the team understands.
Compare at the Case Level
Average scores are useful summaries and dangerous conclusions.
Imagine version B raises the overall score from 0.82 to 0.86. That sounds like a win. But the case-level view may show that it improved simple English questions and regressed every multilingual case. Or it may have gained style points while failing one high-risk policy scenario.
For each candidate, examine:
- overall and category-level results;
- cases fixed;
- new regressions;
- evaluator disagreement;
- latency percentiles;
- token use and cost;
- output length;
- and tool-call behavior.
Make the release threshold reflect product risk. A formatting feature may tolerate a small number of recoverable failures. A workflow that controls payments or provides health-adjacent guidance needs much stricter review.
Review the Pareto Frontier
There is rarely one prompt that wins every metric.
A more detailed prompt may improve instruction following while increasing input tokens. A reasoning-heavy model may improve difficult cases while adding latency. A terse response policy may reduce cost and user patience while omitting useful context.
The team is choosing a tradeoff, whether it acknowledges it or not.
Plotting or comparing quality, latency, and cost together helps identify candidates that are clearly dominated. If version B costs more and scores worse, reject it. If version C produces a meaningful quality gain for a small latency increase, the product owner can make an informed decision.
The right unit is often cost or latency per successful outcome, not per model call. An apparently cheap configuration can retry more often and cost more at the workflow level.
Release With Identity
When the candidate ships, production traces should record which prompt version generated each result.
Without that identity, the experiment ends at deployment. The team can see aggregate quality move but cannot confidently attribute it to a version, user cohort, model, or other simultaneous change.
Depending on product scale and risk, release strategies can include:
- a small internal cohort;
- a percentage rollout;
- an A/B comparison;
- a shadow evaluation where the candidate runs without serving its output;
- or a full release with a defined rollback threshold.
The important part is deciding the observation window and failure threshold before enthusiasm for the new prompt takes over.
Evaluate Production, Then Feed It Back
Offline tests contain known cases. Production contains the changing distribution.
After release, run the relevant evaluations on real traces—either all eligible traffic or a meaningful sample. Compare the new version against the baseline by quality, latency, cost, user segment, and task type. Review user feedback alongside automated scores.
When a new failure appears, do not only patch it. Promote it into the dataset. The next prompt candidate should be tested against both the old suite and the case that escaped it.
This is the production-to-evaluation loop:
- Trace what users experienced.
- Score and review the outcome.
- Turn meaningful failures into cases.
- Compare the next candidate against those cases.
- Release with version identity.
- Verify the improvement in production.
A Practical Release Checklist
Before shipping a prompt change, I want clear answers to these questions:
- What behavior is this change intended to improve?
- Which dataset represents that behavior?
- Does the baseline reproduce the known problem?
- Which evaluators are deterministic, model-based, and human-reviewed?
- Where did the candidate improve and regress?
- What happened to latency, tokens, cost, and tool use?
- What version identifiers will appear in production traces?
- How will the rollout be limited or reversed?
- Which production signal will confirm success?
- How will new failures enter the regression suite?
If those answers exist, the prompt is no longer “vibe tested.” It has evidence behind it.
Confidence Is Not Certainty
No evaluation suite can guarantee that a generative system will behave perfectly on every future input. Production will always contain surprises.
The goal is not certainty. It is justified confidence: knowing what you tested, what improved, which tradeoffs you accepted, how the release is identified, and how quickly a new failure will become a permanent test.
Prompt engineering becomes real engineering when changes are observable, comparable, and reversible.
That is the difference between editing text and operating an AI product.