Most AI teams already have the pieces required to improve their product. They have application logs, model calls, user feedback, prompt files, test scripts, analytics, and support conversations.
The problem is that the pieces do not form a loop.
A user reports a hallucination in Slack. An engineer finds a related request in the logs. Someone edits the prompt. A few examples look better in a playground, so the change ships. Two weeks later, a similar failure appears and nobody knows whether the old fix regressed or this is a new problem.
The team is working hard, but the system is not learning.
I built Currai around a simple idea: production traces, evaluations, prompts, experiments, cost, and user outcomes should connect. Every failure should make the test suite stronger, and every change should be evaluated against the reality users created.
The Linear Workflow Breaks
The default AI development workflow is linear:
- Write a prompt.
- Test a few inputs.
- Deploy it.
- Watch aggregate metrics.
- React when users complain.
This works for a prototype. It breaks as soon as the application gains multiple prompts, models, retrieval steps, tools, and user segments.
Aggregate metrics tell you that latency increased or token cost rose. They rarely tell you why a particular answer was unsupported. A screenshot shows the output, but not the retrieved context or tool response that shaped it. A playground recreation uses today’s prompt, not necessarily the version that ran in production.
The missing element is continuity. The production event needs to stay connected to the debugging, evaluation, and release decision that follows it.
Step 1: Capture a Useful Trace
A trace should represent one meaningful product run: a user request, an agent turn, or a workflow execution.
Inside that trace, generations represent model calls and spans represent non-model work such as retrieval, reranking, database access, or tool execution. The trace can carry input, output, model, prompt version, tokens, latency, cost, user or session identifiers, environment, and relevant metadata.
The goal is not to collect data for its own sake. It is to preserve cause and effect.
When the final answer is wrong, the team should be able to inspect the path that created it. When a run is slow, they should see which step consumed the time. When cost spikes, they should identify the prompt, model, feature, or retry behavior responsible.
Start with one important AI path. Instrument it end to end before tracing every experimental call in the codebase. A complete trace of the workflow users depend on is more useful than partial telemetry everywhere.
Step 2: Attach Outcome Signals
A trace tells you what happened. It does not automatically tell you whether the result was good.
Outcome signals can come from several places:
- deterministic validators;
- automated rubric-based evaluations;
- LLM judges;
- human review;
- explicit user feedback;
- and implicit product outcomes such as resolution, correction, abandonment, or escalation.
These signals answer different questions. A schema validator can confirm the output is structurally usable. A groundedness evaluator can check support against retrieved context. A human reviewer can catch nuance the automated rubric misses. A product event can show whether the user accepted the answer or immediately tried again.
The most valuable view keeps those signals beside the trace. A score without the prompt, context, and output becomes another isolated number.
Step 3: Find a Cohort, Not Just an Anecdote
One bad response matters, but it may not explain the shape of the problem.
Filter related traces by prompt version, agent name, model, tag, evaluation score, latency, cost, environment, user segment, or time range. The question is whether the failure is isolated or systematic.
For example, a support assistant may have a groundedness problem only for one document collection. An agent may fail after the third tool call, not on simple tasks. A new prompt may improve English responses while reducing quality in French. Average scores can hide each of these patterns.
Cohort analysis turns “this answer is bad” into “this prompt version fails this class of requests.” That statement is specific enough to guide a fix.
Step 4: Promote the Failure Into a Test Case
This is the step most teams skip.
After debugging the trace, preserve it as an evaluation example. Keep the input and the context required to reproduce the behavior. Add an expected outcome, reference answer, rubric, or deterministic assertion depending on the failure.
The example now serves two jobs:
- It proves the current system has a known weakness.
- It prevents future changes from reintroducing the same weakness silently.
Production cases are especially valuable because they contain real ambiguity. They reflect how users phrase questions, what retrieval actually returned, and which tool states existed at the time.
Privacy still matters. Remove data that is unnecessary for reproduction, control access, and apply retention policies. “Use real traces” should never mean “copy sensitive data into every test environment.”
Step 5: Compare the Candidate Change
Now test the new prompt, model, retrieval configuration, or agent policy against the dataset.
Do not compare only one headline quality score. A change can improve correctness while making responses slower, more expensive, or less consistent. Review multiple dimensions:
- task success or rubric score;
- regressions by category;
- latency distribution;
- token use and estimated cost;
- tool-call count;
- formatting validity;
- and human preference on ambiguous cases.
This is where prompt versioning becomes operationally meaningful. A version is not just a saved string. It is a candidate with measured behavior against a defined set of cases.
The release decision can then be explicit: ship only if the candidate improves the target failure without crossing agreed limits elsewhere.
Step 6: Observe the Same Behavior After Release
Offline evaluation answers, “How did this candidate perform on our known cases?” Production evaluation answers, “How is it performing on the distribution arriving now?”
You need both.
After release, group new traces by the deployed prompt and model version. Run suitable evaluations on sampled or targeted traffic. Watch the failure dimension the change was meant to improve and the operational signals it might affect.
If performance holds, the team has evidence that the fix transferred beyond the test set. If it does not, new traces reveal the gap and become the next cases in the dataset.
Now the workflow is a loop:
production → trace → evaluation → dataset → experiment → release → production
Why Unification Matters
It is possible to build this loop with separate logging, experiment, prompt, annotation, and monitoring tools. Many teams do.
The cost is not only integration work. Context gets lost at every handoff. A trace ID in one system does not automatically carry the human label in another. The prompt in the repository may not match the prompt that generated the production answer. An offline score may use different logic from the production monitor.
Currai’s approach is to keep the trace as the central record. Evals can run against real traces. Failed traces can become dataset cases. Prompt candidates can be compared on quality, latency, and cost. The result remains tied to the run that produced it.
That does not remove the need for engineering judgment. It makes the evidence easier to follow.
A Small-Team Version of the Loop
You do not need thousands of examples or a dedicated evaluation team to begin.
A practical starting point is:
- Trace the most important user-facing AI workflow.
- Define the three failure modes that would most damage the product.
- Create 10–20 representative examples for each.
- Add deterministic checks wherever possible.
- Use a clear rubric and human review for subjective dimensions.
- Review failed production traces every week.
- Add the valuable failures to the dataset before changing the prompt.
- Require an evaluation comparison for every meaningful prompt or model release.
This process is intentionally small. The habit matters more than the initial scale.
The Compounding Advantage
An AI product with no feedback loop resets its knowledge after every incident. The team fixes symptoms, but its tests do not become more representative.
A connected loop compounds. Every meaningful failure improves the dataset. Every dataset improvement makes experiments safer. Every measured release creates better production evidence. Over time, the product becomes harder to regress because its evaluation suite reflects the difficult situations users have already discovered.
That is the real promise of AI observability. It is not another dashboard for watching model calls.
It is the infrastructure that turns what happened into what you improve next.