Most AI failures pass the test suite.
The endpoint returns 200. The schema validates. The model answers. The product still makes the wrong decision.
This is the trap in testing AI like normal software.
Traditional software is mostly expected to be deterministic. Given the same state and the same input, we expect the same output. Tests can assert that a calculation equals a known value, a permission blocks the wrong user or an API returns the right fields.
An AI system behaves differently. Its output can vary, several answers can be acceptable and a response can look correct while the reasoning behind it is fragile. Once tools, retrieval and multi-step agents are involved, the result depends on an entire trajectory of decisions rather than a single function call.
The solution is not to abandon conventional testing.
It is to add a testing discipline that measures behaviour, quality and risk across many representative runs.
Traditional tests are still essential
AI products are still software products.
Authentication should be deterministic. Permissions should be enforced. Data should be validated. Tool arguments should match their schemas. Retries should stop. Audit records should exist. A payment should never be made twice because a model produced the same action again.
These are excellent candidates for unit, integration and end-to-end tests.
If the code can be tested with an exact assertion, test it that way.
The mistake is assuming those tests also prove that the intelligent part works well.
A conventional test can confirm that an agent called the customer lookup tool with a valid account ID. It cannot, by itself, tell you whether the agent chose the right account, used the evidence correctly or should have escalated the case to a person.
That requires evaluation.
A correct answer can hide an incorrect system
Imagine an agent that reviews support requests and decides whether to issue a refund.
It receives a customer message, checks the order, reads the refund policy and produces a decision.
For one test case, it returns the expected answer: approve the refund.
That output alone does not tell us enough.
The agent may have retrieved the wrong policy and guessed correctly. It may have ignored a fraud signal. It may have tried five unnecessary tools before arriving at the result. It may approve the same request only six times out of ten. It may perform well in English and fail when the customer writes in Portuguese.
One correct answer can conceal a weak system.
This is why teams need to inspect both the outcome and the path that produced it.
Capability is not reliability
A demo usually asks one question: can the system do this?
Production asks a harder one: how often can it do this correctly, safely and within acceptable cost and latency?
That distinction matters because AI performance is a distribution, not a single result.
Run the same task repeatedly and you may see several valid paths, a few inefficient ones and an occasional failure. Change the phrasing, add irrelevant context or remove one document and the distribution can shift again.
Anthropic’s guidance on agent evaluations makes this practical with two useful measures. pass@k asks whether at least one of several attempts succeeds. It is useful when the system can retry or generate alternatives. pass^k asks whether every attempt succeeds. It is much closer to what a business needs when consistency matters.
An agent that succeeds once in five attempts may look impressive in a curated demo.
It is not reliable enough to approve refunds.
The AI testing stack
A useful evaluation system has several layers. Each one catches a different class of failure.
Layer 1: Deterministic correctness
Start with the parts that have clear answers.
Test permissions, schemas, calculations, data transformations, tool contracts, state transitions and business rules with normal automated tests.
These tests should be fast, strict and run on every change.
Do not ask a model to judge something that code can verify exactly.
Layer 2: Outcome
Now measure whether the system completed the intended task.
Did it identify the correct account? Did it produce a valid recommendation? Did it create the requested report? Did the customer receive the right resolution?
Some outcomes have an exact answer. Others need a rubric with explicit criteria.
The key is to evaluate the business result, not whether the response resembles a preferred sentence.
Layer 3: Trajectory
For agents, the route matters.
OpenAI’s agent evaluation guidance recommends evaluating traces, the record of tool calls and decisions made during a run. Google’s Agent Development Kit and Microsoft’s agent evaluators support the same idea.
Inspect whether the agent:
- Selected the right tool.
- Used the correct arguments.
- Retrieved relevant evidence.
- Repeated an action unnecessarily.
- Respected the required sequence.
- Stopped when the task was complete.
- Escalated when confidence or authority was insufficient.
A good result produced through a dangerous path is still a failure.
Layer 4: Quality
Quality is rarely one number.
An answer may be factually correct but unclear. It may be concise but incomplete. It may include every required fact while using a tone that damages the customer relationship.
Define the dimensions that matter for the product: factuality, relevance, completeness, clarity, tone, groundedness and instruction following are common examples.
Then write a scoring rubric with concrete descriptions for each level.
“Good answer” is not a rubric.
“Cites the applicable policy, explains the decision in plain language and does not invent account details” is much closer.
Layer 5: Risk
Quality and safety are related, but they are not the same.
Test for prompt injection, data leakage, harmful content, unsupported claims, excessive agency and actions outside the user’s permission. Include cases designed to confuse the system, not only cases designed to help it succeed.
NIST’s Generative AI risk profile recommends measuring risks with empirically validated methods and recognising that benchmark results may not transfer directly into a real deployment context.
This is especially important for systems connected to email, finance, customer data or operational tools. A fluent answer is not evidence that the action was safe.
Layer 6: Production reality
Offline evaluations are controlled. Users are not.
Production introduces unusual phrasing, incomplete information, new documents, changing policies, tool outages and objectives the test suite never anticipated.
Monitor real outcomes, human overrides, escalation rates, latency, token usage, tool failures and repeated attempts. Sample traces for review. Turn surprising failures into new evaluation cases.
The evaluation suite should evolve with the product.
Build the first evaluation suite
You do not need thousands of examples to begin.
Start with 30 to 50 cases drawn from real work. Include ordinary tasks, important edge cases, known failures and adversarial inputs. Remove sensitive data, but preserve the structure that made the case difficult.
For each case, record:
- The input and relevant context.
- The expected business outcome.
- Required and forbidden actions.
- The evidence the system should use.
- The quality rubric.
- The conditions that require human escalation.
Run every case more than once.
Track the distribution of results instead of keeping only the best output. Compare changes against a stable baseline. A model or prompt update should not ship because three examples looked better in a notebook.
It should ship because the evaluation set improved without creating unacceptable regressions elsewhere.
Do not trust the judge blindly
Models can help evaluate model outputs, especially when the task requires semantic judgement. They can compare responses against a rubric, identify missing evidence and explain why a result failed.
But a model judge is another probabilistic system.
Research on LLMs as judges has documented biases related to response position, verbosity and the judge’s own preferences. A polished answer can receive a better score than a shorter, more accurate one.
Calibrate automated judges against human review.
Use clear rubrics. Hide irrelevant presentation differences. Test whether swapping response order changes the verdict. Review disagreements. Keep deterministic checks outside the judge whenever possible.
The judge accelerates evaluation.
It does not remove accountability.
Turn failures into infrastructure
The most valuable evaluation cases often come from production incidents.
When the system fails, do more than patch the prompt.
Capture the case. Identify the failure class. Add it to the regression set. Decide whether the solution belongs in the model instructions, retrieval layer, tool design, deterministic code, permission system or human approval flow.
This changes failure from an anecdote into an asset.
Over time, the evaluation suite becomes a map of what the business has learned about the system. It protects that knowledge when models, prompts, tools and teams change.
Test the product, not only the model
The model is one component.
Production quality emerges from the full system: context, retrieval, tools, instructions, permissions, interface, fallback behaviour and human oversight.
A stronger model cannot compensate for a tool with ambiguous parameters. A better prompt cannot repair missing access controls. A high benchmark score cannot prove that a customer received the right outcome.
The right question is not:
“Did the model answer correctly?”
It is:
“How reliably does the product produce the intended outcome, through an acceptable path, under realistic conditions?”
That question is harder to test.
It is also the one that matters.
At Syftware, we help teams turn promising AI prototypes into observable, evaluated and production-ready products. The goal is not a perfect demo. It is a system the business can understand, improve and trust.