LLM Evaluation and AI Agent Evals: How to Know Your AI System Is Ready for Production

The gap matters.

Observability records behavior. Evals judge it.

An eval is a repeatable test with:

1. 1an input;
2. 2expected behavior or grading criteria;
3. 3a system run;
4. 4one or more graders;
5. 5a recorded result.

The grader can be deterministic code, an exact match, a model-based judge, a human, or a combination.

An agent eval can grade several surfaces.

OpenAI's current agent-evaluation guidance highlights trace grading as a way to inspect workflow-level behavior such as tool selection, handoffs, and instruction violations.

Anthropic's 2026 agent-eval guidance similarly stresses that agents require richer evaluation because they operate across turns, call tools, modify state, and adapt to intermediate results.

Historical cases are the strongest starting point.

Create categories.

The dataset should resemble production.

Suppose an agent handles refund requests.

A weak eval asks:

A stronger eval asks:

• Was the right customer identified?
• Was current policy retrieved?
• Was eligibility correct?
• Was the amount correct?
• Was approval triggered when required?
• Was the refund tool called with the correct arguments?
• Did the account state change?
• Did the final message match the actual state?

This turns a language test into a workflow test.

Use deterministic grading where the correct answer can be expressed exactly.

Examples include:

• JSON schema validity;
• correct tool name;
• exact account ID;
• amount tolerance;
• required approval flag;
• database state;
• regex or field checks.

Deterministic graders are cheap, repeatable, and easy to interpret.

A model grader is useful when correctness requires judgment.

Examples include:

• whether a summary preserves important evidence;
• whether a response is grounded in supplied sources;
• whether an escalation rationale follows policy;
• whether a trajectory contains unnecessary steps.

Model graders need their own validation. Compare them against trusted human labels on a representative sample.

Human review remains important when:

• the task is subjective;
• policy is nuanced;
• errors are costly;
• grader quality is uncertain;
• the system is new;
• evaluation criteria are still being developed.

The IBM-led 2026 production study found that 74% of surveyed practitioners primarily relied on human evaluation.

That figure also shows how much room remains for repeatable automated testing.

Model behavior can vary between runs.

A test that passes once can fail on the next attempt.

For stochastic tasks, run the same case multiple times and measure the distribution.

Possible measures include:

• pass@k;
• pass rate;
• average grader score;
• worst-case behavior;
• variance;
• failure-category frequency.

The required number of repetitions depends on the task and consequence.

A trace captures the execution path.

A simplified trace looks like this:

user request
→ retrieve policy
→ retrieve account
→ choose refund tool
→ request approval
→ execute refund
→ verify account state
→ respond

A trace grader can identify a system that reached the correct answer through an unacceptable path.

That matters because accidental success is fragile.

RAG needs its own test set.

Each query should have known relevant evidence.

Measure:

• recall@k;
• precision@k;
• ranking;
• freshness;
• permission filtering;
• citation accuracy.

Do not let final-answer quality hide retrieval defects.

Tool evaluation should test both selection and arguments.

Offline evals run against a controlled dataset.

Online evals measure production traffic.

Online measures can include:

• task completion;
• escalation;
• human correction;
• tool failure;
• policy violation;
• user abandonment;
• latency;
• cost;
• sampled model grading;
• incident rate.

Production evaluation needs privacy and sampling policies appropriate to the application.

A production failure is valuable evidence when it becomes reproducible.

The team should:

1. 1preserve the trace;
2. 2classify the failure;
3. 3reproduce it;
4. 4add a test;
5. 5fix the responsible layer;
6. 6rerun the complete suite;
7. 7release only after the regression passes.

A release gate defines which eval results block deployment.

Example:

BLOCK RELEASE IF:
- policy violation > 0 on high-consequence suite
- tool argument accuracy < 99.5%
- retrieval recall@5 < target
- task completion drops > allowed tolerance
- known regression reappears

Thresholds should follow the use case and consequence. Do not copy generic numbers into a production policy without evidence.

Changing the model can change:

• reasoning;
• tool choice;
• argument formatting;
• latency;
• cost;
• refusal behavior;
• retrieval query generation;
• output style.

Run the existing eval suite before changing the production model.

The same applies to prompt changes, tool-description changes, retrieval changes, and workflow changes.

NIST's Generative AI Profile frames AI risk management across governance, mapping, measurement, and management.

For an eval program, that translates into clear ownership.

Define:

• who owns the dataset;
• who approves grading criteria;
• who reviews failed releases;
• who can change thresholds;
• how incidents are added;
• how sensitive test data is handled;
• how long results are retained.

Charts based on survey findings identify their source in the caption. Mathematical examples are labeled as illustrations. Architecture diagrams and decision frameworks are original explanatory assets created for this post.