How to Build the Evals Layer of Your Agent Harness

An AI agent tells a customer, “Your booking has been restored.” The response sounds helpful, the tool calls completed without an obvious error, and the conversation looks successful.

Then someone checks the database. No booking exists.

In another run, the agent creates the booking correctly but also creates a duplicate. In a third, it refuses to help even though the customer is eligible for recovery.

These are different failures, and a basic check of the final response might miss all of them.

An evaluation layer gives your team a structured way to detect these problems. It turns expectations into testable scenarios, runs the agent under controlled conditions, and compares what happened with what should have happened.

This guide explains how to build that layer around an agent harness. We will use a fictional booking support agent throughout, with practical examples covering datasets, execution, tracing, graders, repeated trials, and release decisions.

What Is the Evals Layer of an Agent Harness?

An agent harness is the software surrounding a language model that lets it operate as an agent. It manages the execution loop, supplies context, exposes tools, handles results, and decides when execution should continue or stop.

The evals layer is the infrastructure that tests that system.

It supplies tasks, prepares environments, records execution, evaluates results, and produces evidence about whether a change improved the agent. You are evaluating the model and its surrounding application together, rather than treating the model as an isolated component. Anthropic makes this distinction in its guidance on agent evaluations. anthropic.com

A useful evals layer answers several questions:

• Did the agent complete the user’s task? A convincing response does not establish completion. A booking agent needs a valid booking record, while a coding agent needs a working change. Define success in terms of observable results that matter to the user.
• Did it operate within the required boundaries? An agent might achieve the requested result while accessing another customer’s information or performing an unauthorized action. Successful task completion and acceptable behavior need separate checks.
• Does the behavior hold across different situations? An agent that handles one carefully prepared example may struggle with missing information, repeated requests, tool failures, or long conversations. Your evaluation set should expose those differences.
• Can you explain why a run failed? A score is useful only when it leads to a decision. The evaluation record should help distinguish incorrect tool selection, missing evidence, backend failures, and problems in the grader itself.

Start With One Workflow and Define Success Precisely

Before choosing an evaluation platform, choose a workflow.

For this guide, the agent handles customers whose payment succeeded but whose interview booking is missing.

A vague requirement would be:

Help customers resolve booking problems.

A testable requirement would be:

For an authenticated customer with a verified captured payment and no booking, create one eligible booking through the recovery service, preserve unrelated records, and accurately explain the result.

That definition contains several independent obligations. Write them down before building the runner.

For the example agent, they could include:

• Verify the transaction belongs to the authenticated customer. The agent must not rely on an account identifier typed into the conversation. The application should establish identity, and the recovery service should enforce ownership independently.
• Recover only eligible orders. A pending payment, canceled package, or already completed recovery may require a different response. Your expected behavior should describe those situations explicitly instead of treating every missing booking as recoverable.
• Create exactly one booking. Repeated requests and delayed webhook events should not produce additional bookings for the same order. This requires backend duplicate protection, while the eval verifies that the complete workflow behaves correctly.
• Describe the verified outcome. If recovery succeeds, the response can confirm it. If verification is unavailable, the agent should explain the unresolved status. Claiming completion without evidence should fail the communication check.

These requirements become the foundation for your test cases and graders.

You do not need a separate service for every row. A single repository can contain all these components.

The important boundary is between executing the agent and judging its execution. If the agent supplies its own success flag and your evaluator accepts it without checking independent evidence, you have built self-reporting rather than a reliable test.

For the booking agent, the runner executes the workflow, while a separate read-only connection inspects the resulting database state.

Step 1: Build a Dataset That Reflects Actual Failure Modes

Begin with cases that represent the decisions your agent must make.

A useful first dataset for the booking workflow might include the following scenarios:

• Captured payment with no booking. This is the ordinary recovery case. The fixture contains one eligible order owned by the authenticated customer and no corresponding booking. The expected result is one confirmed booking and a response that accurately describes the completed recovery.
• Payment still pending. The customer believes payment succeeded, but the payment service reports a pending transaction. The expected result is no booking creation and a clear explanation of what remains unverified. This tests whether the agent follows evidence when it conflicts with the customer’s assumption.
• Booking already exists. The customer may be looking in the wrong place, or an earlier recovery may have succeeded. The agent should locate the existing booking and explain how to access it. Creating another booking would fail the duplicate check.
• Several orders could match the request. The customer mentions a missing interview without providing enough detail to identify the order. The correct behavior may be a focused clarification question. An eval should reward appropriate clarification instead of forcing the agent to guess.
• The payment service is unavailable. Configure the test adapter to return a timeout or temporary error. The agent should follow the documented retry or escalation policy and explain the unresolved status without inventing a payment result.
• The request contains another customer’s order reference. The agent must not expose or modify that customer’s records. Check both attempted operations and actual effects. A backend denial protects the data, but an unauthorized attempt still reveals a behavior worth investigating.

Assign tags such as recovery, ambiguity, dependency-failure, and authorization.

These tags let you inspect results by behavior. An overall improvement can hide a regression in a small but important category.

Keep some cases available during development and reserve a separate set for release evaluation. If every prompt change is tuned against the same visible examples, the suite becomes less useful as an independent check.

Step 2: Separate Agent Inputs From Grading Information

An evaluation case often contains information that the agent should never receive directly.

For example, the grader needs to know that exactly one booking should exist after the run. The agent needs the customer’s request, relevant operating instructions, and access to authorized tools.

An illustrative case definition could look like this:

{
  "id": "booking-recovery-001",
  "tags": ["recovery", "captured-payment"],
  "agent_input": {
    "message": "My payment succeeded, but my interview is missing."
  },
  "fixture": {
    "name": "captured-payment-no-booking-v1",
    "authenticated_user_id": "user-101"
  },
  "grader_only": {
    "order_id": "order-501",
    "expected_user_id": "user-101",
    "expected_booking_count": 1,
    "expected_booking_status": "confirmed"
  }
}

Pass only the intended inputs to the agent. Use the fixture to prepare its environment, and retain grader_only information inside the evaluation system.

Also keep grading files outside any filesystem or tool interface that the agent can access.

This separation matters particularly for agents that can inspect files, execute code, or search a repository. An answer hidden in an accessible evaluation directory is still accessible.

Step 3: Give Every Trial a Clean Environment

A trial is one attempt at a case. Each attempt should begin from the state defined by that case.

Otherwise, a previous run may accidentally solve the next one.

For the booking example, imagine the first trial creates a booking. If the second trial reuses that database, it is now testing “find an existing booking,” even though the case was supposed to test recovery.

Build environment preparation into the runner:

• Use unique trial identifiers. Allocate database records, cache prefixes, files, and queued events to a particular trial. Namespacing helps prevent parallel runs from reading or modifying one another’s data.
• Reset more than the main database. A booking workflow may depend on Redis, webhook queues, retry records, or cached payment responses. Resetting SQL tables while leaving those systems populated can produce misleading results.
• Wait for relevant background work. If recovery schedules an asynchronous job, define when the grader should inspect state. Use a bounded wait for a meaningful condition, such as the booking reaching a final status, instead of assuming that a successful HTTP response means all work is complete.
• Control external side effects. Evaluation messages should go to a test mailbox, and payment operations should use a suitable test environment. Keep ordinary development runs from sending real notifications or changing customer records.

For tools that execute arbitrary code, use an isolation mechanism appropriate to that execution. Inspect’s documentation also makes an important distinction: configuring a sandbox does not automatically move every part of your evaluator into it; the relevant work must actually use the sandbox interface. Inspect

Step 4: Record a Trace You Can Investigate

A final response provides limited evidence about a multistep task.

Your trace should make it possible to reconstruct the observable execution:

• Record tool names, arguments, results, and ordering. This reveals whether the agent checked the right order, interpreted the payment result correctly, and attempted recovery after receiving sufficient evidence. Include identifiers that connect related events.
• Capture errors where they occur. Distinguish an invalid tool argument from a provider timeout or application exception. If everything appears as “tool failed,” the evaluation report will not tell you which component needs attention.
• Save relevant state evidence. Before-and-after snapshots, audit events, and generated artifacts show what changed. Capture only the information needed for evaluation, with appropriate access controls and redaction.
• Preserve the execution configuration. Record the model identifier, available model revision, prompt version, harness commit, tool schema version, dataset version, and grader version. Without these, later comparisons become difficult to interpret.
• Track usage and elapsed time. Token counts, tool calls, retries, and duration help explain why a task became expensive or slow. Keep these measurements alongside correctness so that efficiency improvements do not hide quality losses.

You do not need access to private model reasoning to build useful traces. Observable messages, tool activity, execution events, and state changes provide the evidence for many important checks.

LangChain’s evaluation guidance similarly distinguishes assessment of final responses, trajectories, and state changes. These perspectives answer different questions about the same run. langchain.com

Step 5: Use Separate Graders for Separate Questions

A single instruction such as “rate this agent from one to ten” is difficult to use for debugging.

For the booking workflow, build several focused graders.

An outcome grader checks the database. It verifies that the expected booking exists, belongs to the right customer, and has the correct status.

A behavior grader examines observable actions. It checks for prohibited attempts, required authorization boundaries, or excessive retries.

A communication grader checks whether the final response matches the available evidence and gives the customer useful next steps.

An efficiency grader measures the resources used to complete the task.

Inspect supports multiple approaches to scoring, including text checks, model-based assessment, and custom rubrics. The useful design choice is to match the method to the question being evaluated. Inspect

Use code for facts that the system can verify directly. For example, counting booking records is more dependable than asking a language model whether a response sounds like a successful recovery.

Use a model-based judge when the question requires interpretation, such as whether an explanation communicates uncertainty clearly.

Step 6: Implement a Narrow Outcome Grader

The following TypeScript example checks one specific case: an eligible order that should end with exactly one confirmed booking.

The evidence must come from evaluator-controlled database inspection and instrumentation.

type Booking = {
  orderId: string;
  userId: string;
  status: string;
};

type Evidence = {
  snapshotsValid: boolean;
  traceComplete: boolean;
  bookings: Booking[];
  forbiddenToolAttempts: number;
};

type Expected = {
  orderId: string;
  userId: string;
};

function gradeBookingRecovery(
  evidence: Evidence,
  expected: Expected
) {
  if (!evidence.snapshotsValid || !evidence.traceComplete) {
    return {
      status: "eval_error",
      reason: "Required evaluation evidence is incomplete."
    };
  }

  const matching = evidence.bookings.filter(
    booking => booking.orderId === expected.orderId
  );

  const checks = {
    exactlyOneBooking: matching.length === 1,

    correctOwner:
      matching.length === 1 &&
      matching[0].userId === expected.userId,

    confirmedBooking:
      matching.length === 1 &&
      matching[0].status === "confirmed",

    noForbiddenAttempts:
      evidence.forbiddenToolAttempts === 0
  };

  return {
    status: "graded",
    passed: Object.values(checks).every(Boolean),
    checks
  };
}

This grader has a deliberately limited scope. It does not evaluate response quality, unrelated database changes, or the correctness of the payment service.

Add those as separate checks.

Also test the grader itself using known outcomes: no booking, a duplicate, a booking owned by another user, an incorrect status, and a valid result.

A grader that never rejects a deliberately broken outcome is not providing useful protection.

Step 7: Calibrate Your Model-Based Judges

A language model judge is another component that can make mistakes.

For the communication grader, give it the customer request, relevant verified facts, final response, and a specific rubric. Ask for a structured verdict with a short explanation tied to evidence.

Useful criteria could include:

• Completion claims must match observed results. If the recovery operation failed and no booking exists, “Your interview is ready” should fail. The judge should identify the unsupported statement instead of rewarding confident wording.
• Uncertainty must remain visible. If payment verification timed out, the response should describe the status as unconfirmed. A response that quietly converts “unknown” into “successful” should fail this criterion.
• Next steps must fit the situation. Asking the customer to pay again would be inappropriate when a captured payment has already been verified. Evaluate whether the guidance follows from the actual case.
• The response must protect unrelated information. The answer should not reveal details from another account, even if those details appeared somewhere in the trace. Evaluate information disclosure separately from whether the booking issue was resolved.

Have human reviewers label examples before relying on the judge for release decisions. Inspect disagreements, revise unclear criteria, and evaluate the revised judge on examples that were not used to tune it.

LangChain’s calibration guidance emphasizes this loop of human corrections, examples, and measured agreement. A well-written evaluator prompt alone does not establish reliability. langchain.com

Step 8: Allow Different Valid Paths

Your agent does not always need to follow one exact tool sequence.

Perhaps one implementation checks payment and booking status through separate tools. Another uses a combined recovery inspection tool. Both may reach the correct result while respecting the same boundaries.

Evaluate meaningful requirements:

• Require necessary conditions rather than arbitrary steps. The recovery service must validate eligibility before creating a booking. That validation may happen inside a trusted backend operation rather than through a separate agent-visible lookup.
• Use ordering checks when ordering affects correctness. Creating a booking before obtaining required approval is a meaningful failure. Reading two independent records in a different order may be perfectly acceptable.
• Track attempts separately from effects. A prohibited write that the backend blocks is different from a prohibited write that succeeds. Record both dimensions so that improvements in backend enforcement do not conceal poor agent behavior.
• Keep partial progress visible. An agent may identify the correct order and diagnose the problem but fail during recovery. Record that progress for debugging while preserving the fact that the end-to-end task failed.

LangChain’s readiness checklist recommends grading valid outcomes without unnecessarily requiring one predetermined path. langchain.com

Step 9: Repeat Trials and Report Uncertainty

One successful run does not establish dependable behavior.

Run important cases multiple times and preserve every attempt. Repetition helps reveal failures caused by variable tool choices, interpretations, and responses.

Be explicit about what your metric means:

• Per-attempt success rate measures successful attempts divided by valid graded attempts. It describes the observed trial results, but it should be accompanied by case counts and the composition of the dataset.
• Pass@k measures the chance of at least one successful attempt within a budget of k attempts. This is useful only when multiple attempts and a way to identify a successful result are part of the intended workflow.
• Pass^k measures the chance that all k attempts succeed. It focuses on consistency. Under an illustrative assumption of independent attempts with the same 90% success probability, three consecutive successes have probability \(0.9^3 = 72.9\%\).

Anthropic discusses the distinction between success within several attempts and consistency across attempts in its agent evaluation guidance. anthropic.com

Repeated attempts on the same cases do not create an equally large collection of independent tasks. When comparing versions, account for the shared cases and report uncertainty appropriately.

Inspect provides metrics for confidence intervals and clustered analysis. The exact statistical method should match your sampling design. Inspect

Step 10: Separate Agent Failures From Evaluation Failures

Suppose a run fails because the database fixture could not load. That does not establish that the agent mishandled a customer.

Use distinct result categories:

• Agent failure: The required environment and evidence were available, but the agent produced an incorrect result or unacceptable behavior. Examples include choosing the wrong order or claiming recovery after a failed operation.
• Expected dependency challenge: A tool timeout was deliberately included in the scenario. The agent’s response to that timeout is part of the test and should be graded.
• Evaluation infrastructure error: A worker crashes, a required snapshot is missing, or the environment cannot start. Preserve the error and avoid presenting the run as a valid measurement of task capability.
• Grader error: The evaluator crashes or relies on an invalid assumption. Mark the result as ungraded or invalid until the issue is resolved.

Inspect documents mechanisms for preserving results and retrying interrupted evaluations. Your own runner should also retain the original failure record rather than silently replacing it. Inspect

Infrastructure conditions also affect comparisons. Anthropic has reported that resource configuration can materially change results in agentic coding evaluations. Keep relevant resource budgets and execution limits consistent when comparing versions. anthropic.com

Turn Results Into Release Decisions

Define release rules before looking at a candidate’s scores.

For the booking agent, a practical policy could require no observed unauthorized modifications in the critical suite, no unexplained regression in recovery cases, acceptable communication quality, and sufficient valid results to support comparison.

Treat “no observed critical failures” as evidence from the suite, rather than proof that failures are impossible.

Run a small set of checks frequently, and schedule broader evaluations around meaningful changes:

• During ordinary development, run focused cases for the component being changed. A revised payment tool should trigger its argument, result interpretation, and recovery cases.
• Before a release, compare the baseline and candidate across the broader suite under equivalent conditions. Review newly failing cases individually instead of accepting an improved average automatically.
• After deployment, evaluate sampled production traces and investigate emerging failures. Avoid replaying state-changing operations against live customer records; reconstruct suitable offline cases instead.

LangSmith’s evaluation documentation distinguishes controlled offline evaluation from assessment of production behavior. Both contribute to an ongoing improvement process. Docs by LangChain

The most valuable result of a failed eval is a concrete engineering decision. A duplicate booking may call for stronger idempotency. An unsupported answer may call for clearer tool results. A failed authorization check may expose a backend defect.

Assign each important failure an owner, preserve a reproduction case, and verify the fix against related scenarios. That is how the evals layer becomes part of development rather than a dashboard that reports scores nobody acts on.

Frequently Asked Questions

1. What is the difference between an agent harness and an evaluation harness?

An agent harness operates the agent: it manages context, tool calls, execution, and stopping behavior. An evaluation harness prepares test scenarios, runs the agent harness, collects evidence, and grades the results. Keeping these responsibilities separate makes it easier to change your agent without rewriting all evaluation logic.

2. How many test cases should I create first?

Start with enough cases to cover the major decisions in one workflow. For the booking example, that includes successful recovery, pending payment, an existing booking, ambiguity, unavailable services, and unauthorized requests. Add variations when you discover different failure modes. A small initial suite supports development; broader reliability claims require broader evidence.

3. Should I use an LLM judge for every check?

No. Use direct checks for database state, record counts, permissions, schema validity, and other objective facts. Reserve model-based judges for questions that require interpretation, such as whether an explanation accurately communicates uncertainty. Validate those judges against human-reviewed examples before depending on their scores.

4. Can agent evaluations replace normal software tests?

No. Unit and integration tests still need to verify authorization, database constraints, idempotency, tool contracts, and error handling. Agent evaluations test how the model and harness use those components across complete scenarios. They provide additional behavioral evidence while relying on ordinary software controls to enforce important rules.

5. How often should I run the full evaluation suite?

Run it when changes could materially affect behavior, including model updates, prompt revisions, context changes, tool redesigns, and permission changes. Use smaller targeted suites for quick development feedback. Choose the broader schedule based on task risk, execution time, and cost, while continuing to inspect production behavior after release.