Closing the Loop, Part 2: Evaluating Agent Failures and Improving the Harness

Closing the loop for agent improvement

Customer-support agents can create value across retail, airline, telecom, and other domains by answering policy questions, retrieving account context, updating records, routing requests, and executing customer-facing workflows. But reliability is still the deployment bottleneck: failures can update the wrong record, skip a required confirmation, violate policy, or promise an action that never executes.

Agent builders can edit prompts, add guardrails, collect data, or fine-tune the model. Those interventions can help, but they are often ad hoc when they start only from the fact that an agent failed, not from a diagnosis of why it failed or which part of the system should change.

Closing the loop makes improvement systematic. We run the agent on realistic tasks, identify failures, improve the agent based on those failures, and re-run the same eval to check whether the failure moved.

Diagram of the closed loop: a user simulator plays the human turns, the harness wraps the model, the environment supplies tools, and the evaluator's findings flow back into the agent. — **Figure 1.** Closing the loop: run the agent, evaluate the trajectory, diagnose failures, improve the system, and re-evaluate.

Two things make the loop hard. First, most graders return only pass or fail, but not a repair signal. Second, even with a diagnosis, it is not obvious which system change should follow. Our previous blog post used failure analysis to generate targeted data and train a better model; here, we focus on a more cost-effective way of improving agent performance by improving the harness around a fixed model.

We address both challenges with an evaluator that identifies the first unrecovered mistake, failure type, trajectory evidence, and expected behavior, then use those diagnoses to update prompt memory and reviewer checks.

Two branches out of the same eval step: train the model on synthetic data, or identify and tune the harness components behind each failure. Both branches feed a re-eval. — **Figure 2.** The previous project used the loop to train the model. Here, we use the same loop to improve the harness around a fixed model.

The evaluator: where, why, proof

The evaluator takes a failed trajectory and returns a structured diagnosis. It answers three questions a customer or agent builder actually needs:

Where did the agent first fail?
Why did that failure happen?
What proof in the trajectory supports the diagnosis?

That moves evaluation from reporting to repair. The evaluator output becomes an engineering artifact that can drive the next harness change.

Pipeline diagram: a failed trajectory passes through grounded checks, whose evidence feeds a taxonomy-guided LLM diagnosis, producing a diagnosis output of where, why, and proof. — **Figure 3.** The evaluator combines grounded checks with taxonomy-guided LLM diagnosis to produce where, why, and proof.

The evaluator uses 51 grounded checks, split into two types. Deterministic checks handle what a program can verify: final state, required actions, tool arguments, legal transitions, and spending limits. LLM-judge checks are reserved for semantic questions, such as whether the customer clearly confirmed an action or whether the agent's explanation matched the tool output.

The LLM diagnosis then maps the failure into a maintained taxonomy, separating agent failures such as comprehension, planning, and tool execution from task or system gaps. This lets us aggregate failures across runs instead of treating every failed trajectory as a one-off example.

The final output records the following components:

Expected behavior: describes what the agent should have done in that specific situation.
Actual behavior: summarizes what the agent actually did, which led to the failure.
Supporting evidence: provides specific details or excerpts from the trajectory that substantiate the diagnosis.
Confidence: indicates the level of certainty the evaluator has in its diagnosis.

This is the repair signal that tells us what kind of fix the system should consider.

A concrete failure example: misreading tool output

Here is one failure in the shortest useful form. The customer asked the agent to count the available T-shirt options before changing two pending T-shirt orders. The agent had the product data, but counted the available variants incorrectly.

Side-by-side view: the trajectory snippet shows the customer request, the get_product_details tool output supporting 10 available variants, and the agent answering 12; the evaluator diagnosis flags a tool-output computation error with the first wrong step and expected handling. — **Figure 4.** A compact trajectory view: the left side shows the customer request, relevant tool output, and wrong answer; the right side shows the evaluator's failure flag and repair signal.

Evaluator benchmark result

We benchmarked the evaluator on the 29 public AgentRx tau-bench failed trajectories. Our evaluator is competitive on failure type and stronger on critical-step localization.

Grouped bar chart comparing AgentRx and our evaluator on failure type (40.2% vs 41%), exact critical step (54% vs 62%), and critical step within plus or minus one (59.8% vs 79%). — **Figure 5.** On the AgentRx benchmark, our evaluator matches failure-type attribution and improves critical-step localization.

From diagnosis to reflection

The evaluator gives us the raw repair signal. To make that signal reusable, we convert evaluator analyses into reflections: short lessons that describe the failure pattern, the trigger, and the expected handling. A reflection is not just "the agent failed." It is a compact instruction such as: when counting product variants, filter by the availability field before answering.

We use those reflections in two harness methods. The first method checks risky writes at runtime. The second method puts lessons into the agent prompt before the task starts.

Four-step reflection loop: run eval on the agent's trajectories, reflect on the failed checks, carry the reflection forward as a memory file, then re-run the same eval suite to verify the failure mode is reduced. — **Figure 6.** Prompt Memory turns failed trajectories into reusable lessons in the agent prompt.

Method 1: Pre-Write Reviewer

Diagram: past failed trajectories become evaluator analyses, which form a reflection bank grouped by tool; when the agent proposes a write, the reviewer retrieves relevant reflections and either allows the call or blocks it with feedback for a retry. — **Figure 7.** Pre-Write Reviewer retrieves lessons from evaluator analysis when a write tool call is proposed.

The reviewer is a runtime guard for state-changing actions. A basic reviewer checks the proposed write with the task context and a generic review prompt. A reflection-guided reviewer also injects lessons from prior evaluator analyses into the reviewer prompt, so it can catch recurring failure patterns before the tool executes.

On GPT-5.4 mini, the basic reviewer improves airline pass³ from 11.3% to 30.0%. The reflection-guided reviewer improves all three domains: retail from 26.9% to 29.4%, airline from 11.3% to 23.7%, and telecom from 5.0% to 14.4%.

Grouped bar chart of pass^3 by domain for baseline, basic reviewer, and reflection-guided reviewer across retail, airline, and telecom. — **Figure 8.** On GPT-5.4 mini, the Pre-Write Reviewer improves pass³ across all three domains.

Method 2: Prompt Memory

Prompt Memory uses the same reflection source, but applies it before the task starts. Instead of checking one write call at a time, we distill evaluator analyses into lessons and place those lessons in the agent's system prompt. This is useful when the failure is a recurring behavior, such as misreading a tool field, skipping a confirmation, or applying the wrong policy branch.

On Qwen3.5-9B retail, Prompt Memory was the stronger harness fix. Reviewer-style gating did not transfer as well, but carrying lessons in the agent prompt lifted pass³ from 65.0% to 69.2%.

Line chart of pass^K for K = 1, 2, 3 on Qwen3.5-9B retail: Prompt Memory above baseline at every K, with Pre-Write Reviewer below baseline. — **Figure 9.** On Qwen3.5-9B retail, Prompt Memory outperforms both baseline and reviewer across pass^k.

The evaluator also tells us what changed underneath the score. In the Qwen3.5-9B retail comparison, comprehension failures dropped sharply after Prompt Memory, while the remaining failures concentrated more in planning. That is still useful: the next loop now has a clearer target.

Horizontal bar chart of failure counts by category for baseline and reflection: comprehension halves from 14 to 7 while planning rises from 4 to 7. — **Figure 10.** Failure distribution comparison. Prompt Memory reduces comprehension failures, revealing planning as the next major target.

The same task after harness improvement

The same T-shirt task now succeeds after Prompt Memory adds the reflection to the agent prompt. The reflection tells the agent to filter the raw tool output by the requested field before counting; based on that lesson, the agent reports the exact count, asks for confirmation, and completes both pending order updates.

Side-by-side view: the successful trajectory snippet shows the agent filtering by availability, reporting 10 options and asking for confirmation; the reflection panel shows the lesson carried in prompt memory. — **Figure 11.** The reflection tells the agent how to count filtered tool outputs; with that lesson in prompt memory, the agent reports 10 available options and completes the confirmed order updates.

What this means for customers

The business value is not just a higher benchmark number. It is a faster path from a production failure to a targeted fix.

Support teams get root cause instead of a mystery score. They can see the exact turn, evidence, and expected behavior.
Engineering teams can improve without immediate retraining. Prompt Memory and Pre-Write Reviewer are harness changes that can be deployed around a fixed model.
Risky actions get special treatment. State-changing writes can be reviewed before they touch a database or payment workflow.
The next iteration is measurable. After the fix, the same evaluator shows whether the score and the failure distribution moved.

Closing the loop

The principle is the same as the previous training work: assess first, then fix. The difference is where the fix lands. Training uses failure diagnoses to change the model. Harness improvement uses the same diagnoses to change the system around the model.

The takeaway is to find where the agent actually fails first. Turn that failure into a diagnosis. Use the diagnosis to make a targeted harness change. Then re-run the same eval and check whether the gap moved.

If you are putting agents into production, that is the loop worth running: every failed trajectory becomes a repair signal, every repair is re-tested against ground truth, and the next failure mode becomes clearer.

If you want to know where your agents are quietly failing and which harness change would close the gap, let's talk.