Closing the Loop: Finding and Fixing a Customer-Service Agent's Failure Modes

Customer-service agents are moving into production, where every action mutates a live order database and moves real money. "Sounds right" isn't the bar; being right is. We took a small baseline agent, assessed exactly where it failed, generated verifiable data aimed at those failures, trained on it, and re-measured against the same ground truth. The loop holds because every task is graded by deterministic logic, never authored by a model. The reward can't be gamed; the only way to earn it is to solve the task.

Two data pipelines supply the tasks, depending on what's available: (1) synthesize them from a grounded retail world when there are no successful trajectories to learn from, or (2) curate and verify existing ones when there are. Both feed a short post-training run.

Fabrication and tool-misreads largely collapse. Plan adherence is the honest residual a deterministic eval refuses to hide.

Assess first, then fix

A retail support agent acts for one logged-in customer, and its actions are real: it edits the order database, issues refunds, changes payment methods. When something goes wrong, the cost is concrete: a refund that shouldn't have gone out, a payment moved to the wrong card. A good agent should be clear and pleasant to deal with. On top of that, what we measure is whether it completes the task correctly, and does so reliably rather than in one lucky run.

The approach is simple: you can't fix a failure you haven't pinned down. We ran the baseline retail agent in an environment that matches the stakes of production, studied where and how it broke, built training data targeting those failures, and retrained to close them. Every task is scored against the real state of the world at the end, so a convincing but wrong answer earns nothing.

Gap analysis

We started with a baseline model pointed at our retail environment. Before doing anything else, we ran it through a set of tasks to understand where it fails before trying to fix it. Three failure modes showed up consistently:

• Plan-adherence failure. The agent agrees to a plan, then doesn't follow through: authentication skipped, a write never issued, a confirmed action that never touched the database.
• Invention / hallucination. The agent introduces facts that came from nowhere (a fabricated order total, a made-up tracking number) and then acts on them. This is the dangerous one: the invented value becomes the argument to a write action.
• Tool-output misinterpretation. The agent misreads what the tools returned and acts on a wrong state. A delivered order is treated as pending, then a cancellation is attempted on it, an illegal transition that traces back to a single misread.

These three failure modes became our target and gave clarity on what data we needed to curate.

What gets checked, and who plays the customer

Grading runs in two layers, split by what's deterministically checkable. Static code checks the boring, verifiable stuff (legal transitions, dollar ceilings, tool-call counts, the final database state) and per-task checks pin the exact outcome (which order was cancelled, which item swapped, what the new address is). A language model judges only the genuinely soft half: was the customer's "yes" actually clear, did the agent read back the right order, was a refusal grounded in real policy. Anything a computer can verify is never handed to a model to judge.

The customer is a strong model working from a script, but it talks like a person and reacts in real time. We kept it plain on purpose: personas are simple (terse, chatty, formal, anxious) and chosen the same way every time, and intent stays honest. Real requests, no jailbreaks, no trick questions. The difficulty comes from normal messes, not attacks: a vague item, a forgotten email, a mid-conversation "oh, can you also." A hard customer, not a hostile one.

Two experiments, one principle

We ran two experiments to close the gaps, each designed for a different starting point: one for when you have no existing data to learn from, one for when you do. Both are built on the same idea: the correct answer is always computed from the world state, never generated by a model.

Experiment 1: Synthetic data generation

Now that we know what the model gets wrong, we can generate data that specifically targets those gaps. The generator maintains a taxonomy of templates that guarantees coverage. Given a failure mode and a template, it generates new tasks focused on that failure mode, with adjustable difficulty parameters. The key design decision: the gold output is still verifiable.

Every generated task goes through a cascade of LLM judges that check executability, validity, and policy adherence. This matters because it means the reward can't be gamed. The only way to earn it is to actually solve the task. We then trained the model on these synthetic tasks, and pass¹ went from 42.5% to 75%.

Experiment 2: Data curation

The second experiment starts from a different place: what if you already have trajectories? We had roughly 6,000 successful trajectories. The question was which ones are actually worth training on. If the model already solves a task easily, training on it is just rehearsal. If the task is too far outside what the model can do, it doesn't learn well from it either. The signal lives in between: the tasks the model sometimes gets right and sometimes gets wrong.

So we curated a mix: tasks the model succeeds on (to anchor existing skills) and tasks it fails on (to target the gaps). Getting that ratio right is the lever. We landed on roughly 1,200 carefully selected samples. As a yardstick, we also trained on the full 6,000-trajectory set; the curated 1,200 matched it, reaching the same benchmark performance from one-fifth the data. Beyond the targeted subset, extra volume is mostly rehearsal. Volume isn't the bottleneck. Targeting is.

Results

The pass¹ lift is the headline: an average attempt now succeeds about three times in four, up from fewer than one in two. The pass⁴ lift is the point. pass⁴ counts only the tasks the agent solves on all four independent attempts, and it nearly doubled, from 27.5% to 52.5%. So the agent didn't just get luckier on average; it got steadier. That distinction is the whole game in production: a live agent serves one customer, once, and has to get that interaction right. It doesn't get to be correct "on average." After training, more than half the benchmark is solved every single time.

Where the gaps close

The two grounding failures collapse. Invention, where the agent makes up a value and feeds it to a write, fell from 17.5% of the benchmark to 5%. That's the mode we most wanted gone: a fabricated order total or payment id isn't just a wrong sentence, it becomes the argument to an action that moves money. Training pushed the agent to act on what the tools actually returned, and tool-output misreads more than halved for the same reason.

Plan adherence is the stubborn one. Invention and misreads are perception problems; the fix is getting the model to attend to what's already in front of it. Plan adherence is a follow-through problem: the agent has to hold a commitment across many turns (confirm the change, then issue the write) without getting pulled into the customer's next request. That's harder to instill by imitation, and it's the failure a deterministic eval refuses to hide. The transcript reads fine and the agent "said" it would update the address, but the database never changed. A judge reading the words would pass it; a check on the final state will not. So it's our largest remaining gap, and the mode we generate against next.

Before and after: two failures, fixed

Aggregate numbers persuade managers. Trajectory-level diffs persuade people who build evals. Here are two, lifted from the benchmark, each showing the exact tool call that flips the outcome.

Closing the loop

Start to finish, this is one loop. The eval named the three failure modes. We generated data aimed at exactly those modes, with gold no model authored. Training on that data moved the numbers, and the same eval confirms the close. Because augmentation conditioned each task on the measured failures, a small targeted set beat raw volume: a few hundred synthetic trajectories, or replay-verified curated tasks, took pass¹ from 42.5% to 75%. And the gains landed where we aimed: invention dropped from 17.5% of the benchmark to 5%, and tool-output misreads more than halved. The modes the data targeted are the modes that closed, which is the signature of a loop that is actually closing rather than a model that got luckier.

The takeaway: next time you're improving an agent, don't start by adding data or training longer. Find where it actually fails first, then generate data targeted at those gaps, graded against ground truth no model can author. Measure, target, train, re-measure. That's the loop we run at Reinforce Labs, and plan-adherence is where we point it next.

If you're putting agents into production and want to know where yours is quietly failing, let's talk.