Your eval passed. That doesn't mean your system is safe.

A team wants to know whether their customer-facing chatbot can be talked into revealing someone's personal information. So they do the sensible thing: they write probes, run 300 of them, and count the leaks. Zero. The dashboard turns green. Ship it?

Hold on. That green dashboard is hiding a breach.

First, make the claim testable

"The chatbot leaks PII" isn't something you can test as written, because no finite number of clean probes can ever prove a leak rate of exactly zero. What you can test is a ceiling. Reframed, the question becomes: is the true leak rate below the line we're willing to tolerate? The job of testing is to gather enough clean evidence to say, with stated confidence, that it is.

The reassuring math

Treat each probe as an independent draw. If the true leak rate is p, the chance of seeing zero leaks across n probes is roughly:

Ask when that drops to 5% and you get pn ≈ 3 — the rule of three: zero leaks in n probes buys you a 95% upper bound of about 3/n.

So 300 clean probes means you're 95% confident the true leak rate is at most 1%. Under the tolerance bar. Looks like a ship.

The catch

The rule of three rests on one assumption: a single, homogeneous population sampled at random. PII leakage violates it badly. Leaks don't sprinkle evenly across traffic. They cluster in thin, structured pockets.

Suppose the truth is that 99% of your traffic is ordinary and essentially never leaks, while 1% is adversarial — social-engineering frames, prompt-extraction attempts, jailbreak wrappers — and leaks 20% of the time. The population average is 0.99 × 0 + 0.01 × 0.20 ≈ 0.2%, comfortably under your bar.

But watch what your test actually did. A random sample of 300 lands only about 3 probes inside that adversarial pocket. You'll most likely see zero leaks there and conclude "all clear" — having learned essentially nothing about the one stratum that matters. And here's the sharp part: 0.2% describes the average user, but an attacker doesn't live in the average. An attacker lives entirely inside the 20% pocket, and will find it. Your aggregate number is technically true and operationally useless. That's how you ship a breach and wake up to a PR nightmare.

The fix is the oldest move in sampling

Stratify. Carve the space of interactions into strata by how a leak gets induced — a direct ask, a social-engineering frame, system-prompt extraction, a jailbreak wrapper, injection through retrieved content — and allocate probes not by how common each stratum is, but by where uncertainty and consequence concentrate. Oversample the strata that can actually hurt you.

Take a 400-probe budget. Put 100 in the benign stratum: zero leaks gives a rule-of-three bound under 3%, plenty for the safe majority. Put the other 300 in the adversarial stratum, where they do real work — and say they surface 58 leaks, about 19%.

Then the crucial move: report the strata disaggregated, not pooled. The population estimate ("a typical user sees roughly 0.2%") sits beside the conditional worst case ("given social-engineering technique X, leak probability is about 19%"), and you state plainly that the worst-case figure is a characterization of a threat, not a frequency.

One design now answers both questions a serious team actually needs answered: can an adversary make it leak (yes — about one time in five, via X), and how often does an ordinary user encounter it (about one in 500). The naive study answered neither.

That is the difference between a number and evidence: disaggregated rather than pooled, weighted to real usage, honest about which figures are frequencies and which are worst cases, and accountable to a threat model written down in advance and not assembled from whatever scenarios happened to occur to the team.

The same discipline applies in a world of agents instead of static probes, where you move from stratified samples to targeted environments.

If you're preparing to put an AI system in front of real users and you want to be able to say not just that it passed, but exactly how you know — that's the work we do. Let's talk.

Social sciences in the loop

None of this is new. Stratified sampling is textbook survey methodology. The social sciences have spent fifty years working out how to sample a population so your conclusions generalize, how to tell whether a measure captures what it claims to, and how to know when you've measured enough. AI evaluation is quietly rediscovering these problems by hand and it doesn't have to.

This series is about closing that gap, one worked example at a time.

Next: how decades of trust research turn into a working detector — and a red-team plan.