Test an Email Agent Without Letting It Reach Customers

Consider one support thread about a missing package. The customer says the carrier marked it delivered, the address on the order is correct, and the parcel is not at the building desk. A human agent previously checked the order, opened a carrier lookup, drafted a response, and requested approval for a replacement.

An email agent replaying that thread appears competent at first. It routes the message to delivery support, finds the right order, and writes a clear reply. At step four, however, it calls the replacement-order tool before approval. The sandbox blocks the call, but the attempt is the important result: a pleasant draft concealed a policy failure.

That missing-package thread is a better evaluation unit than a prompt asking the model to answer an isolated email. Email work depends on prior messages, customer records, tool results, permissions, and state changes. A useful test has to recreate those conditions while making real-world effects impossible.

Build the sandbox around completed threads

Start with historical threads that represent work the agent may eventually handle. Each test case should contain the messages available at the moment the agent would take over, along with the relevant account state and tool outputs from that time. Remove later messages from the input. Otherwise, the test leaks the answer by showing the agent how the case ended.

A sandbox is an isolated environment where the agent can read realistic data and attempt actions without touching production systems. Copy only the fields needed for the workflow, replace customer names and contact details, and remove free-text material that does not affect the decision. Redaction needs inspection, since signatures, forwarded messages, attachments, tracking links, and internal notes can preserve personal information after the obvious address fields are gone.

Do not connect the sandbox to a real mailbox with a rule that says “do not send.” Give it a fake mail server, sometimes called an SMTP sink, that accepts outgoing messages and stores them for review without delivering them. Block general network access as well. An agent that cannot use the approved mail function may try a browser, a customer-relationship platform, or another integration that can still contact someone.

Production credentials do not belong in this environment. Create test identities with narrow permissions, use nonproduction service accounts, and deny any route to live order, billing, calendar, or messaging systems. Network isolation is the last control, not the first one; the agent should encounter explicit permission failures that also appear in its trace, the ordered record of its model decisions and tool calls.

For the missing-package case, the sandbox might contain a fictionalized order record, a stored carrier response showing “delivered,” and a replacement tool that cannot create anything. That tool is a stub, a test substitute that validates the requested action and returns a simulated result. It should record the proposed order identifier, reason code, shipping address, and approval token, including whether any required value was absent.

Replay the decision point, not the final answer

Feed the agent the thread exactly where work remained unfinished. Let it classify the message, retrieve context, choose tools, and produce either a draft or a proposed action. The harness, which is the code that runs and records the test, should capture the route selected, every tool request and response, the final text, permission denials, retries, and the stopping condition.

Historical handling is evidence, not an unquestionable answer key. The original employee may have chosen an acceptable route rather than the only acceptable one, written an awkward reply, or worked under a policy that has since changed. Record a set of allowed outcomes where several paths are valid, and tie policy-dependent expectations to the policy version used for the test.

Replay cases more than once when the model can vary its output. A single pass may miss an intermittent tool call or a draft that changes meaning after a different retrieval result. Repeated runs consume model tokens and add evaluation time, especially when long threads include attachments, but they reveal unstable behavior that a polished demonstration tends to hide.

Keep the initial set narrow enough to inspect. Cases should include ordinary work as well as boundary conditions such as ambiguous identity, missing order data, conflicting instructions inside a forwarded email, and a request that requires approval. The goal is not to reproduce the entire inbox before learning anything. It is to cover the decisions that can create customer harm or irreversible work.

Keep four scores instead of one

A single “good response” score collapses distinct failures. The missing-package agent could receive high marks for its prose even though it selected a forbidden action. Maintain separate results for routing, draft quality, tool selection, and unauthorized actions, then set deployment gates for each category rather than averaging them into one reassuring number.

Routing asks whether the agent sent the case to an allowed queue or workflow. Compare its choice with the accepted destinations for that thread, and distinguish a nearby but inefficient route from one that bypasses a specialist or required review. If delivery support and general order support can both take the case, the rubric should accept both instead of pretending the historical label is perfect.

Draft quality should emphasize factual and operational requirements before style. Check whether the draft uses facts available in the thread, avoids unsupported promises, includes required instructions, and matches the action state. In the anchor case, a message saying “your replacement has been ordered” is wrong if the tool only requested approval, no matter how courteous the wording sounds.

Some draft checks can run as deterministic rules, meaning the same input always produces the same result. A rule can detect whether the draft contains a real email address, claims an unperformed refund, or omits a required disclosure. Human reviewers are still useful for ambiguity and tone, but they should receive a rubric and review the draft without seeing which model produced it. Free-form preference alone is too noisy for a release gate.

Tool selection measures whether the agent chose an allowed tool for the task, supplied arguments supported by the record, and used the result correctly. Score an unnecessary carrier lookup differently from a replacement call against the wrong order. Also preserve the sequence: the same calls can be acceptable after identity verification and unacceptable before it.

Unauthorized action is a safety result, not a subset of tool quality. Count attempted sends, purchases, refunds, account changes, data exports, and approval bypasses even when the sandbox blocks them. Record the requested scope and whether the agent retried through another path. A denied replacement call followed by an attempted customer email is more serious than a single denied lookup, yet both need to remain visible rather than disappearing inside an average.

Turn the failed fourth step into a regression test

The missing-package run should leave a compact artifact: input thread, policy version, retrieved records, chosen route, draft, complete tool trace, and category results. Preserve the attempted replacement call with its missing approval token. After changing the prompt, model, tool descriptions, or orchestration code, rerun that same case and compare behavior at step four.

Do not repair the test by adding the exact thread to the system prompt or writing a special rule for its wording. That teaches the agent one example. Repair the control boundary instead: require the replacement stub to validate an approval token, limit the agent identity to proposal-only permission, and make the orchestrator hand the request to a person when approval is absent.

This distinction matters because prompts describe policy while permissions enforce it. A prompt can reduce bad attempts, which improves workflow quality and reviewer load, but it cannot make an authorized production credential harmless. The test environment should show both whether the agent follows the instruction and whether the surrounding system contains it when it does not.

Set gates that match the consequence

Before deployment, decide which results block release. Any attempted high-impact action outside permission may be an automatic failure, while minor routing errors may trigger more testing or a narrower launch. Draft quality can tolerate stylistic variation. Claims about completed refunds, cancellations, replacements, or account changes need a stricter rule because customers may act on them.

Then run the agent in shadow mode, where it processes current work but cannot send messages or execute production tools. Compare its proposed routes and drafts with live human handling, while keeping the same outbound blocks used in the sandbox. Shadow mode catches changes in customer language and current systems that historical data misses, though it costs inference money and creates logs that need retention and access controls.

The first live permission should remain narrower than the tested capability. An agent may route low-risk messages automatically while saving drafts for review, or it may retrieve order status without receiving replacement authority. Expand one permission at a time and keep the missing-package regression in the release suite. If a model or workflow update brings back the step-four call, the gate should stop deployment before a customer becomes the test case.

Questions people ask

Can historical email threads be used without exposing customer data?

They can be minimized and redacted, but removing names and email addresses is not enough. Inspect attachments, signatures, quoted replies, tracking links, internal notes, and unusual order details. Restrict access to the evaluation set, document retention, and use synthetic records where the original content is not needed for the decision.

Should the sandbox use the agent’s real tools?

It should preserve the real tool schemas and permission checks, but action tools should point to test systems or stubs. Read-only production access still creates exposure and can make results hard to reproduce. A faithful stub records arguments, simulates expected errors, and never sends, refunds, purchases, or changes a customer account.

How many historical threads are enough before deployment?

There is no universal count. Coverage matters more than a large undifferentiated sample: include common routes, policy boundaries, missing data, adversarial content, and every action with a meaningful consequence. Add cases whenever production review reveals a new failure pattern, and rerun the full set after changes to models, prompts, tools, or permissions.

Can a good draft compensate for a wrong tool call?

No. Draft quality and action safety measure different behavior, so averaging them can hide a blocked refund, replacement, or send attempt. Keep the scores separate and make unauthorized high-impact actions release blockers. The missing-package reply can read well while its step-four replacement call still disqualifies the run.