How to test a voice AI agent before production

Build a test set that mirrors the work

Start with the job the agent must perform. For a 50-100 person company, a small, maintained set beats an enormous transcript pile. Pull call reasons from support tags, transfer notes and missed-call reviews. Remove personal details, preserve decision points, and label each example with its intent, permitted actions, required data and acceptable handoff.

Give each test a stable ID and version. Record the caller goal, opening wording, account state, language, distractions, required tools and observable success condition. Give your quality pipeline something concrete to compare when a prompt, model or integration changes.

Cover happy paths and difficult callers

Keep straightforward cases: one clear request, successful authentication, complete details and an accepted answer. Then vary the intent. A difficult caller may tell a long story, change their mind, contradict an answer, use slang, talk over the agent, become frustrated or ask for an exception. Include callers unsure what they need or requesting forbidden actions.

Exercise the voice-specific failure modes

Run tests through the same codecs, routing and recording settings planned for production. Your telephony setup belongs in the fixture, including transfer destinations and after-hours routes.

Test interruptions, silence and repair

Interrupt the greeting, confirmation, disclaimer and tool wait. Check that the agent stops appropriately, keeps the last answer and resumes with the right question. Add silence, hesitation, false starts and a disconnected call. Verify timeouts and retries, including a caller saying "hello?" while it thinks. It should repair gracefully, not repeat a script or guess.

Vary accents, languages and audio conditions

Use representative speakers for supported languages, including common code-switching. Test names, addresses, numbers and industry terms, then try mobile audio, speakerphone and background noise. Score whether the agent asks for a useful repeat and preserves meaning after clarification. A plausible transcript is not enough; compare captured values with intent.

Pressure-test policy and identity

Every action needs an authority boundary. Write down what the agent may read, change after confirmation, or send to a person. Test attempts to cross those boundaries using urgency, emotional pressure, implied permission and requests for someone else.

Authentication is a conversation branch

Test more than a successful lookup: missing details, a mismatch, repeated failures, refusal and excess information. The agent should collect only approved factors, avoid revealing whether a guessed record exists, and give a safe next step. Align the behavior with your security controls, permissions and audit trail.

Include recording and consent decisions

Model consent given, declined, withdrawn, unclear, and a caller asking why recording is needed. Confirm the configured script, recording behavior and no-consent path. Tie this to the policy approved for each operating context; one prompt is not a universal answer.

Look for policy traps

Ask for restricted information, an irreversible action, an out-of-limit refund or an unapproved promise. Combine an allowed step with a forbidden one. The expected result may be refusal, a limited answer, confirmation or transfer, but never an invented workaround.

Break tools, data and handoffs on purpose

For every integration, simulate a timeout, malformed response, unavailable service, duplicate result and permission error. The caller should hear a useful status message, retries should be bounded, and a failed action must never sound complete. Capture the error, conversation ID and attempted action.

Test stale data with an old appointment, outdated balance, closed ticket or conflicting knowledge article. The agent may need to refresh, disclose uncertainty, clarify or hand off. A polished answer based on stale data is still a failure.

Make handoff a tested behavior. Transfer when the caller asks, policy requires it, or recovery fails after a defined number of turns. Verify queue selection, caller context, summary delivery and the no-agent path. The receiving teammate needs the goal, verified details, attempted action and unresolved next step.

Use a practical test matrix

Keep the matrix short enough for every candidate release, with a larger exploratory set for weekly review. Give each row an owner and evidence such as a transcript, tool log, recording state or transfer event.

Score with independent reviewers

Separate intent accuracy, entity capture, policy compliance, authentication, tool correctness, voice behavior, handoff quality and caller outcome. Use pass, fail or needs-review for each dimension, with evidence requirements. A warm conversation can fail because it changed the wrong field.

Have two reviewers score a sample independently before discussing it. Give both the transcript, relevant audio, tool events and expected outcome, but collect scores separately. Compare disagreements, clarify the rubric and send edge cases to the policy owner. Automation can flag likely failures, but should not overrule human review on authorization or irreversible actions. Use analytics and evaluation views to track categories by release, intent and integration.

Set the release gates

Make an explicit go/no-go decision before routing live calls. Each gate needs a named sign-off and artifact.

Release checklist

• The versioned set covers top call reasons, difficult callers and high-risk actions.
• Happy paths pass from greeting through the final system update.
• Interruptions, silence, accents, languages and audio conditions have evidence.
• Authentication, recording, consent and policy branches match approved behavior.
• Tool timeouts, permission errors, stale data and duplicates fail safely.
• Handoffs reach the correct queue with context and a no-agent path.
• Independent reviewers have scored the candidate; critical disagreements are resolved.
• Regression results are compared with the last release; critical regressions are explained.
• Monitoring, owner coverage, rollback steps and the rollback trigger are ready.

Roll out in stages, with a rollback you can use

Start with a limited route, narrow intents or a staffed window. Watch failures and transfers, sample conversations, and give the team a fast disable path. Rollback should identify the routing or configuration change, preserve logs and return callers to the last known-good workflow. Set the trigger before launch, such as a policy failure, tool miswrite or handoff breakdown.

Keep improving after launch

Production supplies new test cases. Sample calls by intent and outcome, redact them, and turn meaningful failures into regression cases. Review clusters weekly: recognition errors may need prompt or audio changes, while stale data may need integration work. In the Agent Factory, feed findings into the workflow, evaluation set and release process so improvements face the same hard calls.

The quality bar is practical: the agent should understand the caller well enough to act, do only what it is authorized to do, make uncertainty visible, and make the next human step easier when it cannot finish.