Building an Automated Voice AI Testing Framework for Welcomr.ai

How we built a scenario-driven test runner in Go that replays pre-recorded human speech into live agent endpoints and validates entire conversation flows, turn by turn

The Problem

Welcomr.ai is a multi-tenant Agentic Voice AI platform for hospitality — guests call or connect via web widget and speak to an AI concierge that handles bookings, room service, and inquiries in real-time. The system supports two voice channels: WebRTC (browser-based, Opus codec at 48kHz) and Twilio (phone calls, mulaw codec at 8kHz).

Testing this manually meant someone had to actually call the agent, speak scripted lines, and verify responses — every time we pushed a change. That doesn't scale. We needed automated end-to-end conversation testing that exercises the full pipeline: audio encoding → network transport → speech-to-text → LLM reasoning → tool execution → text-to-speech → audio playback.

The Approach

We built a scenario-driven test runner in Go that replays pre-recorded human speech into live agent endpoints and validates the entire conversation flow, turn by turn.

1. Scenario Design

Each test scenario is a folder containing:

• scenario.yaml — metadata (name, channels, expected tools, turn count)
• Per-turn .meta.yaml — the "script" with timing, barge-in flags, and assertions (expected STT keywords, expected tool calls, expected response content)
• Pre-recorded WAV files — real human speech at two sample rates (48kHz for WebRTC, 8kHz for Twilio)
• encode.sh — ffmpeg pipeline that converts source recordings to both formats with EBU R128 loudness normalization

Our first scenario: a 6-turn hotel room booking in English — check availability, ask for the most expensive room (with barge-in interruption), confirm, provide personal details across two turns, and say goodbye.

2. The Test Runner Architecture

Orchestrator
  ├── Loads scenario + turn metadata
  ├── Creates channel-specific client (WebRTC or Twilio)
  ├── For each turn:
  │     ├── Waits for configured delay
  │     ├── Streams pre-recorded audio (Opus RTP or base64 mulaw)
  │     ├── Waits for agent response completion
  │     └── Captures STT transcript, agent response, events
  └── Runs assertions against captured data

The key insight: both clients implement the same ConversationClient interface, so the orchestrator is channel-agnostic. The protocol differences are encapsulated:

• WebRTC client: SDP offer/answer exchange, Opus encoding, RTP packet streaming, data channel event parsing (transcript, agent_response, tts_playback, interrupted, session_ended)
• Twilio client: WebSocket connection mimicking Twilio Media Streams protocol (connected → start → media events with base64 mulaw chunks → mark events for TTS completion)

3. Hard Problems We Solved

VAD end-of-speech detection

The agent's Voice Activity Detector needs silence after speech to know the user stopped talking. A recording file just... ends. No silence follows. The VAD never fires speech_ended. Fix: append 1.5–2 seconds of encoded silence frames after each recording.

Barge-in (interruption) testing

Turn 2 interrupts the agent mid-sentence. The orchestrator uses WaitForAgentStart to detect the agent has begun speaking (via tts_playback started on WebRTC, first outbound media event on Twilio), then immediately streams the next turn's audio. The client handles the cascade of interrupted/completed events without leaking stale results into subsequent turns.

Multi-sentence response detection

The agent's response to "what rooms are available?" can be 5 sentences long. On WebRTC, we track tts_started.total_sentences and count tts_playback completed events. On Twilio, we use a settling-timeout approach — wait for mark events, and when none arrive for 10 seconds, consider the response complete.

Audio loudness

Raw recordings from a phone were too quiet (−34 dB) for the STT energy threshold (400k). We added EBU R128 loudness normalization via ffmpeg's loudnorm filter, boosting to −16 dB mean without clipping.

Recording at the right fidelity

The agent's STT model (gpt-4o-transcribe) works with resampled audio internally, but the transport must match the channel exactly — Opus at 48kHz for WebRTC (the browser codec), mulaw at 8kHz for Twilio (telco standard). Same human speech, two encodings, validated against the same assertions.

4. What We Test

Per turn:

• STT accuracy: Does the transcript contain expected keywords? ("availability", "April 1", "April 10")
• Tool execution: Did the agent call the right tools? (check_availability, create_reservation)
• Response content: Does the agent's reply contain expected information? ("confirmation", room details)
• No errors or timeouts: Did the full turn complete within bounds?

Across the scenario:

• End-to-end booking flow works (availability → selection → personal details → payment → confirmation → goodbye)
• Barge-in interruption is handled correctly
• The agent calls end_call on farewell

5. Results

The framework runs as a standard go test with flags:

go test ./tests/conversations/... -args -channel=both -target=local

Our hotel booking scenario: 6/6 turns passing on both WebRTC and Twilio. The full run takes ~2.5 minutes per channel (real-time audio streaming + agent processing). It exercises: audio codec encoding, WebRTC/Twilio transport, speech-to-text, LLM reasoning with tool calling (MCP protocol to a hotel PMS), text-to-speech, and audio playback — all automated, no human in the loop.

The Bigger Picture

This isn't unit testing. It's not mocking. It's streaming real audio over real protocols to a real agent, and validating that a human-like conversation flows correctly end to end. It catches:

• STT regressions (model changes, VAD tuning, energy threshold drift)
• LLM behavior changes (prompt sensitivity, tool calling reliability)
• TTS pipeline issues (latency, silence gaps, playback completion)
• Protocol-level bugs (WebRTC data channel event ordering, Twilio mark timing)
• Integration failures (MCP server connectivity, database config loading)

The scenarios are extensible — adding a new test is recording yourself speaking, writing a YAML file, and running encode.sh.