The Hardest Bug in Voice AI: When Your Agent Hears Itself Talk

In full-duplex voice AI, the agent's own speech plays through a speaker. The microphone picks it up. Now your system "hears itself talking" and triggers false speech detection, phantom transcriptions, and broken barge-in.

Traditional browser echo cancellation helps, but it's not enough. It varies wildly across devices, degrades with Bluetooth, and has no idea about your application state.

So we built something different at Welcomr.ai .

The core insight

We don't fight echo in the signal domain. We sidestep it.

We control the TTS playback, so we know the precise start time, end time, and duration of every utterance. We use that knowledge to surgically suppress the microphone input only when it would contain echo.

Three layers of protection

Layer 1: Grace period The first 500ms after TTS playback starts is a "grace period" where barge-in detection is blocked entirely. This handles the initial echo burst, the loud sharp onset when audio first hits the speaker, which is the most common source of false barge-in triggers.

Layer 2: Playback suppression When TTS is actively playing, incoming microphone audio is attenuated by 20dB for VAD and barge-in detection. The raw (unsuppressed) audio still flows to the STT buffer, so if the user genuinely speaks during playback, their words aren't lost. The suppression only affects the decision-making path, not the transcription path.

Layer 3: Adaptive echo tail After TTS playback ends, residual echo lingers, especially in reverberant environments like hotel lobbies and conference rooms. Rather than using a fixed delay, the suppression window adapts to the TTS duration: 10% of playback length, clamped between 300ms and 1000ms. A 2-second response gets 300ms of post-suppression. A 10-second explanation gets the full second.

The barge-in dance

The trickiest part is the handoff between echo prevention and barge-in. When a user genuinely interrupts mid-sentence, we need to get the mic hot again immediately, not 300ms later when some post-suppression timer expires.

Here's the sequence:

1. The Interruption Manager fires: cancels the current LLM context, clears the TTS queue, stops playback
2. It then fully resets the echo prevention state (not just EndPlayback, but a hard Reset)
3. This is critical: EndPlayback would leave the post-suppression timer running, which would swallow the user's next words, the very words they interrupted to say
4. The hard reset ensures the microphone is immediately hot again

We also guard against a race condition: if a barge-in fires during TTS audio decoding (between "prepare audio" and "send to speaker"), we check an interrupt flag before calling StartPlayback. Otherwise the freshly-reset echo prevention would be re-contaminated, and the system would go silent.

Why not just use browser AEC?

Browser AEC (via WebRTC's built-in echo cancellation) handles the acoustic domain, subtracting the reference signal from the microphone. It works well in ideal conditions. But:

• It varies wildly across browsers and devices
• It degrades with Bluetooth headsets, external speakers, speakerphone mode
• It doesn't know about our application state (is the agent speaking? is this a barge-in?)
• Residual echo after AEC is still enough to trigger a sensitive VAD

Our timing-based system is a complementary layer. It operates at the application level, using knowledge the browser doesn't have: the exact timing and duration of every TTS utterance. Browser AEC removes most of the acoustic echo; our system prevents the residual from causing bad decisions.

The numbers

• Suppression depth: 20dB during active playback
• Grace period: 500ms (blocks false barge-in from echo onset)
• Post-playback tail: 300ms to 1000ms (adaptive, 10% of TTS duration)
• Gain release: gradual (linear interpolation over the suppression window)
• Thread-safe: all state behind sync.RWMutex, safe for concurrent audio processing

The takeaway

The hardest problems in voice AI aren't in the models. They're in the plumbing.

Echo prevention is invisible when it works. When it doesn't, the conversation breaks. And the fix wasn't more sophisticated signal processing. It was using information we already had: when we're talking.

Simple insight. Surprisingly hard to ship.

Originally published on LinkedIn.