Barge-In: Letting Humans Interrupt an AI Mid-Sentence
Last week I wrote about echo prevention in voice AI. This week: the other half of the problem.
In a voice conversation, the agent can speak for several seconds, reading out a hotel reservation, explaining check-in policies, listing amenities. During that time, the user might want to cut in: "Wait, I meant tomorrow, not today."
This is barge-in. The ability for a human to interrupt the AI mid-speech and take back the conversational turn.
Getting it right is deceptively hard. Get it wrong, and you either:
- Never detect real interruptions, making the agent feel like a lecture
- False-trigger on echo and ambient noise, making the agent stutter and cut itself off constantly
The Detection Pipeline
Barge-in detection lives in the AudioProcessor, running on every 20ms audio frame. The check fires inside the VAD state handler, when speech is detected and TTS is playing simultaneously:
case audio.VADStateSpeech:
ap.consecutiveSpeechFrames++
// Only check for barge-in after sustained speech
if ap.consecutiveSpeechFrames >= ap.bargeInThreshold {
ap.checkForBargeIn()
}
The bargeInThreshold defaults to 10 frames = 200ms of sustained speech. A single frame of detected speech is ignored. This is the first line of defense against transient sounds: a door closing, a cough, a brief echo spike.
Three Protection Layers Inside checkForBargeIn()
Once the VAD has counted enough consecutive speech frames, we still don't immediately fire the barge-in. Three more checks gate the decision:
Protection 1: Grace Period
The first 500ms after TTS starts playing is a "no-interrupt zone." This is when the initial echo burst is loudest, the speaker just turned on, the room hasn't settled, and the microphone is picking up the loudest reflection of the agent's voice.
echoPrevention := ttsStreamer.GetEchoPrevention()
if echoPrevention != nil && echoPrevention.IsInBargeInGracePeriod() {
return // Block — too early, likely echo onset
}
Protection 2: Barge-In Lock
Certain utterances are non-interruptible by design. The welcome message, for instance, is locked. The agent should always be able to greet the user without being cut off by its own echo in an unfamiliar acoustic environment:
if bargeInLocked {
return // Blocked — non-interruptible playback (welcome message)
}
The lock is set before TTS and released after:
// In conversation_welcome.go
cm.session.BargeInLocked = true
// ... generate and play welcome TTS ...
cm.session.BargeInLocked = false
Protection 3: Dual Energy Threshold
This is the most nuanced check. We maintain two energy thresholds and switch between them depending on echo suppression state:
if echoPrevention != nil && echoPrevention.ShouldSuppressAudio() {
// TTS actively playing — ONLY trust absolute threshold
if ap.currentEnergy < ap.absoluteEnergyThreshold {
return // Blocked — energy too low, likely echo
}
} else {
// Normal mode — barge-in if energy exceeds EITHER threshold
if ap.currentEnergy < ap.absoluteEnergyThreshold &&
ap.currentEnergy < relativeThreshold {
return
}
}
Why two modes? During active TTS playback, the speaker's output leaks back into the microphone. This raises the rolling average energy (recentAverageEnergy), which raises the relative threshold (1.3x average). But echo energy adapts with the rolling average, the relative threshold becomes useless because it tracks the echo up.
So during playback, we fall back to the absolute threshold (default 150,000 RMS), which stays constant and reliably separates echo (~5k-10k energy) from real human speech (~50k+ energy).
In normal mode (no TTS playing), we use both thresholds as an OR. Either one being exceeded is sufficient. This makes normal barge-in sensitive, while playback-mode barge-in is strict.
The Interrupt Cascade
When all checks pass, the system fires a carefully orchestrated cascade:
Inside InterruptionManager.Interrupt(), five things happen atomically under a mutex:
func (im *InterruptionManager) Interrupt() {
im.mu.Lock()
defer im.mu.Unlock()
// 1. Debounce — ignore if < 500ms since last interrupt
if time.Since(im.lastInterruptTime) < time.Duration(im.debounceMs)*time.Millisecond {
return
}
// 2. Cancel the current LLM/TTS context
if im.currentCancel != nil {
im.currentCancel()
}
// 3. Clear TTS queue + stop playback
ttsStreamer.ClearQueue()
ttsStreamer.StopPlayback()
// 4. CRITICAL: Hard-reset echo prevention
ttsStreamer.ResetEchoPrevention()
// 5. Clear pending transcripts so only the new input is processed
conversationMgr.ClearTranscriptQueue()
}
Step 4 is the most important and the one we got wrong initially. StopPlayback() internally calls EndPlayback(), which starts the post-suppression timer (300-1000ms). That timer would swallow the user's next words, the very words they interrupted to say. The hard Reset() zeroes everything: no suppression, no timer, microphone is immediately hot.
The Context Cancellation Chain
The context.Context propagation is what makes the entire pipeline responsive. When the InterruptionManager calls cancel(), it ripples through every in-flight operation:
InterruptionManager.cancel()
│
├─► LLM streaming response: ctx.Done() → stop reading chunks
│
├─► TTS HTTP request: ctx cancelled → abort synthesis
│
└─► TTSStreamer.streamMP3() playback loop:
for i := 0; i < len(pcm); i += chunkSize {
select {
case <-ts.session.Context.Done():
return // ← Stop sending audio frames
default:
}
// Also check manual stop flag for faster response:
if !ts.isPlaying || ts.isStopped {
return // ← Barge-in detected
}
audioProcessor.SendPCMAudio(chunk)
}
The TTS playback loop checks for cancellation every chunkSize samples (1 second of audio at 48kHz). This means worst-case latency from barge-in detection to audio silence is ~1 second. In practice it's much faster because the isStopped flag is checked at the same frequency.
The Race Condition
There's a subtle race between barge-in and TTS startup. Consider this sequence:
T=0ms TTS chunk arrives, decodeMP3() starts (CPU-bound, ~50ms)
T=10ms User speaks, VAD fires, InterruptionManager.Interrupt() runs
T=10ms ResetEchoPrevention() clears all state
T=50ms decodeMP3() finishes, about to call StartPlayback()
← If we call StartPlayback() here, echo prevention re-activates
← User's speech is suppressed. System appears dead.
The fix is a pre-flight interrupt check between decode and playback:
func (ts *TTSStreamer) streamMP3(mp3Data []byte) error {
pcm, err := ts.decodeMP3(mp3Data)
// ...
// Check if interrupted during decode
ts.queueMutex.Lock()
interrupted := !ts.isPlaying || ts.isStopped
ts.queueMutex.Unlock()
if interrupted {
return nil // Abort — don't contaminate echo prevention
}
// Safe to proceed
ts.echoPrevention.StartPlayback()
// ... stream audio ...
}
Re-Trigger Prevention
After a barge-in, the system generates a new LLM response and starts new TTS. But the consecutiveSpeechFrames counter might still be above threshold from the user's interrupting speech. Without resetting it, checkForBargeIn() fires again at T=0ms of the new TTS, killing the new response before the first audio frame is sent.
// After all checks pass and barge-in is confirmed:
ap.consecutiveSpeechFrames = 0 // Prevent immediate re-trigger
go interruptMgr.Interrupt()
The Full Barge-In Sequence
T=0ms — Agent TTS is playing audio via WebRTC. Echo prevention is suppressing mic input for VAD.
T=120ms — User starts speaking ("wait, no"). Opus decoder converts speech to PCM. Echo prevention suppresses the PCM for the VAD path, but raw audio still flows to the STT buffer.
T=140ms — VAD detects speech. consecutiveSpeechFrames = 1.
T=320ms — consecutiveSpeechFrames reaches 10 (200ms of sustained speech). checkForBargeIn() runs: Grace period passed? ✓ BargeIn locked? ✗ Energy above threshold? ✓ → BARGE-IN CONFIRMED.
T=320ms — The cascade fires. AudioProcessor resets consecutiveSpeechFrames to 0. InterruptionManager cancels LLM context, clears TTS queue, stops playback, hard-resets echo prevention, and sends "interrupted" event to client. TTSStreamer sets isStopped = true. Audio goes silent. Mic is immediately hot.
T=~500ms — STT transcribes the buffered user speech: "wait, I meant tomorrow."
T=~600ms — LLM generates new response with the user's correction.
T=~900ms — New TTS plays the corrected answer.
Total latency from interruption to silence: ~200ms. From interruption to new response: ~600-800ms.
Configurable Parameters
All timing values are exposed in the agent configuration, tunable per-deployment (only by us at the moment, not exposed customer side):
Sustained speech threshold — Default: 10 frames (200ms). Config: barge_in.threshold_frames. How long speech must persist before checking.
Absolute energy threshold — Default: 150,000. Config: barge_in.absolute_energy_threshold. Fixed energy floor for barge-in during TTS.
Relative multiplier — Default: 1.3x. Config: barge_in.relative_threshold_multiplier. Ratio above rolling average (normal mode).
Debounce interval — Default: 500ms. Config: barge_in.debounce_ms. Minimum time between interrupts.
Grace period — Default: 500ms. Config: echo_prevention.barge_in_grace_period_ms. No-interrupt zone after TTS starts.
Suppression window — Default: 10% of TTS duration. Auto-calculated. Post-TTS echo tail suppression.
Crossfade samples — Default: 480 (10ms). Config: tts.streaming.crossfade_samples. Smooth audio transition on interrupt.
The Key Insight
Barge-in is not a single detection event. It's a multi-layered decision pipeline with carefully coordinated cleanup.
The detection part (VAD + energy + timing) is only half the story. The other half is the teardown: cancelling contexts, resetting echo prevention, clearing queues, preventing re-triggers, and handling race conditions between concurrent goroutines that are all operating on shared session state under mutexes.
Simple concept. A lot of mutexes.
Originally published on LinkedIn.