Voice interface for home robotics
Models used: Noise suppression, speech enhancement, target speaker isolation, speech-to-text
AirCaps Audio Research Lab explores how intelligent systems can better understand and interact with complex, real-world acoustic environments in real time.
Our work spans multichannel audio processing, spatial filtering, source separation, sound localization, target-speaker extraction, automatic speech recognition, and speech synthesis.
We focus specifically on messy audio found only in the physical world: far-field, reverberant, low signal-to-noise ratio (diffuse and directional noise), overlapping speech (babble), and multi-speaker scenarios.
01
We've been developing voice technologies for consumer electronics for 8 years. Before the era of ElevenLabs, or even OpenAI. We realized early on that all the leading speech models (at the time, Google and Microsoft) were optimized for the "digital" realm: transcripts for call centers, meeting notes for Zoom calls, captions for podcasts and movies. As soon as you deploy them in the "messy" physical world—far-field, reverberant, low SNR, overlapping speech—they degrade rapidly.
Part of the problem is the scarcity of real-world, noisy audio data. The largest datasets and evaluation benchmarks for speech models are not even close to the complexity of the real world.
Loading waveform…
A sample from Librispeech, the de-facto reference benchmark for ASR. Carefully segmented, clean, single-speaker audiobook speech.
Loading waveform…
Real-world sample from our proprietary dataset. Background noise, distance, interfering speakers all compete with the target voice.
The gap remains surprisingly wide - Ayllon et al. (2026) evaluated leading 40 ASR models and found that median word error rate (WER) reached 20.45% on speech with background noise - the same models report 2-3% on clean benchmarks. This study also found evidence of optimization for benchmarks ("benchmark-maxxing").[1]
02
Speech recognition ("ASR" or "STT") answers one question: what words were spoken? Simply slapping an STT model onto a machine does not make it good at comprehending real-world audio. A useful system must also infer who spoke, where the voice is, whether it was directed at the device, which speaker to focus on, all while enhancing speech and suppressing interfering noise. This is known as the "cocktail-party problem." Humans can do this in a few milliseconds. Machines have not been able to do this well, despite efforts since 1953[2].
We don't just develop STT models. We also build dedicated models for:
03
Human-machine interaction cannot wait for an upload. A real-time system must be causal: at time t, it can use only samples at or before t. It never sees future context that would make the current decision easier. Thus, batch (async / offline) model accuracy therefore overstates what an interactive (streaming) system can do.
Causality changes architecture, training, and evaluation. Every stage must keep bounded state, recover continuously, and process audio faster than it arrives (a real-time factor "RTF" below 1.0) to keep end-to-end latency low enough for natural turn-taking.
04
Intelligent machines should not lose the ability to communicate when they lose internet. Your car's voice assistant shouldn't stop working in a tunnel; voice dictation on a smartphone shouldn't be handicapped when you're on a hike. Nor should every spoken interaction incur a metered charge per word.
Edge intelligence used to lose on two fronts: devices lacked compute, and capable models were too large. Both constraints are moving quickly. Apple's A19 Pro (iPhone 17 Pro) combines a 16-core Neural Engine with Neural Accelerators in every GPU core and reports up to 40% better sustained performance than its predecessor. Novel specialized audio model architectures concentrate capacity on one task, delivering high intelligence density where the microphones are. Recent wave-based simulation methods can generate controlled mixtures of rooms, motion, reverberation, noise, and overlapping speakers, leading to higher quality training data.
All our models are optimized for edge deployments on consumer grade devices (modern smartphones, laptops, wearable NPUs) while maintaining RTFs < 1. This enables private, low-latency, and cost-effective voice AI that works everywhere, anytime.
01
Existing providers tackle real-world environments by training an STT model on noisy audio, which works well up to a degree. But in highly reverberant, far-field (when the target sound is far from the microphone) noisy conditions (SNR < 5 dB), their performance degrades significantly.
Instead, we build specialized models that first process raw audio and output enhanced audio before feeding it to the STT model. We train the enhancement and STT models jointly. Our models utilize spatial cues from multi-channel audio signals to "zoom in" on speech frequencies from a noisy mixture. Watch the difference for yourself:
02
Often, a machine must be able to distinguish desired speech (a user) vs unwanted speech (e.g., a group sitting at nearby table at a restaurant, a TV playing in the background). Existing models are trained to simply focus on the most "dominant" voice in the mixture, which leads to either jumbled transcripts or missing the target speaker completely.
This is a hard problem. Another voice is harder to remove than ordinary noise: it has the same structure as the signal we want to keep.
Our work combines spectral (what the voice sounds like) and directional (where the sound is coming from) information to isolate a target voice and suppress competing speech.
03
In many cases, simultaneous overlapping speech must not be discarded as all participants are targets. Across natural conversations, simultaneous speech occupies as much as 20.8% of total conversation time[3].
Group meetings lose interruptions, confirmations, and decisions; interviews and live panels lose the quieter speaker; cars, smart rooms, and multi-user voice agents discard target speakers entirely. A single isolation or suppression model would extract only a single target, what is needed here is multi-speaker separation.
Our models separate multi-speaker mixtures into stable, speaker-consistent streams so transcription, speaker identification, and intelligent systems can retain and attribute several target voices.
Models used: Noise suppression, speech enhancement, target speaker isolation, speech-to-text
Models used: Background speech suppression, target speech enhancement, speech-to-text
Models used: Sound source localization, directional speech isolation, speaker identification and verification, speech-to-text
Models used: speech enhancement & dereverberation, noise suppression, directional speech extraction, speaker verification, personalized voice activity detection, speech-to-text