OmniEcho teaches an omni-modal LLM to hear where things are by adding a spatial-audio pathway that processes 4-channel First-Order Ambisonics (FOA) recordings alongside the model’s existing mono audio and vision encoders, letting an embodied agent answer spatial questions and navigate toward sound sources.
Imagine a home robot: a person calls out from another room, or a smoke alarm chirps behind a wall. A camera-only agent has no way to know a direction, because the source is off-screen. Humans handle this trivially using two ears. Today’s omni-modal LLMs (models that jointly process audio, vision, and text like Qwen3-Omni) largely ignore spatial audio. They typically ingest a single-channel waveform, which preserves what was said but discards where it came from.
The paper’s baseline is that state of affairs: strong pretrained omni models score barely above chance on spatial-audio questions in the authors’ benchmark. There is also prior work on audio-visual navigation (SoundSpaces and follow-ups), but those systems were built on rendered acoustics in simulators and don’t ingest FOA. The closest recent effort at spatial-audio understanding in an MLLM, Spatial-Omni, trained without visual frames and without a navigation task. So there was no unified way to evaluate, or model, spatial audio for embodied agents.
The core idea: don’t replace the pretrained audio encoder, add a second, spatially-aware one next to it, then let the language model see both streams of tokens in time order.
Concretely, each FOA clip goes down two paths. Path one: the omnidirectional channel is downmixed to mono and fed to the frozen Qwen3-Omni audio tower, giving normal semantic audio tokens (“a dog is barking”). Path two: the full 4-channel clip is turned into a 5-channel feature map (a log-mel spectrogram plus DirAC intensity features intensity vectors and a diffuseness scalar) and fed to a small custom FOA encoder that produces spatial tokens. A trainable projector maps those into the LLM’s embedding space, and they’re spliced in right after the corresponding semantic audio tokens.
The FOA encoder is trained in three stages before the LLM ever sees it:
1.
Semantic alignment. Train the encoder from scratch to match CLIP text embeddings of sound labels, using a SigLIP objective. This gives it an open-vocabulary sense of what sounds are present.
2.
Query-conditioned localization. Fine-tune with a cross-attention head that, given a text query naming a source, regresses that source’s azimuth, elevation, and distance. Angles are predicted as (sin, cos) pairs to avoid the ±180° wraparound problem.
3.
Integration. Freeze the FOA encoder and the audio/vision towers. Only the projector and the LLM update, trained jointly on spatial QA and sound-guided navigation data.
# per FOA clip, at inference
semantic_tok = frozen_audio_tower(downmix_to_mono(foa))
spatial_feat = build_5ch_feature(foa) # mel(W) + intensity + diffuseness
spatial_tok = projector(frozen_foa_encoder(spatial_feat))
spatial_tok = resample_to_grid(spatial_tok, rate=7) # match audio-token rate
tokens = interleave(semantic_tok, spatial_tok) # spatial after semantic, in time
answer = llm(tokens, video_tokens, text_prompt)
For navigation, the model output is not text but a list of 8 future waypoints as (forward_m, left_m, yaw_rad) relative to the current pose, following the InternNav / InternVLA-N1 formulation. A discrete controller turns each waypoint into forward-step or turn-in-place actions.
Two benchmarks, both introduced by the paper. OmniEchoBench-QA has 2,972 spatial QA pairs over 197 real-recorded audio-visual scenes with FOA audio. OmniEchoBench-Nav has 900 navigation samples across 30 real-scanned indoor environments, with a dense grid of FOA recordings per scene.
On spatial QA (audio+vision setting): OmniEcho reaches 28.5% overall accuracy versus 18.5% for the base Qwen3-Omni it’s built on, and versus 11.4% for SO-7B (the Spatial-Omni baseline that also uses FOA). Sub-tasks include camera rotation, source direction, 3D localization, source motion, and a bird’s-eye “cognitive map” localization task. Gains are consistent across sub-tasks, with the largest jump on the cognitive-map task (22.3 → 47.5).
An ablation is the most informative result. Removing the vision input barely hurts source-direction, 3D-localization, and source-motion accuracy, so those tasks are being solved mostly from FOA. Camera-rotation accuracy collapses without vision, which makes sense: FOA tells you where the world is, not which way you turned. Another ablation shows the pre-extracted 5-channel feature outperforms feeding the raw 4-channel waveform (17.9° vs 30.5° azimuth error), and skipping the Stage-1 semantic pretraining measurably degrades localization.
On sound-guided navigation (OmniEchoBench-Nav val-unseen, 1 m success radius): OmniEcho reaches 16.2% success rate using only FOA audio as guidance, versus 17.8% for InternVLA-N1 using full text instructions and 5.4% for a SoundSpaces model given binaural audio. On the classic VLN-CE R2R R2R benchmark, with sound synthesized at the destination, OmniEcho reaches 22.2% SR, comparable to older text-instruction baselines but well below current text-guided systems.
One caveat the authors surface themselves: an oracle-direction experiment (feeding ground-truth source bearing at each step) raises SR to 23.7%, and among episodes that visit the goal region, the success rate jumps from ~31% to ~69%. So a large chunk of the remaining error is in bearing estimation and in knowing when to stop, not in path planning.
•
If you’re building an embodied agent with a microphone array, the two-pathway trick is the transferable idea. Keep the pretrained mono audio encoder for semantics, add a spatial encoder on the multichannel signal, and interleave the tokens temporally. It preserves whatever the base omni model already knew about sound events.
•
For spatial-audio research, the benchmark is probably the artifact worth grabbing. OmniEchoBench-QA uses real FOA recordings of scripted human performances (not simulated acoustics), which the paper shows exposes a real sim-to-real gap: a small fine-tune on 1,000 real samples lifts overall accuracy from 28.5 to 34.9. Code and data are promised at GitHub.
•
If you want to try this on a hosted omni model without training, the paper doesn’t establish that path. Their approach requires injecting new token streams into the LLM’s context and updating LLM weights (Stage 3 trained on 32 A100s for ~4 days). Prompting a black-box API with a text description of FOA features is a plausible next experiment but not evaluated here.
•
Treat the navigation numbers as “proof of signal,” not deployment-ready. Success rate is 16%, distance estimation MAE is 2.2 m, and roughly 37% of episodes visit the goal but fail to stop there. Useful for research comparisons, not for shipping a delivery robot.
•
Most training data is synthetic: videos generated by Seedance with LLM-scripted trajectories, then rendered into FOA. The authors align distributions with the real benchmark, but the sim-to-real experiment confirms residual bias, and only spatial QA is tested there, not navigation.
•
The claim that OmniEcho is “close to traditional VLN” holds against older baselines (Seq2Seq, CMA). Modern text-guided systems like InternVLA-N1 are still meaningfully ahead on the same simulated benchmark (50.6% vs 22.2% SR on VLN-CE R2R).
•
Comparisons to prior audio-visual navigation methods are imperfect. Those systems don’t accept FOA input, so the authors convert to binaural for SoundSpaces, which the paper notes cannot fully match the task formulation.
•
Fine-grained localization (exact degrees, exact metric distance) remains a stated open problem. The model gets coarse direction right but often misses distance and precise angles, as the failure-case visualizations show.