When a speech or audio model produces a wrong answer, answer-only scoring cannot tell you which of two very different things went wrong. The system may have generated a confident response from a waveform that never contained the requested evidence — silence, noise, or the wrong speaker. Or the waveform may have been perfectly good while overall accuracy hardly moves when that audio is removed or swapped. Two archived studies give you controls for each case: SURE-Challenge formalizes a rejection decision made before the backbone runs, and Tracing Audio Grounding and Answer Selection in Audio LLMs supplies silence, mismatch, and Attention knockout tests for how much aggregate accuracy depends on admitted audio and for which internal pathways carry that dependence. I would use them for two separate checks: whether the input contains usable audio, and what changes when usable, answerable audio is removed or replaced. The papers test different tasks, datasets, and models; they do not validate a combined pipeline.
Why answer accuracy is not a diagnosis
Picture a transcription-plus-QA endpoint. One request arrives as a clean utterance; another is a four-speaker babble mix with the prompt “what did the main speaker say?”; a third is a length-matched tone. If your only metric is whether the emitted text matched a reference, all three failures collapse into one number. But they demand different fixes. In the babble and tone cases, there was no admissible evidence, and the correct behavior was abstention or a clarification request. In the clean case, the interesting question is whether the model’s answer was actually a function of what it heard.
SURE makes the first distinction explicit by scoring the admission decision itself rather than the generation. Each example pairs audio with a prompt and an expected response label; supported cases include clean transcription, additive noise, filtering, reverberation, speed perturbation, and first-word QA, while unsupported cases are length-matched silence, colored noise, synthetic tones, and off-source babble that asks for a “main speaker” who is absent from the mix. The scoring follows the decision: transcriptions count as correct at normalized WER ≤ 0.25, unsupported inputs require abstention or clarification (a generic [Music] tag does not satisfy a transcript prompt), and abstaining on supported speech is counted as an error. The reported metrics are supported accuracy, unsupported rejection, WER on answered supported examples, and the number of downstream calls. Note the construction caveat the authors themselves flag: examples are generated from 40 LibriSpeech validation-clean utterances (Core) and 120 train-clean-100 utterances (Extended), so counts from the same source are dependent rather than independent observations.
Does the input contain usable audio?
The Front-end admission rule rule in SURE is deliberately dull. It drops extremely short or near-silent clips, decodes the rest with Whisper-small, averages the per-step maximum token probability into a score, and abstains below a threshold chosen on clean development data (τ = 0.70) by maximizing unsupported rejection subject to zero supported false rejects. Because the decision uses only pre-backbone signals, the same frozen decisions can be replayed in front of six different backbones.
The numbers that matter here are the contrast, not the headline. Without a front end, raw Qwen2-Audio refused 15 of 204 unsupported inputs on the screened SURE-Extended test; with the fixed rule, 196 of 204 were rejected, and downstream calls fell by 41%. The tempting shortcut — just prompt the model to abstain when unsure — was measured too, and it failed in an instructive way: a self-abstention prompt rejected all 70 unsupported Core examples but left supported accuracy at 0.033 for Qwen2-Audio and 0.156 for Qwen2.5-Omni. The refusal gain came partly from suppressing valid generations. If you only log refusal rate, that looks like a win.
This is exactly why unsupported rejection must always be reported beside supported retention at a stated operating point. SURE’s external evaluations pull apart configurations that tie on its own generated split. The main rule retained 554/600 supported English clips and rejected 917/1160 unsupported external clips; VAD+score retained 552/600 while rejecting 999/1160. Tightening the Whisper threshold to τ = 0.90 pushed all-unsupported rejection to 0.964 but cut supported English retention to 0.505 — half the good audio thrown away. Wrong-source speech remained the hardest condition: the leakage-screened babble seed gave 46/54 rejections, and regenerated no-speed babble only 18–24/54. The authors are direct about scope: dysarthric and child speech were not evaluated, multilingual retention depended strongly on decoder language selection, and external corpora are referenced by recipe rather than redistributed. There is no universally best threshold in this evidence, only a retention–rejection exchange whose right point depends on which error costs you more.
One boundary deserves emphasis because it defines where gate one stops. SURE’s stated main limitation is that audio-only admission cannot determine whether recognized speech answers a particular prompt. Their semantically unanswerable speech prompts left Qwen2-Audio’s refusal rate unchanged after filtering. Admission tells you the clip passed an audio-level gate; it does not tell you that any recognized speech answers the prompt.
Does removing usable audio change accuracy?
Suppose your gate is calibrated and a clean, paired waveform passes. Now the failure mode shifts. The grounding study attacks this with a perturbation design on multiple-choice AudioQA, where the model sees audio, a question, and candidate answers, and must emit only the option letter. Three conditions are compared: the original paired audio; a silent condition where the waveform becomes an all-zero signal of the same duration, removing informative content while preserving an audio-shaped input; and a mismatch condition where a natural waveform from another sample in the same benchmark replaces the original, preserving audio realism but breaking correspondence with the question. The net accuracy drop relative to the original, averaged over both perturbations, defines Behavioral acoustic-evidence sensitivity. Be precise about what that statistic is: a comparison of aggregate scores, not of individual predictions. To see why the difference matters, imagine a hypothetical 100-question set where a model answers items 1–50 correctly with the real audio and items 51–100 correctly under silence; both runs score 50%, yet every answer flipped. A small sensitivity value therefore means the aggregate score held up, not that the predictions were unchanged, and comparing per-example transitions between the two runs is a separate check.
The logic of averaging is worth pausing on. Silence and mismatch fail in different ways — one removes informative content while keeping an audio-shaped input, the other supplies a natural waveform that does not match the question — and the authors average the two conditions specifically to reduce dependence on either perturbation alone. Across Qwen2-Audio and Qwen2.5-Omni, LoRA-adapted on 19,480 DCASE 2026 AudioMCQ samples and evaluated on held-out ADQA-Bench, MMAU-test-mini, MMAR, and MMSU, every one of the six model–dataset pairs where accuracy improved also showed increased sensitivity. MMAU went the other way: negative sensitivity change alongside weak or negative accuracy change. That is explicit evidence against assuming tuning uniformly increases either accuracy or audio use, and it is why the authors restricted their pathway analysis to the benchmarks where both moved together. The authors also state plainly that this score measures behavior, not mechanism.
Inside the second gate: availability is not use
Behavioral sensitivity leaves an obvious next question. If removing the audio lowers accuracy, where in the model does that dependence live? Attention knockout answers by zeroing, within a chosen layer window, the attention from one token group to another — audio tokens, question tokens, option tokens, and the first generated answer position — while leaving all parameters untouched.
The pattern splits cleanly by stage. Blocking audio→options reduced correct-answer probability in early-to-middle layers, strongest around layers 8–12 in Qwen2-Audio and 10–16 in Qwen2.5-Omni, and this effect was already present before training. Blocking options→answer did almost nothing in the zero-shot models but, after training, produced a clear reduction at later layers, around 18–22 and 20–27 respectively. The authors’ reading is that acoustic evidence reaches candidate-answer representations early, largely independent of tuning, while training mainly strengthens how those representations feed the final prediction. Their conclusion states the distinction directly: making acoustic evidence available to answer representations and using those representations for the final decision are distinct stages. A finer split reinforced it — in Qwen2.5-Omni, blocking the correct option’s contribution to the answer produced a pronounced post-training decrease around layers 18–25, while blocking wrong options changed little.
These are causal but narrow results: specific checkpoints, a specific LoRA configuration, and the first generated option letter. They do not establish a universal circuit, and nothing here transfers automatically to free-form speech generation.
Evaluate input filtering and dependence on audio separately. For each test, state which inputs are included in the score.
First, calibrate a pre-generation admission rule and report both error types at your chosen operating point: false accepts over unsupported inputs and false rejects over supported ones. Freeze the threshold on development data, then hold it. Choose the point by your own error costs — a medical dictation service and a voice-search toy sit at different places on that curve, and neither paper establishes a cost-optimal setting.
Second, apply silence and mismatched-audio replacements only to inputs that passed admission and are genuinely answerable. Mixing unsupported clips into this denominator would let a model that ignores audio score well simply because ignoring a tone is harmless.
Third, if you are tuning a multiple-choice model and want to know why sensitivity moved, add targeted interventions on the path from option representations to the emitted answer. Knockout is appropriate where token groups and answer structure are well defined; it is not a drop-in probe for arbitrary generative architectures.
The studies test different settings. SURE evaluates frozen pre-backbone decisions over transcription, first-word QA, non-speech, and source-ambiguous speech across six backbones; the grounding study evaluates zero-shot and LoRA-adapted Qwen models on multiple-choice AudioQA. No supplied experiment runs both sets of tests on the same examples, and their metrics share no denominator, so their numbers should never be compared side by side. What they jointly support is narrower and still useful: answer-only scoring cannot distinguish an input-admission failure from weak aggregate dependence on admitted audio, and two separately reported gates can. Neither gate, on its own, tells you how individual answers moved.
The practical shift is where you look when an audio system disappoints. If unsupported rejection is near zero — as raw refusal was for all six backbones SURE replayed — and if your evaluation set contains unsupported inputs, your accuracy number is partly measuring confident answers to them. A pre-generation gate addresses that directly, and SURE showed that the obvious alternative of prompting for self-abstention bought refusals by suppressing valid answers too; it did not test whether some other form of tuning could achieve the same admission behavior.
If admission is solid but silence and mismatch barely move accuracy on admitted inputs, treat that net result as a starting point rather than a verdict: an unchanged aggregate score is also consistent with answers shifting in both directions and cancelling out, so the next check is to compare per-example predictions between the original and perturbed runs. Even then the score describes behavior rather than internal mechanism.
If a fine-tune raises accuracy, the grounding results give you a cheap way to check whether aggregate accuracy became more or less sensitive to removing the paired audio: MMAU shows those two changes can move in opposite directions for the same model–dataset pair when the original instruction-tuned checkpoint is compared with its LoRA-adapted counterpart. Knowing which of the three descriptions fits your system is the difference between fixing an input filter, changing training data, and discovering the benchmark was answerable from text all along.