Convex Language Detection (CLD) replaces the language-ID head of an Automatic Speech Recognition system with a two-layer ReLU network trained as a Convex reformulation of two-layer ReLU networks, giving global optima from ~100 training samples per dialect and keeping accented speech from being decoded in the wrong language.
You’ve shipped a voice agent on top of Whisper or a similar speech model. A Singaporean user speaks English; the model guesses “Malay,” and the decoder dutifully produces Malay-looking text. Your downstream LLM now sees garbage and the conversation collapses. The usual fix is fine-tuning the encoder on more dialect audio, which is expensive, leaks into other languages via catastrophic forgetting, and needs thousands of labeled clips you probably don’t have. CLD attacks just the language-ID step in front of the decoder, treats it as a small convex problem, and trains it on a few hundred utterances.
The pipeline keeps the frozen Whisper encoder and decoder. In between, the authors insert a lightweight classifier that picks the language token before decoding starts. The novelty is how that classifier is trained. A normal two-layer ReLU network has a non-convex loss, so training depends on learning rates, initialization, and luck. Prior theory shows that such a network has an exact Convex reformulation of two-layer ReLU networks: enumerate the possible ReLU activation patterns the data could induce, and the original problem becomes a convex program whose global optimum matches the non-convex one. CLD samples a manageable number of those patterns, pools the encoder’s per-frame features into one utterance vector, and solves the resulting program with Alternating Direction Method of Multipliers in JAX across multiple GPUs. Because the program is convex, there is no learning-rate grid search, and training converges in polynomial time. The authors also derive a Variation norm bound directly from the trained weights, which yields a certified radius: perturbations to the encoder features smaller than that radius cannot flip the predicted language.
# Offline training
for x, y in dataset:
h = whisper_encoder(x) # frozen features
H.append(masked_mean_pool(h)); Y.append(y)
D = sample_relu_activation_patterns(H, P)
v, w = admm_solve_convex_program(H, Y, D, beta) # global optimum
# Online inference
h = masked_mean_pool(whisper_encoder(x))
lang = argmax(convex_head(h, v, w))
text = whisper_decoder(x, init_token=lang)
The prevailing fix for accented-speech failures is to fine-tune the whole ASR stack on more dialect data, hoping gradient descent finds a better basin. This paper shows the opposite. Freeze the big encoder, isolate the language-ID decision as a small convex problem, and you get a global optimum from a handful of samples plus a printable robustness certificate. The evidence that matters is not the headline accuracy, it is the flatness of CLD’s accuracy curve from 100 to 10,000 training samples while every baseline climbs from poor to mediocre.
The load-bearing finding is sample-efficiency flatness. CLD on Whisper-Small lands between 96.94% and 99.14% language-ID accuracy across training sizes of 100, 500, 1000, and 10,000 samples per language. A vanilla neural head on the same encoder features needs the full 10,000 samples to approach that range and collapses at 100. The mechanism (convex global optimum on frozen features) is what removes the data-size dependence.
Secondary numbers reinforce this:
•
On multiclass detection across English, Chinese, Indonesian, Malay, Hindi with 24 sub-accents, swapping in CLD drops Word Error Rate from 139.37 to 31.74 on Whisper-Small. Most of that gap is cross-lingual decoding failures avoided, not better within-language transcription.
•
On the Min Dong Chinese dialect, default Whisper gets 9.86% language-ID accuracy and the neural baseline gets 25.35%. CLD hits 88.73%.
•
Training time is 64.45 seconds, roughly 7.7% of the neural baseline’s runtime, with about 13x fewer TFLOPs, because Alternating Direction Method of Multipliers parallelizes cleanly and there is no hyperparameter sweep.
•
The certified margin radius is computable from the trained weights, so each prediction comes with a feature-space invariance guarantee rather than a black-box confidence score.
Reach for this when you run a voice frontend on a frozen ASR model and your users speak a dialect that the base model mis-routes to a neighboring language. Instead of fine-tuning the encoder (slow, regression-prone, data-hungry), collect a few hundred labeled clips per dialect, extract pooled encoder embeddings once, and train a CLD head to override the model’s built-in language detector before the decoder runs. The decoder stays frozen and your existing pipeline stays intact.
The authors release a pip-installable jaxcld package and code on GitHub. It plugs in front of Whisper or Massively Multilingual Speech (MMS-1B) decoders. The audio datasets used (Common Voice, LAHAJA, MUSAN augmentation, and Singapore’s National Speech Corpus) are mostly public; the Singapore corpus requires institutional access.
When a giant frozen model gets one small decision wrong, fix that decision with a convex head, not the whole model. The convex reformulation only buys you this much when the upstream features are already strong. The trick works for Whisper-class encoders precisely because their embeddings already separate languages linearly-ish; on a weak encoder, a convex head over bad features will not save you.
•
The robustness certificate lives in encoder-feature space. Translating it back to an audio-space guarantee needs a Lipschitz bound on the encoder, which for deep Transformers is loose enough that the authors call end-to-end audio certificates “conservative diagnostics.”
•
CLD fixes language routing, not transcription quality within a language. The big WER drops come from stopping cross-lingual decoding failures; dialectal mis-transcriptions inside the correct language remain.
•
The head is a two-layer ReLU network on pooled utterance features. Tasks that need temporal structure (code-switching mid-utterance, speaker diarization) sit outside what this design can express.