A compact MEG speech decoder is built so its weights map back to cortical sources and rhythms, and paired input swaps show retrieval leans on acoustic state, phonetic classes, and narrative structure, using ~20× fewer parameters than the Défossez et al. brain decoder baseline.
You’ve shipped a model that reads brain signals and picks which 3-second audio clip a person was listening to. It scores well. Your neuroscientist collaborator asks: which part of the brain drove that, and which parts of the sound did the model actually latch onto? Today you shrug. The network’s weights don’t correspond to anything an electrophysiologist can name, and the decision rule is opaque.
The dominant approach here, Défossez et al. brain decoder, hits strong retrieval numbers on Magnetoencephalography (MEG) but its spatial-attention layer operates over a flattened 2D projection of sensors and its weights don’t localize to cortex. That leaves accuracy without mechanism. This paper redesigns the front end so that both questions, where in cortex and what in the stimulus, become answerable from the trained model.
The idea is to make each channel of the network stand in for one neural population, with its own location on the cortex and its own preferred rhythm. Concretely, the decoder’s front end is a bank of branches, and each branch is a spatial filter (which sensors to weight) followed by a temporal filter (what frequencies to pass). Because both are learned jointly, the paper uses the interpretation recipe of Petrosyan interpretation framework to map each branch’s weights back to a cortical source topography and a frequency spectrum.
Four concrete changes to the Défossez et al. brain decoder pipeline:
•
Replace 2D Fourier spatial attention with a layer parameterized by Spherical harmonics, since MEG sensors sit on a roughly spherical helmet.
•
Add trainable per-branch temporal filters (150 ms kernels) so each branch is matched in time as well as space.
•
Shrink the subject-specific representation from 270 channels to K=25 branches, a physiologically plausible bottleneck.
•
Remove ocular and cardiac Independent component analysis (ICA) components before training, since eye and heart activity track linguistic structure and would be a shortcut.
The rest of the pipeline is standard: a small convolutional decoder produces embeddings aligned to wav2vec 2.0 audio embeddings via a CLIP-style contrastive loss on the MEG-MASC dataset.
for meg_segment, audio_segment in batch:
x = spherical_harmonic_attention(meg_segment) # 208 sensors -> 270 virtual
x = unmix_1x1(x) # shared mixing
x = subject_projection[subj_id] @ x # -> K=25 branches
x = depthwise_temporal_filter(x) # 150 ms per branch
meg_emb = conv_decoder(x) # -> 768-dim, aligned to audio
loss = clip_contrastive(meg_emb, wav2vec(audio_segment))
To ask what the model uses, the authors do paired MEG occlusion: for each candidate stimulus feature (silence, vowels, high surprisal, etc.), they replace the MEG during feature-present intervals with either (a) real MEG from another feature-present interval or (b) real MEG from a feature-absent interval, and compare retrieval rank. The paired design controls for generic damage from replacing signal.
The prevailing move in brain decoding is to chase accuracy with a large opaque network, then run post-hoc explanation on latent features. This paper shows the opposite. Build the front end from the physics of the measurement and the physiology of the sources, and interpretability becomes a property of the trained weights rather than a separate analysis, without losing accuracy. The clearest evidence isn’t the headline retrieval score. It is that the K=25 branches, mapped through the Petrosyan interpretation framework recipe, land on the known speech-perception network, and that paired occlusion cleanly separates which stimulus features the decoder actually uses from those it ignores.
The load-bearing result is the paired occlusion contrast. 15 of 19 stimulus features produced a significant positive effect after familywise correction. The largest rank shifts came from silence (Δr=75.6), high loudness (60.8), vowels (38.7), and strong acoustic onsets (36.2), all positive in every one of 27 participants. Phoneme classes, word onsets, and high-surprisal words also contributed, while predictive entropy, silence onsets, and rare words did not.
One odd finding sharpens the interpretation. For random word lists, the contrast went negative (Δr=−17.8): swapping in MEG from coherent narrative improved retrieval on word-list segments. Brain activity during structureless word lists carries less recoverable information than activity during coherent speech. Contextual structure isn’t just another decodable feature; it supports the tracking the decoder relies on.
Secondary evidence for the mechanism:
•
39.75 ± 0.34% Top-1 among 1005 candidates across six seeds, with roughly 20× fewer trainable parameters than the Défossez et al. brain decoder. Within the authors’ own ablation grid (same data, same test set), a K=270, five-block variant is 14.8× larger and scores 3.60 pp lower in Top-1.
•
Mapping the front-end weights through RAP-MUSIC localizes sources to bilateral auditory cortex, superior temporal gyrus, supramarginal region, and left inferior frontal cortex, matching the canonical speech-perception network.
•
Left-lateralized branches carry an extra faster spectral component (~13 Hz) on top of the shared slow (<10 Hz) rhythm, consistent with the Asymmetric sampling in time account of auditory lateralization.
•
Compressing the wav2vec 2.0 target from 768 to ~12 learned feature dimensions leaves retrieval essentially unchanged, but heavy temporal compression (global pooling) collapses to near-chance. The MEG-decodable subspace is thin along features and thick along time.
Reach for this design when you’re building any decoder over sensor-array biosignals (MEG, EEG, ECoG) and your stakeholders want more than a leaderboard number. The pattern generalizes: put a physically motivated spatial layer first, factorize spatial and temporal filtering into per-source branches with a small K, and use the Petrosyan interpretation framework interpretation recipe on the learned weights. You get a model whose channels correspond to something a domain expert can name. The paired-occlusion protocol is also directly reusable: any time you have a decoder over time-series input with annotated stimulus features, swapping feature-present intervals with matched feature-absent donors from the same recording gives you a cleaner attribution than gradient-based saliency.
Artifacts: a project page is listed at LISA. The paper builds on the public MEG-MASC corpus (27 participants, ~1 hour each of naturalistic story listening) and uses standard MNE-Python and FreeSurfer tooling for source localization. The paper doesn’t specify a code release license in the text provided.
If you want a model to explain itself, put the domain’s physics into the first layer rather than reaching for post-hoc attribution on an opaque one. Here that means spherical-harmonic spatial attention plus per-source spatial-temporal branches; the payoff is that trained weights map to named cortical sources at no cost to accuracy.
•
Source maps use one template brain shared across all 27 participants because most anatomical scans were truncated by defacing. Individual anatomies would sharpen localization and could shift some cluster assignments.
•
Effect magnitudes across occlusion features are confounded by mask duration: silence intervals are long, word onsets are brief, so the ordering shows which features the decoder uses, not how much each contributes.
•
Results are from one dataset and one language. The participant sign-flip test generalizes across listeners on the held-out audio, not to new narratives, architectures, or evaluation protocols.