Get Started
Home
Topics
Search
Library
Research questionWhen can factual recall in sparse MoE language models be attributed to one expert rather than an expert set?Restoring a mixture-of-experts block can recover a corrupted factual prediction, but the block output combines multiple expert contributions. The central difficulty is distinguishing single-expert causal effects from effects that arise only from the routed group.
AI
Machine Learning
Mechanistic Interpretability
Natural Language Processing
Latest papersRecent research connected to this question, newest first.Expert-Aware Causal Tracing of Factual Recall in Sparse MoE Language ModelsThe evidence concerns single-token COUNTERFACT contrasts in Qwen3-30B-A3B-Base and Mixtral-8x7B-v0.1. It uses subject-embedding corruption, clean block restoration, and clean-minus-noised expert-update restoration under fixed routing; results indicate individual-expert specificity in some cases and routed top-2-set specificity in others.research paper · Sep 3, 2026
Related questions
How does sparse mixture-of-experts routing trade off approximation, learning error, and computation under misspecification and evolving experts?How should experts be pruned in over-dispersed MoE routing when router importance and perplexity mislead?How can residual sparsification reduce MoE LLM memory use without accuracy loss from coupled expert projections?How can inference prune activated MoE experts without confounding compute savings with output rescaling?