Get Started
Topic · 9 recaps
Mechanistic Interpretability
Reverse-engineering neural networks at the level of circuits and features — figuring out what specific weights and activations compute, not just what the model outputs.
Play all
...
Posts
Questions
Home
Topics
Search
Library
Questions researchers are working on
Follow a question through Rcap’s explanations and the latest papers addressing it.
Search
Can chain-of-thought monitoring detect consequential computation hidden in semantically irrelevant filler tokens?
Language models may gain task performance from semantically irrelevant filler tokens without making the relevant computation interpretable in their visible reasoning. This complicates the use of chain-of-thought as evidence of what a model has computed.
Can intermediate LLM activations guide faster jailbreak search without weakening attack effectiveness?
Refusal behavior may be represented in transformer activations before the model produces its output. The difficulty is using that signal to reduce the cost of prompt search without losing the effectiveness of the resulting attacks.
Do language models generalize phonological rules or rely on memorized forms when selecting allomorphs?
Tokenization obscures sound structure, yet models often select morpheme forms according to phonological context. It is difficult to determine whether this behavior reflects memorized forms or a rule-like representation that causally influences selection.
Do multilingual attention heads that retrieve context also control transitions into the target language during reasoning?
In multilingual Transformers, retrieving information from context and producing reasoning in a specified target language may involve different internal components. It is unclear whether the same attention heads perform both roles.
Do pretrained language models encode a reusable truthfulness signal for detecting misinformation without external evidence?
Misinformation detectors can rely on wording patterns rather than whether a claim is true, while evidence-retrieval pipelines depend on outside sources. The central issue is whether pretrained models contain an internal signal that supports transferable truthfulness judgments.
Do refusal representations transfer across language-model architectures, and where should safety interventions read them?
Architectures differ in how they mix and update token information, so a refusal signal identified in one model may not be directly usable in another. Safety tooling must determine both whether the representation transfers and where the relevant computation is exposed.
How can AI-generated text detectors expose evidence users can inspect and act on?
Binary labels and confidence scores do not show which properties of a passage drove a detector’s decision. Users therefore need evidence that connects the prediction to observable textual signals and can be examined for faithfulness.
How can Arabic LLMs generate accurate target dialects from MSA prompts without fine-tuning?
Arabic LLMs receive far less dialectal data than Modern Standard Arabic, so they often default to MSA instead of producing an accurate dialect. It remains unclear whether dialectal information is localized in a few internal features or distributed across the model, and how that affects controllability.
How can attention heads be pruned in text-to-image diffusion transformers without losing prompt-specific object identity?
During denoising, semantic information may be maintained by structural template tokens and image-to-text interactions rather than by the prompt tokens that initially encode it. This makes it difficult to identify redundant attention computation without disrupting object identity.
How can attention-head contributions be measured in prompt-injection classifiers across circuit and output scales?
Many attention heads can jointly shape a classifier’s logits, while global output behavior can obscure which local circuit components drove the decision. The challenge is to connect fine-grained head behavior with the model’s final classification.
How can audio-video diffusion models preserve intended conditioning when biased cross-modal attention reroutes semantics?
In audio-video diffusion generation, cross-attention among text, audio, and video can route semantics bidirectionally rather than respecting intended conditioning. Learned biases may cause one modality to override prompts, producing visually canonical but semantically incorrect outputs.
How can encoder Transformers preserve document-scale structure in positional encodings during masked language modeling?
Transformers need positional signals to represent sequence order, but these signals can become entangled with semantic representations and lose document-scale structure. This complicates long-context understanding and retrieval.
How can evaluators distinguish missing knowledge from miscalibrated outputs in language models?
A model may encode a correct judgment while an output threshold produces the wrong answer. Observing only the final response therefore cannot reliably distinguish missing knowledge from a faulty readout.
How can inference-time hidden-state steering of LLMs become efficient and extensible for deployment?
Inference-time steering changes an LLM’s hidden states to alter its behavior without retraining. Existing frameworks can be computationally inefficient and difficult to extend or control, complicating applications such as mitigating overthinking and hallucinations.
How can language-model attention remain reliable beyond its training context?
Attention and positional mechanisms can become unreliable when inference contexts exceed the lengths seen during training. Their weighting can also lack a clear account of how information evolves across positions.
How can large language models allocate reasoning computation to preserve accuracy under limited training and inference budgets?
Extended reasoning can improve performance on difficult tasks but may consume substantial computation through slow or unnecessary steps. More efficient reasoning can reduce cost but may sacrifice accuracy.
How can LLM sandbox security remain reliable when linguistic monitoring misrepresents internal computation?
An LLM’s verbal self-reports and linguistically defined internal probes may not faithfully reflect its underlying activation-space computations. This makes it difficult to guarantee sandbox safety using language as the primary window into model behavior.
How can LLMs avoid unsafe pharmacological inferences driven by drug-name affixes?
Drug-name affixes can trigger class-level pharmacological responses even for fictitious drugs, leading models to produce confident but unsupported clinical content. Models may also fail to reveal when affix cues, rather than the full drug name, drive their interpretation.
How can LLMs remove undesirable knowledge while preserving utility with limited retention and unlearning data?
Removing targeted knowledge from a pretrained language model can damage unrelated capabilities, while preserving those capabilities often requires substantial retention or unlearning data. The practical challenge is balancing reliable forgetting against model utility when such data are limited.
How can local loss-landscape geometry reveal algorithmic structure in learned Turing-machine computations?
A learned machine’s algorithm may not be obvious from its behavior or parameters alone. Local geometry of the learning objective may provide indirect signatures of the computation being implemented.
Previous
1 / 4
Next