Get Started
Topic · 10 recaps
Alignment & Safety
Research on getting models to do what we actually want — honest, harmless, and helpful behavior — covering RLHF, constitutional methods, adversarial robustness, and broader safety theory.
Play all
...
Posts
Questions
Home
Topics
Search
Library
Questions researchers are working on
Follow a question through Rcap’s explanations and the latest papers addressing it.
Search
Can automated alignment research mitigate multiple measurable safety failures without sacrificing general model capability?
Alignment failures such as deception, sycophancy, and jailbreaks can be measured, but reducing several simultaneously may interfere with a model’s broader capabilities. It is also unclear whether automated researchers can develop effective interventions without extensive human guidance.
Can black-box attackers identify and reconstruct prompts supposedly removed from language models without knowing them in advance?
Machine unlearning may suppress responses to removed data without eliminating signals that reveal what was removed. The difficulty is determining whether an attacker can use those signals to discover and reconstruct forgotten prompts that are initially unknown.
Can chain-of-thought monitoring detect consequential computation hidden in semantically irrelevant filler tokens?
Language models may gain task performance from semantically irrelevant filler tokens without making the relevant computation interpretable in their visible reasoning. This complicates the use of chain-of-thought as evidence of what a model has computed.
Can chain-of-thought monitoring detect preferences received through tools or inferred from raw artifacts?
Chain-of-thought monitoring assumes that a model’s reasoning trace reveals the information influencing its answer. Preferences delivered through tool returns or inferred from unprocessed artifacts may affect answers without being clearly verbalized in the trace.
Can intermediate LLM activations guide faster jailbreak search without weakening attack effectiveness?
Refusal behavior may be represented in transformer activations before the model produces its output. The difficulty is using that signal to reduce the cost of prompt search without losing the effectiveness of the resulting attacks.
Can small targeted grayscale patches force chosen semantics in infrared vision-language models across tasks?
Localized perturbations may cause an infrared multimodal system to produce a selected class, caption, or answer instead of reflecting its input. The extent to which this vulnerability transfers across tasks and model architectures is unclear.
Do more capable language models exhibit stable task preferences that conflict with helpful, honest behavior?
Language models may show consistent choices among tasks rather than merely following explicit instructions. Such dispositions can favor shorter, more agreeable, or self-congruent tasks, potentially making their behavior less helpful or honest in some situations.
Do refusal representations transfer across language-model architectures, and where should safety interventions read them?
Architectures differ in how they mix and update token information, so a refusal signal identified in one model may not be directly usable in another. Safety tooling must determine both whether the representation transfers and where the relevant computation is exposed.
Do source labels bias human and LLM judgments of logical fallacies differently?
Labels about who produced a comment can influence judgments of its credibility and logical quality independently of the argument itself. It remains unclear whether humans and language models are affected by these source cues in the same way.
How can active preference learning obtain scalable, calibrated uncertainty for neural reward models without full Bayesian inference?
Active preference learning must choose which comparisons to request, but reliable uncertainty estimates become expensive for neural reward models when inference considers all parameters. Poorly calibrated uncertainty can lead to less informative queries and inefficient reward learning.
How can adversarial perturbations transfer to unseen semantic-segmentation models while accounting for dense spatial and class-wise structure?
A perturbation crafted on a surrogate segmentation model must mislead an unseen target model. This is difficult because dense prediction depends on spatially organized and class-specific representations, not only output scores.
How can agent decisions be reconstructed for auditing and controlled replay when tool state and authorization context are missing?
An agent’s final output does not reveal the evidence, tool state, authorization, or action path that produced a committed decision. Missing or unobserved execution state makes it difficult to distinguish a faithfully replayed decision from an unexplained divergence.
How can agents make their intentions and internal perspective interpretable to observers?
Observers must infer an agent’s hidden intentions and perspective from its behavior and any explanations it provides. This is difficult when the agent’s internal reasoning is not directly accessible.
How can AI agent harnesses prevent trusted plugin updates from triggering host-privileged attacker commands through lifecycle hooks?
Lifecycle hooks can bind shell commands to routine agent events and execute them with host privileges, including at times the model may not observe. A malicious update can therefore transform benign plugin configuration into host-side behavior without obvious agent involvement.
How can AI agents adapt execution routes as runtime evidence invalidates their planned continuation?
Agents often commit to a multi-step execution route before intermediate outcomes are known. When those outcomes change what should happen next, stale actions or broad replanning can waste computation and discard useful progress.
How can AI augment computational design while preserving domain grounding, verification, and scientific judgment?
AI can expand the search space of computational design, but its contributions may be difficult to ground in domain requirements or verify and reproduce. Researchers therefore need ways to keep AI-assisted design decisions traceable and scientifically accountable.
How can alignment systems infer the multiple criteria behind human pairwise preferences?
Pairwise preference labels record which option people choose, but not the interacting considerations behind that choice. This makes it difficult to build preference models that are both faithful to judgments and interpretable.
How can assistants identify when a request conflicts with implicit user-specific circumstances?
A seemingly reasonable request may become inappropriate because of facts about the user that the request does not mention. The assistant must connect those user-specific facts to the request before determining whether a conflict exists.
How can asynchronous AI agents resist false claims gaining authority simply because they arrive first?
When agents exchange messages asynchronously, the first claim received can shape later answers even when it is wrong. Context and provenance may be lost across handoffs, making verification harder.
How can attention-head contributions be measured in prompt-injection classifiers across circuit and output scales?
Many attention heads can jointly shape a classifier’s logits, while global output behavior can obscure which local circuit components drove the decision. The challenge is to connect fine-grained head behavior with the model’s final classification.
Previous
1 / 10
Next