Get Started
Home
Topics
Search
Library
Research questionHow many attention heads are necessary for Boolean computation in one-layer attention-only models, even with unlimited dimension and precision?A one-layer attention-only model may gain expressive power from additional heads, but it is unclear whether very large embeddings or precise numerical values can compensate for too few heads. The key issue is isolating the computational role of head count from those other resources.
AI
Machine Learning
Research Paper
Technology
Latest papersRecent research connected to this question, newest first.The Head Complexity of Boolean Functions in Single-Layer AttentionThe supplied results concern Boolean functions in a one-layer attention-only model. They establish an exact hierarchy for parity, compactness bounds for dimension and precision, and near-matching general bounds: 2^n heads suffice for every n-bit function, while almost all such functions require Ω(2^n/n^2) heads.research paper · Sep 3, 2026
Related questions
How can attention-head contributions be measured in prompt-injection classifiers across circuit and output scales?How can FP4 attention exploit Blackwell tensor cores when softmax overhead dominates?How can analog compute-in-memory attention perform softmax without costly analog-to-digital conversion?How can associative-memory capacity be compared across Hopfield and attention-like models without conflating assumptions?