Blind-Spots-Bench is a 235-question benchmark of tasks humans find trivial but frontier models flunk (draw a five-legged dog, count chess pieces, repeat a string N times), showing a ~10 pp gap between closed and open-weight models even when they tie on standard benchmarks.
You’re shipping a multimodal assistant that helps users generate marketing images and answer questions about product photos. It nails your eval suite. Then a user asks for “a logo with exactly four stars” and gets five, or asks how many items are in a shelf photo and gets a confident wrong number. These are the failures aggregate benchmarks like MMMU or GPQA don’t surface, because those benchmarks reward broad knowledge and reasoning, not the boring precision skills (counting, character-level control, spatial layout) that break user trust in production.
The authors ran a graduate AI course at EPFL in October 2025 and asked each student to submit five prompts that current frontier chatbots got wrong. That yielded ~287 raw items, which they cleaned down to 235 by removing duplicates, items too hard for humans, and ambiguous phrasings. Each surviving item gets a structured reference solution listing the expected answer, correctness conditions, and known failure modes observed during collection.
They then group items along two axes. Format: text-only (46%), image-generation (36%), and image+text-to-text (18%). Skill taxonomy: three top-level buckets (object-centric, abstract reasoning, language-and-knowledge) subdivided into 12 sub-tasks such as perceptual counting, spatial reasoning, attribute binding, character-level manipulation, and irrelevant-context robustness.
Evaluation is a two-stage LLM-as-judge pipeline built on Inspect AI. A solver model answers with no in-context examples and no chain-of-thought prompting. A grader (gemini-3-flash with code execution enabled) then reads the question, the solver’s answer, and the reference solution, and emits a binary correct/incorrect. The authors hand-audit >100 grader calls and report 96.6% agreement with humans on text outputs and 90.9% on image outputs, and check that the Google-family grader doesn’t inflate scores for Google-family solvers.
for item in bench_235:
answer = solver(item.prompt) # no CoT prompt, no tools by default
verdict = grader( # gemini-3-flash + code exec
question=item.prompt,
response=answer,
reference=item.structured_solution,
) # returns True/False
log(item.task_type, item.subtask, verdict)
# Repeat k=4 times for text tasks, k=1 for image-gen (cost).
They run this over 32 LLMs/VLMs and 6 image-generation models spanning GPT-5 through 5.5, Gemini 2.5 through 3.1, Qwen3/3.5, GLM 4.7\u20135.2, Kimi K2.5/2.6, DeepSeek-V4, Gemma-4, and GPT-OSS.
The prevailing story about frontier progress is that near-saturation on benchmarks like MMMU, GPQA, or SWE-bench means the models are approaching general competence. This paper’s stress test says the opposite: models that look tied on aggregate leaderboards can differ by ~10 pp on trivially-human tasks, and no model wins across all skill types. The load-bearing evidence isn’t the headline leaderboard, it’s the per-subtask breakdown where every evaluated model sits at 30\u201360% on perceptual counting and attribute recognition.
•
Fine-grained visual perception is the shared floor. No model exceeds ~60% on attribute/pattern recognition or perceptual counting. Many sit at 30\u201340% on counting objects in an image. This holds across GPT-5.5, Gemini-3.1-Pro, and every open-weight model.
•
Closed vs open gap ~10 pp even at matched general intelligence. Best closed model (Gemini-3.1-Pro) hits 83.3% on text-only, 66.9% on multimodal. Best open-weight (GLM-5.2) tops out at ~74% text-only. Plotting against Artificial Analysis Intelligence Index scores shows open-weight models sit below the closed-model trend line at the same general-intelligence score, hinting that public benchmarks may overstate open-weight robustness on underrepresented skills.
•
No single winner per subtask. GPT excels at arithmetic reasoning (88.6% on GPT), Kimi-2.6 tops arithmetic at 83%, DeepSeek-V4 leads character-level manipulation at 75%. GPT-5.5 wins text overall (~84%) but drops sharply on multimodal (~58.7%).
•
Scaling within a family isn’t monotonic. Gemma-4-26B trails the tiny Gemma-4-E2B by ~8 pp on irrelevant-context robustness. Qwen3.5-122B drops 10\u201314 pp below Qwen3.5-35B on several abstract-reasoning subtasks; the 397B recovers some but doesn’t beat 35B there.
•
Tools help unevenly. Adding a Python execution environment lifts Gemini-3.1-Flash while hurting GPT-5.4 and Qwen3.5-397B, often because models copy-paste inputs incorrectly into their own code.
•
Cost curves flatten fast. Going from 60% to 70% on text-only costs roughly 10\u00d7 more per sample; DeepSeek-V4, Qwen3.5, and GLM-5.2 sit on a favorable frontier.
Reach for this when you’re picking between two models that tie on your internal eval and you need to know which one will embarrass you first. If your product does image generation with counting constraints, spatial layout requirements, or exact string manipulation, run your candidates through the object-centric and character-level subsets before shipping. The per-subtask heatmaps in the appendix let you match model strengths to your task mix rather than picking on aggregate score.
Artifacts: the 235-item dataset is on Hugging Face under CC-BY-4.0, and the evaluation harness is on GitHub, built on Inspect AI (MIT). Because grading uses code execution to verify hard constraints like character counts, you can drop in new solver models without rewriting verifiers.
Aggregate benchmarks tell you which model is smart on average; blind-spot benchmarks tell you which model will fail in front of a user. Two models that tie on MMMU can be 10 pp apart on “draw exactly four stars,” and that’s the gap your support team will hear about.
•
Only 235 items, unevenly distributed across 12 subtasks; some subtask cells have single-digit sample sizes, so per-subtask deltas are noisy.
•
Questions were adversarially collected against frontier chatbots available in October 2025, so the benchmark is biased toward the specific weaknesses of that snapshot and will drift as models change.
•
No human baseline is reported. The claim that these tasks are “easy for humans” rests on annotator judgment during curation, not measured human accuracy.
•
The grader is gemini-3-flash. Authors check for pro-Google bias and find none in agreement rate, but note higher false-positive rates on non-Google outputs, which would slightly understate the closed-vs-open gap they report.