SynthDocBench is a synthetic long-document benchmark where charts are generated with a paired hidden data manifest, letting the authors independently vary length, page position, modality, and reasoning depth to attribute VLM failures to specific causes rather than an opaque bundle of confounds.
You’ve shipped a doc-Q&A feature that answers questions over 50-page investor reports with embedded charts. Accuracy is mediocre, and you can’t tell whether the bottleneck is chart pixel-reading, cross-page retrieval, or multi-hop reasoning. Real benchmarks like MMLongBench-Doc confound all of these because their questions come from real PDFs where difficulty axes co-vary. SynthDocBench decouples them by building documents from scratch with known ground truth.
The trick is a dual-layer document. Every chart is emitted twice from the same source: once as a rendered D3.js visualization the model sees, and once as a hidden structured metadata block (axes, exact data points, derived insights) the model never sees but the benchmark uses to derive ground-truth answers. That removes the annotation bottleneck and makes chart-value answers deterministic by construction.
A topic seed drives an LLM pipeline that produces a semantic outline, samples a layout archetype (magazine, dashboard, academic, etc.), synthesizes the paired chart-plus-metadata objects, and assembles an HTML report rendered to PDF via Playwright. A second stage walks the manifest to generate three question families: chart-reading, cross-modal (chart plus text from a non-adjacent section), and complex multi-hop (2\u20134 evidence units, difficulty L1\u2013L5).
Evaluation is strictly vision-only: models see rasterized 5-page strips at 144 DPI, never the HTML or manifest. A GPT-5 judge scores each answer 0\u201310 against the deterministic reference; correct means \u22656, aligned with MMLongBench-Doc’s convention.
for topic in seeds:
outline = llm_outline(topic)
layout = sample_archetype(topic)
charts = []
for section in outline:
viz_svg, viz_meta = render_d3_and_metadata(section)
charts.append((viz_svg, viz_meta))
pdf = assemble_html_to_pdf(outline, charts, layout)
qa = generate_qa(manifest=[m for _, m in charts])
images = rasterize(pdf, dpi=144, pages_per_strip=5)
for q in qa:
pred = vlm(images, q.text) # vision only
score = gpt5_judge(pred, q.gold) # 0..10
The prevailing way to stress-test long-context VLMs is to scale up real-document benchmarks and watch aggregate accuracy fall. This paper shows the opposite move is more informative. Build the documents yourself so length, position, modality, and reasoning depth are independent knobs, and the same aggregate drop resolves into three distinct, separable failure modes. The load-bearing evidence is not any single leaderboard number but the fact that a text-only OCR-plus-LLM baseline flips the ranking depending on subset, isolating which failures are truly visual.
The finding that makes the thesis work is the OCR-versus-vision asymmetry. Feed extracted page text to GPT-4o with no images and complex multi-hop ACC jumps to 0.798, versus 0.360 for the same model reading pixels. On chart-reading it reverses: OCR gets 0.297, vision 0.457. So “long-doc reasoning” as usually measured is largely a text-retrieval task in disguise; the genuinely visual bottleneck is chart-value extraction, and even the top model (Gemini 3.1 Pro at 0.759 chart ACC) leaves a 46 pp gap over the OCR chart baseline.
Secondary findings that ride on the controlled axes:
•
Cross-modal is uniformly hardest. Every model scores lowest on the cross-modal subset (Gemini 0.628, Qwen3.5-VL-122B 0.561, Claude Sonnet 4.5 0.250), with a 13\u201316 pp gap versus chart-reading in every topic domain.
•
Positional sensitivity is real and non-uniform. The middle third of a document is hardest for 5 of 8 models on chart-reading. Claude-Sonnet-4.5 declines -11.7 pp monotonically Early\u2192Late; Qwen3.5-VL-122B drops -18.5 pp Early\u2192Middle then partly recovers; Gemini shows a U-shape reminiscent of the Lost in the middle effect.
•
Depth degradation. All models except Gemini fall monotonically L1\u2192L5; Claude drops 23 pp, Gemini stays roughly flat (0.784\u21920.670).
•
Visual hallucination dominates hard failures. On the 109 questions where every model scores \u22643, models return plausible chart values that simply aren’t there, concentrated on dumbbell plots and dense multi-series charts.
•
Ablation sanity. Increasing pages-per-strip from 1 to 10 lifts Gemini overall ACC from 0.369 to 0.792, with cross-modal gaining most, confirming that multi-page context (not just per-page fidelity) is what’s being measured. Judge choice barely matters: GPT-5 and Gemini-as-judge agree within 3.5 ACC points (Pearson r \u2265 0.94).
Reach for this when you’re shipping a system that answers questions over long reports with embedded charts and you want to know why it fails, not just how often. Run your model on the three subsets separately: if chart-reading is fine but cross-modal collapses, you have a retrieval/alignment problem, not a perception problem. Position-bucketed accuracy tells you whether to chunk-and-rerank or trust native long context. The OCR-plus-LLM baseline is worth replicating on your own docs, since it will tell you how much of your “multimodal” pipeline is really doing text work.
Code: ServiceNow/SynthDocBench on GitHub. Dataset: ServiceNow-AI/SynthDocBench on Hugging Face. 200 reports, 1,788 questions, averaging 51 pages and 17 charts per document, across 24 chart types and 6 layout archetypes. The generation pipeline itself is the reusable artifact if you want to synthesize documents matching your own domain.
When a long-context multimodal benchmark saturates or confuses, the fix isn’t harder real documents. It’s synthetic ones with the difficulty axes pulled apart. Only then can “the model got 45%” become “the model can read charts but can’t align them with distant text,” which is the sentence you can actually act on.
•
Rendering-familiarity confound. All charts are D3.js/HTML-rendered, which may favor models whose training distribution overlaps that style. The authors flag this as a possible reason for Gemini’s 13.9 pp lead and recommend future work with alternative rendering backends.
•
Synthetic content is not real content. Tight, unimodal distributions of page count and word count are the point (they enable clean ablations) but mean SynthDocBench doesn’t capture heavy-tailed real-world layouts, scanned pages, or OCR noise. Rankings correlate only moderately with MMLongBench-Doc (Spearman \u03c1 = 0.657).
•
Judge is an LLM. Reference answers are deterministic, but scoring is done by GPT-5. Cross-judge validation with Gemini agrees within 3.5 ACC points; Claude-as-judge is systematically lenient by 11\u201316 pp and was excluded. If a future model games GPT-5’s rubric specifically, the benchmark will need a re-check.