Canopy compresses retrieved text, tables, and videos for Retrieval-Augmented Generation by building a region hierarchy per item and keeping any child whose learned similarity to the query matches its parent’s, cutting reader-input evidence tokens by 14\u201328% at comparable accuracy versus the same pipeline without compression.
You’ve built a question-answering system that retrieves from a mixed pile of Wikipedia passages, tables, images, and video segments, then stuffs the top-k results into a vision-language model’s prompt. Two problems bite. Passing whole items wastes context on irrelevant sentences, rows, or frames, and long distractor text is known to hurt answer quality. But chopping items down uniformly, say to one sentence or one row, strips away surrounding context that the reader needs to interpret the hit.
Prior compressors handle this one modality at a time. RECOMP extracts or summarizes sentences, LongLLMLingua prunes tokens by a small LM’s probabilities, TableRAG picks table cells, VideoRAG picks frames. Each has its own unit, its own scoring trick, and its own budget knob. The paper’s framing: there isn’t a shared recipe that says, item by item and region by region, how much to keep when the retrieved pile is heterogeneous. That’s the gap Canopy targets, and the authors argue it’s a post-retrieval problem distinct from choosing a retrieval granularity up front (as UniversalRAG does by routing queries to paragraph vs document corpora, clip vs full video).
The core move is to treat each retrieved item as a tree of its own original regions. For text, leaves are sentences and higher nodes are groups of sentences up to the whole passage. For tables, leaves are rows (headers kept at every level). For videos, leaves are 30-second clips. Images stay as a single node. Nothing is summarized or rewritten; nodes just point at spans of the original item.
For each node they compute cosine similarity between the query embedding and a node embedding. The query encoder is frozen (Qwen3-VL-Embedding-2B). The node encoder starts from the same model but is fine-tuned with a LoRA adapter so that parent and child scores can be compared to each other. The fine-tuning signal is built from gold-evidence annotations: at each node that overlaps gold evidence, children that overlap gold should score at least as high as the parent, children that don’t should score lower. A standard pairwise ranking loss enforces those preferences.
At inference time there is no LLM call to decide what to prune. The procedure, called parent-relative refinement, is: start at the root; if any child scores \u2265 its parent, descend into every such child and throw away the others; if no child qualifies, stop and keep the parent whole. Different branches stop at different depths, so one table can yield, say, two specific rows plus one intact section, with no fixed “keep N units” budget.
Compression alone can’t fix missing evidence, which matters for multi-hop questions where you only know what to look up next after reading hop one. So Canopy wraps the compressor in a loop: an LLM critic reads the accumulated compressed evidence against the original question, and either answers or writes a targeted follow-up query. New items get compressed before joining the pile. The loop stops when the critic is satisfied or after R=3 rounds. One extra wrinkle: for aggregate questions over tables (“how many\u2026”, “average\u2026”), an LLM classifier flags the question once and the table is kept whole, since row-level pruning would break aggregation.
def canopy_item(query, item):
tree = build_hierarchy(item) # sentences / rows / 30s clips
selected = {tree.root}
for v in internal_nodes_bfs(selected):
q_v = cos(f0(query), f_theta(v))
kids = [(u, cos(f0(query), f_theta(u))) for u in v.children]
qualifying = [u for u, s in kids if s >= q_v]
if qualifying:
selected.remove(v)
selected.update(qualifying) # recurse into each
# else: keep v whole, stop this branch
return spans_covered_by(selected)
They evaluate on five QA benchmarks mixed into a ~33M-item corpus: NQ (single-hop text), HotpotQA (multi-hop text), OTT-QA (tables + text), MMQA (text + tables + images), and LVBench (long video). Reader is Qwen3-VL-8B-Instruct or InternVL3.5-8B. Baselines include Vanilla RAG, UniversalRAG, IRCoT, plus per-modality compressors.
•
Headline accuracy. With Qwen3-VL-8B-Instruct, Canopy has the best average EM/accuracy across the five benchmarks and is top on the four non-video ones. Its gain over IRCoT on HotpotQA (+0.3 F1) is not statistically significant by the authors’ CI.
•
Where the accuracy actually comes from. An ablation turns the pieces off independently. Removing iterative retrieval drops HotpotQA EM by 10.1 and OTT-QA EM by 14.9. Removing the hierarchical compressor barely moves accuracy (all five benchmark CIs include zero). The authors’ reading: the critic-loop drives multi-hop gains, and compression is doing a token-saving job, not an accuracy job.
•
Token savings from compression. In the unrouted Qwen3-VL-8B-Instruct setting, hierarchical compression cuts reader-input evidence tokens by 14.2\u201327.7% vs the same iterative pipeline that passes whole items, at essentially unchanged answer quality.
•
Cross-modality baselines. Against LongLLMLingua and RECOMP on their native text benchmarks, Canopy matches or beats them at similar budgets. S2G-RAG reaches comparable F1 with fewer evidence tokens but spends 5.6 LLM calls per question vs 2.8 for Canopy, because Canopy’s node-level pruning is pure embedding comparison.
•
Encoder fine-tuning matters for the mechanism. Correct parent\u2013child decisions rise from 28\u201343% (frozen encoder) to 34\u201372% (fine-tuned), and the share of items where some content is pruned jumps from ~30\u201340% to ~85\u201396%.
•
Doesn’t win everywhere. On LVBench with InternVL3.5-8B, routed Canopy actually underperforms UniversalRAG (34.0 vs 37.2 accuracy) while using more tokens. Routing helps only on video; elsewhere unrouted Canopy is better.
•
If you run a multimodal RAG stack and your reader prompt is bloated with full retrieved items, Canopy’s mechanism is a plausible drop-in: fine-tune a region encoder with LoRA on gold evidence (if you have it), then prune at serve time with cosine comparisons only. The paper shows this cuts tokens ~15\u201328% at matched accuracy on five QA benchmarks. You need gold spans to train the encoder; the authors used ~4k questions pooled across benchmarks.
•
If your pain is multi-hop questions missing a second fact, the ablation suggests the win is the critic loop, not the compressor. A simpler iterative-retrieval baseline with a sufficiency-checking LLM might capture most of the accuracy gain without the hierarchy machinery. Worth A/B-ing.
•
For long-video retrieval, the paper’s own results are mixed: on LVBench, modality routing matters more than Canopy’s region pruning, and with one reader Canopy loses to the UniversalRAG routing baseline. Don’t generalize “Canopy wins” to video without testing.
•
The compressor skips LLM calls for node pruning, but still spends LLM calls on the critic, the aggregate-question classifier for tables, and (optionally) a modality router. If your bottleneck is LLM calls per query rather than prompt tokens, the savings story is weaker.
•
The headline 14.2\u201327.7% token reduction is measured in the unrouted Qwen3-VL-8B-Instruct setting against the same pipeline minus compression. In the routed LVBench setting with a different reader, compression costs 1.8 accuracy points for a 33% token saving. The trade-off is dataset- and setup-specific.
•
Fine-tuning the node encoder needs localized gold evidence, which many practical corpora don’t have. The authors drop questions whose gold can’t be mapped. How well the method works with weaker or synthetic supervision isn’t tested here.
•
Accuracy gains on multi-hop benchmarks come from the critic loop, not from the compression mechanism that is the paper’s named contribution. Several key compression-only effects have 95% CIs that include zero.
•
The aggregate-vs-lookup table classifier is an LLM call per table-containing question and is a point failure: if it mislabels an aggregate question as lookup, row pruning can delete rows the answer depends on.
•
Gold-fragment recall after refinement averages ~79%, so compression does drop some gold spans; the pipeline tolerates this because surrounding retained context often still supports the answer, but it’s not lossless.