Rubric4Setwise picks documents for a RAG context by scoring candidate sets against query-specific rubrics covering nine quality dimensions, hitting the best downstream answer quality with ~2.66 docs instead of a fixed top-5 in short-form QA.
You’re building a RAG pipeline. Your retriever returns five documents that all score perfectly on relevance, and your evaluation dashboard is green. The LLM still answers wrong, because three of the five say roughly the same thing, two of them quietly contradict each other, and none of them covers the second hop the question needs. Standard retrieval metrics like nDCG can’t see any of this. They score each document in isolation and add up the scores, treating a set as a bag of independent items. This paper argues that in the RAG era the set is the unit that matters, and builds both a benchmark and a selection method around that idea.
The authors split the contribution into an evaluation piece and a selection piece that share the same machinery.
The evaluation piece is SetwiseEvalKit. For each query plus reference answer, they use frontier LLMs (GPT 5.1, Gemini 3.1-Pro-Preview, DeepSeek-V4-Pro) to generate query-specific rubrics: pointed yes/no evaluation questions like “do the documents jointly identify Pam’s husband as Jim?” that reference concrete entities from the query. Rubrics span nine dimensions grouped into three levels: per-document (relevance, authenticity, quality), inter-document (complementarity, redundancy, conflict), and whole-set (completeness, density, reachability). A judge LLM reads the selected document set and scores each rubric 0-4; averaging per dimension gives a coverage percentage. The benchmark covers a short-form setting built on multi-hop QA datasets like HotPotQA and MuSiQue, and a long-form setting where a search agent (DR.Tulu-8B) runs multi-turn retrieval on ResearchQA.
The selection piece, Rubric4Setwise, notices that the same rubrics that measure set quality can guide set construction. Given a candidate pool, a qwen3-8b backbone reasons about which subset best satisfies the rubrics, adaptively choosing set size rather than always returning top-k.
rubrics = generate_rubrics(query, reference_answer) # ~10-20 yes/no criteria
candidates = retrieve(query, k=20) # BM25 or search API
selected = qwen3_8b.select_subset(
query, candidates, rubrics, # chain-of-thought over rubrics
stop_when="rubrics satisfied" # size not fixed
)
answer = generator(query, selected)
One honest caveat baked into the design: rubrics are generated with access to the reference answer, so Rubric4Setwise is an oracle setting. The authors are explicit that this establishes an upper bound and flag distilling rubric preferences into a reference-free reward as future work.
The prevailing assumption in retrieval evaluation is that set quality equals the sum of per-document relevance scores, which is why nDCG over independently-judged documents has held on into the RAG era. This paper shows the opposite. Once an LLM is the consumer, what matters is how documents interact: whether they complement each other, duplicate each other, or contradict each other, and whether the union supports the full reasoning chain. The load-bearing evidence is that rubric coverage correlates with downstream answer quality at Pearson r=0.92, while sets with perfect relevance still fail on redundancy, conflict, and reachability.
•
The evaluation dimensions themselves separate: doc-level rubric scores predict downstream generation quality strongly (relevance r=0.94, quality r=0.90), but set-level dimensions like complementarity and conflict show near-zero variance across the 12 rerankers tested. All existing methods are homogeneously weak at cross-document coordination.
•
No reranker breaks 45% overall coverage, and none leads in both scenarios: setwise methods (SetR, Rank4Gen) dominate short-form multi-hop QA, while reasoning-enhanced methods (ReasonRank, Rearank) lead long-form research QA.
•
Rubric4Setwise wins downstream generation in both settings. Short-form: EM 26.10 vs 25.13 for the next-best method, using 2.66 documents on average instead of a fixed 5. Long-form: LLM-judge score 70.57, beating the runner-up with fewer documents (20.52 vs 29.23) and fewer search rounds (4.52 vs 4.73).
•
In multi-turn search, all rerankers’ per-round rubric scores degrade monotonically across turns. The authors read this as a shift from broad acquisition to narrow gap-filling, not reranker failure, which motivates their union-scoring protocol.
Reach for this when you’re building a RAG system where context is expensive (long documents, small window, or paid tokens) and where relevance-only reranking still leaves the generator confused. The pattern to steal: at evaluation time, generate a handful of query-specific yes/no criteria that reference concrete entities from the expected answer, then have a judge LLM score whether the retrieved set collectively satisfies them. This catches redundancy and missing-hop failures your nDCG dashboard can’t see.
The artifacts are released: GitHub repo, a project page, and the ~28K-rubric SetwiseEvalKit dataset on Hugging Face. The rubric-generation prompts for all nine dimensions are in the appendix, so you can adapt the recipe to your own domain without retraining anything.
Score the set, not the documents. In RAG, the LLM eats the whole context at once, so redundancy, conflict, and missing hops are first-class failures. A retrieval eval that only measures per-document relevance is measuring the wrong object.
•
Rubric4Setwise generates rubrics using the reference answer, so the reported numbers are an oracle upper bound. In production you’d need a reference-free rubric generator, which the paper leaves as future work.
•
The judge is an LLM (DeepSeek-V4-Pro) scoring rubrics written by other LLMs. The r=0.92 correlation with downstream quality is reassuring but the evaluation loop is entirely model-based, and long-form scores are meaningfully less reproducible across passes than short-form.
•
The selection step calls a reasoning model over the candidate pool per query, so latency and cost are higher than a cross-encoder rerank. The efficiency win is fewer downstream tokens (smaller context, fewer search rounds), not cheaper reranking.