Conformal Relevance replaces per-task hand-written LLM prompts for content selection with a mean ensemble of four In-Context Learning-example retrieval strategies, cutting up to ~50% more irrelevant sentences at the same coverage guarantee across seven NLP tasks.
You’ve shipped a Retrieval-Augmented Generation pipeline or a document-review tool that needs to pull the sentences a downstream LLM (or a human reviewer) actually cares about. Today you probably write a task-specific prompt: “score each sentence 0-1 based on whether it’s PHI” or “…whether it supports the answer.” Every new domain gets a new prompt, and nothing tells you how much relevant content you’re dropping.
The dominant principled fix is Conformal Prediction applied to content selection: calibrate a scoring function on ~100 labeled examples so the returned set is guaranteed to contain, say, 80% of the truly relevant sentences 90% of the time. Prior work like Conformal Importance gives that guarantee, but the scoring function is still a hand-crafted LLM prompt per task. This paper removes the per-task prompt engineering.
The key idea: don’t describe relevance in a prompt, demonstrate it with a few labeled documents and let the LLM infer the criterion. Then build several such scorers that fail on different sentences and average their scores.
Each scorer is an LLM call whose prompt contains k=2 labeled example documents (sentences marked positive or negative) plus the target document to score. What varies across the four scorers is which labeled examples get retrieved from a shared pool of ~50 labeled docs:
•
anchor_dpp: pick the pool doc most similar to the query, then add diverse companions via a Determinantal Point Process.
•
pattern_dpp: for each pool doc, compute the embedding direction from negative-sentence centroid to positive-sentence centroid; pick docs whose relevance directions are maximally different.
•
bm25: classic lexical retrieval over the pool.
•
random: uniform sampling, acts as a regularizer.
The four sentence-level scores are averaged, then fed through standard split-conformal calibration. Averaging is critical: the paper proves that at K=2, the ensemble floor (worst-scoring true positive) beats the better individual scorer exactly when complementarity (how much the two scorers disagree on which positive they under-rate) exceeds the gap between their individual floors. Adding more scorers gives diminishing returns bounded by 1/(K+1).
for x in calibration_set:
scores = []
for sigma_j in [anchor_dpp, pattern_dpp, bm25, random]:
icl_examples = sigma_j(pool, x, k=2)
scores.append(LLM_score(x, icl_examples)) # per-sentence in [0,1]
R_bar = mean(scores) # per-sentence
S[x] = min_over_positives(R_bar) # conformal score
q_hat = quantile(S, alpha)
# At test time: keep sentences with R_bar(c; x_test) >= q_hat
The prevailing move when you want an LLM to score relevance is to write a careful prompt describing what “relevant” means for your task. This paper shows the opposite. Skip the description entirely. Show the model a handful of labeled examples selected four different ways, average the scores, and let disagreement between retrieval strategies do the work that prompt engineering used to do. The load-bearing evidence is that a single fixed configuration beats task-tuned prompts on all seven datasets, and that matched-compute controls (same LLM call budget, temperature-varied ensembling of the hand-written prompt) don’t close the gap.
The key finding is that retrieval-induced diversity, not extra compute or extra labels, drives the gains. When the authors match the LLM call budget by ensembling four temperature-varied calls of the hand-written prompt baseline, Ens4 still wins on 6 of 7 datasets. When they match the ICL budget by giving a single best strategy all 8 examples, Ens4 wins by +0.070 to +0.119 MAP everywhere. When they train a supervised classifier on the full label pool, Ens4 wins on 6 of 7.
•
On the headline metric mean Average Precision, Ens4 beats the hand-written prompt by +12% to +59% across seven datasets (financial, medical, legal, encyclopedic, general QA).
•
No single retrieval strategy wins everywhere. Each of the four (including random) is best on at least one dataset.
•
Empirical coverage stays within 1 percentage point of the target across all datasets, confirming the theory holds.
•
Diminishing returns match the 1/K bound: K=3→4 gains are strictly smaller than K=2→3 on all seven datasets.
•
The theoretical complementarity condition (Comp > floor gap) predicts actual ensemble improvement with >99% agreement on samples where it holds.
•
One failure mode: on PhysioNet PHI Detection, removing the one-line task hint crashes MAP from 0.879 to 0.549, because PHI detection is defined by a regulatory rule that examples alone don’t convey.
Reach for this when you’re building a content-filter stage in front of an LLM (RAG retrieval refinement, evidence extraction, PII redaction, clause selection for legal review) and you want a coverage guarantee without writing a new prompt per task. Budget ~150-440 labeled examples per task (a shared pool of ~50 plus a 100-sample calibration set), pay 4x the LLM cost per document versus a single prompt call (parallelizable), and you get a scoring function that works across tasks with the same config.
Code and prompts are released at github.com/layer6ai-labs/conformal-relevance. The paper evaluates on seven public datasets including HotPotQA, ECTSum, ContractNLI, and Evidence Inference, all reformulated to sentence-level binary relevance with recipes in the appendix. Main results use Gemini-2.5-Flash-Lite; robustness checks on Llama3-8B, Qwen3-8B, and GPT-5.6-terra show the same pattern.
•
The 4x LLM cost per document is real. On long-document tasks like SubSumE the measured serial latency multiplier hit 18x (parallelizable, but you pay in tokens either way).
•
The framework assumes a static relevance criterion between calibration and test. Drift breaks the exchangeability assumption that underwrites the coverage guarantee.
•
Non-semantic tasks whose relevance is defined by external rules (PHI regulations, statistical-evidence conventions) still need a task hint. Pure example-driven inference collapsed MAP by 33 points on PhysioNet.
•
Coverage is marginal, not conditional. If you need equal coverage across intents, document lengths, or protected groups, you need Mondrian Conformal Prediction or similar on top.