PaceMaker handles user requests that look fine on the surface but conflict with hidden personal facts by planning likely conflict angles, expanding a k-NN document graph via multi-hop search, then filtering for decisive evidence, lifting conflict-case pass rates by roughly 4–11 points over strong retrieval baselines.
Imagine you’re shipping a personal assistant that books restaurants, schedules meetings, or files expenses on a user’s behalf. A user says “book the seafood place downtown for Saturday.” Sounds trivial. But their calendar has a conflicting flight, or their dining companion has a shellfish allergy noted three months ago in a chat log. Today’s Retrieval-Augmented Generation pipelines retrieve documents that look semantically similar to the query. Nothing about “book seafood restaurant” retrieves “friend allergic to shrimp,” so your assistant confidently books the reservation and creates a mess.
Most safety and personalization benchmarks assume the risky signal is in the request (“help me build a bomb”) or explicitly labeled in a preference field. The PACE paper argues the harder, more realistic case is when the blocker is buried as one atomic fact among thousands in a personal knowledge base, and it doesn’t share vocabulary with what the user just asked.
The paper contributes two things: a dataset (PACE) and a retrieval agent (PaceMaker).
PACE is a synthetic benchmark of ~3,249 queries across 185 personas. Each persona has an Egocentric Knowledge Base of about 2,035 atomic facts. Queries are written to look ordinary; whether they’re actually feasible depends on ~4 gold facts scattered across the KB. Labels come in three flavors: Temporal (schedule clashes), Personal (health, preferences, alter constraints), and State (external conditions like closures). Roughly half are true conflicts, half are compatible, so a model can’t just refuse everything.
PaceMaker is a training-free multi-agent retrieval pipeline. The key move: don’t just embed the query and search. First ask an LLM planner what dimensions of conflict this request might have (timing? companion needs? venue state?), and generate counter-view queries targeting those hidden angles. Then run hybrid dense + BM25 retrieval on all views, fuse ranks with weighted reciprocal-rank fusion, walk a pre-built k-Nearest Neighbor Document Graph over documents to reach evidence that wasn’t lexically close to the query, and finally have another agent filter the pool down to the most decision-relevant facts.
plan = conflict_planner(query, ref_date) # up to 3 conflict angles
views = [query] + counter_view_generator(plan) # original + counter views
hits = hybrid_retrieve(views, dense_index, bm25) # weighted RRF
seeds = pre_hop_filter(hits, n=10)
pool = bfs_graph_traversal(seeds, knn_graph, depth=5, neighbors=3)
evidence = post_hop_filter(pool, n=10)
answer = generate(query, evidence)
The planner and filters are LLM calls; indexing uses no LLM calls, which matters for personal KBs that get refreshed often.
The prevailing approach in personalized Retrieval-Augmented Generation is to retrieve documents that look semantically close to the user’s request and trust that relevance equals decision-usefulness. This paper shows the opposite. The evidence that determines whether a request is safe to execute is often the evidence least similar to the request itself, because a well-formed query hides exactly the constraint it violates. You have to actively hypothesize what could go wrong and search for the blockers, not for the topic. The load-bearing evidence for this thesis is the traversal ablation: killing the multi-hop graph walk hurts conflict-case accuracy more than killing either the query planner or the final filter.
The most telling result is the gap between the Oracle setting (model gets only the gold facts) and Full KB (model gets everything). Even with the entire KB in context, GPT-5.4-mini scores 73.10% pass vs 87.13% with only gold facts. More context isn’t the answer; the model gets distracted by plausible irrelevancies. That gap is what makes retrieval quality the actual bottleneck.
Secondary findings anchoring that story:
•
On conflict queries specifically, PaceMaker beats the strongest non-oracle baseline by +11.40, +3.47, and +4.02 points across the Qwen, GPT, and Gemini configurations, while staying comparable on non-conflict queries. That asymmetry is the point: it catches hidden problems without over-refusing.
•
Ablations: removing multi-hop traversal drops overall pass from 75.35% to 71.65% and conflict pass from 59.17% to 52.41%. Removing query planning is nearly as damaging. Removing the final selection filter barely moves the needle.
•
Compared to structured retrieval baselines GraphRAG and HippoRAG 2, PaceMaker’s conflict-pass advantage is much larger (54.42% vs 43.57% and 34.98%) than its overall-pass advantage, meaning existing graph-RAG methods retrieve topically related stuff but miss the decisive blockers.
•
Temporal conflicts are the hardest category across all configurations, because the required facts (routines, travel times, prior commitments) are distributed and individually unremarkable.
Reach for this when you’re building a personal assistant, scheduling agent, or booking flow that runs against a large per-user memory store, and the failure mode you fear is confident execution of a request that violates something the user already told you months ago. The pattern to steal is the counter-view planner: before retrieval, have an LLM enumerate 2–3 ways the request could conflict with the user’s state, generate retrieval queries for those angles, and merge results with higher weight on the counter views. This is cheap, model-agnostic, requires no training, and adds only a handful of LLM calls per query.
Code and dataset are released at GitHub. The dataset is fully synthetic (personas seeded from MSC and Synthetic-Person-Chat, generated with GPT-5.4-mini, human-validated at 93.3% agreement on feasibility labels), so it’s usable as a drop-in eval for any assistant that consumes a personal KB. Note that the benchmark judges feasibility decisions, not end-to-end task execution, so it won’t tell you whether your downstream scheduling or booking logic actually works.
When retrieval decides whether an action is safe to take, search for the blockers, not the topic. A query about booking dinner should trigger retrieval for allergies, calendar conflicts, and venue closures, not for other dinner bookings. Semantic similarity is a decent default for question-answering; it’s the wrong default for personal-assistant execution.
•
The dataset is entirely LLM-generated and validated by another LLM plus crowdworkers. Real personal KBs are messier: contradictory, stale, gossipy, full of ambiguity that a synthetic pipeline smooths over.
•
The benchmark isolates a binary feasibility judgment. It doesn’t measure whether the assistant proposes a good alternative, negotiates with the user, or handles borderline cases where a human would just ask a clarifying question.
•
PaceMaker adds four LLM calls per query and a cold-start indexing cost. For a chatty assistant handling thousands of small requests per user per day, that latency and token bill is not free, and the paper’s cost analysis is on a single 2,056-fact instance.