KoNA teaches vision-language models to refuse or correct only the bad part of a mixed question while still answering the good part, closing most of the roughly 50-point accuracy drop that base VLMs suffer when a bad premise is embedded inside an otherwise answerable question.
You’ve shipped a multimodal support assistant. A user uploads a photo of their broken appliance and asks: “The red button on the front is stuck. What’s the serial number on the back label?” The photo shows a blue button, and the back label isn’t visible. What should the assistant do? Ideally: correct the color, admit it can’t read the label, and stop there. In practice, today’s VLMs do one of three wrong things. They confabulate a serial number, they silently accept the “red button” premise, or they refuse the whole request.
Most prior work on model refusal treats a query as one atomic thing: comply, or don’t. That framing breaks the moment real users mix a valid ask with a false premise, an unsafe subtask, or something that isn’t visible. The Visual Question Answering literature has long known models hallucinate under bad premises; this paper argues the harder skill is doing surgery on a compound query.
The contribution has two pieces: a benchmark and a training recipe.
The benchmark, KoNA, defines five reasons a VLM should withhold compliance on some part of a question: False Premise (the question asserts something the image contradicts), Visual Inaccessibility (occlusion, blur, or lighting makes the answer unreadable), Universal Unknown (the answer isn’t derivable from any image, like a stranger’s intent), Task Feasibility (the model literally can’t do it, e.g. perform a physical action), and Safety (the request is harmful). For each image, KoNA builds three paired queries: a single query that triggers one non-compliance reason, a compound query that mixes that reason with an answerable component, and a fully-answerable control. Pairing is the key design choice. It lets you measure whether a model that correctly refuses in isolation still refuses selectively when the trigger is hidden inside a longer ask. The set totals 9,300 QA pairs over 3,100 images from MS COCO and Open Images V7, generated by GPT-5 and Gemini-2.5-Flash and human-verified on the test split.
The training recipe fine-tunes small open VLMs (InternVL3-2B, Qwen2.5-VL-3B) in two stages. First, supervised fine-tuning on a mix of compound non-compliance examples and fully-answerable examples, so the model learns both when to refuse a component and when to just answer. Second, Group Relative Policy Optimization (GRPO) with GPT-5-mini as an LLM judge providing the reward. The reward is deliberately asymmetric: full credit only if the model correctly handled the non-compliant component, then a small penalty (λ=0.3) if the answerable component was factually wrong.
# per training example (compound query)
resp = model(image, compound_query)
r_non = judge_component_noncompliance(resp) # PASS/FAIL
r_fac = judge_factuality(resp) # PASS/FAIL
if r_non == "PASS":
reward = 1.0 - 0.3 * (r_fac == "FAIL")
else:
reward = 0.0 # no partial credit for getting only the easy half
The asymmetry matters: getting the refusal wrong zeros out the reward, so the model can’t game the objective by always answering fluently and ignoring the trap.
The prevailing fix for VLM over-compliance is either safety-tuning the whole response (refuse the query) or better prompting (“think step by step”). This paper shows the opposite. Alignment for real queries is component-level, not query-level. A model needs to answer and refuse in the same response, and this skill has to be trained explicitly because prompting alone doesn’t produce it. The load-bearing evidence is that inference-time tricks close very little of the single-vs-compound gap, while training on paired compound examples closes almost all of it.
The single most load-bearing result is the size of the single-vs-compound gap in base models and how prompting fails to close it. On the strongest base configuration (Qwen2.5-VL-72B with Behavior Guidance prompting), single-query accuracy is 0.87 but compound-query accuracy is 0.79, and for weaker models the gap is much larger: GPT-5 default drops from 0.55 single to 0.42 compound, InternVL3-2B from 0.25 to 0.10. Chain-of-thought produces “modest and inconsistent changes” and Behavior Guidance helps mainly on Safety and Task Feasibility, categories with stereotyped refusal patterns.
Against this, KoNA fine-tuning is the finding that proves the thesis. The tuned Qwen2.5-VL-3B goes from 0.29 / 0.11 (single / compound, default) to 0.91 / 0.87, and InternVL3-2B goes from 0.25 / 0.10 to 0.92 / 0.90. The single-compound gap essentially closes. Answerable-set accuracy dips slightly (Qwen 0.73 → 0.71, InternVL 0.77 → 0.70), and factuality on answerable components actually improves (0.84 → 0.89).
The ablation isolates why: SFT on compound examples only (no answerable set) crashes answerable accuracy to 0.38–0.53 as the model over-refuses everything. Adding the answerable set restores it. GRPO on top further lifts answerable accuracy while preserving non-compliance gains. General-capability benchmarks (TextVQA, MMBench, POPE) stay flat, and over-refusal on the MOSSBench safe-but-scary-looking probe stays near baseline (0.04–0.05 refusal rate). The behavior transfers to four external benchmarks reformatted into compound form (HaloQuest, MM-SafetyBench, R-Bench, Unsolvable Problem Detection) and to human-originated VizWiz-VQA questions the pipeline never rewrote.
Reach for this when you’re shipping any multimodal assistant that fields real user questions: a visual support bot, an accessibility app, a shopping assistant. Users routinely ask questions that mix a valid request with an unverifiable premise, an unreadable detail, or an unsafe subtask. The dominant fix today is a safety filter that either passes the whole query or blocks it. KoNA’s recipe suggests a cheaper alternative: fine-tune a small open VLM on ~1,200 paired examples spanning your five failure modes, keeping ~10% fully-answerable data as an anchor, then run a short GRPO pass with an LLM-judge reward that gives zero credit unless the model handled the tricky component. The paper does this with 3B-scale models and gets closed-source-competitive selective behavior.
Code and dataset are at github.com/mz-kim/KoNA. The 9,300-pair dataset is directly usable as an eval suite for any VLM you already ship; the generation prompts and filtering prompts are documented in the appendix, so you can extend the pipeline to your own image domain (product photos, medical images) rather than MS COCO scenes.
Refusal is a per-component skill, not a per-query verdict, and prompting won’t teach it. If your product ever sees questions that are half-valid, you have to train for the mixed case explicitly, with both the refuse-this-part and the answer-normally examples in the same batch. Otherwise the model will either hallucinate the bad half or refuse the good half.
•
Both the training labels and the GRPO reward come from LLM judges (GPT-5-mini agrees with humans ~95% of the time, but that’s still a shared-bias risk with GPT-5 as one of the generators). The whole loop could be learning the judge’s preferences more than “correct” behavior.
•
Evaluated only on ≤3B open VLMs and two closed ones; no test that this recipe scales up cleanly, and no comparison to alternatives like DPO or constitutional-AI-style multimodal alignment.
•
Fine-tuned models still show three residual failure modes the authors highlight: over-refusing informational follow-ups, dropping answerable sub-questions while refusing others, and generating confidently-wrong corrections to false premises. The mechanism teaches when to refuse, not how to ground the correction.