OmniVChat trains and evaluates omni models that take raw audio-visual input and reply in text, using a multi-agent synthesis pipeline to build both a rubric-scored benchmark and an RL training set that lifts held-out rubric scores on real human recordings from 0.402 to 0.632 without training on them.
Imagine you’re building a voice-and-camera assistant on a phone. The user points the camera at a mug and asks “what pattern is on it?” The obvious pipeline runs speech recognition, sends the transcript plus a video caption to an LLM, and speaks the answer back. That cascade loses tone of voice, adds latency, and can’t ground the reply in what the user is looking at without a separate captioner.
Native audio-visual dialogue means the model consumes the raw audio and video streams together and writes a reply directly. Two things block research on this: nobody has a large corpus of people actually using their devices this way, and even if you did, grading a reply that has to notice the user’s environment, gesture, and emotion can’t be done by keyword matching. The authors’ response is to synthesize the dialogues and grade with tiered rubrics checked by an LLM judge.
The contribution has three pieces that share one data format.
OmniVChat-Studio is a multi-agent pipeline that turns a text corpus into audio-visual dialogue clips with reference replies and rubrics. Four agents play fixed roles: a Director writes and reviews text, a Renderer generates the actual audio-visual clip, a Reviewer captions the rendered clip and answers targeted questions about it, and a deterministic Validator checks format rules. Each dialogue subcategory (say, “user pauses mid-question”) only needs custom prompts; the framework skeleton stays the same. A key design choice: observation is separated from judgment. The Reviewer describes what actually got rendered, and the reply is grounded in that description, not in the original script, because video generation drifts.
Tiered rubrics replace keyword matching. Each instance has criteria stacked in tiers. Tier 0 checks language and acts as a gate: fail it, score zero. A criterion in a later tier only earns its point if every earlier tier is fully satisfied. The score is credited points over total non-gate criteria. In plain terms: the model has to get the basics right before detail-level credit counts.
OmniVChat-RL fine-tunes Qwen3-Omni-Instruct with GSPO using LoRA adapters. The reward has four parts: rubric correctness (same scoring as evaluation), a format check, a within-group efficiency term (rubric credit per word, ranked across the sampled replies for one prompt), and a style term where a judge LLM checks seven criteria for natural conversational tone. Efficiency is compared only within the group of replies to the same input, so there’s no fixed length target.
for step in range(1000):
prompts = sample_batch(train_set, n=32)
for x in prompts:
replies = policy_old.sample(x, n=4)
r_correct = [grade_rubric(y, x.rubric) for y in replies]
r_style = [grade_style(y) for y in replies]
r_eff = rank_within_group(r_correct, lengths=[len(y) for y in replies])
R = [c + 0.5*f + 0.1*e + 0.5*s for c,f,e,s in zip(...)]
advantages = [Ri - mean(R) for Ri in R] # GRPO-style group baseline
update_lora(policy, advantages, clip=(1-eps_lo, 1+eps_hi))
On the synthesized OmniVChat-Bench (2,800 instances, 17 subcategories), training lifts Qwen3-Omni-Instruct from 0.465 to 0.652 subcategory-mean rubric score. On OmniVChat-Bench-Human, 360 Chinese recordings of real people that the model never saw during training, the score rises from 0.402 to 0.632. Mean reply length falls from 91.5 to 36.6 words, and rubric credit per thousand words rises from 3.37 to 17.18. The paper reads the matched gains on synthetic and recorded sets as evidence that the synthetic distribution is close enough to real dialogue for the training signal to transfer.
Against twelve released systems under one shared system prompt, Gemini-3.5-Flash leads the synthetic mean at 0.667 and Gemini-3.7-Flash leads the human-recorded mean at 0.575. The trained model beats every open-source baseline on the human set and ties or leads the closed-source frontier. The authors flag that the synthetic and recorded leaders differ, so synthetic rank does not fully determine real-world rank.
The reward ablations are the most informative result. Removing the efficiency term raises the synthetic mean to 0.697 and the recorded mean to 0.691 but blows reply length from 36 to 99 words and cuts efficiency from 18.38 to 7.12. Removing the style term drops the style pass rate from 0.992 to 0.788 with rubric scores mostly unchanged. So the three reward components genuinely trade off, and the full recipe is a compromise, not a Pareto win on rubric accuracy.
A judge-sensitivity check regrades one fixed Gemini-3.7-Flash reply set with six different judges. Pooled scores range from 0.643 to 0.678, a spread about five times the bootstrap noise level, so different judges assign systematically different strictness even though their subcategory profiles correlate above 0.93. This means absolute rubric numbers depend on which judge you use; comparisons within one judge are more trustworthy than cross-paper comparisons.
If you’re evaluating an omni model on live audio-visual dialogue, the tiered rubric idea is worth borrowing directly. The gate on language plus a stacked order of criteria prevents a reply from getting partial credit for florid style when it missed the basic fact, which is a common failure of pointwise LLM-as-judge setups.
If you’re doing RL on open-ended replies, the within-group efficiency reward is a clean way to push brevity without picking a length target. Because it’s ranked within the four replies to one prompt, the group-mean baseline in Group Relative Policy Optimization (GRPO)-style advantage estimation cancels out any constant bias. The ablation shows this specific term is what compresses replies from 91 to 36 words; if you don’t include it, expect verbosity to grow as rubric quality grows.
The transfer result (synthetic training, recorded evaluation) is real but narrow. The recorded probe is 360 Chinese single-turn clips with no live interruption and no multi-turn dependencies. Extending this claim to English recordings, multi-turn dialogue, or true streaming with barge-in is worth testing before assuming it holds. The paper is explicit that the recorded set does not test live turn-taking latency.
Code and the benchmark are promised for release; the paper does not link a repository in the supplied text.
Everything in training and most of evaluation is model-generated. The video renderer, the caption Reviewer, the rubric grader, and the style grader are all LLMs or foundation models, and several are from the same family (qwen3.7-max grades rubrics during both training and evaluation, and the base model is also Qwen). The authors partly address this with the six-judge sensitivity study, but training-grader and evaluation-grader alignment is not independently controlled.
The multi-turn extensions were not verified on human recordings; the human probe is single-turn only. Rubric scores across the development set, synthetic benchmark, and human benchmark are noted by the authors as “not directly comparable,” so read the 0.402→0.632 number as a within-set improvement, not an absolute quality measure.
The reward-weight choice (correctness 1.0, format 0.5, efficiency 0.1, style 0.5) is presented without a sweep, and the ablations show it sits at a real trade-off point rather than a global optimum on rubric accuracy. If your application cares more about correctness than brevity, the “no efficiency term” variant is stronger on both benchmarks.