Ovis-Embedding turns a pretrained omni-modal understanding model into a retrieval encoder by pooling the last token of a shared Qwen-Omni backbone, so text, image, video, and audio queries land in one comparable vector space without bolt-on modality adapters.
If you’ve built a search index for a support agent, you’ve probably shipped separate encoders: a text embedder for tickets, CLIP for images, maybe a speech encoder for call recordings. When a user shows up with an audio clip plus a screenshot and asks “which manual page matches this?”, you can’t just cosine-compare the vectors. They come from different geometries. Your workarounds (routing to modality-specific indexes, or projecting everything through a text bridge) leak accuracy.
The standard fix in the literature is to take a strong vision-language embedder and retrofit an audio pathway onto it: align an audio encoder to the existing text-image space through contrastive training. The paper’s complaint is that audio then lives in a geometry designed without it, which caps cross-modal precision. The alternative Ovis pursues: start from a model where all four modalities were already fused during pretraining, and adapt that for retrieval.
The contribution is mostly a what to start from and a training recipe, not a new architecture. The team takes Qwen2.5-Omni-3B (their omni variant) or Qwen3.5 (their VL variants), strips the speech-generation head, and uses the final hidden state at the last non-padding token as the embedding. No projection head, no modality-specific output. The embedding dimension is whatever the backbone’s hidden size is (2048 or 4096).
Training runs in four stages:
1.
Low-rank contrastive pretraining. Standard InfoNCE over a candidate pool gathered across all data-parallel workers, so every query sees cross-modal negatives. Two twists: a difficulty-aware focal reweighting that concentrates gradient on queries whose positive isn’t yet well-separated, and an Embedding Distillation loss that matches the full similarity distribution of a stronger teacher over the candidate set (forward KL). The teacher’s ranking is precomputed offline. Crucially, they use LoRA here, not full fine-tuning. Their observation: when embeddings are still “chaotic,” full-parameter updates thrash the pretrained semantics before anything stabilizes. Low-rank updates act as a stabilizing prior.
2.
Full-parameter homogeneous finetuning. Now unfreeze everything, but change the sampling: each micro-batch is drawn from a single dataset, and candidates are deduplicated by hash. This kills modality shortcuts (the model can’t tell positives from negatives by “this one is audio, that one is text”) and forces fine-grained discrimination within a task.
3.
Annealing embedding distillation. Filter the training data to examples the teacher solves, upsample the ones the student still misses, and apply a per-query KL weight that scales with student uncertainty. Confident queries get weak teacher supervision, hard queries get strong teacher supervision.
4.
Elastic embedding inference. Instead of MRL, which they say hurts full-width quality when trained jointly, they attach a post-hoc adapter: a shared PCA rotation fitted on a modality-balanced mix of candidates, plus a zero-initialized linear residual per target width. Truncating the rotated vector to its first d coordinates gives a compact embedding at widths {128, 256, 512, 1024, 2048}. Because everything is linear, one width folds into one matrix multiply at serve time.
# Stage-1 per-batch loss (simplified)
e_q, e_c = encoder(queries), encoder(candidates) # last-token pool, L2-normed
sims = e_q @ e_c.T / tau # NxN(1+K) similarity matrix
pi = softmax(sims, dim=-1)[:, positive_idx] # per-query positive prob
focal_w = stop_grad((1 - pi) ** gamma)
focal_w = focal_w / focal_w.mean() # unit-mean normalize
L_focal = -(focal_w * log(pi)).mean()
L_dist = kl_div(teacher_dist, softmax(sims)) # teacher precomputed offline
loss = lam * L_focal + lam * L_dist
Headline results are on MMEB-v3, a 190-task suite spanning image, video, visual-document, text, audio, and agent retrieval. Ovis-Embedding-Omni-3B scores 58.46 overall, versus 53.27 for the next-best baseline (Tianmu-Emb-Uni, 8B), and ranks first on the aggregate of every one of the six modality groups. The audio margin is the largest: +7.04 points over the runner-up. It also leads MAEB (audio-focused, 30 tasks) and MVEB (video-focused, 23 tasks), and on the text-only RTEB it edges out Qwen3-Embedding-4B (67.35 vs 67.27), a model dedicated to text.
A few things worth flagging about what these numbers do and don’t show:
•
The audio gain is where the “native omni backbone” story is most persuasive, because that’s the modality the baselines had to retrofit. The paper argues, but does not causally isolate, that starting from Qwen-Omni (rather than adding an audio branch) is what produces it. There’s no ablation swapping the backbone while holding the recipe fixed.
•
The elastic-embedding table is a genuine ablation. Cutting from 2048 to 128 dims (16x reduction) retains 93.2% of average quality, versus 85.8% for naive truncation of the same encoder. The PCA rotation carries most of the gain at 512+; the residual adapter carries more of it at 128. VisDoc and Agent suffer most from compression (roughly 7 and 6 points at d=128), which the authors read as those tasks needing lower-variance directions that short prefixes discard.
•
Not everything wins. On MultiConIR (text queries with multiple simultaneous constraints), Omni-3B trails by ~8 points. Memory retrieval in the agent group is also a second-place finish.
•
Footnote 1 of the paper says training-set sizes are placeholder estimates for the current data freeze. Take the “~50M pairs” figure and any per-modality data claims as provisional.
•
If you’re building any-to-any retrieval and today you’re stitching together separate encoders, the case for a single omni backbone is now stronger. The Omni-3B and VL checkpoints are promised open-source; try them on your own query mix before deciding whether to keep modality-specific towers.
•
The homogeneous-source sampling trick is orthogonal to the rest of the paper and easy to steal. If you contrastively finetune an embedder and your batches mix datasets/modalities, in-batch negatives can be trivially rejectable by surface cues. Restricting each micro-batch to one source, plus hash-dedup of candidates, is a small change with reported broad gains. Worth testing on your own multi-task setup.
•
The elastic-embedding recipe (fit PCA on a modality-balanced candidate mix, then train a zero-init linear residual per target width against a similarity-preservation objective) is applicable to any frozen encoder you already have. You do not need to retrain the encoder to get variable-width embeddings; the residual is unsupervised and fits on candidates alone. If your index storage is the bottleneck and you can tolerate ~7% quality loss at 16x compression, this is a cheap experiment.
•
If your retrieval workload is document-localization or agent-state disambiguation (VisDoc, Agent in their taxonomy), be more cautious with aggressive compression. The paper’s own results show these degrade fastest.
•
If you need multi-condition text retrieval specifically, this model is not the pick based on the MultiConIR result. That’s a modality-of-text weakness rather than an omni-modal one.
•
The training-corpus statistics are explicitly marked as placeholders for the camera-ready. Any specific data-mix number in the paper should be treated as tentative.
•
No ablation isolates the contribution of the Qwen-Omni initialization from the training recipe. The paper’s central claim (“native beats retrofit”) is supported by end-to-end leaderboard wins, not by a controlled swap.
•
Baselines are evaluated with their released checkpoints and default settings, so comparisons reflect out-of-the-box quality, not what a well-tuned retrofit approach could reach.
•
The elastic-embedding retention numbers are averages weighted by MMEB-v3 task counts. Per-suite drops at d=128 range from under 1 point (audio, video) to nearly 7 (VisDoc, Agent). Don’t assume the aggregate retention applies to your workload without checking the relevant suite.
•
The paper’s teacher for distillation is described only functionally (“a stronger teacher,” “complementary experts”). Which specific models were used as teachers per modality is not spelled out in the excerpts here, so reproducing Stage-1 exactly requires details the paper does not fully specify.