ViQ turns images into a small set of discrete codes (like text tokens) while keeping both semantic meaning and pixel detail, by pre-aligning the visual encoder to language and then squeezing the feature space gradually through a bottleneck before quantization. The payoff: 20\u201370% faster VLM training with multimodal accuracy on par with continuous encoders.
If you ship a vision-language product, you probably feed images through something like SigLIP2 or a CLIP-style encoder, get back a long sequence of high-dimensional continuous vectors, and pipe those into your LLM. That encoder forward pass is expensive, the vectors are bulky to cache, and they live in a fundamentally different representational world than your text tokens. Prior attempts to convert images into discrete tokens (so they look like text to the LLM) such as QLIP and UniTok cut storage and unify the pipeline, but lose enough information that benchmark scores collapse, particularly on OCR and document tasks. ViQ is the first quantized visual encoder the authors show closing that gap against strong continuous baselines like InternViT and AIMv2.
The core idea is that you cannot quantize a high-dimensional semantic feature directly without losing too much. You have to walk the feature space down to a small dimension gradually, and you have to align it with language before you start cutting bits.
Stage 1 trains a vision transformer (initialized from SigLIP2-g) by attaching a small LLM (Qwen2.5-0.5B) and supervising image-question-answer triples with a cross-entropy text loss. A self-distillation loss against the original fixed-resolution SigLIP2-g keeps the encoder from forgetting its pretraining. The encoder is adapted to native resolution inputs by swapping in resizable position embeddings, following the NaViT recipe.
Stage 2 compresses and quantizes. The 1536-dim feature passes through a bottleneck to 128 dims, then is projected to the surface of a hypercube via an L-infinity norm constraint (every coordinate’s max absolute value is forced to 1). This is the “proximal representation”: before quantizing, push features into a bounded, regular shape so the quantizer’s bins line up well with the data. Then a second bottleneck drops to 6 dims, and Finite Scalar Quantization (FSQ) discretizes each dimension independently into a small set of levels (here [8,8,8,5,5,5], giving a 64,000-entry codebook with no learned codebook vectors). A 2D Rotary Position Embedding (RoPE) layer injects position before quantization so spatial relationships survive. A multi-head attention block expands each patch into 2\u00d72 sub-codes for capacity. A separate reconstruction loss pushes the recovered feature to match the latent of a frozen pretrained image VAE (Qwen-Image’s encoder), which is cheaper than pixel reconstruction.
# Stage 2 forward, schematically
f = vit(image) # [B, N, 1536]
f1 = Linf_normalize(bottleneck(f)) # [B, N, 128], on hypercube
f2 = down_proj(f1) # [B, N, 6]
f2 = rope_2d(f2, h, w)
f2 = multihead_attn_expand(f2) # [B, 4N, 6]
z = fsq_round(f2) # discrete codes
f_hat = up_proj(z) # back to LLM input dim
loss = text_ce + distill_cos + mse_to_vae_latent(f_hat)
The prevailing recipe for discrete visual tokenizers is to train a VQ-VAE end-to-end with a learnable codebook and reconstruct pixels. This paper shows the opposite works better for understanding. Align with language first, then quantize with a fixed (non-learned) scalar grid after gradually constraining the feature space; do not let the codebook learn itself, and do not reconstruct pixels. The ablations carry this claim: FSQ with a frozen 64k grid beats learnable SimVQ codebooks at 32k\u2013128k, and predicting a pretrained VAE latent beats both MSE+LPIPS and a diffusion-style reconstruction head while being faster.
The load-bearing finding is in the ablations: removing proximal representation (going straight from continuous SigLIP features to SimVQ) drops the 8-benchmark average from 68.7 to 60.9. Adding a bottleneck plus L2 norm recovers to 66.6; switching to L-infinity and FSQ recovers to 68.7. That ~8 point swing is what makes quantized representations competitive at all. Position encoding matters similarly: removing RoPE drops the average to 65.3.
With that mechanism in place, the headline numbers follow:
•
Against continuous encoders on a 9-benchmark average: ViQ scores 57.2 with Qwen2.5-1.5B (vs 57.0 for the 6B-parameter InternViT-2.5) and 63.9 with Qwen2.5-7B (vs 63.8 for InternViT-2.5-6B).
•
Against prior quantized encoders: QLIP scores 29.7 and UniTok scores 33.0 on the same 1.5B setup. ViQ’s 57.2 is a roughly 24+ point jump, the gap the field had been stuck on.
•
Training speedup vs SigLIP2-g, because ViQ codes can be precomputed offline and loaded as integers: 70\u201378% forward-pass speedup on Qwen2.5-0.5B, 46\u201365% on 7B, with full-iteration speedups of 20\u201340%.
•
Storage: an image becomes HW/32 bytes, about 1/96 the raw size, with reconstruction PSNR 22.73 and rFID 0.62, competitive with the best discrete tokenizers.
•
The honest weak spot: on OCRBench, ViQ still trails some smaller continuous encoders, which the authors attribute to high-frequency detail loss inherent to aggressive discretization.
Reach for this when you’re shipping a VLM-based product where images are processed offline (document understanding pipelines, chart QA, catalog tagging) and you can precompute visual features at ingest time. Today you cache continuous SigLIP or InternViT vectors as float arrays. Swap to ViQ and you cache uint16 codes instead: ~1/96 the disk footprint, and training/fine-tuning iterations get 20\u201370% faster because you skip the visual encoder forward pass entirely and only pay for a small projection into the LLM.
The paper does not link a code repo or release weights in the text provided. The authors say checkpoints and recipes will follow but do not specify a URL or license. If you want to replicate today, you would need to rebuild from SigLIP2-g plus FSQ plus the Qwen-Image VAE latent target yourself; the architectural pieces are all from public prior work.
If you want discrete visual tokens that an LLM can actually use, align to language first, then narrow the feature space step by step before you let the quantizer touch it. End-to-end VQ with a learnable codebook is the obvious move and the wrong one for understanding tasks; a frozen scalar grid on a pre-shaped feature space wins, and the resulting codes are small enough to make training-time visual encoding essentially free.
•
The detail gap is real, not closed. On OCRBench, ViQ trails continuous encoders with fewer parameters. If your product is receipt parsing or dense small-text extraction, the headline averages flatter the method more than your use case will.
•
All experiments cap at Qwen2.5-7B. The authors explicitly flag that behavior with 70B+ backbones is untested, and quantization losses can interact with scale in non-obvious ways.
•
The recipe leans on several frozen pretrained pieces (SigLIP2-g initialization, Qwen-Image VAE as the reconstruction target, Qwen2.5-0.5B as text supervisor). The paper does not ablate how much of the gain comes from these specific choices versus the proximal-FSQ mechanism, so portability to a from-scratch or different-ecosystem setup is unclear.