Get Started
Home
Topics
Search
Library
Research questionWhen does increasing representation rank improve class discrimination rather than dilute it in asymmetric knowledge distillation?Asymmetric distillation can force a small student's representation into a lower-dimensional geometry, but adding dimensions does not guarantee better downstream features. Expansion may dilute class-discriminative structure, causing representation rank and linear-probe performance to move in opposite directions.
AI
Computer Vision
Machine Learning
Research Paper
Statistical Machine Learning
Latest papersRecent research connected to this question, newest first.Breaking the Geometric Bottleneck: Contrastive Expansion in Asymmetric Cross-Modal DistillationThe evidence covers CLIP ViT-B/32 teachers distilled into 0.5M–8.0M-parameter local-receptive-field CNNs on CIFAR-10 and CIFAR-100. It compares cosine, class-blind InfoNCE, and supervised contrastive objectives using centered-SVD Effective Rank, class-structure analysis, and linear probes, with capacity and temperature sweeps limited to the reported settings.research paper · Sep 2, 2026
Related questions
How can contrastive distillation efficiently transfer representations to smaller students without memory banks or fixed temperatures?How can autoregressive logit distillation preserve local next-token preferences when teacher and student rankings disagree?How can weight-inheritance distillation train the full deployed student matrix rather than a restricted slice?How should on-policy distillation select examples for data-efficient reasoning post-training?