UNICOM & MLCD

Why cluster discrimination为什么用聚类判别

CLIP learns by instance discrimination: an image is pulled towards its own caption and pushed away from every other image in the batch, even one that shows the same thing. Two wedding photos become a negative pair, and the semantic structure of the data never enters the encoder. UNICOM and MLCD use cluster discrimination instead. They cluster the training set once, offline, and then train the encoder to classify each image into its clusters, so images with the same content share a class and move together (Figure 1).

CLIP 用实例判别来学习：一张图被拉向它自己的文本，同时被推离同一批次里的其他所有图像，哪怕那张图画的是同样的内容。两张婚礼照片因此成了负样本对，数据中的语义结构进不了编码器。UNICOM 和 MLCD 改用聚类判别：先离线把训练集聚类一次，再训练编码器把每张图分到它所属的类，内容相同的图像落在同一类、一起靠近（图 1）。

UNICOM: one class per imageUNICOM：每张图一个类别

UNICOM averages the CLIP image and text features of each LAION-400M sample and runs k-means once. With feature quantization the whole dataset clusters in about ten minutes, into one million classes of roughly 400 images each. Classes found this way conflict: a pair of birds lands in a wedding-couple class, and close-ups of wedding flowers also turn up in a wedding-venue class, because one image carries several labels and the granularity of a class is arbitrary.

UNICOM 对每个 LAION-400M 样本取 CLIP 图像特征与文本特征的平均，只做一次 k-means。借助特征量化，整个数据集约十分钟即可聚完，得到一百万个类，每类约 400 张图。这样聚出的类别彼此冲突：一对鸟被分进“新人合影”类，婚礼鲜花的特写也出现在“婚礼场地”类中，原因是一张图常带有多个标签，类别的粒度也没有统一标准。

Two random selections let the classifier train on such labels (Figure 2). At each step the margin-based softmax (ArcFace, margin 0.3, scale 64) sees the positive class and a random 10% of the other class prototypes, so a conflicting class is ten times less likely to push the image away. A second selection keeps a random subset of the feature dimensions, the same subset for the whole mini-batch and for both the embeddings and the prototypes. Unlike Dropout, this trains each sub-space of the embedding on its own, which keeps short descriptors useful for retrieval: on CUB, after one extra epoch at r₂ = 0.5, UNICOM beats the unsupervised retrieval method STML at every length from 256 down to 32 dimensions, although STML trains on CUB and UNICOM does not.

两次随机选择让分类器能在这样的标签上训练（图 2）。每一步，带间隔的 softmax（ArcFace，间隔 0.3，尺度 64）只看到正类和随机 10% 的其他类原型，冲突类把图像推开的概率因此降为十分之一。第二次选择随机保留一部分特征维度，整个 mini-batch 共用同一组维度，嵌入和原型同时截取。与 Dropout 不同，这样嵌入的每个子空间都单独得到训练，短描述子依然适合检索：在 CUB 上，用 r₂ = 0.5 多训练一个 epoch 后，UNICOM 在 256 到 32 维的每个长度上都超过无监督检索方法 STML，尽管 STML 在 CUB 上训练过，UNICOM 没有。

MLCD: several classes per imageMLCD：每张图多个类别

One pseudo-label rarely covers a web image. The wedding photo in Figure 3 shows a couple, white roses and a gown, while its caption names only the gowns. MLCD keeps UNICOM’s clustering, one million classes from CLIP ViT-L/14 features of LAION-400M, but ranks all class centres by similarity to each image and keeps the eight closest as positives. Every other class is a negative, and a random 10% of them enter each step.

一个伪标签很难概括一张网络图片。图 3 中的婚礼照片里有一对新人、白玫瑰和婚纱，而它的文本只提到婚纱。MLCD 沿用 UNICOM 的聚类，即用 CLIP ViT-L/14 特征把 LAION-400M 聚成一百万类，但会把所有类中心按与图像的相似度排序，取最近的 8 个作为正类。其余类都是负类，每一步随机取其中 10% 参与计算。

The loss follows Circle loss. A plain multi-label loss (MLC) only shrinks every gap sⱼ − sᵢ between a negative and a positive similarity, which leaves the decision boundary ambiguous: the pairs {sⱼ, sᵢ} = {0.1, 0.4} and {0.5, 0.8} meet the same margin of 0.3, yet the second pair’s negative scores higher than the first pair’s positive. MLCD adds one term that raises every sᵢ and one that lowers every sⱼ, and the loss then splits into a positive part and a negative part. In the paper’s training curves, MLCD’s mean similarity to positive class centres rises well above MLC’s, while its mean similarity to negative centres stays near zero.

损失函数参照 Circle loss。普通的多标签损失（MLC）只缩小每一对负类与正类相似度之差 sⱼ − sᵢ，决策边界因此含糊：{sⱼ, sᵢ} = {0.1, 0.4} 与 {0.5, 0.8} 都满足 0.3 的间隔，但第二组的负类比第一组的正类得分还高。MLCD 再加两项，一项抬高每个 sᵢ，一项压低每个 sⱼ，损失于是分成正类和负类两部分。论文给出的训练曲线显示，MLCD 与正类中心的平均相似度明显高于 MLC，与负类中心的平均相似度则保持在 0 附近。

Results结果

UNICOM: retrieval and linear probingUNICOM：检索与线性探测

UNICOM’s encoders are trained from scratch for 32 epochs on the clustered LAION-400M. Its ViT-L/14 averages 69.9 on seven unsupervised retrieval benchmarks, 7.5 points above OpenCLIP’s ViT-L/14 and 5.4 above OpenCLIP’s larger ViT-H/14 (Table 5). In linear probing on 13 datasets it is ahead of OpenCLIP, trained on the same data, by 3.6, 2.7 and 1.4 points on average for ViT-B/32, B/16 and L/14 (Table 6). Fine-tuned for retrieval, its largest model is above the best earlier result on all ten metrics of Table 7, results that come from various backbones. Fine-tuned on ImageNet-1K, its ViT-L/14 reaches 88.3, above ViT-L/16 pre-trained on JFT-300M (87.8) or IG-3.6B (88.1) (Table 8).

UNICOM 的编码器在聚类后的 LAION-400M 上从零训练 32 个 epoch。它的 ViT-L/14 在 7 个无监督检索基准上平均 69.9，比 OpenCLIP 的 ViT-L/14 高 7.5，比更大的 OpenCLIP ViT-H/14 还高 5.4（表 5）。在 13 个数据集的线性探测上，它平均领先使用相同数据的 OpenCLIP：ViT-B/32、B/16、L/14 分别高 3.6、2.7、1.4（表 6）。为检索微调后，它最大的模型在表 7 的十项指标上都超过此前最好的结果，这些结果来自各不相同的骨干网络。在 ImageNet-1K 上微调后，它的 ViT-L/14 达到 88.3，超过在 JFT-300M（87.8）或 IG-3.6B（88.1）上预训练的 ViT-L/16（表 8）。

MLCD: against CLIP, OpenCLIP, FLIP and UNICOMMLCD：对比 CLIP、OpenCLIP、FLIP 与 UNICOM

MLCD’s ViT-L/14 is trained for 32 epochs on LAION-400M; for zero-shot tasks a text encoder is then trained against the frozen image encoder (LiT). Over 26 linear-probe datasets it averages 84.6, against 82.3 for OpenCLIP and 83.3 for UNICOM on the same data and 83.5 for CLIP reproduced on WIT-400M; it is ahead of OpenCLIP on 25 datasets and of UNICOM on 23 (Figure 4). Over 25 zero-shot datasets it averages 67.5, against 66.0 for FLIP and 63.6 for OpenCLIP; it is ahead of OpenCLIP on 23 and of FLIP on 15 (Table 9). In zero-shot retrieval it ties FLIP on Flickr30k text R@1 and is best on every other metric except Flickr30k text R@5 and R@10 (Table 10). On ImageNet it reaches 87.1 fine-tuned, 84.6 linear and 75.6 zero-shot, and it is the most robust LAION-400M model on four of the five shifted test sets; FLIP stays ahead on ImageNet-R (Table 11).

MLCD 的 ViT-L/14 在 LAION-400M 上训练 32 个 epoch；零样本任务所需的文本编码器随后在冻结的图像编码器上训练（LiT）。在 26 个线性探测数据集上，它平均 84.6，相同数据下 OpenCLIP 为 82.3、UNICOM 为 83.3，在 WIT-400M 上复现的 CLIP 为 83.5；它在 25 个数据集上领先 OpenCLIP，在 23 个上领先 UNICOM（图 4）。在 25 个零样本数据集上，它平均 67.5，FLIP 为 66.0，OpenCLIP 为 63.6；其中 23 个领先 OpenCLIP，15 个领先 FLIP（表 9）。零样本检索中，它在 Flickr30k 文本检索 R@1 上与 FLIP 持平，除 Flickr30k 文本检索的 R@5、R@10 外其余指标都最好（表 10）。在 ImageNet 上，它的微调、线性探测和零样本结果分别为 87.1、84.6、75.6，五个分布偏移测试集中有四个是 LAION-400M 模型里最鲁棒的，ImageNet-R 上 FLIP 仍领先（表 11）。

As the vision tower of LLaVA-1.5, an MLCD ViT-L/14 pre-trained on LAION-400M and COYO-700M, then trained one more epoch at 336 px, replaces CLIP ViT-L/14 under the LLaVA-1.5 recipe. With Qwen2-7B it is better on 12 of the 18 scores, with one tie; with Qwen2-72B it is better on 15 (Table 13). It loses on the OCR-heavy TextVQA and AI2D with both language models, which the paper plans to address by adding OCR models to the clustering.

作为 LLaVA-1.5 的视觉编码器，在 LAION-400M 与 COYO-700M 上预训练、再在 336 px 下训练一个 epoch 的 MLCD ViT-L/14，按 LLaVA-1.5 的流程替换 CLIP ViT-L/14。搭配 Qwen2-7B 时，它在 18 项分数中有 12 项更好、1 项持平；搭配 Qwen2-72B 时有 15 项更好（表 13）。两种语言模型下，它在偏重 OCR 的 TextVQA 和 AI2D 上都落后，论文计划在聚类时加入 OCR 模型来弥补。

The patch features carry semantics without dense supervision. After one epoch at 448 px, thresholding the first PCA component of the 32×32 patch tokens separates objects from the background, and mapping three components to RGB colours objects of the same category alike (Figure 5).

它的 patch 特征不需要稠密监督就带有语义。在 448 px 下训练一个 epoch 后，对 32×32 个 patch token 的第一主成分取阈值即可把物体与背景分开，再把三个主成分映射为 RGB，同类物体呈现相近的颜色（图 5）。

After the papers论文之后

The repository kept scaling MLCD as a vision tower for multimodal LLMs. ViT-bigG/14 models at 336 and 448 px add 2D RoPE; inside LLaVA-NeXT with Qwen2.5-7B, the 448-px model scores 73.80 on ChartQA, 83.34 on DocVQA and 582 on OCRBench, above CLIP ViT-L/14 and SigLIP SO400M (Table 14). In April 2025 MLCD ViT-bigG was merged into Hugging Face Transformers as MLCDVisionModel.

代码仓库继续把 MLCD 扩展为多模态大模型的视觉编码器。336 px 和 448 px 的 ViT-bigG/14 模型加入了 2D RoPE；在搭配 Qwen2.5-7B 的 LLaVA-NeXT 中，448 px 模型在 ChartQA、DocVQA、OCRBench 上分别得到 73.80、83.34、582，高于 CLIP ViT-L/14 和 SigLIP SO400M（表 14）。2025 年 4 月，MLCD ViT-bigG 以 MLCDVisionModel 的形式并入 Hugging Face Transformers。

The objective also outlived the two papers. Extended from whole images to regions it became RICE (ICCV 2025 highlight), the vision encoder of LLaVA-OneVision-1.5. OneVision-Encoder is trained by cluster discrimination over more than one million concepts, and Mage-ViT uses negative-sampling cluster discrimination with the same 10% ratio.

这个训练目标也延续到了两篇论文之后。从整张图扩展到区域后，它成为 RICE（ICCV 2025 Highlight），也就是 LLaVA-OneVision-1.5 的视觉编码器。OneVision-Encoder 在一百多万个概念上做聚类判别训练，Mage-ViT 也采用带负采样的聚类判别，负采样比例同样是 10%。

Open-source resources开源资源

Code and models are public. The repository is MIT-licensed; on Hugging Face the MLCD ViT-B/32 and ViT-L/14 weights are Apache-2.0 and the ViT-bigG/14 weights MIT.

代码与模型全部公开。代码仓库采用 MIT 许可；Hugging Face 上 MLCD 的 ViT-B/32 与 ViT-L/14 权重采用 Apache-2.0，ViT-bigG/14 权重采用 MIT。

Citation引用

@inproceedings{anxiang_2024_mlcd,
  title     = {Multi-label Cluster Discrimination for Visual Representation Learning},
  author    = {An, Xiang and Yang, Kaicheng and Dai, Xiangzi and Feng, Ziyong and Deng, Jiankang},
  booktitle = {ECCV},
  year      = {2024}
}
@inproceedings{anxiang_2023_unicom,
  title     = {Unicom: Universal and Compact Representation Learning for Image Retrieval},
  author    = {An, Xiang and Deng, Jiankang and Yang, Kaicheng and Li, Jiawei and Feng, Ziyong and
               Guo, Jia and Yang, Jing and Liu, Tongliang},
  booktitle = {ICLR},
  year      = {2023}
}