Research研究

UniME-V2

MLLM-as-a-Judge for universal multimodal embedding learning以多模态大模型为评判者的通用多模态嵌入学习

UniME-V2 trains a multimodal LLM as a universal embedding model and lets a second MLLM grade the training data. For every training query, VLM2Vec retrieves 50 candidates from the whole training pool, and Qwen2.5-VL-7B scores how well each one meets the query. The scores remove false negatives, choose the hard negatives and become soft labels: the embedding model’s similarity distribution is trained to match the judge’s instead of a one-hot target. A reranker trained on the same negatives then re-orders what the embedding model retrieves. On LLaVA-OneVision-7B, UniME-V2 averages 71.2 over the 36 MMEB datasets and 66.7 over its 16 out-of-distribution ones, the best out-of-distribution score in the paper’s comparison.

UniME-V2 把多模态大模型训练成通用嵌入模型,并让另一个多模态大模型为训练数据打分。对每个训练查询,VLM2Vec 先从整个训练池中检索 50 个候选,再由 Qwen2.5-VL-7B 评估每个候选是否满足查询。这些分数用来剔除假负样本、挑选困难负样本,并作为软标签:嵌入模型的相似度分布要对齐评判者给出的分布,而不是 one-hot 目标。在同一批负样本上训练的重排模型,再对嵌入模型检索到的结果重新排序。以 LLaVA-OneVision-7B 为底座时,UniME-V2 在 MMEB 的 36 个数据集上平均 71.2,在其中 16 个分布外数据集上为 66.7,是论文对比中最高的分布外成绩。

Why let a judge pick the negatives为什么让评判者来挑负样本

A universal multimodal embedding model maps queries and candidates of any kind (an image, a text, an image with a question or an instruction) into one space, so that retrieval, classification, visual question answering and visual grounding all become a search for the nearest candidate. Recent models build it from an MLLM. VLM2Vec fine-tunes one with a contrastive InfoNCE loss on the MMEB training sets, and UniME adds several hard negatives per query, chosen inside each batch. Both choose negatives by embedding similarity, which leaves two gaps. Negatives drawn from a batch are few and alike. And an embedding that is still being trained cannot tell a hard negative from a false negative: a candidate that also meets the query but is not the labelled target. InfoNCE then treats every negative alike, with probability one for the target and zero for everything else, however close a candidate comes.

通用多模态嵌入模型把任意形式的查询和候选(一张图、一段文字、一张图加一个问题或指令)映射到同一个空间,检索、分类、视觉问答和视觉定位于是都变成寻找最近的候选。近来的模型多由多模态大模型改造而来。VLM2Vec 在 MMEB 训练集上用对比学习的 InfoNCE 损失微调多模态大模型,UniME 则在每个批次内为每个查询挑选多个困难负样本。两者都按嵌入相似度选负样本,留下了两个缺口:批次内的负样本数量少、彼此相似;而一个仍在训练中的嵌入模型分不清困难负样本和假负样本,即那些同样满足查询、只是没有被标为目标的候选。InfoNCE 又对所有负样本一视同仁:目标的概率为 1,其余一律为 0,无论某个候选有多接近。

UniME-V2 hands both decisions to a judge. An MLLM reads each query–candidate pair and returns a semantic matching score between 0 and 1. The scores choose the negatives, dropping any candidate that scores almost as high as the target, and they replace the one-hot target with a soft one, so that a near miss is pushed away less than an unrelated candidate (Figure 1).

UniME-V2 把这两件事交给一个评判者。多模态大模型逐一阅读查询–候选对,给出 0 到 1 之间的语义匹配分数。分数决定负样本的取舍,得分几乎与目标一样高的候选会被剔除;分数还把 one-hot 目标换成软目标,使“差一点就对”的候选比无关候选被推开得更少(图 1)。

Global retrieval: VLM2Vec (Qwen2-VL-7B) ranks the whole training pool for this query; candidates at or above a similarity threshold δ are dropped and the top 50 of the rest kept (five drawn).全局检索:VLM2Vec(Qwen2-VL-7B)为这个查询对整个训练池排序,相似度达到阈值 δ 的候选被丢弃,其余的保留前 50 个(图中画出 5 个)。

Figure 1. One training query through UniME-V2’s mining pipeline and loss. The query, its target caption and three candidates with their judge scores (target 0.998; 0.893, 0.407, 0.294) are from the paper’s figure of mined training data (Figure 3 below). The other two candidates, every embedding similarity x, the scores of the four hard negatives not drawn in the list and the model’s distribution P are illustrative. The settings are the paper’s: top 50, β = 0.01, ten candidates per query, k = 8 hard negatives in the loss, τ = 0.02. Q is computed from the drawn scores: at τ = 0.02 only a negative within a few hundredths of the target’s score keeps visible weight.图 1. 一个训练查询经过 UniME-V2 的挖掘流程和损失。查询、目标文本以及三个候选及其评判分数(目标 0.998;0.893、0.407、0.294)取自论文中挖掘后训练数据的示例图(见下方图 3)。另外两个候选、所有嵌入相似度 x、列表中未画出的四个困难负样本的分数,以及模型分布 P 均为示意。设置取自论文:前 50 个、β = 0.01、每个查询十个候选、损失中 k = 8 个困难负样本、τ = 0.02。Q 由图中的分数算出:τ = 0.02 时,只有与目标分数相差几个百分点以内的负样本还能分到看得见的权重。

Mining hard negatives with an MLLM judge用多模态大模型评判者挖掘困难负样本

Mining starts from the whole training set rather than the batch. An existing embedding model, VLM2Vec on Qwen2-VL-7B, embeds every query and candidate. For each query, candidates whose cosine similarity reaches a threshold δ are dropped as likely false negatives, and the 50 most similar of the rest form the potential hard-negative set (Figure 2). The paper does not report δ.

挖掘从整个训练集出发,而不是从批次出发。现成的嵌入模型 VLM2Vec(Qwen2-VL-7B)为所有查询和候选计算嵌入。对每个查询,余弦相似度达到阈值 δ 的候选被视为可能的假负样本而丢弃,其余候选中最相似的 50 个构成潜在困难负样本集(图 2)。论文没有给出 δ 的取值。

The MLLM-as-a-Judge pipeline: queries from retrieval, classification and VQA tasks; an off-the-shelf embedding model retrieves from the candidate pool, filters false negatives by similarity and keeps the top 50 candidates; an MLLM judge scores each query–candidate pair; hard-negative sampling filters false and easy negatives
Figure 2. The MLLM-as-a-Judge mining pipeline, from the paper. An off-the-shelf embedding model retrieves candidates for each query from the global pool, filters likely false negatives by similarity and keeps the top 50. The MLLM judge scores every query–candidate pair, and hard-negative sampling uses the scores to drop false and easy negatives.图 2. MLLM-as-a-Judge 挖掘流程(取自论文)。现成的嵌入模型从全局候选池中为每个查询检索候选,按相似度过滤可能的假负样本并保留前 50 个;多模态大模型评判者为每个查询–候选对打分,困难负样本采样再依据分数剔除假负样本和简单负样本。

Each of the 50 pairs then goes to the judge, Qwen2.5-VL-7B, with one instruction. The semantic matching score s = ey / (ey + en) is computed from the logits of the Yes and No tokens. On mined training data the target scores close to 1, partly relevant candidates fall in between and unrelated ones near 0 (Figure 3).

随后,这 50 对逐一交给评判者 Qwen2.5-VL-7B,使用同一条指令。语义匹配分数 s = ey / (ey + en) 由 Yes 和 No 两个 token 的 logits 算出。在挖掘后的训练数据上,目标的分数接近 1,部分相关的候选介于两者之间,无关候选接近 0(图 3)。

judge-prompt.txt
I will provide you with a query and a candidate. Please evaluate whether the candidate meets the requirements of the query. If it does, respond with 'Yes'; if it doesn't, respond with 'No'. Query:<Query>, Candidates:<Candidate>.
Three mined training examples: each query with an instruction, its positive candidate scored near 1.0 and three hard-negative candidates with intermediate or low judge scores
Figure 3. Mined training examples with their judge scores, from the paper’s supplementary material. In each row the first candidate, marked with a thumbs-up, is the target. The first row is the example of Figure 1.图 3. 挖掘后的训练样例及其评判分数(取自论文补充材料)。每行第一个候选(带点赞图标)是目标。第一行就是图 1 中的例子。

The scores then choose the negatives. A candidate scoring above α = sq,ct − β, with β = 0.01, is treated as a false negative and removed, so only near ties with the target ct go. For diversity, the rest is sampled cyclically at five-step intervals; if fewer than ten candidates remain, picks are duplicated up to ten. For the under 1% of queries where no candidate qualifies, ten of the 50 are drawn at random and given a score of 1.0. Each query ends up with a hard-negative set and the judge scores that go with it.

接下来由分数决定负样本。得分高于 α = sq,ct − β(β = 0.01)的候选视为假负样本而剔除,因此只有与目标 ct 几乎打平的候选会被去掉。为保证多样性,剩下的候选以五步为间隔循环采样;若不足十个,就重复选取补足十个。对不到 1% 找不到合格候选的查询,从 50 个中随机抽取十个,并把它们的分数都设为 1.0。最终每个查询得到一个困难负样本集及对应的评判分数。

Pool候选池
Ωp = Rank50({x1, …, xn}), xi < δ
Score评分
si = eyi / (eyi + eni), S ∈ ℝnq×50
Filter过滤
drop c if sq,c > α = sq,ct − β, β = 0.01若 sq,c > α = sq,ct − β 则剔除 c,β = 0.01
Sample采样
cyclic, five-step interval, at least 10 per query → Ωh = {c1, …, ck}, Sh = {sq,c1, …, sq,ck}以五步为间隔循环采样,每个查询至少 10 个 → Ωh = {c1, …, ck},Sh = {sq,c1, …, sq,ck}

Training on the judge’s scores用评判分数训练

Training then replaces InfoNCE’s one-hot target. The embedding model is an MLLM fine-tuned with LoRA, and the last token’s hidden state is the embedding of a query or a candidate. For a query q, its target ct and k = 8 hard negatives, the cosine similarities pass through a softmax at temperature τ = 0.02 to give P, the model’s distribution over the k + 1 candidates. The judge scores of the same candidates pass through the same softmax to give Q, and the loss averages KL(P‖Q) and KL(Q‖P) over the batch (Figure 4, left).

训练时,InfoNCE 的 one-hot 目标被替换掉。嵌入模型是用 LoRA 微调的多模态大模型,最后一个 token 的隐状态就是查询或候选的嵌入。对查询 q、目标 ct 和 k = 8 个困难负样本,余弦相似度经温度 τ = 0.02 的 softmax 得到 P,即模型在这 k + 1 个候选上的分布;同一组候选的评判分数经过同样的 softmax 得到 Q;损失是批次内 KL(P‖Q) 与 KL(Q‖P) 的平均(图 4 左)。

InfoNCE asks for probability one on the target and zero everywhere else, so a near miss is pushed away as hard as an unrelated caption. Q keeps the judge’s ranking among the negatives. At τ = 0.02 the soft target is still sharp: a negative that scores 0.1 below the target gets e−5 ≈ 0.7% of the target’s weight, so only negatives within a few hundredths of the target keep visible probability (step 4 of Figure 1).

InfoNCE 要求目标的概率为 1、其余全为 0,“差一点就对”的候选和完全无关的文本被推开得一样狠。Q 则保留了评判者对负样本的排序。在 τ = 0.02 时,软目标依然很尖锐:比目标低 0.1 分的负样本只得到目标权重的 e−5 ≈ 0.7%,只有与目标相差几个百分点以内的负样本才保有看得见的概率(图 1 第 4 步)。

Model模型
P(eq, Ec) = exp(cos(eq, ec+)/τ) / [exp(cos(eq, ec+)/τ) + Σi=1…k exp(cos(eq, eci−)/τ)]
Judge评判
Q(Sc) = exp(sq,ct/τ) / [exp(sq,ct/τ) + Σi=1…k exp(sq,ci/τ)]
Loss损失
L = ½ [ (1/N) Σi KL(Pi ‖ Qi) + (1/N) Σi KL(Qi ‖ Pi) ]
Setting设置
Ωc = {ct, c1, …, ck}, k = 8, τ = 0.02
Left: UniME-V2, a LoRA-tuned MLLM whose normalized last-token embeddings give a similarity matrix P that is aligned with the judge's semantic matching score matrix Q in both KL directions. Right: UniME-V2-Reranker, an MLLM trained with pairwise and listwise reranking
Figure 4. The training framework, from the paper. Left: UniME-V2 takes the normalized last-token embedding of a LoRA-tuned MLLM; its similarity matrix P is supervised by the judge’s score matrix Q in both KL directions. Right: UniME-V2-Reranker is trained with pairwise and listwise reranking.图 4. 训练框架(取自论文)。左:UniME-V2 取 LoRA 微调的多模态大模型最后一个 token 的归一化嵌入,其相似度矩阵 P 在两个 KL 方向上接受评判分数矩阵 Q 的监督。右:UniME-V2-Reranker 用成对和列表两种重排方式训练。
Training details (Table 1)训练细节(表 1)

UniME-V2 follows UniME’s settings. The training data are the 662K pairs of the 20 in-distribution MMEB training sets, which cover classification, VQA, retrieval and visual grounding. Each model trains for 2,000 steps with LoRA rank 16, GradCache and an accumulated batch of 1,024 on 8 × A800 (80 GB); the implementation section adds DeepSpeed ZeRO stage 2. The backbones are Qwen2-VL-2B, Qwen2-VL-7B and LLaVA-OneVision-7B. The implementation section gives 336 × 336 images for every model; the supplementary table, followed here, gives 672 for Qwen2-VL and 336 for LLaVA-OneVision.

UniME-V2 沿用 UniME 的设置。训练数据是 MMEB 20 个分布内训练集的 662K 对样本,覆盖分类、VQA、检索和视觉定位。每个模型在 8 × A800(80 GB)上训练 2,000 步,LoRA 秩为 16,使用 GradCache,累积批大小为 1,024;实现部分还提到了 DeepSpeed ZeRO stage 2。底座为 Qwen2-VL-2B、Qwen2-VL-7B 和 LLaVA-OneVision-7B。实现部分写的是所有模型都用 336 × 336 的图像,补充材料中的表格则是 Qwen2-VL 用 672、LLaVA-OneVision 用 336,此处采用后者。

Table 1Training hyperparameters of UniME-V2, from the paper’s supplementary material.表 1UniME-V2 的训练超参数(取自论文补充材料)。

Hyperparameter超参数Qwen2-VL-2B / 7BLLaVA-OneVision-7B
Training samples训练样本662K662K
Batch size批大小10241024
Learning rate学习率1×10⁻⁴2×10⁻⁵
LoRA rankLoRA 秩1616
Training steps训练步数20002000
Optimizer优化器AdamWAdamW
Infrastructure训练框架GradCacheGradCache
Max length最大长度40964096
Temperature τ温度 τ0.020.02
Hard negatives k困难负样本 k88
Image resolution图像分辨率672336
Precision精度BF16BF16
GPUsGPU8 × A8008 × A800
Random seed随机种子4242

UniME-V2-Reranker

The same mined negatives also train a reranker: Qwen2.5-VL-7B, fine-tuned with LoRA in two ways at once (Figure 4, right). Pairwise, it sees the target and the hardest negative ch separately and must answer YES for the first and NO for the second. Listwise, the target is inserted at a random position among the top x hard negatives by judge score, and the reranker must output the target’s position. The loss is the sum of the two cross-entropies; the paper does not give x.

同一批挖掘出的负样本还用来训练一个重排模型:Qwen2.5-VL-7B,用 LoRA 以两种方式同时微调(图 4 右)。成对训练时,它分别看到目标和最难的负样本 ch,要对前者回答 YES、对后者回答 NO。列表训练时,目标被随机插入按评判分数排在前 x 位的困难负样本中,重排模型要输出目标的位置。损失是两个交叉熵之和;论文没有给出 x。

Pairwise成对
Lpair = Lce(YES, η(q, ct)) + Lce(NO, η(q, ch))
Listwise列表
Llist = Lce(Ict, η(q, ct, {c1, …, cx}))
Total总计
L = Lpair + Llist

At inference the two models run in sequence. UniME-V2 embeds the query and the candidates and retrieves the ten nearest by cosine similarity; the reranker reads the query with the numbered list and answers with the number of the best candidate:

推理时两个模型串联。UniME-V2 为查询和候选计算嵌入,按余弦相似度检索最近的十个;重排模型读入查询和编号后的候选列表,回答最佳候选的编号:

rerank-prompt.txt
I will provide you with a query followed by multiple candidates in the format: (1) candidate1 (2) candidate2, etc. Each candidate is independent of the others. Evaluate each candidate against the query, and respond with the number corresponding to the candidate that best meets the requirements of the query. Query:<Query>, Candidates:<Candidate list>.
Reranker training and the pairwise prompt (Table 2)重排模型的训练与成对提示词(表 2)

The reranker follows LamRA’s setup: Qwen2.5-VL-7B with LoRA rank 128, one epoch of about 2,000 steps on the lmms-finetune code base, sequences of up to 4,096 tokens. The listwise prompt is the inference prompt above; the pairwise prompt is the judge’s instruction with “Candidate” in place of “Candidates”:

重排模型沿用 LamRA 的设置:Qwen2.5-VL-7B,LoRA 秩 128,在 lmms-finetune 代码库上训练一个 epoch(约 2,000 步),序列最长 4,096 个 token。列表训练的提示词就是上面的推理提示词;成对训练的提示词与评判者的指令相同,只是把 “Candidates” 换成了 “Candidate”:

pairwise-prompt.txt
I will provide you with a query and a candidate. Please evaluate whether the candidate meets the requirements of the query. If it does, respond with 'Yes'; if it doesn't, respond with 'No'. Query:<Query>, Candidate:<Candidate>.

Table 2Training hyperparameters of UniME-V2-Reranker, from the paper’s supplementary material.表 2UniME-V2-Reranker 的训练超参数(取自论文补充材料)。

Hyperparameter超参数Qwen2.5-VL-7B
Training samples训练样本662K
Batch size批大小64
Learning rate学习率2×10⁻⁵
LoRA rankLoRA 秩128
Training epochs训练轮数1
Optimizer优化器AdamW
Infrastructure训练框架lmms-finetune
Max length最大长度4096
Precision精度BF16
DeepSpeed stageDeepSpeed 阶段2
GPUsGPU8 × A800
Random seed随机种子42

Results结果

MMEBMMEB

MMEB has 36 datasets in four meta-tasks: 10 classification, 10 VQA, 12 retrieval and 4 visual grounding. The 20 in-distribution datasets supply the training pairs and the other 16 are held out; every dataset is scored by Precision@1. With the same data and settings, UniME-V2 improves on VLM2Vec by 3.5 points on Qwen2-VL-2B (63.6 against 60.1) and by 2.2 on Qwen2-VL-7B (68.0 against 65.8). On LLaVA-OneVision-7B it reaches 71.2 overall, 0.5 above QQMM and UniME and 0.9 above LLaVE, and 66.7 on the out-of-distribution datasets, the highest in Table 3. It does not lead everywhere: LLaVE keeps the best grounding (91.9) and in-distribution (75.0) scores, QQMM and UniME the best classification score (66.8), and on Qwen2-VL-7B UniME-V2 trails UniME by 0.7 on classification and 0.2 in distribution. Table 4 lists all 36 datasets.

MMEB 包含四类元任务共 36 个数据集:10 个分类、10 个 VQA、12 个检索、4 个视觉定位。20 个分布内数据集提供训练对,其余 16 个不参与训练;每个数据集都以 Precision@1 计分。在相同的数据和设置下,UniME-V2 在 Qwen2-VL-2B 上比 VLM2Vec 高 3.5 分(63.6 对 60.1),在 Qwen2-VL-7B 上高 2.2 分(68.0 对 65.8)。以 LLaVA-OneVision-7B 为底座时,它的总分为 71.2,比 QQMM 和 UniME 高 0.5,比 LLaVE 高 0.9;分布外得分 66.7,是表 3 中最高的。它并非处处领先:视觉定位(91.9)和分布内(75.0)的最佳成绩仍属于 LLaVE,分类的最佳成绩(66.8)属于 QQMM 和 UniME;在 Qwen2-VL-7B 上,UniME-V2 的分类比 UniME 低 0.7,分布内低 0.2。表 4 列出了全部 36 个数据集。

Table 3MMEB, average Precision@1 (%). The number under each heading counts its datasets; IND: in-distribution, OOD: out-of-distribution. UniME-V2 rows shaded, with the change from UniME on the same backbone under each score. Blue marks the best score in each column, ties included (the paper bolds only UniME’s 66.8 in classification, not QQMM’s equal score). Sizes are the paper’s labels.表 3MMEB 结果,平均 Precision@1(%)。每个表头下的数字为该项包含的数据集数;IND:分布内,OOD:分布外。UniME-V2 行加底色,每个分数下方是相对同底座 UniME 的变化。蓝色为每列最佳,并列者都标出(论文只把分类一栏 UniME 的 66.8 加粗,没有标出与之相同的 QQMM)。模型规模沿用论文的标注。

Model模型Size规模Per meta-task各类任务Average平均
Classification分类10VQAVQA10Retrieval检索12Grounding定位4IND分布内20OOD分布外16Overall总体36
Zero-shot on MMEBMMEB 零样本
CLIP (ViT-L)0.4B42.89.153.051.837.138.739.2
OpenCLIP (ViT-L)0.4B41.56.944.653.532.836.036.6
MagicLens (ViT-L)0.4B38.88.335.426.031.023.727.1
SigLIP (So/14)0.9B40.38.431.659.532.338.035.0
BLIP2 (ViT-L)1.2B27.04.233.947.025.325.128.0
CLIP (ViT-bigG/14)2.5B52.314.050.560.338.945.844.3
EVA-CLIP8B56.010.449.258.938.145.643.7
E5-V (Phi3.5-V)4.2B39.19.638.057.633.131.936.1
E5-V (LLaVA-1.6)7B39.710.839.460.234.233.437.5
Fine-tuned on MMEB在 MMEB 上微调
CLIP (ViT-L)0.4B55.219.753.262.247.642.847.6
VLM2Vec (Qwen2-VL)2B59.049.465.473.466.052.660.1
VLM2Vec (Qwen2-VL)7B62.657.869.981.772.257.865.8
LLaVE (LLaVA-OV)7B65.765.470.991.975.064.470.3
QQMM (LLaVA-OV)7B66.866.870.590.474.765.670.7
UniME (Qwen2-VL)2B59.053.464.969.665.554.660.6
UniME (Qwen2-VL)7B64.759.071.682.772.261.467.4
UniME (LLaVA-OV)7B66.866.670.590.974.665.870.7
UniME-V2 (Qwen2-VL)2B62.1+3.156.3+2.968.0+3.172.7+3.167.4+1.958.9+4.363.6+3.0
UniME-V2 (Qwen2-VL)7B64.0−0.760.1+1.173.1+1.582.8+0.172.0−0.263.0+1.668.0+0.6
UniME-V2 (LLaVA-OV)7B65.3−1.567.6+1.072.9+2.490.2−0.774.8+0.266.7+0.971.2+0.5

Table 4MMEB per dataset, Precision@1 (%), from the paper’s supplementary material. The paper copies the BLIP2 to UniME columns from the UniME paper; E5-V, VLM2Vec and UniME there use LLaVA-1.6-7B, not the backbones of Table 3. UniME-V2 columns shaded; blue marks the best score on each dataset; OOD marks the 16 held-out datasets. Summary rows are printed as in the paper. For UniME-V2 on Qwen2-VL-7B the per-dataset scores average to 62.8, 59.0, 72.9 and 81.5 per meta-task (71.6 IND, 61.7 OOD, 67.2 overall), below the printed summaries, which match Table 3.表 4MMEB 各数据集结果,Precision@1(%),取自论文补充材料。BLIP2 到 UniME 各列由论文从 UniME 论文中抄录;其中 E5-V、VLM2Vec 和 UniME 用的是 LLaVA-1.6-7B,而不是表 3 中的底座。UniME-V2 列加底色;蓝色为每个数据集上的最佳;OOD 标出 16 个分布外数据集。汇总行按论文原样列出。对 Qwen2-VL-7B 上的 UniME-V2,各数据集分数按元任务平均为 62.8、59.0、72.9、81.5(分布内 71.6、分布外 61.7、总体 67.2),低于论文给出的汇总值;汇总值与表 3 一致。

Dataset数据集BLIP2MagicLensEVA-CLIPE5-VLLaVA-1.6VLM2VecLLaVA-1.6UniMELLaVA-1.6UniME-V2Qwen2-VL-7BUniME-V2LLaVA-OV-7B
Classification (10 datasets)分类(10 个数据集)
ImageNet-1K10.348.075.040.566.571.380.378.8
N24News36.033.733.831.576.479.566.966.6
HatefulMemes49.649.049.349.360.964.665.965.3
VOC200752.151.644.376.784.090.484.992.0
SUN39734.557.062.752.373.275.978.978.7
Place365OOD21.531.538.732.042.145.642.542.9
ImageNet-AOOD3.28.054.818.239.945.553.748.0
ImageNet-ROOD39.770.995.456.774.678.487.989.3
ObjectNetOOD20.631.667.834.234.336.435.073.1
Country-211OOD2.56.238.75.916.118.732.319.8
All classification分类平均27.038.856.039.756.860.664.065.3
VQA (10 datasets)VQA(10 个数据集)
OK-VQA8.712.79.915.166.568.359.371.9
A-OKVQA3.22.92.84.754.958.732.371.4
DocVQA2.63.07.49.164.467.691.292.6
InfographicsVQA2.05.96.08.734.837.063.963.5
ChartQA0.50.91.54.233.133.456.955.8
Visual7W1.32.52.24.549.851.760.162.5
ScienceQAOOD6.85.214.19.637.340.544.554.0
VizWizOOD4.01.74.38.639.942.747.453.7
GQAOOD9.743.544.734.157.363.655.869.5
TextVQAOOD3.34.610.89.565.765.278.484.5
All VQAVQA 平均4.28.310.410.850.452.960.167.6
Retrieval (12 datasets)检索(12 个数据集)
VisDial18.024.820.457.675.379.783.484.2
CIRR9.839.136.041.051.352.264.065.5
VisualNews t2i48.150.782.443.970.774.879.977.3
VisualNews i2t13.521.188.246.875.278.883.579.2
MSCOCO t2i53.754.165.368.669.974.977.779.1
MSCOCO i2t20.340.067.254.867.773.873.075.2
NIGHTS56.558.10.20.163.366.269.368.1
WebQA55.443.070.933.783.689.891.590.6
FashionIQOOD9.311.216.111.215.216.528.526.4
Wiki-SS-NQOOD28.718.746.761.063.466.668.871.2
OVENOOD39.51.61.80.549.655.771.268.0
EDISOOD54.462.695.653.873.786.284.488.2
All retrieval检索平均33.935.449.239.463.367.973.172.9
Visual grounding (4 datasets)视觉定位(4 个数据集)
MSCOCO28.922.135.841.777.076.569.378.2
RefCOCOOOD47.422.859.962.285.989.388.494.6
RefCOCO-matchingOOD59.535.670.074.983.890.689.791.4
Visual7W-pointingOOD52.023.470.261.883.684.178.793.8
All visual grounding视觉定位平均47.026.058.960.282.685.182.890.2
Final scores总分
Overall (36)总体(36)28.027.143.737.563.366.668.071.2
In-distribution (20)分布内(20)25.331.038.134.264.968.472.074.8
Out-of-distribution (16)分布外(16)25.123.745.633.453.957.963.066.7

Short, long and compositional retrieval短文本、长文本与组合式检索

Five zero-shot benchmarks test the embeddings outside MMEB, all scored by Recall@1: short captions on Flickr30K and COCO, long captions on ShareGPT4V and Urban1K, and SugarCrepe, where an image has to be matched to its caption rather than a version with objects, attributes or relations replaced, swapped or added. Against UniME on the same backbone, text-to-image retrieval on Flickr30K and COCO improves by 2.2 to 9.7 points, while image-to-text moves between −0.7 and +1.6. Every long-caption score improves, by 1.2 to 12.3 points, and every SugarCrepe score by 4.5 to 9.2. On LLaVA-OneVision-7B, UniME-V2 beats the 8B EVA-CLIP on all four long-caption scores and on SugarCrepe by 2.7, 3.4 and 3.8 points (Table 5).

五个零样本基准在 MMEB 之外检验嵌入,都以 Recall@1 计分:Flickr30K 和 COCO 是短文本,ShareGPT4V 和 Urban1K 是长文本;SugarCrepe 则要求把图像与其正确的文本配对,而不是与替换、交换或添加了物体、属性或关系的改写版本配对。与同底座的 UniME 相比,Flickr30K 和 COCO 上的文到图检索提高 2.2 到 9.7 分,图到文检索的变化在 −0.7 到 +1.6 之间。长文本的每一项都有提高,幅度为 1.2 到 12.3 分;SugarCrepe 的每一项提高 4.5 到 9.2 分。以 LLaVA-OneVision-7B 为底座时,UniME-V2 在全部四项长文本检索上都超过 8B 的 EVA-CLIP,在 SugarCrepe 上分别高出 2.7、3.4、3.8 分(表 5)。

Table 5Zero-shot retrieval, Recall@1 (%). T→I: text query, image candidates (qt → ci in the paper); I→T the reverse. UniME-V2 rows shaded, with the change from UniME on the same backbone under each score; blue marks the best in each column. The paper prints +0.8 for UniME-V2 (Qwen2-VL-7B) on ShareGPT4V T→I; 94.3 − 86.5 is +7.8, shown here. The paper’s text puts the short-caption gains in image-to-text retrieval; its table puts them in text-to-image, and this page follows the table.表 5零样本检索,Recall@1(%)。文→图:文本查询、图像候选(论文中记为 qt → ci);图→文反之。UniME-V2 行加底色,每个分数下方是相对同底座 UniME 的变化;蓝色为每列最佳。论文在 UniME-V2(Qwen2-VL-7B)的 ShareGPT4V 文→图一格写的是 +0.8,而 94.3 − 86.5 = +7.8,此处取后者。论文正文把短文本上的提升归于图到文检索,表格显示的却是文到图,本页以表格为准。

Model模型Size规模Short caption短文本Long caption长文本Compositional组合
Flickr30KCOCOShareGPT4VUrban1KSugarCrepe
T→I文→图I→T图→文T→I文→图I→T图→文T→I文→图I→T图→文T→I文→图I→T图→文Replace替换Swap交换Add添加
OpenCLIP (ViT-L)0.4B67.387.237.058.181.884.047.047.079.562.774.9
CLIP (ViT-bigG/14)2.5B79.592.951.367.390.193.677.880.786.568.988.4
EVA-CLIP8B80.394.552.070.193.191.280.477.885.970.386.7
E5-V (Phi3.5-V)4.2B72.279.644.753.486.088.583.883.688.266.675.3
E5-V (LLaVA-1.6)7B77.385.749.157.685.182.188.983.286.368.766.9
VLM2Vec (Qwen2-VL)2B69.389.640.062.578.188.278.783.967.246.566.4
VLM2Vec (Qwen2-VL)7B80.094.249.268.578.590.494.094.270.051.772.2
UniME (Qwen2-VL)2B74.990.644.063.583.688.683.383.265.645.265.7
UniME (Qwen2-VL)7B80.892.750.969.886.593.895.394.068.853.069.8
UniME (LLaVA-OV)7B83.394.454.874.093.989.394.395.580.565.582.2
UniME-V2 (Qwen2-VL)2B79.8+4.989.9−0.753.7+9.765.1+1.691.6+8.094.2+5.695.6+12.392.2+9.070.9+5.351.2+6.070.2+4.5
UniME-V2 (Qwen2-VL)7B84.6+3.893.5+0.857.3+6.470.3+0.594.3+7.895.2+1.497.2+1.996.3+2.377.8+9.062.2+9.279.0+9.2
UniME-V2 (LLaVA-OV)7B85.5+2.293.7−0.760.9+6.174.1+0.195.1+1.294.1+4.896.3+2.096.7+1.288.6+8.173.7+8.290.5+8.3
The five retrieval benchmarks (Table 6)五个检索基准(表 6)

All five are evaluated zero-shot under each benchmark’s standard protocol, with the instruction “Represent the image/text” on both sides. In SugarCrepe each image chooses between two captions.

五个基准都按各自的标准协议做零样本评测,图像和文本两侧都使用指令 “Represent the image/text”。SugarCrepe 中每张图像在两条文本之间二选一。

Table 6The retrieval benchmarks of Table 5, from the paper’s supplementary material.表 6表 5 所用的检索基准(取自论文补充材料)。

Benchmark基准Zero-shot零样本Test queries测试查询数Candidates per query每个查询的候选数
Flickr30K✓1K5K
COCO✓5K25K
ShareGPT4V✓1K1K
Urban1K✓1K1K
SugarCrepe✓7.5K2

Reranking重排

Table 7 compares UniME-V2-Reranker with LamRA’s reranker, both on Qwen2.5-VL-7B with the same training settings, reranking the top five candidates of UniME-V2 (the inference pipeline above reranks ten; this comparison uses five). Both rerankers improve every score of the embedding model alone. UniME-V2-Reranker, trained on 0.6M examples against LamRA’s 1.1M, ties LamRA’s short-caption score after the 2B model and leads everywhere else, by 0.5, 0.4, 0.3 and 7.4 points after the 7B model. The gap is widest on SugarCrepe, 94.8 against 87.4. SugarCrepe offers only two candidates per query, and each reranker scores the same there after either embedding model.

表 7 比较 UniME-V2-Reranker 与 LamRA 的重排模型:两者都基于 Qwen2.5-VL-7B、训练设置相同,对 UniME-V2 检索出的前五个候选重排(上文的推理流程重排前十个,这组对比用的是前五个)。两个重排模型都让嵌入模型单独使用时的每一项得分有所提高。UniME-V2-Reranker 只用 0.6M 训练样本(LamRA 为 1.1M),在 2B 嵌入模型之后的短文本检索上与 LamRA 持平,其余各项都领先;在 7B 嵌入模型之后分别领先 0.5、0.4、0.3、7.4 分。差距最大的是 SugarCrepe:94.8 对 87.4。SugarCrepe 每个查询只有两个候选,两个重排模型在它上面的得分都与前面接哪个嵌入模型无关。

Table 7Reranking the top five candidates retrieved by UniME-V2. RShort, RLong and RCompos average the short-caption, long-caption and SugarCrepe columns of Table 5. UniME-V2-Reranker rows shaded; blue marks the best in each block, ties included (the paper bolds only UniME-V2-Reranker’s 76.4).表 7对 UniME-V2 检索出的前五个候选重排。RShort、RLong、RCompos 分别是表 5 中短文本、长文本和 SugarCrepe 各列的平均。UniME-V2-Reranker 行加底色;蓝色为每组最佳,并列者都标出(论文只把 UniME-V2-Reranker 的 76.4 加粗)。

Embedding model嵌入模型Reranker重排模型Reranker data重排训练数据MMEBRShortRLongRCompos
2B embedding models2B 嵌入模型
UniME (Qwen2-VL-2B)––60.668.384.758.8
UniME-V2 (Qwen2-VL-2B)––63.672.193.464.1
UniME-V2 (Qwen2-VL-2B)LamRA (Qwen2.5-VL-7B)1.1M67.376.496.487.4
UniME-V2 (Qwen2-VL-2B)UniME-V2-Reranker (Qwen2.5-VL-7B)0.6M67.676.496.994.8
7B embedding models7B 嵌入模型
UniME (Qwen2-VL-7B)––67.473.692.463.9
UniME-V2 (Qwen2-VL-7B)––68.076.495.873.0
UniME-V2 (Qwen2-VL-7B)LamRA (Qwen2.5-VL-7B)1.1M69.178.397.287.4
UniME-V2 (Qwen2-VL-7B)UniME-V2-Reranker (Qwen2.5-VL-7B)0.6M69.678.797.594.8

Analysis分析

Ablations on Qwen2-VL-2B separate the two uses of the judge. Mining hard negatives with it, still trained contrastively, adds 1.5 points on MMEB and 5.5, 7.6 and 3.7 on short, long and compositional retrieval; training on the soft scores adds 2.0, 3.2, 3.6 and 0.4 more (Table 8). The judge itself matters: with InternVL3-8B as judge MMEB drops to 58.5, InternVL3-14B recovers 63.2 and Qwen2.5-VL-7B gives 63.6, a gap the paper attributes to the instruction data each judge was fine-tuned on (Table 9). Eight hard negatives beat four, six and ten (Table 10), and τ = 0.02 is best on MMEB and on short and long retrieval, though not on SugarCrepe (Table 11).

在 Qwen2-VL-2B 上的消融把评判者的两种用途分开来看。只用评判者挖掘困难负样本、仍做对比学习时,MMEB 提高 1.5 分,短文本、长文本和组合式检索分别提高 5.5、7.6、3.7 分;再用软分数训练,又分别提高 2.0、3.2、3.6、0.4 分(表 8)。评判者本身也很关键:换成 InternVL3-8B 时 MMEB 降到 58.5,InternVL3-14B 回升到 63.2,Qwen2.5-VL-7B 为 63.6;论文把差距归因于各评判模型在指令微调阶段所用数据的分布不同(表 9)。8 个困难负样本优于 4、6、10 个(表 10);τ = 0.02 在 MMEB、短文本和长文本检索上最好,但在 SugarCrepe 上不是(表 11)。

Ablation: judge-mined negatives and soft scores (Table 8)消融:评判者挖掘的负样本与软分数(表 8)

The first row is the paper’s plain contrastive baseline. Its MMEB, RLong and RCompos equal those of VLM2Vec (Qwen2-VL-2B) in Tables 3 and 5; its RShort (63.4) does not, since VLM2Vec’s four short-caption scores in Table 5 average 65.4.

第一行是论文中的普通对比学习基线。它的 MMEB、RLong、RCompos 与表 3、表 5 中 VLM2Vec(Qwen2-VL-2B)的数值相同;RShort(63.4)则不同,表 5 中 VLM2Vec 的四项短文本得分平均为 65.4。

Table 8The two components on Qwen2-VL-2B. RShort, RLong, RCompos as in Table 7. The full model shaded; blue marks the best in each column.表 8两个组成部分的消融(Qwen2-VL-2B)。RShort、RLong、RCompos 同表 7。完整模型加底色;蓝色为每列最佳。

Judge-mined hard negatives评判者挖掘的困难负样本Soft judge scores评判者软分数MMEBRShortRLongRCompos
––60.163.482.260.0
✓–61.668.989.863.7
✓✓63.672.193.464.1
Ablation: which MLLM judges (Table 9)消融:用哪个多模态大模型做评判(表 9)

The three judges score the same mined candidates under the same inference settings; all three score files are released with the training data. Qwen2.5-VL-7B is best on MMEB and long-caption retrieval, InternVL3-14B on short-caption retrieval (72.9), and InternVL3-8B ties Qwen2.5-VL-7B on SugarCrepe (64.1).

三个评判模型在相同的推理设置下为同一批挖掘出的候选打分,三份打分文件都随训练数据一同发布。Qwen2.5-VL-7B 在 MMEB 和长文本检索上最好,InternVL3-14B 在短文本检索上最好(72.9),InternVL3-8B 在 SugarCrepe 上与 Qwen2.5-VL-7B 持平(64.1)。

Table 9The judge model, with UniME-V2 on Qwen2-VL-2B. The paper’s judge shaded; blue marks the best in each column, ties included (the paper bolds only Qwen2.5-VL-7B’s 64.1).表 9评判模型的选择(UniME-V2,Qwen2-VL-2B)。论文采用的评判模型加底色;蓝色为每列最佳,并列者都标出(论文只把 Qwen2.5-VL-7B 的 64.1 加粗)。

Judge评判模型MMEBRShortRLongRCompos
Qwen2.5-VL-7B63.672.193.464.1
InternVL3-8B58.570.291.364.1
InternVL3-14B63.272.993.163.2
Ablation: number of hard negatives (Table 10)消融:困难负样本的数量(表 10)

Going from four to eight hard negatives adds 2.3, 2.9, 2.4 and 1.7 points. Ten negatives bring in easier ones and score slightly lower (63.0 on MMEB), while RLong holds at 93.4.

困难负样本从 4 个增加到 8 个,四项分别提高 2.3、2.9、2.4、1.7 分。用 10 个时会混入更简单的负样本,得分略有下降(MMEB 为 63.0),RLong 保持 93.4。

Table 10Hard negatives per query, with UniME-V2 on Qwen2-VL-2B. The paper’s setting shaded; blue marks the best in each column, ties included.表 10每个查询的困难负样本数(UniME-V2,Qwen2-VL-2B)。论文采用的设置加底色;蓝色为每列最佳,并列者都标出。

Hard negatives k困难负样本数 kMMEBRShortRLongRCompos
461.369.291.062.4
661.870.891.761.2
863.672.193.464.1
1063.072.093.463.4
Ablation: temperature (Table 11)消融:温度(表 11)

τ sets how sharp both P and Q are. 0.02 is best on MMEB and on short and long retrieval. On SugarCrepe both 0.01 (66.5) and 0.03 (65.8) score higher than 0.02 (64.1), although the paper’s text calls 0.02 optimal on every metric and its table bolds 0.02 there.

τ 同时决定 P 和 Q 的尖锐程度。0.02 在 MMEB、短文本和长文本检索上最好。在 SugarCrepe 上,0.01(66.5)和 0.03(65.8)都高于 0.02(64.1),尽管论文正文说 0.02 在所有指标上都最优,表格也把这一栏的 0.02 加粗。

Table 11Temperature of the loss, with UniME-V2 on Qwen2-VL-2B, from the paper’s supplementary material. The paper’s setting shaded; blue marks the best in each column.表 11损失的温度(UniME-V2,Qwen2-VL-2B),取自论文补充材料。论文采用的设置加底色;蓝色为每列最佳。

Temperature τ温度 τMMEBRShortRLongRCompos
0.0361.970.692.065.8
0.0263.672.193.464.1
0.0162.170.191.166.5

A smaller modality gap更小的模态鸿沟

Figure 5 projects text and image embeddings with t-SNE. EVA-CLIP-8B, a dual encoder, keeps the two modalities in separate clouds; UniME-V2 on LLaVA-OneVision-7B mixes them. The paper links this smaller modality gap to its more robust long-caption retrieval than EVA-CLIP.

图 5 用 t-SNE 投影文本和图像的嵌入。双塔结构的 EVA-CLIP-8B 把两种模态分成两团;基于 LLaVA-OneVision-7B 的 UniME-V2 则把它们混在一起。论文把这种更小的模态鸿沟与它在长文本检索上比 EVA-CLIP 更稳健的表现联系在一起。

t-SNE of text and image embeddings: EVA-CLIP places text and images in two separate clusters, UniME-V2 interleaves them
Figure 5. t-SNE of text (blue) and image (orange) embeddings from EVA-CLIP-8B (left) and UniME-V2 on LLaVA-OneVision-7B (right), from the paper.图 5. EVA-CLIP-8B(左)与基于 LLaVA-OneVision-7B 的 UniME-V2(右)的文本(蓝)和图像(橙)嵌入的 t-SNE 可视化(取自论文)。

Retrieve, then rerank先检索,再重排

Figure 6 shows the two models together on three MMEB tasks. For the bear photo, UniME-V2 retrieves the black-bear, brown-bear and sloth-bear labels, and the reranker puts brown bear first; for the skier it picks “Skiing down the mountain.” over two answers about a lodge, and for the grounding query the crop that shows the motorcycle.

图 6 展示两个模型在三个 MMEB 任务上的配合。对那张熊的照片,UniME-V2 检索出黑熊、棕熊和懒熊三个标签,重排模型把棕熊排在第一;对滑雪者的照片,它在两个关于小屋的答案和 “Skiing down the mountain.” 之间选了后者;对视觉定位查询,它选中了显示摩托车的那块区域。

Three queries with their retrieved candidates: a bear photo with black bear, brown bear and sloth bear labels; a skier with three location answers; a street scene with three motorcycle crops. The reranker's top-1 is boxed
Figure 6. Retrieval and reranking on classification, VQA and visual grounding, from the paper. Each row shows a query with its instruction and the candidates UniME-V2 retrieved; the red dashed box marks the reranker’s top-1.图 6. 分类、VQA 和视觉定位上的检索与重排(取自论文)。每行是一个查询及其指令,以及 UniME-V2 检索出的候选;红色虚线框是重排模型选出的第一名。
More retrieval and reranking examples (Figure 7)更多检索与重排示例(图 7)
Further examples: a weasel photo classified among mink, weasel and ferret labels; a question about what a toy bear and the floor have in common; a grounding query for a car, each with the retrieved candidates and the reranker's choice boxed
Figure 7. Further retrieval and reranking results, from the paper’s supplementary material; the red dashed box marks the reranker’s top-1.图 7. 更多检索与重排结果(取自论文补充材料);红色虚线框是重排模型选出的第一名。

Open-source resources开源资源

The weights, the judge-scored training data and the code are public. On Hugging Face the four models and the dataset are Apache-2.0; the repository is MIT-licensed and holds the training, evaluation and demo code for both models.

模型权重、带评判分数的训练数据和代码全部公开。Hugging Face 上的四个模型和数据集采用 Apache-2.0 许可;代码仓库采用 MIT 许可,包含两个模型的训练、评测和演示代码。

Citation引用

citation.bib
@misc{gu2025unimev2mllmasajudgeuniversalmultimodal,
  title         = {UniME-V2: MLLM-as-a-Judge for Universal Multimodal Embedding Learning},
  author        = {Tiancheng Gu and Kaicheng Yang and Kaichen Zhang and Xiang An and Ziyong Feng and
                   Yueyi Zhang and Weidong Cai and Jiankang Deng and Lidong Bing},
  year          = {2025},
  eprint        = {2510.13515},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  url           = {https://arxiv.org/abs/2510.13515}
}

Authors作者

Tiancheng Gu*, Kaicheng Yang*, Kaichen Zhang, Xiang An, Ziyong Feng, Yueyi Zhang†, Weidong Cai, Jiankang Deng†, Lidong Bing

* Equal contribution. † Corresponding author. MiroMind AI, The University of Sydney, M.R.L. Team, LMMs-Lab Team, Imperial College London.* 同等贡献。† 通讯作者。MiroMind AI、悉尼大学、M.R.L. Team、LMMs-Lab Team、帝国理工学院。