Mage-VL

Why codec-native为什么要编解码器原生

Large vision–language models reason well over a handful of offline frames but struggle with continuous video. Uniform frame sampling re-encodes static background in every sampled frame even when only a small part of the scene changes, so cost climbs quickly with video length, and an offline model has no notion of when to respond. Video codecs already solve the first half of this problem: they spend bits only where a frame changes. Mage-VL turns that bit allocation into an importance prior applied before the visual encoder, and adds an event gate for the second half (Figure 1).

大型视觉语言模型擅长对少量离线帧做推理，却难以处理连续视频：均匀抽帧时，即使画面只有一小部分在变，静止背景也要在每一帧里重复编码，计算量随视频变长迅速增加；离线模型也不知道何时该回应。视频编解码器早已解决了前一半问题——它只在画面变化的地方花比特。Mage-VL 把这种比特分配变成视觉编码之前的重要性先验，再用一个事件门控解决后一半问题（图 1）。

Mage-ViT: reading video like a codecMage-ViT：像编解码器一样读视频

Mage-ViT is a 24-layer ViT-L/16 (hidden size 1024, 16 heads; about 316M parameters in the release) trained from scratch without captions. Images and short clips are clustered on MetaCLIP features, and the encoder learns to tell each sample’s cluster from sampled negatives. Pre-training uses about 560M unlabeled images and 100M video frames: first variable-resolution images (224–448 px), then images together with 64-frame clips, mixed as 50% codec, 40% chunk-wise and 10% collage inputs.

Mage-ViT 是一个 24 层的 ViT-L/16（隐藏维度 1024、16 个注意力头，发布版本约 3.16 亿参数），不用任何文本描述从零训练：先用 MetaCLIP 特征对图像和短视频聚类，编码器再学习在采样的负类中判别每个样本所属的聚类。预训练使用约 5.6 亿张无标注图像和 1 亿个视频帧，先训练可变分辨率图像（224–448 px），再联合训练图像与 64 帧视频片段，输入按 50% 编解码器、40% 分块、10% 拼贴混合。

The patchifier estimates how many bits the codec spends on each 16×16 patch: with HEVC, a weighted mix of motion-vector magnitude and residual energy; with the neural codec DCVC-RT, the patch’s negative log-likelihood. Every anchor-frame patch is kept, and the top-k predicted-frame patches by this score fill the rest of a token budget, so a 64-frame clip goes from 16,384 to 4,096 tokens. The kept patches are packed into canvases, and a shared 3D RoPE gives each one its original (t, h, w) position (Figure 2). At operating point tc-N, salient patches from 8N frames fill N canvases, the nominal workload of N dense frames.

patch 化模块估计编解码器在每个 16×16 patch 上花费的比特：使用 HEVC 时，是运动矢量幅度与残差能量的加权组合；使用神经编解码器 DCVC-RT 时，是该 patch 的负对数似然。锚点帧的 patch 全部保留，其余 token 预算由所有预测帧中得分最高的 top-k 个 patch 填满，一段 64 帧的视频因此从 16,384 个 token 降到 4,096 个。保留的 patch 被打包进画布，共享的 3D RoPE 为每个 patch 保留原始的 (t, h, w) 位置（图 2）。在 tc-N 工作点下，8N 帧中的显著 patch 填满 N 张画布，名义计算量相当于 N 张稠密帧。

Without ever seeing a caption, Mage-ViT is competitive with SigLIP2 (Table 1). It keeps improving as the budget grows to 676 tokens per image, where SigLIP2 peaks at 256, and with codec input it reaches 64.14 on Diving-48 against 62.30 for SigLIP2 (Table 2). Trained with HEVC only, it accepts DCVC-RT at inference without retraining: 57.7 against 58.0 averaged over twelve video benchmarks, using 92% of the canvases.

Mage-ViT 从未见过文本描述，却与 SigLIP2 不相上下（表 1）；随着每张图的 token 预算增加到 676 仍持续提升，而 SigLIP2 在 256 时即达峰值；使用编解码器输入时，它在 Diving-48 上达到 64.14，SigLIP2 为 62.30（表 2）。它只用 HEVC 训练，推理时可直接换用 DCVC-RT 而无需重训：在十二个视频基准上平均 57.7 对 58.0，所用画布为 HEVC 的 92%。

Proactive streaming: System 1 and System 2主动流式交互：System 1 与 System 2

Mage-VL streams with one model and two systems. System 1 is a light event gate on top of Mage-ViT: the features of each causal codec window pass through an event-preserving feature extractor with a recurrent memory, and the gate emits silent or speak. Generation starts only when p_speak ≥ τ. System 2 is the vision–language model itself, which then answers from the most recent codec segments; a user can also ask a question at any time. Only the gate is trained in the final stage, on about 3.35M streaming samples whose speak labels come from the start times of timestamped captions.

Mage-VL 用一个模型、两套系统完成流式交互。System 1 是建在 Mage-ViT 之上的轻量事件门控：每个因果编解码窗口的特征经过带循环记忆的事件保持特征提取器，门控输出“静默”或“发言”，只有 p_speak ≥ τ 时才开始生成。System 2 就是视觉语言模型本身，被触发后根据最近的编解码片段作答；用户也可以随时提问。最后一个训练阶段只训练门控，使用约 335 万条流式样本，“发言”标签取自带时间戳描述的起始时刻。

Mage-VL beats StreamMind on SoccerNet-Caption response timing, on both TriggerAcc (79.21 vs 52.18) and TimVal (55.54 vs 47.36), although StreamMind was trained in-distribution on SoccerNet-Caption; JoyAI’s high TriggerAcc comes with a TimVal of 19.25 (Table 3). On OVO-Bench, without streaming-specific fine-tuning, it has the best overall average in Table 4 (64.00 against 63.00 for Qwen3-VL-4B), carried by real-time perception (79.84); backward tracing stays behind the best (48.15 vs 53.1). In a 2026 World Cup broadcast the gate stays silent from 2.16 s to 24.50 s and fires at 29.36 s; the live demo with its transcript is on the project page.

在 SoccerNet-Caption 响应时机评测中，Mage-VL 的 TriggerAcc（79.21 对 52.18）和 TimVal（55.54 对 47.36）都高于在该数据集上做域内训练的 StreamMind；JoyAI 的 TriggerAcc 很高，但 TimVal 只有 19.25（表 3）。在 OVO-Bench 上，未做流式专门微调的 Mage-VL 取得表 4 中最高的总平均分（64.00，Qwen3-VL-4B 为 63.00），主要来自实时感知（79.84），回溯追踪则落后于最佳（48.15 对 53.1）。在一场 2026 年世界杯直播中，门控从 2.16 秒到 24.50 秒保持静默，在 29.36 秒触发；带实时字幕的演示见项目主页。

Data and training数据与训练

Mage-VL connects Mage-ViT to Qwen3-4B-Instruct-2507 through a two-layer MLP; in the released code it first merges each 2×2 group of patches into one token. It is trained in five stages (Table 5) on about 350M recaptioned image–caption pairs, 54M image instructions, 7.95M video captions and 3.35M streaming samples. Stage 4 feeds long videos as codec windows: with about one eighth of the tokens, sequences can be eight times longer than in Stage 3.

Mage-VL 通过一个两层 MLP 把 Mage-ViT 接到 Qwen3-4B-Instruct-2507 上；发布的代码中，这个 MLP 先把每 2×2 个 patch 合并为一个 token。训练分五个阶段（表 5），数据包括约 3.5 亿对重新生成描述的图文对、5400 万条图像指令、795 万条视频描述和 335 万条流式样本。第 4 阶段以编解码窗口输入长视频：token 只需约八分之一，序列长度可达第 3 阶段的八倍。

The captions themselves come out of an AI4AI loop (Figure 3). A frozen Qwen3-VL-32B captions a 1,000-image rollout batch, a GPT-5 rubric scorer grades completeness, redundancy, coherence and OCR fidelity, and a GitHub Copilot agent proposes minimal edits to the prompt and to the harness code. An edit is kept only if it passes a 200-image validation batch and a human reviewer. After ten iterations, recaptioning with the optimized prompt improves all nine benchmarks tested, most on InfoVQA (+5.62), OCRBench (+3.80) and RealWorldQA (+3.79). On video, the evaluation agent found that GPT-4.1 often misplaced events in time and that prompt edits alone could not fix it, so the harness now draws timestamps onto the sampled frames, which reduces temporal hallucinations.

描述数据本身也来自一个 AI4AI 闭环（图 3）。冻结的 Qwen3-VL-32B 为 1,000 张图的批次生成描述，GPT-5 评分器按完整性、冗余、连贯性和 OCR 准确度打分，GitHub Copilot 智能体对提示词和数据处理代码提出最小修改；修改只有通过 200 张图的验证批次和人工审核才会保留。迭代十轮后，用优化后的提示词重新生成描述，测试的九个基准全部提升，其中 InfoVQA +5.62、OCRBench +3.80、RealWorldQA +3.79。在视频上，评估智能体发现 GPT-4.1 常把事件的时间定错，只改提示词解决不了，于是数据处理代码改为把时间戳直接画在采样帧上，减少了时间上的幻觉。

Results结果

Mage-VL-4B is compared with Qwen3-VL-4B and two Phi-4 models. On images it is ahead of Qwen3-VL-4B on 22 of 35 benchmarks, by the widest margins on CrossPoint (80.00 vs 26.90), CharXiv-Reason (35.20 vs 26.10), ChartQAPro (32.57 vs 26.79), MMStar (67.32 vs 62.04) and EmbSpatial (82.67 vs 77.50); Qwen3-VL-4B is ahead on the other 13 (Table 6), most clearly on CC-OCR Doc (39.69 vs 32.25), ERQA (42.30 vs 36.00), DUDE (50.98 vs 46.44) and TextVQA (80.55 vs 77.28). On video it is ahead on 16 of 19: long-video QA (VideoMME +4.3, MLVU-dev +7.2, LVBench +2.6), temporal grounding, VSI-Bench (+11.0) and tracking. It trails on MV-Bench, VideoMME with subtitles and, by 0.1, VideoMME-V2 (Table 7).

Mage-VL-4B 与 Qwen3-VL-4B 及两个 Phi-4 模型对比。图像任务上它在 35 个基准中有 22 个领先 Qwen3-VL-4B，领先最多的是 CrossPoint（80.00 对 26.90）、CharXiv-Reason（35.20 对 26.10）、ChartQAPro（32.57 对 26.79）、MMStar（67.32 对 62.04）和 EmbSpatial（82.67 对 77.50）；Qwen3-VL-4B 在其余 13 个上领先（表 6），差距最大的是 CC-OCR Doc（39.69 对 32.25）、ERQA（42.30 对 36.00）、DUDE（50.98 对 46.44）和 TextVQA（80.55 对 77.28）。视频任务上它在 19 个基准中 16 个领先：长视频问答（VideoMME +4.3、MLVU-dev +7.2、LVBench +2.6）、时序定位、VSI-Bench（+11.0）和目标跟踪；在 MV-Bench、带字幕的 VideoMME 上落后，在 VideoMME-V2 上落后 0.1（表 7）。

Codec input also makes inference cheaper (Table 8). At tc8, Mage-VL finishes the NExT-QA evaluation in 415 s against 1,460 s for Qwen3-VL-4B, with higher accuracy (80.8 vs 79.8): 3.5× faster. On the other benchmarks tc8 is 1.1–2.7× faster, except VSI-Bench, where Qwen3-VL-4B is quicker. tc16 beats Qwen3-VL-4B on VideoMME (61.8 vs 59.7) in less time (439 s vs 463 s).

编解码器输入也让推理更省（表 8）。在 tc8 下，Mage-VL 完成 NExT-QA 评测用时 415 秒，Qwen3-VL-4B 用时 1,460 秒，准确率还更高（80.8 对 79.8），快 3.5 倍；在其他基准上 tc8 快 1.1–2.7 倍，只有 VSI-Bench 上 Qwen3-VL-4B 更快。tc16 在 VideoMME 上以更短的时间（439 秒对 463 秒）超过 Qwen3-VL-4B（61.8 对 59.7）。

Seven findings七个发现

1. Web-scale pre-training is not essential for a VLM encoder. Mage-ViT, trained on about 560M unlabeled images and 100M frames, is competitive with SigLIP2, which used more than 10B image–text pairs: ImageNet linear probe 85.69 vs 85.92, CIFAR-10 99.33 (the best in Table 1), HMDB-51 85.17 vs 84.61.VLM 编码器并不一定需要网络规模的预训练。Mage-ViT 只用约 5.6 亿张无标注图像和 1 亿个视频帧训练，就与使用了超过 100 亿图文对的 SigLIP2 不相上下：ImageNet 线性探测 85.69 对 85.92，CIFAR-10 99.33（表 1 最佳），HMDB-51 85.17 对 84.61。
2. Variable-resolution pre-training scales monotonically. Mage-ViT keeps improving from 196 to 676 tokens per image (above 96.1% on Food-101 and 86.3% on ImageNet), whereas SigLIP2 peaks at 256 tokens and degrades beyond it.可变分辨率预训练带来单调的缩放。Mage-ViT 从每张图 196 个 token 到 676 个持续提升（Food-101 超过 96.1%，ImageNet 超过 86.3%），而 SigLIP2 在 256 个 token 时达到峰值，之后下降。
3. Codec-native input is more efficient than frame sampling. At the same token budget it scores higher than uniform frame sampling, most clearly at 16 canvases or fewer, and tc8 runs NExT-QA 3.5× faster than Qwen3-VL-4B with higher accuracy.编解码器原生输入比抽帧更高效。在相同 token 预算下，它的得分高于均匀抽帧，预算不超过 16 张画布时差距最明显；tc8 在 NExT-QA 上比 Qwen3-VL-4B 快 3.5 倍且准确率更高。
4. Dense video captions can stand in for long-VideoQA SFT. With only short-video SFT plus detailed video captions, Mage-VL beats Qwen3-VL-4B on VideoMME (+4.3), MLVU-dev (+7.2), LongVideoBench (+3.6) and VideoEval-Pro (+24.5); dedicated long-video SFT brought no noticeable gain.稠密视频描述可以替代长视频问答 SFT。只用短视频 SFT 加上详细视频描述，Mage-VL 就在 VideoMME（+4.3）、MLVU-dev（+7.2）、LongVideoBench（+3.6）和 VideoEval-Pro（+24.5）上超过 Qwen3-VL-4B；专门的长视频 SFT 没有带来明显提升。
5. Dynamic video training goes with spatial intelligence. Mage-VL is strongest on CV-Bench-3D, EmbSpatial, CrossPoint and VSI-Bench (+11.0 over Qwen3-VL-4B). The paper notes that this is cross-model evidence, not a single-variable ablation.动态视频训练与空间智能相辅相成。Mage-VL 在 CV-Bench-3D、EmbSpatial、CrossPoint 和 VSI-Bench（比 Qwen3-VL-4B 高 11.0）上表现最强。论文指出这是跨模型的证据，而非单变量消融。
6. AI-driven data pipelines pay off downstream. Recaptions from the optimized prompt improve all nine benchmarks tested, most on InfoVQA (+5.62), OCRBench (+3.80) and RealWorldQA (+3.79).AI 驱动的数据管线能提升下游表现。用优化后的提示词重新生成的描述让测试的九个基准全部提升，其中 InfoVQA +5.62、OCRBench +3.80、RealWorldQA +3.79。
7. Skipping visual SFT unlocks multimodal RL. On the LLaVA-OV-1.5 Quick Start data, replacing visual SFT with pure-text reasoning SFT before RL raises the 24-benchmark average by 5.33 points (54.28 vs 48.96), wins 19 of 24 and needs 50.7% fewer RL steps (Table 9).跳过视觉 SFT 能释放多模态 RL。在 LLaVA-OV-1.5 Quick Start 数据上，RL 之前用纯文本推理 SFT 替代视觉 SFT，24 个基准的平均分提高 5.33（54.28 对 48.96），24 个中赢下 19 个，所需 RL 步数减少 50.7%（表 9）。

Limitations局限

Mage-VL still trails Qwen3-VL on complex agentic tasks, and the paper also lists mathematical reasoning as a weak spot. It traces both to limited compute, too little high-quality text data with no joint text–multimodal pre-training, and no RL post-training. Its proposed remedy is Zero-Vision SFT followed by RL (finding 7), which it calls a compute-efficient path; that study used the LLaVA-OV-1.5 Quick Start setup, and applying it to Mage-VL is left to future work.

Mage-VL 在复杂智能体任务上仍落后于 Qwen3-VL，论文也把数学推理列为短板。论文把原因归结为算力有限、高质量文本数据不足且缺少文本与多模态联合预训练，以及没有做 RL 后训练。论文提出的改进办法是先做 Zero-Vision SFT 再做 RL（发现 7），并称之为一条计算高效的路线；这项研究在 LLaVA-OV-1.5 Quick Start 设置上进行，用到 Mage-VL 上留待后续工作。

Open-source resources开源资源

The model, the encoder and the inference code are public. The paper is CC BY 4.0, the code and Mage-VL weights Apache-2.0, and the Mage-ViT weights MIT.

模型、编码器与推理代码全部公开。论文采用 CC BY 4.0，代码与 Mage-VL 权重采用 Apache-2.0，Mage-ViT 权重采用 MIT 许可。

Citation引用

@article{yang2026mage,
  title   = {Mage-VL: An Efficient Codec-Native Streaming Multimodal Foundation Model},
  author  = {Yang, Senqiao and Zhang, Kaichen and Jia, Zhaoyang and Guo, Jinghao and
             Shen, Yifei and Zhang, Xinjie and Zhang, Xiaoyi and Wang, Haoqing and
             Li, Xiao and Zhang, Peng and others},
  journal = {arXiv preprint arXiv:2607.24904},
  year    = {2026}
}