RL Data Strategy强化学习数据策略

The RL data is not new annotation. It is drawn from public sets (ViRL39K^[50], Ref-L4^[51], VigoRL-SA and VigoRL-SAT^[52], PixmoCount^[16], WebCode2M^[53], UniSVG^[54], InfoVQA^[7] and AI2D^[1]) and filtered with the instruct model itself. Two filters decide what stays:

RL 数据不是新标注的，而是取自公开数据集（ViRL39K^[50]、Ref-L4^[51]、VigoRL-SA 与 VigoRL-SAT^[52]、PixmoCount^[16]、WebCode2M^[53]、UniSVG^[54]、InfoVQA^[7] 和 AI2D^[1]），再用指令模型本身筛选。两道筛选决定哪些样本留下：

• Discrepancy-driven selection. A large gap between Pass@N and Pass@1 means correct answers already appear among the model’s samples but not reliably on the first try. RL can turn that latent ability into consistent behaviour, so these tasks are kept; tasks the model already solves (high Pass@1) or cannot solve (low Pass@N) are dropped.差异驱动筛选。Pass@N 与 Pass@1 之间差距大，说明正确答案已经出现在模型的采样中，只是第一次作答做不稳。RL 能把这种潜在能力变成稳定的行为，因此保留这类任务；模型已经会做（Pass@1 高）或做不出来（Pass@N 低）的任务被剔除。
• Reward-based sampling. The base model answers each prompt several times and every answer is scored. Prompts whose average reward is near 1 (trivial) or near 0 (unsolvable) are removed, which leaves medium-difficulty prompts with a useful learning signal.基于奖励的采样。基础模型对每个题目作答多次并逐一打分。平均奖励接近 1（太简单）或接近 0（不可解）的题目被移除，留下能提供有效学习信号的中等难度题目。

67K examples remain, 58% of them STEM (Figure 8a). Each example also records whether it is posed answer-only or with chain-of-thought, and the two training stages draw on that: stage 1 uses 19.9K answer-only examples, three quarters of them grounding; stage 2 uses 49.2K chain-of-thought examples, 79% of them STEM.

最终留下 6.7 万条样本，其中 58% 为 STEM（图 8a）。每条样本还记录了它是仅答案形式还是链式思维形式，两个训练阶段据此取数：阶段一使用 1.99 万条仅答案样本，其中四分之三是定位任务；阶段二使用 4.92 万条链式思维样本，其中 79% 为 STEM。

Reward System Architecture奖励系统架构

Rewards come from rules, not from a learned preference model. Each answer type has its own checker in the reward/ module (Table 4): math answers are parsed and checked for symbolic equivalence, so ½, 0.5 and 2/4 all match; boxes are scored by IoU; code by token and tag overlap, plus rendering similarity for SVG. Each response gets one scalar. A format reward applies to every task: exactly one <think> block, at least one \boxed{}, and boxed content no longer than 20% of the response.

奖励来自规则，而不是学习得到的偏好模型。每种答案类型在 reward/ 模块中都有自己的校验器（表 4）：数学答案先解析再做符号等价判断，因此 ½、0.5 和 2/4 都算正确；定位框按 IoU 打分；代码按 token 与标签重叠度打分，SVG 还加上渲染相似度。每条回答得到一个标量奖励。格式奖励作用于所有任务：恰好一个 <think> 块、至少一个 \boxed{}，且框内内容不超过回答长度的 20%。

def math_reward_fn(completions, answer):
    model_answer = extract_answer(completion, format_strict=True)
    gold_parsed = parse(gt_answer, extraction_config=[
        StringExtractionConfig(), LatexExtractionConfig(), ExprExtractionConfig(),
    ])
    correct = verify(answer_parsed, gold_parsed)
    if not correct:
        correct = is_equal(gt_answer, model_answer)  # SymPy fallback
    return 1.0 if correct else 0.0

def bbox_reward_fn(completions, answer):
    xA = max(predicted[0], gt[0]); yA = max(predicted[1], gt[1])
    xB = min(predicted[2], gt[2]); yB = min(predicted[3], gt[3])
    inter = max(0, xB-xA) * max(0, yB-yA)
    iou = inter / float(areaA + areaB - inter)
    return iou  # ∈ [0, 1]

def multiplechoice_reward_fn(completions, answer):
    predicted = extract_boxed_content(completions)[-1]
    predicted = predicted.strip().strip('.()')
    predicted = predicted[0].upper() if predicted else ""
    return 1 if predicted == answer.upper() else 0

def number_reward_fn(completions, answer):
    answer_str = extract_boxed_content(completions)[-1]
    match = re.findall(r"([0-9\\.]+)", answer_str)
    count = match[-1] if match else ""
    return float(count.strip() == answer.strip())

def html_reward_fn(completions, answer):
    token_score = calculate_token_overlap(gen, ref)
    structure_score = calculate_tag_structure_similarity(gen, ref)
    reward = 0.6 * token_score + 0.4 * structure_score
    return max(0.0, min(1.0, reward))

def svg_reward_fn(completions, answer):
    token_score = calculate_token_overlap(gen, ref)
    structure_score = calculate_structure_similarity(gen, ref)
    image_score = calculate_image_similarity(gen_png, ref_png)  # SSIM
    reward = 0.5 * image_score + 0.25 * (token_score + structure_score)
    return reward  # ∈ [0, 1]

def ocr_reward_fn(completions, answer):
    dist = levenshtein_distance(gt, det)
    length = max(len(target), len(det))
    reward = 1 - min(values)
    return reward if reward >= 0.5 else 0  # threshold

Two-Stage Training Procedure两阶段训练流程

Both stages run GRPO in the asynchronous AReaL framework (below). Stage 1 trains the instruct model on short answers so that grounding, counting and OCR become reliable without long outputs. Stage 2 starts from the stage-1 checkpoint and asks the model to reason inside <think> before answering, mostly on STEM problems.

两个阶段都在异步 AReaL 框架中运行 GRPO（见下文）。阶段一用简短回答训练指令模型，让定位、计数与 OCR 在不输出长文本的情况下变得稳定。阶段二从阶段一检查点出发，要求模型先在 <think> 中推理再作答，训练数据以 STEM 题为主。

Performance Results性能结果

The RL model can answer in two modes: thinking (chain-of-thought) and fast (direct answer). Table 5 averages the 26 benchmarks of Table 6 by group. Thinking mode lifts reasoning the most, from 45.8 to 51.8 (MathVision 25.6 → 34.4, MMMU-Pro_vision 25.2 → 35.7), at a small cost on OCR and charts. Fast mode keeps OCR and charts at 84.6 and raises the remaining group from 69.0 to 71.6, mainly through counting (PixmoCount 62.2 → 71.1). Both modes improve general VQA.

RL 模型可以用两种模式作答：thinking（链式思维）和 fast（直接作答）。表 5 按类别对表 6 中的 26 个基准求平均。Thinking 模式对推理的提升最大，从 45.8 到 51.8（MathVision 25.6 → 34.4，MMMU-Pro_vision 25.2 → 35.7），代价是 OCR 与图表略有下降。Fast 模式把 OCR 与图表保持在 84.6，并把“其他”类别从 69.0 提升到 71.6，主要来自计数（PixmoCount 62.2 → 71.1）。两种模式都提升了通用 VQA。

Figure 9 adds spatial reasoning, grounding and coding benchmarks and compares with Qwen2.5-VL. Fast mode gives the best grounding (Ref-L4 IoU 57.1 → 61.6, RefCOCO IoU 64.0 → 68.8), while thinking mode lowers it. Thinking mode helps web coding (WebCode 55.6 → 58.3, Design2Code 60.1 → 63.9); on UniSVG, fast mode is best (50.4 → 57.5).

图 9 补充了空间推理、定位与代码基准，并与 Qwen2.5-VL 对比。Fast 模式的定位最好（Ref-L4 IoU 57.1 → 61.6，RefCOCO IoU 64.0 → 68.8），而 thinking 模式会使其下降。Thinking 模式提升网页代码生成（WebCode 55.6 → 58.3，Design2Code 60.1 → 63.9）；在 UniSVG 上 fast 模式最好（50.4 → 57.5）。

GRPO Algorithm & AReaL Async FrameworkGRPO 算法与 AReaL 异步框架

GRPO removes the critic. Each prompt gets G = 16 sampled answers, and each answer’s advantage is its reward minus the group mean, divided by the group standard deviation. The run uses no KL penalty and an asymmetric clip on the probability ratio (0.2 below, 0.28 above).

GRPO 去掉了 critic。每个提示采样 G = 16 个回答，每个回答的优势等于它的奖励减去组内均值、再除以组内标准差。训练不使用 KL 惩罚，对概率比采用非对称裁剪（下界 0.2，上界 0.28）。

AReaL runs generation and training at the same time on separate GPUs. Rollout workers keep sampling with a slightly older policy while the actor updates, and new weights are pushed to them after every step. A sample may be at most four steps stale, and an importance weight, capped at 5, corrects for the gap between the policy that generated it and the current one. Removing the wait between generation and training raises throughput 2.77×.

AReaL 在不同的 GPU 上同时进行生成与训练。actor 更新的同时，rollout worker 用稍旧的策略持续采样，每一步结束后再把新权重推送给它们。样本最多落后当前策略四步，并用上限为 5 的重要性权重修正生成样本的策略与当前策略之间的差异。消除生成与训练之间的等待使吞吐提升 2.77 倍。

# utils/functional.py — ppo_actor_loss_fn
ratio = torch.exp(logprobs - proximal_logprobs)
clipped_ratio = torch.clamp(ratio, 1.0 - eps_clip, 1.0 + eps_clip_higher)
pg_loss = torch.max(-advantages * ratio, -advantages * clipped_ratio)

# Behavior importance weight (off-policy correction)
behav_kl = proximal_logprobs - old_logprobs
behav_imp_weight = torch.clamp(behav_kl.exp(), max=behav_imp_weight_cap)
pg_loss = pg_loss * behav_imp_weight

# trains/grpo.py — main training loop
for global_step in range(start_step, max_steps):
    batch = rollout.prepare_batch(train_dataloader, workflow=workflow)
    batch["prox_logp"] = actor.compute_logp(batch)
    actor.compute_advantages(batch)
    actor.ppo_update(batch)

    rollout.pause()
    actor.update_weights(weight_update_meta)
    rollout.set_version(global_step + 1)
    rollout.resume()

Acknowledgements致谢

We thank the following projects and frameworks:

我们感谢以下项目与框架：

• AReaL: Lightning-Fast RL for LLM Reasoning and Agents面向大模型推理与智能体的高吞吐强化学习框架
• sglang: Fast serving framework for LLMs and vision language models面向大语言模型与视觉语言模型的高性能服务框架
• lmms-eval: Standardized evaluation framework标准化评测框架
• LLaVA: Large Language-and-Vision Assistant大型语言-视觉助手
• LLaVA-NeXT: Next-generation multi-modal assistant新一代多模态助手

Open-Source Resources开源资源

The RL code, the trained model and the 67K training examples are all public.

RL 代码、训练好的模型和 6.7 万条训练数据全部公开。

Citation引用

@inproceedings{LLaVA-OneVision-1.5,
  title={LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Training},
  author={An, Xiang and Xie, Yin and Yang, Kaicheng and Zhang, Wenkang and Zhao, Xiuwei and Cheng, Zheng and Wang, Yirui and Xu, Songcen and Chen, Changrui and Wu, Chunsheng and Tan, Huajie and Li, Chunyuan and Yang, Jing and Yu, Jie and Wang, Xiyao and Qin, Bin and Wang, Yumeng and Yan, Zizhen and Feng, Ziyong and Liu, Ziwei and Li, Bo and Deng, Jiankang},
  booktitle={arXiv},
  year={2025}
}

@inproceedings{xie2025region,
  title={Region-based Cluster Discrimination for Visual Representation Learning},
  author={Xie, Yin and Yang, Kaicheng and An, Xiang and Wu, Kun and Zhao, Yongle and Deng, Weimo and Ran, Zimin and Wang, Yumeng and Feng, Ziyong and Miles, Roy and Elezi, Ismail and Deng, Jiankang},
  booktitle={ICCV},
  year={2025}
}

@article{lillava,
  title={LLaVA-OneVision: Easy Visual Task Transfer},
  author={Li, Bo and Zhang, Yuanhan and Guo, Dong and Zhang, Renrui and Li, Feng and Zhang, Hao and Zhang, Kaichen and Zhang, Peiyuan and Li, Yanwei and Liu, Ziwei and Li, Chunyuan},
  journal={Transactions on Machine Learning Research},
  year={2024}
}