3DThinker & 4DThinker

3DThinker: spatial reasoning from a few views3DThinker：从少数几个视角做空间推理

Robots and cars see their surroundings through a handful of cameras, each covering part of the scene. Answering “what is behind the plant, seen from the second photo?” means imagining the whole scene from those partial views. Vision-language models pretrained on image–text data are still weak at this. The 3DThinker paper names two causes: they cannot extract the 3D geometry that the images contain, and their capacity for spatial imagination is limited.

机器人和汽车通过少数几个相机观察周围，每个相机只拍到场景的一部分。要回答“从第二张照片的视角看，植物后面是什么”，就得根据这些局部视角想象出整个场景。在图文数据上预训练的视觉语言模型在这类问题上仍然很弱。3DThinker 论文指出两个原因：模型无法提取图像中蕴含的 3D 几何信息，空间想象能力也有限。

Earlier fixes fall into two groups (Figure 2). The first reasons in pure text or with 2D visual cues, which cannot hold a complex layout. MindCube trains models to draw cognitive maps of the layout, but building them takes bird’s-eye-view annotations; Ego3D-VLM generates them automatically with GroundingDINO and Depth Anything V2, and fails where those models fail, on low-resolution or uncurated images. The second group feeds extra inputs such as point clouds or camera parameters, or calls external encoders and tools for depth maps or 3D tokens. That rules out settings with only monocular images, adds inference cost, and is not a skill of the model itself.

此前的改进分为两类（图 2）。第一类用纯文本或 2D 视觉线索推理，难以容纳复杂的空间布局。MindCube 训练模型画出布局的认知地图，但构建这些地图需要鸟瞰图（BEV）标注；Ego3D-VLM 借助 GroundingDINO 和 Depth Anything V2 自动生成认知地图，这些外部模型失效的地方（低分辨率或未经筛选的图像）它也随之失效。第二类额外输入点云、相机参数等模态，或调用外部编码器和工具获取深度图、3D token。这样就用不到只有单目图像的场景，推理开销也随之增加，而且这并不是模型自身的能力。

3DThinker sets three goals instead: learn 3D geometry directly from a few 2D images (3D-imaginable), without densely annotated data (annotation-free), and with no external prior or auxiliary model at inference (intrinsic). The model imagines the scene inside its own reasoning, as a block of latent tokens tied to the features of a 3D foundation model (Figure 1).

3DThinker 因此设定了三个目标：直接从少量 2D 图像中学习 3D 几何（可想象 3D），不依赖密集标注的数据（无需标注），推理时不借助外部先验或辅助模型（内生）。模型在自己的推理过程中想象场景，形式是一段与 3D 基础模型特征绑定的潜在 token（图 1）。

The idea: a 3D latent block inside the reasoning思路：推理中的一段 3D 潜在表示

People asked where something is in a room they glimpsed picture the room and look around in it. 3DThinker gives a vision-language model a place to do the same. Its output is split as o = o_pre ⊕ t_3D ⊕ o_post, where t_3D is a short run of special tokens standing for the imagined scene. The last-layer hidden states at those positions, F_latent, are what the model “sees” in its mind. They are generated autoregressively like any other token, conditioned on the question, the images and what came before.

人们被问到只匆匆看过一眼的房间里某样东西在哪里时，会在脑中想象出房间，再在里面四处“看看”。3DThinker 给视觉语言模型留出做这件事的位置。它的输出被拆成 o = o_pre ⊕ t_3D ⊕ o_post，其中 t_3D 是一小段代表想象场景的特殊 token。这些位置最后一层的隐状态 F_latent 就是模型在脑中“看到”的东西。它们和其他 token 一样自回归生成，以问题、图像和此前的输出为条件。

The closest earlier work, Mirage, also continues a reasoning trajectory with latent embeddings, but it supervises them with ground-truth image embeddings and stays within “thinking with images”. 3DThinker supervises its latents with a 3D foundation model instead: VGGT reads the same views during training and supplies the target features, so no 3D data has to be built. Training has two stages (Figure 3). Stage 1 teaches the model to write the latent block and aligns it with VGGT; Stage 2 refines the whole trajectory, latents included, from outcome rewards alone.

最接近的已有工作 Mirage 同样用潜在嵌入延续推理轨迹，但它用真实图像的嵌入作监督，停留在“用图像思考”的范式里。3DThinker 改用 3D 基础模型来监督潜在 token：训练时 VGGT 读取同一组视角并给出目标特征，因此无需构建任何 3D 数据。训练分两个阶段（图 3）。阶段 1 教模型写出潜在 token 段并与 VGGT 对齐；阶段 2 只依靠结果奖励优化包括潜在 token 在内的整条轨迹。

Training data with a 3D placeholder带 3D 占位符的训练数据

A vision-language model only writes text, so it first needs examples of reasoning that contain a 3D step. Given the images, the question and the ground-truth answer of an existing question–answer pair, GPT-4.1 writes the step-by-step reasoning and puts an <output_3D> placeholder at the start, followed by <think>…</think><answer>…</answer>. A trace in the wrong format is regenerated, and a sample that still fails after ten attempts is dropped. GPT-4.1 only places the placeholder; the geometry behind it comes from VGGT. That is what the paper means by annotation-free: no manual geometric labels such as ground-truth point clouds.

视觉语言模型只会写文本，因此先要有包含 3D 步骤的推理示例。对于已有的问答对，GPT-4.1 根据图像、问题和标准答案写出逐步推理，在开头放一个 <output_3D> 占位符，后面接 <think>…</think><answer>…</answer>。格式不对的推理会重新生成，尝试 10 次仍不合格的样本被丢弃。GPT-4.1 只负责放置占位符，占位符背后的几何信息来自 VGGT。论文所说的“无需标注”就是这个意思：不需要真实点云这类人工几何标注。

Two training sets appear in the paper. MindCube-only uses 10,000 MindCube question–image–answer pairs: Stage 1 trains on their generated reasoning, and Stage 2 on the plain question, images and answer, with no annotated cognitive maps at any stage. The large-scale curated set draws on MM-Spatial, SPAR, MindCube and SpatialLadder; Gemini 2.5 Pro rated the data quality and the single- and multi-image shares were balanced, leaving 237K samples. Stage 1 trains on all of it for one epoch, Stage 2 on a 57K subset. Results marked * below use the large-scale set.

论文用到两套训练数据。MindCube-only 取 MindCube 中 10,000 个问题–图像–答案三元组：阶段 1 在生成的推理链上训练，阶段 2 只用问题、图像和答案，各阶段都不用标注的认知地图。大规模精选数据来自 MM-Spatial、SPAR、MindCube 和 SpatialLadder；先由 Gemini 2.5 Pro 评估数据质量，再平衡单图与多图样本的比例，最终保留 237K 条。阶段 1 在全部数据上训练一个 epoch，阶段 2 用其中 57K 条。下文标 * 的结果使用这套大规模数据。

You are an advanced AI reasoning assistant with expertise in 3D spatial
understanding. Your task is to complete the intermediate reasoning process
using the provided Question, Images, and Answer. To assist your reasoning,
you should mentally construct a 3D layout of the scene.

Wrap your reasoning process with <think></think> tags and your final result
with <answer></answer> tags. The mental 3D scene you imagine will be
represented by the special token <output_3D>, which should be presented at
the beginning of your response. Only one <output_3D> is allowed, and
</output_3D> should not be included.
# … followed by one worked example

python preprocessing/feature/extract_vggt_feature.py   # VGGT features + resized images
python preprocessing/produce_cot.py                    # chain-of-thought data
python preprocessing/clean.py                          # remove non-compliant data
python preprocessing/remove.py                         # filter useless data
python preprocessing/jsonl_add_idx.py                  # match VGGT indices

images = load_and_preprocess_images(new_image_paths)
images = images.to(device, non_blocking=True).to(dtype).unsqueeze(0)
with torch.no_grad():
    with torch.cuda.amp.autocast(enabled=True, dtype=dtype):
        aggregated_tokens_list, ps_idx = model.aggregator(images)
feature = aggregated_tokens_list[-1].cpu().numpy()   # last aggregator layer
np.savez_compressed(save_path, feature=feature, ps_idx=ps_idx_np)

Stage 1: align the imagined scene with VGGT阶段 1：让想象的场景对齐 VGGT

The obvious way to teach 3D reasoning, making the model output point clouds, needs 3D labels and a model that can generate geometry explicitly. Stage 1 distils instead. The last layer of VGGT’s aggregator gives geometry features F_3D for the input views; the image encoder gives patch features F_images. A projector (Figure 4) turns the latent hidden states, together with the patch features, into F_proj of the same shape as F_3D, and a Frobenius loss pulls them together. Cross-entropy on the text before and after the latent block keeps the reasoning coherent.

教模型做 3D 推理，最直接的办法是让它输出点云，但这需要 3D 标注，还要求模型能显式生成几何。阶段 1 改用蒸馏。VGGT 聚合器的最后一层为输入视角给出几何特征 F_3D，图像编码器给出 patch 特征 F_images。投影器（图 4）把潜在隐状态与 patch 特征一起转换为与 F_3D 同形状的 F_proj，再用 Frobenius 损失拉近两者。潜在 token 段前后的文本仍用交叉熵训练，保证推理通顺。

For the MindCube-only models, Stage 1 uses an MLP depth of 6, a learning rate of 1e-4, twelve latent tokens and ten epochs. In the released code the vision encoder stays frozen while the language model and the projector are trained. The projector maps the VLM’s latents into VGGT’s space rather than VGGT’s features into the VLM’s, so the imagined scene can be decoded back into geometry with VGGT’s own heads (see What 3DThinker’s latents encode below).

对于 MindCube-only 模型，阶段 1 的 MLP 深度为 6，学习率 1e-4，潜在 token 12 个，训练 10 个 epoch。开源代码中视觉编码器保持冻结，训练语言模型和投影器。投影方向是把 VLM 的潜在表示映射到 VGGT 的空间，而不是把 VGGT 的特征压缩到 VLM 的空间，因此想象出的场景可以用 VGGT 自己的预测头解码回几何（见下文“3DThinker 的潜在 token 编码了什么”）。

<|latent_start|><|latent_pad|> … <|latent_pad|><|latent_end|>   # 12 × <|latent_pad|>
<think> … </think><answer> … </answer>

feature_proj = model.projector_model(shift_predict_embeddings, image_embeddings)
data = np.load('../../data/feature_vggt/' + str(idx[0]) + '/vggt.npz')
feature_3d = torch.tensor(data['feature']).to(device=..., dtype=...).squeeze()
feature_sim = ((feature_proj - feature_3d.detach()) ** 2).sum(dim=-1)
sim_loss = feature_sim.mean()*0.0005
loss = 0.1 * ce_loss + sim_loss

python src/main.py \
    --model <path>/Qwen2.5-VL-3B-Instruct --epochs 10 \
    --task mindcube \
    --latent_size 12 \
    --per_device_train_batch_size 1 \
    --learning_rate 1e-4 \
    --warmup_steps 10 \
    --weight_decay 0.01 \
    --logging_steps 20 \
    --save_steps 2000 \
    --save_total_limit 1 \
    --stage stage1 \
    --data_path ../../data/example.jsonl \
    --log_file ./log.txt \
    --save_model_path ../../models/3DThinker-S1-Qwen2.5-VL-3B_mlp6_lr1e-4_latent12 \
    --wandb_name 3DThinker-S1-Qwen2.5-VL-3B_mlp6_lr1e-4_latent12

Stage 2: refine the whole trajectory from outcomes阶段 2：仅凭结果优化整条轨迹

After Stage 1 the model writes formatted latent blocks and the projector is trained. Stage 2 uses only outcome signals to improve the sampled trajectories and, with them, the imagined scene. It runs outcome-based GRPO: for each question and its images, the current policy samples N completions (eight in the paper), each completion’s reward is normalised against its group, and the policy maximises the clipped objective with a KL penalty to the reference model. The projector is frozen in this stage, and VGGT’s features still constrain the latents through a reward rather than a loss.

阶段 1 之后，模型已经能写出格式正确的潜在 token 段，投影器也训练好了。阶段 2 只用结果信号来改进采样轨迹，并随之改进想象出的场景。它采用基于结果的 GRPO：对每个问题及其图像，当前策略采样 N 条回答（论文中为 8 条），每条回答的奖励在组内归一化，策略最大化带截断的目标函数，并对参考模型施加 KL 惩罚。这一阶段投影器冻结，VGGT 的特征不再作为损失，而是通过奖励继续约束潜在 token。

Three rewards are summed with weight 1 each. The 3D reward r_3D runs each rollout’s latent hidden states through the frozen projector and compares the result with VGGT’s features by cosine similarity, mapped to [0, 1]. The format reward r_format is 1.0 when the output follows …<|latent_start|><|latent_pad|>…<|latent_end|>…<think>…</think><answer>…</answer>, and the answer reward r_ans is 1 when the chosen option matches the ground truth and 0 otherwise. Stage 2 uses a learning rate of 1e-5 and eight rollouts per question.

三项奖励各以权重 1 相加。3D 奖励 r_3D 把每条轨迹的潜在隐状态送入冻结的投影器，与 VGGT 特征计算余弦相似度，并映射到 [0, 1]。格式奖励 r_format 在输出符合 …<|latent_start|><|latent_pad|>…<|latent_end|>…<think>…</think><answer>…</answer> 时为 1.0；答案奖励 r_ans 在所选选项与标准答案一致时为 1，否则为 0。阶段 2 的学习率为 1e-5，每个问题采样 8 条轨迹。

decoder_feature = projector_model(extracted_emb.to(target_dtype).to("cuda:0"),
                                  image_embeddings.to(target_dtype).to("cuda:0"))
decoder_feature_norm = decoder_feature / decoder_feature.norm(dim=-1, p=2, keepdim=True)
data = np.load('../../data/feature_vggt/' + str(idx) + '/vggt.npz')
feature_3d = torch.tensor(data['feature']).to(...).squeeze()
feature_3d_norm = feature_3d / feature_3d.norm(dim=-1, p=2, keepdim=True)
cos_sim = torch.nn.functional.cosine_similarity(decoder_feature_norm, feature_3d_norm).mean()
reward = (cos_sim + 1) / 2

model_path="../../models/3DThinker-S1-Qwen2.5-VL-3B_mlp6_lr1e-4_latent12"
export EXP_NAME="3DThinker-S1+S2-Qwen2.5-VL-3B_lr1e-5_gen8"
torchrun --nproc_per_node="8" --nnodes="1" --node_rank="0" \
    --master_addr="127.0.0.1" --master_port="12346" \
  src/open_r1/grpo_jsonl.py \
    --use_vllm True \
    --model_name_or_path $model_path \
    --is_reward_customized_from_vlm_module True \
    --task_type rec \
    --per_device_train_batch_size 1 \
    --gradient_accumulation_steps 2 \
    --num_train_epochs 1 \
    --num_generations 8 \
    --max_completion_length 2048 \
    --reward_funcs format response latent_sim \
    --beta 0.04 \
    --learning_rate 1e-5 \
    --use_peft true --lora_r 64 --lora_alpha 128 --lora_dropout 0.05 \
    --freeze_vision_modules true \
    --deepspeed local_scripts/zero3.json --bf16

Training curves and cost (Figure 5)训练曲线与开销（图 5）

On MindCube-only, the Stage 1 loss converges after about 20k steps at around 0.15; the Stage 2 reward settles much sooner, after about 500 steps at around 2.7 out of 3. The format and answer rewards both dip early, which the paper attributes to exploration at the start of RL, then rise steadily. For Qwen2.5-VL-3B on a single H200 with batch size 1, Stage 1 takes 21.84 h and Stage 2 12.85 h to converge.

在 MindCube-only 数据上，阶段 1 的损失约在 2 万步后收敛到 0.15 左右；阶段 2 的奖励收敛得快得多，约 500 步后稳定在 2.7 左右（满分 3）。格式奖励和答案奖励在早期都有短暂下降，论文认为这是强化学习初期的探索所致，之后稳步上升。对于 Qwen2.5-VL-3B，在单张 H200、批大小为 1 时，阶段 1 和阶段 2 分别需要 21.84 小时和 12.85 小时收敛。

4DThinker: from a still scene to a moving one4DThinker：从静止的场景到运动的场景

3DThinker imagines a static scene from a few photos. Many questions about the physical world are about change instead: how the camera moved, which way a car is heading, whether a person slows down. Answering them from a monocular video means separating the camera’s ego-motion from the motion of objects, which matters for driving and robotics, and recent benchmarks show that even the strongest vision-language models fail at basic cases. Two kinds of fixes exist. One writes the spatial-temporal reasoning out as text, which is verbose and imprecise for complex motion; the other adds external modules, such as 3D foundation models that inject geometric priors or mask decoders for grounding, which makes inference heavier and leaves the skill outside the model. Latent reasoning methods, 3DThinker included, have so far covered static scenes and relied on annotated reference images or distilled foundation models for supervision, which is hard to scale to video.

3DThinker 根据几张照片想象静态场景。但关于物理世界的很多问题问的是变化：相机怎么移动、汽车往哪个方向开、一个人是否在减速。要从单目视频回答这些问题，需要把相机自身的运动和物体的运动区分开，这对自动驾驶和机器人至关重要；近期的基准显示，即使最强的视觉语言模型也会在基本情形上出错。现有的改进有两类：一类把时空推理全部写成文字，面对复杂运动时既冗长又不精确；另一类增加外部模块，例如注入几何先验的 3D 基础模型或用于定位的掩码解码器，这会让推理更重，能力也不在模型自身。包括 3DThinker 在内的潜在推理方法目前只覆盖静态场景，并依赖带标注的参考图像或蒸馏的基础模型作为监督，很难扩展到视频。

4DThinker sets three goals: extend latent visual reasoning to how a scene evolves (imagery-dynamic), keep the skill inside the model with no external geometric module (model-intrinsic), and scale without manual annotation (data-scalable). It takes its cue from how people follow motion: they anchor on static landmarks to judge their own movement and track the trajectories of moving things. 4DThinker highlights such landmarks with mask overlays and treats the highlighted frames as the imagery the model learns to picture in its own hidden space (Figure 6).

4DThinker 设定了三个目标：把潜在视觉推理扩展到场景随时间的演变（动态想象），让能力留在模型内部、不借助外部几何模块（模型内生），并且无需人工标注就能扩展（数据可扩展）。它借鉴了人类追踪运动的方式：以静止的地标为参照判断自身的移动，同时跟踪运动物体的轨迹。4DThinker 用掩码叠加图突出这些地标，并把这些高亮的帧当作模型要在自身隐空间中学会想象的画面（图 6）。

Table 1 lists what changes from 3D to 4D. The biggest change is the latent target. 3DThinker aligns its latents with an external 3D teacher through a trained projector; 4DThinker uses the base model’s own frozen vision encoder, applied to the overlay frames, so no other model is distilled and no projector is needed. The imagery also moves inside the reasoning, one block per moment in time, and the reinforcement stage stops pushing gradients through the latent positions (Figure 7).

表 1 列出了从 3D 到 4D 的变化。最大的变化是潜在 token 的目标：3DThinker 通过训练好的投影器把潜在 token 对齐到外部的 3D 教师模型；4DThinker 则把基座模型自己冻结的视觉编码器作用在叠加图上，不再蒸馏其他模型，也不需要投影器。想象图也移到了推理文本内部，每个时刻一段；强化学习阶段则不再让梯度经过潜在位置（图 7）。

4D reasoning data from raw video, without labels无需标注、直接从原始视频生成 4D 推理数据

Hand-labelling spatial-temporal data is expensive and does not scale, and 4DThinker also needs reasoning traces that contain imagery. Its pipeline builds both from SpatialVID, a collection of 2.7M clips (7,089 hours) whose camera poses were estimated automatically by MegaSaM; nothing in it needs human annotation (Figure 8). Frames are sampled once per second. Gemini-3-pro picks, by fixed rules, one static object (say, a red building) and one moving object (a person on a blue bike) that persist through the video. SAM3 tracks each to give consistent masks, and each frame gets a mask overlay that tints the object, Î_t = (1 − αM_t) ⊙ I_t + αM_t ⊙ c, with α = 0.6 and red as c. Gemini then cross-checks the overlays and keeps only those whose object identity stays consistent.

人工标注时空数据成本高、无法扩展，而 4DThinker 还需要包含想象图的推理链。它的数据流程从 SpatialVID 同时构造这两者：SpatialVID 有 270 万段视频（7,089 小时），相机位姿由 MegaSaM 自动估计，整个流程不需要人工标注（图 8）。视频按每秒一帧采样。Gemini-3-pro 按固定规则选出一个贯穿视频的静止物体（例如一栋红色建筑）和一个运动物体（例如骑蓝色自行车的人）。SAM3 逐帧跟踪，得到一致的掩码，每帧再生成一张给物体染色的叠加图：Î_t = (1 − αM_t) ⊙ I_t + αM_t ⊙ c，其中 α = 0.6，c 为红色。随后 Gemini 交叉核对这些叠加图，只保留物体身份前后一致的帧。

The data is split along two axes. For camera motion, SpatialVID’s per-segment labels of 12 movement types cut the video into segments; Gemini turns a segment’s label into a multiple-choice question, and the imagery is the static object’s overlay at the segment’s first and last frames. A static object does not move, so its apparent shift in the image is caused entirely by the camera. For object motion, questions ask about direction, distance to the camera, speed changes, and direction with a bounding box; Gemini works out the answer by analysing the moving object’s overlays, and the imagery is 2 to 5 of them, always including the first and last frames of the motion. Gemini finally writes the reasoning in the form <think>…<imagery>…<imagery>…</think><answer>…</answer>, and a validator checks the tags, the placeholder count and the answer. This yields about 38K samples for supervised training; the reinforcement stage uses DSR-Train, about 37K question–answer pairs on compound camera and object motion, without reasoning traces.

数据沿两个轴构造。相机运动：SpatialVID 为每个片段给出 12 种运动类型之一的标签，据此把视频切成若干片段；Gemini 把片段的标签改写成选择题，想象图是静止物体在片段首尾两帧的叠加图。静止物体本身不动，它在画面中的位移完全由相机运动造成。物体运动：问题涉及方向、与相机的距离、速度变化，以及带边界框的方向描述；Gemini 通过分析运动物体的叠加图得出答案，想象图取其中 2 到 5 张，且必须包含运动区间的首帧和末帧。最后由 Gemini 写出形如 <think>…<imagery>…<imagery>…</think><answer>…</answer> 的推理，并由校验器检查标签、占位符数量和答案。这样得到约 38K 条监督训练样本；强化学习阶段使用 DSR-Train，约 37K 个关于相机与物体复合运动的问答对，不含推理链。

cd preprocess
export OPENAI_API_KEY=your_api_key
export OPENAI_BASE_URL=https://api.openai.com/v1
export DATA_BASE_DIR=/path/to/your/data
bash run.sh                     # frames + SAM3 masks + object identification
python merge_jsonl.py
python generate_camera_qa.py    # camera motion questions + CoT
python generate_dynamic_qa.py   # object motion questions + CoT
python convert_format.py ./camera_data_qa_all.jsonl ./camera_qa_converted.jsonl
python convert_format.py ./dynamic_data_qa_all.jsonl ./dynamic_qa_converted.jsonl
python check_output_image.py ./camera_qa_converted.jsonl
python check_output_image.py ./dynamic_qa_converted.jsonl

DIFT and 4DRL: learning to think with 4DDIFT 与 4DRL：学会用 4D 思考

Each overlay becomes a short run of latent visual tokens. The base model’s vision encoder f_vis turns the overlay into L patch embeddings of dimension D; the sequence is split into K equal parts and each part is averaged, giving K latent tokens z⁽¹⁾…z^(K), with K = 4. Every imagery placeholder in the reasoning is replaced by a latent block, a start token, the K latent vectors and an end token, so training sequences interleave discrete text with continuous embeddings, and the model reasons through the imagery without leaving autoregressive generation.

每张叠加图都会变成一小段潜在视觉 token。基座模型的视觉编码器 f_vis 把叠加图编码为 L 个 D 维 patch 嵌入；把这个序列等分成 K 份、每份取平均，得到 K 个潜在 token z⁽¹⁾…z^(K)，K = 4。推理中的每个想象图占位符都替换为一段潜在 token：起始 token、K 个潜在向量和结束 token。这样训练序列中离散的文本与连续的嵌入交错出现，模型无需离开自回归生成就能借助想象图推理。

Dynamic-imagery fine-tuning (DIFT). The video frames enter as visual tokens, followed by the question and the reasoning with its latent blocks. The vision encoder is frozen, so the targets stay in a fixed embedding space. Cross-entropy trains the text positions; at the latent positions, a next-embedding objective asks the hidden state before each position, h_t−1, to point in the direction of the target z_t. At inference this becomes a recurrent loop: inside a latent block the model’s input embedding is its own previous hidden state, so each step of imagined motion feeds the next, and no overlay is ever shown.

动态想象微调（DIFT）。 视频帧作为视觉 token 输入，后接问题和带潜在 token 段的推理。视觉编码器保持冻结，使目标始终处在固定的嵌入空间中。交叉熵训练文本位置；在潜在位置上，“下一嵌入预测”目标要求每个位置之前的隐状态 h_t−1 指向目标 z_t 的方向。推理时这变成一个循环：在潜在 token 段内，模型的输入嵌入就是它自己上一位置的隐状态，想象中的每一步运动都传给下一步，整个过程不需要任何叠加图。

4D reinforcement learning (4DRL). DIFT only sees one kind of motion per sample, so 4DRL trains on DSR-Train’s compound motions with a modified GRPO. For each question the policy samples G = 8 answers, each scored by R = λ_accR_acc + λ_fmtR_fmt: 1 for the correct option and 1 for the think-with-4D format, weighted 1.0 and 0.2. Advantages are normalised within the group, and the clipped objective is regularised by a KL term to the frozen DIFT model. The change from standard GRPO is that the loss runs only over text positions: at latent positions the input is a continuous hidden state, not a sampled token, so its log-probability does not describe what happened, and including it adds noisy gradients. Unlike 3DThinker’s Stage 2 there is no reward on the latents.

4D 强化学习（4DRL）。 DIFT 的每条样本只涉及一类运动，因此 4DRL 用改进的 GRPO 在 DSR-Train 的复合运动上训练。对每个问题，策略采样 G = 8 个回答，每个回答的得分为 R = λ_accR_acc + λ_fmtR_fmt：选项正确得 1，符合“用 4D 思考”的格式得 1，权重分别为 1.0 和 0.2。优势在组内归一化，带截断的目标函数再以对冻结 DIFT 模型的 KL 项正则。与标准 GRPO 的区别在于损失只计算文本位置：潜在位置的输入是连续的隐状态，而不是采样出的 token，它的对数概率并不对应实际发生的事情，计入只会带来噪声梯度。与 3DThinker 的阶段 2 不同，这里没有针对潜在 token 的奖励。

# DIFT environment
conda create -n 4dthinker python=3.10 -y
conda activate 4dthinker
pip install -r dift/requirements_dift.txt
cd dift && pip install -e ./transformers/ && cd ..

# 4DRL environment
conda create -n 4dthinker-rl python=3.10 -y
conda activate 4dthinker-rl
pip install -r 4drl/requirements_4drl.txt
cd 4drl
pip install -e ./transformers_rl/
cp -rf ./trl $(python -c "import site; print(site.getsitepackages()[0])")/trl
pip install -e ./src/open-r1-multimodal/
cd ..

# Training
conda activate 4dthinker && bash dift/train.sh
conda activate 4dthinker-rl && bash 4drl/run_scripts/train_4dthinker.sh

# DSR-Bench evaluation
CUDA_VISIBLE_DEVICES=0 python evaluation/dsr_eval.py \
    --model_path model/dift/checkpoints \
    --benchmark_path ./raw_data/DSR_Suite-Data/benchmark.parquet \
    --video_root ./raw_data/DSR-data/bmk_video \
    --latent_size 4

3DThinker results3DThinker 的结果

Multiple-choice questions are scored by exact-match accuracy; numerical answers use the mean relative accuracy of VSI-Bench. MindCube-Tiny has 1,050 questions over three camera motions: Rotation (the camera turns in place), Around (it circles the objects) and Among (it moves among them). Ego3D-Bench has over 8,600 questions on ego-centric multi-view outdoor scenes: absolute distance, relative distance, localisation, motion and travel time, asked from the ego vehicle or from an object.

选择题按完全匹配计算准确率；数值题采用 VSI-Bench 的平均相对准确率（MRA）。MindCube-Tiny 有 1,050 道题，涵盖三种相机运动：Rotation（相机原地旋转）、Around（绕物体环行）和 Among（在物体之间穿行）。Ego3D-Bench 有 8,600 多道题，场景是以自我为中心的多视角户外环境，题型包括绝对距离、相对距离、定位、运动和行程时间，分别从自车和物体的角度提问。

Generalist VLMs on MindCube-Tiny and Ego3D-Bench通用 VLM 在 MindCube-Tiny 与 Ego3D-Bench 上的表现

Table 4 trains eight Qwen2.5-VL and InternVL3 models on MindCube-only and every one improves on both benchmarks. On Qwen2.5-VL-3B, Stage 1 lifts MindCube-Tiny from 33.2 to 62.7 (+88.9%) and Stage 2 to 75.2 (+19.9%); on Ego3D-Bench it goes from 39.1 to 46.7 and then 50.8. Across the eight models, S1+S2 raises the MindCube-Tiny overall score by 58.1% to 126.5% and the Ego3D-Bench average by 18.8% to 40.1%. Ego3D-Bench is out of domain: no Ego3D data is used in training. The best model, InternVL3-78B with both stages, scores 78.9 on MindCube-Tiny and 73.3 on Ego3D-Bench, above every closed model, o3 included (56.6 and 73.0).

表 4 在 MindCube-only 数据上训练了 8 个 Qwen2.5-VL 和 InternVL3 模型，每个模型在两个基准上都有提升。以 Qwen2.5-VL-3B 为例，阶段 1 把 MindCube-Tiny 从 33.2 提高到 62.7（+88.9%），阶段 2 再提高到 75.2（+19.9%）；Ego3D-Bench 从 39.1 提高到 46.7，再到 50.8。在这 8 个模型上，S1+S2 使 MindCube-Tiny 总分提高 58.1% 到 126.5%，Ego3D-Bench 平均分提高 18.8% 到 40.1%。Ego3D-Bench 属于域外测试：训练中没有用到任何 Ego3D 数据。最好的模型是经过两个阶段训练的 InternVL3-78B，在 MindCube-Tiny 和 Ego3D-Bench 上分别得到 78.9 和 73.3，超过所有闭源模型，包括 o3（56.6 和 73.0）。

Travel time is the weak spot. Stage 2 leaves it at or slightly below Stage 1 on six of the eight models, and on Qwen2.5-VL-3B it never moves past the base model (30.9, then 30.8 and 30.8). The paper’s explanation is that mapping normalised 3D representations onto real-world distances and times needs richer context.

行程时间是短板。在 8 个模型中有 6 个，阶段 2 在这一项上与阶段 1 持平或略低；在 Qwen2.5-VL-3B 上，它始终没有超过基座模型（30.9，之后为 30.8 和 30.8）。论文的解释是，要把归一化的 3D 表示对应到真实世界的距离和时间，还需要更丰富的上下文信息。

Against spatial models on six more benchmarks在另外六个基准上对比空间推理模型

Table 5 trains on the large-scale curated set and compares with recent spatial models built on the same base. On Qwen2.5-VL-3B, 3DThinker averages 55.3 after Stage 1 and 60.4 after Stage 2, 11.5% and 21.8% above SpatialLadder-3B (49.6). On Qwen2.5-VL-7B the margin over VILASR-7B (48.4) is 22.7% and 33.7% (59.4 and 64.7). SpatialLadder-3B leads on its own SPBench, while 3DThinker improves on every benchmark. It also stays strong on single-image CV-Bench (78.4 and 81.1), where Video-R1 falls below its base model.

表 5 使用大规模精选数据训练，并与基于同一基座的近期空间推理模型比较。在 Qwen2.5-VL-3B 上，3DThinker 阶段 1 后平均 55.3，阶段 2 后 60.4，分别比 SpatialLadder-3B（49.6）高 11.5% 和 21.8%。在 Qwen2.5-VL-7B 上，相对 VILASR-7B（48.4）的优势为 22.7% 和 33.7%（59.4 和 64.7）。SpatialLadder-3B 在它自己的 SPBench 上领先，而 3DThinker 在每个基准上都有提升。在单图的 CV-Bench 上它同样表现很好（78.4 和 81.1），Video-R1 在这里反而低于其基座模型。

Against Ego3D-VLM across model sizes在不同模型规模上对比 Ego3D-VLM

Ego3D-VLM builds a cognitive map with Grounding-DINO-Base for referring expressions and Depth-Anything-V2-Metric-Large for depth. 3DThinker uses no external module at inference, yet matches or beats it on all eight models (Table 6), by 14.4% on Qwen2.5-VL-3B (50.8 vs 44.4); the two tie on InternVL3-38B.

Ego3D-VLM 借助 Grounding-DINO-Base（指代表达理解）和 Depth-Anything-V2-Metric-Large（深度估计）构建认知地图。3DThinker 推理时不依赖任何外部模块，却在全部 8 个模型上持平或超过它（表 6），在 Qwen2.5-VL-3B 上高出 14.4%（50.8 对 44.4）；两者在 InternVL3-38B 上打平。

Qwen3-VL and LLaVA-OneVision-1.5Qwen3-VL 与 LLaVA-OneVision-1.5

The supplementary material repeats the recipe on newer bases. Qwen3-VL already reserves <think> as a special token, so its reasoning tag becomes <thinking>, and the mixture-of-experts Qwen3-VL-30B-A3B is trained in Stage 2 with sequence-level GSPO instead of token-level GRPO for stability. MindCube-Tiny rises from 26.0 to 75.8 on Qwen3-VL-4B, from near chance to accurate, and Ego3D-Bench by up to 29.9% (43.2 to 56.1 on Qwen3-VL-2B) (Table 7). On LLaVA-OneVision-1.5, the 8B model goes from 37.9 to 63.7 on MindCube-Tiny and from 45.3 to 48.7 on Ego3D-Bench (Table 8).

补充材料在更新的基座上复用了同一套方法。Qwen3-VL 已把 <think> 预留为特殊 token，因此推理标签改为 <thinking>；混合专家模型 Qwen3-VL-30B-A3B 在阶段 2 中改用序列级的 GSPO 代替 token 级的 GRPO，以保持训练稳定。MindCube-Tiny 在 Qwen3-VL-4B 上从 26.0 提升到 75.8，从接近随机猜测变为高准确率；Ego3D-Bench 最多提升 29.9%（Qwen3-VL-2B 从 43.2 到 56.1）（表 7）。在 LLaVA-OneVision-1.5 上，8B 模型的 MindCube-Tiny 从 37.9 提高到 63.7，Ego3D-Bench 从 45.3 提高到 48.7（表 8）。

3DSRBench and general benchmarks3DSRBench 与通用基准

On 3DSRBench, 3DThinker scores 65.6, 8.79% above the best earlier method, SpatialReasoner (60.3) (Table 9). On general image benchmarks the Qwen2.5-VL-7B-based model keeps its base model’s abilities: POPE rises from 85.9 to 88.4 and SEED-Image from 77.0 to 78.9, while MME, mostly 2D tasks such as OCR and numerical reasoning, stays comparable (Table 10).

在 3DSRBench 上，3DThinker 得到 65.6，比此前最好的 SpatialReasoner（60.3）高 8.79%（表 9）。在通用图像基准上，基于 Qwen2.5-VL-7B 的模型保留了基座模型的能力：POPE 从 85.9 提高到 88.4，SEED-Image 从 77.0 提高到 78.9；以 OCR、数值推理等 2D 任务为主的 MME 与基座相当（表 10）。

4DThinker results4DThinker 的结果

Both benchmarks are multiple choice and scored by exact-match accuracy. DSR-Bench measures how geometric quantities evolve in dynamic scenes, in 13 subtasks: distance, direction, orientation, speed, speed comparison and direction prediction, each with an absolute viewpoint fixed at one moment (A.) or a relative one that moves with the observer (R.), plus free-form non-template questions (N-Temp). Dyn-Bench tests semantic understanding of dynamics on 1K videos and 7K questions along three axes: inter-object, object–scene and camera–object; 4DThinker does not use its mask data.

两个基准都是选择题，按完全匹配计算准确率。DSR-Bench 衡量动态场景中几何量如何变化，共 13 个子任务：距离、方向、朝向、速度、速度比较和方向预测，每类又分为固定在某一时刻的绝对视角（A.）和随观察者移动的相对视角（R.），另有非模板的自由提问（N-Temp）。Dyn-Bench 在 1K 段视频、7K 个问题上考察对动态内容的语义理解，分为物体之间、物体与场景、相机与物体三个维度；4DThinker 没有使用其中的掩码数据。

Fine-grained 4D reasoning on DSR-BenchDSR-Bench 上的细粒度 4D 推理

4DThinker improves the average of every base model (Table 11). On Qwen2.5-VL-3B, DIFT adds 6.5 points (24.6 to 31.1) and 4DRL brings it to 34.2; one subtask, R.SpdC, drops there (29.0 to 26.8). The gains grow with scale: Qwen3-VL-32B goes from 28.0 to 62.0 (+34.0), above Gemini-2.5-Pro (31.7) and above DSR Suite-Model (58.9), the previous best, which injects 3D priors through a geometry selection module; InternVL3.5-38B gains 32.7 (26.7 to 59.4). The largest jumps come on absolute subtasks where base models sit near chance, around 20%: A.Dis on Qwen3-VL-32B rises from 14.5 to 84.3. The paper reads this as the 4D latents supplying geometric grounding that language-only reasoning lacks.

4DThinker 提升了每一个基座模型的平均分（表 11）。在 Qwen2.5-VL-3B 上，DIFT 提高 6.5 分（24.6 到 31.1），加上 4DRL 后达到 34.2；其中 R.SpdC 一项有所下降（29.0 到 26.8）。模型越大提升越明显：Qwen3-VL-32B 从 28.0 提高到 62.0（+34.0），超过 Gemini-2.5-Pro（31.7），也超过此前最好的 DSR Suite-Model（58.9），后者借助几何选择模块注入 3D 先验；InternVL3.5-38B 提高 32.7 分（26.7 到 59.4）。提升最大的是基座模型接近随机水平（约 20%）的绝对视角子任务：Qwen3-VL-32B 的 A.Dis 从 14.5 提高到 84.3。论文认为这说明 4D 潜在 token 提供了纯语言推理所缺的几何依据。

Holistic dynamic understanding on Dyn-BenchDyn-Bench 上的整体动态理解

On Dyn-Bench the three best averages all come from 4DThinker (Table 12). Qwen2.5-VL-7B with both stages reaches 65.9 (+11.6), above GPT-5 (61.4), Gemini-2.5-Pro (58.8) and every dedicated spatial model, including SpaceR-7B (59.2), VST-7B-RL (58.8) and SpatialLadder-3B (56.2); Qwen3-VL-32B reaches 75.4 (+10.9), the best result. The camera–object axis, which requires separating ego-motion from object motion, improves on every base model, by 12.0 to 13.6 points.

在 Dyn-Bench 上，平均分最高的三个结果都来自 4DThinker（表 12）。经过两个阶段训练的 Qwen2.5-VL-7B 达到 65.9（+11.6），超过 GPT-5（61.4）、Gemini-2.5-Pro（58.8）以及所有专门的空间模型，包括 SpaceR-7B（59.2）、VST-7B-RL（58.8）和 SpatialLadder-3B（56.2）；Qwen3-VL-32B 达到 75.4（+10.9），为最佳结果。需要把自身运动与物体运动分开的“相机与物体”维度在每个基座上都有提升，幅度为 12.0 到 13.6 分。

What 3DThinker’s latents encode3DThinker 的潜在 token 编码了什么

Compared with other ways to train与其他训练方式的比较

Table 13 trains Qwen2.5-VL-3B on MindCube-only in several ways. With supervised training alone, 3DThinker reaches 62.7, above raw question–answer SFT (52.3), CoT SFT (53.4) and the best cognitive-map SFT from the MindCube paper (60.8, +3.1%). It trails the cognitive-map baselines on Rotation (44.0 against 49.5 and 47.5), which needs dynamic imagination; the paper reads its supervised stage as mainly static spatial understanding, and Stage 2 adds the dynamic part by optimising whole trajectories. With RL, 3DThinker reaches 75.2, 6.4% above both cognitive-map SFT + RL baselines (70.7), while GRPO on the plain model gives 50.6.

表 13 用几种方式在 MindCube-only 数据上训练 Qwen2.5-VL-3B。仅用监督训练时，3DThinker 达到 62.7，高于原始问答 SFT（52.3）、CoT SFT（53.4）以及 MindCube 论文中最好的认知地图 SFT（60.8，+3.1%）。它在 Rotation 上落后于认知地图基线（44.0 对 49.5 和 47.5），这类题需要动态想象；论文认为监督阶段主要学到的是静态空间理解，阶段 2 通过优化整条轨迹补上动态能力。加入强化学习后，3DThinker 达到 75.2，比两种认知地图 SFT + RL 基线（均为 70.7）高 6.4%；直接在原模型上做 GRPO 只有 50.6。

Decoding the imagined scene解码想象出的场景

Because the projector maps into VGGT’s feature space, the latents can be turned back into geometry. At inference the paper takes the last-layer hidden states of the special tokens, projects them, and runs VGGT’s DPT head on the result to get a point cloud (Figure 9). The clouds roughly depict the scene, and the clearer regions tend to be the objects the question asks about, which suggests the latents encode a mental scene steered by the prompt. Both examples are answered correctly. Latent reasoning is usually opaque; here the intermediate state can be inspected.

由于投影器映射到 VGGT 的特征空间，潜在 token 可以还原为几何。推理时，论文取特殊 token 的最后一层隐状态，经投影器投影后交给 VGGT 的 DPT 头，得到点云（图 9）。这些点云大致描绘出了场景，而且较清晰的区域往往正是问题所问的物体，说明潜在 token 编码的是受提示引导的心理场景。两个例子都回答正确。潜在推理通常难以解释，这里的中间状态却可以直接查看。

Are the clear regions really prompt-relevant? (Figure 10)清晰区域真的与提示相关吗？（图 10）

The supplementary material measures it: the density of the reconstructed points inside a 3D box around the object in question is 196,494.67 points per unit³, against 1,332.54 for the whole cloud. The repository’s tests/visual_decoder.py does the decoding: it replaces the last entry of VGGT’s aggregated token list with the projected latents and runs VGGT’s camera, depth and point heads.

补充材料对此做了定量测量：在问题所涉物体周围的 3D 框内，重建点的密度为每立方单位 196,494.67 个点，而整个点云只有 1,332.54。仓库中的 tests/visual_decoder.py 实现了解码：它把 VGGT 聚合 token 列表的最后一项替换为投影后的潜在表示，再运行 VGGT 的相机头、深度头和点云头。

Ablations: latent size, token position, projector direction, rewards and L_3D (Tables 14–16, Figure 11)消融：潜在 token 数、位置、投影方向、奖励与 L_3D（表 14–16、图 11）

Latent size. About twelve tokens work best (Table 14). Fewer limit what the latents can represent; more compromise the model’s natural expression, and it starts repeating <|latent_start|> without reaching an answer: 32 tokens fall to 25.1 and 64 to 15.5.

潜在 token 数。 12 个左右最好（表 14）。太少会限制潜在表示的容量；太多会损害模型自然的表达能力，模型开始反复输出 <|latent_start|>，给不出答案：32 个时降到 25.1，64 个时降到 15.5。

Table 14Number of latent tokens, 3DThinker-S1 on Qwen2.5-VL-3B, MindCube-Tiny overall accuracy %. The released setting, 12, is shaded.表 14潜在 token 数量，Qwen2.5-VL-3B 上的 3DThinker-S1，MindCube-Tiny 总体准确率（%）。开源设置 12 加底色。

Latent tokens潜在 token 数	4	8	12	16	32	64
MindCube-Tiny overallMindCube-Tiny 总体	60.2	60.6	62.7	59.9	25.1	15.5

Where the block goes. Between <think> and </think> the latents can resemble character features, producing garbled text and early stops: 42.0 against 75.2 (Table 15). At the end, after </answer>, they score 74.3; isolated from the text at either end, they work.

放在哪里。 放在 <think> 与 </think> 之间时，潜在表示可能与某些字符特征相似，导致乱码和提前终止：42.0 对 75.2（表 15）。放在末尾（</answer> 之后）得到 74.3；只要与文本隔开，放在开头或末尾都可行。

Projector direction. Compressing VGGT’s features into the VLM’s space, for example by adaptive average pooling, needs no trained projector but cannot be decoded back into 3D, and scores slightly lower (74.1 vs 75.2) (Figure 11).

投影方向。 把 VGGT 的特征压缩到 VLM 的空间（例如用自适应平均池化）不需要训练投影器，但无法解码回 3D，得分也略低（74.1 对 75.2）（图 11）。

Rewards. Dropping the format reward barely matters (74.8). Dropping r_3D costs 6.9 points (68.3), as the latents lose their stable constraint; dropping the answer reward, the only ground-truth signal, costs 11.0 (64.2). Even without r_3D, Stage 2 still improves on Stage 1 by 8.93% (68.3 vs 62.7), so the RL stage does not need the teacher.

奖励。 去掉格式奖励几乎没有影响（74.8）。去掉 r_3D 损失 6.9 分（68.3），因为潜在表示失去了稳定的约束；去掉答案奖励这个唯一的真值信号，损失 11.0 分（64.2）。即使没有 r_3D，阶段 2 仍比阶段 1 高 8.93%（68.3 对 62.7），可见强化学习阶段并不依赖教师模型。

Table 15Design ablations for 3DThinker-S1+S2 on Qwen2.5-VL-3B, MindCube-Tiny overall accuracy %. Each column changes one thing from the full model, which places the block at the beginning, projects VLM → VGGT and uses all three rewards.表 153DThinker-S1+S2（Qwen2.5-VL-3B）的设计消融，MindCube-Tiny 总体准确率（%）。每一列只改动完整模型的一处设计；完整模型把潜在 token 段放在开头，投影方向为 VLM → VGGT，并使用全部三项奖励。

	3D tokens placed3D token 位置		Projector投影方向	Stage-2 rewards阶段 2 奖励			Full完整
	Middle中间	End末尾	VGGT → VLM	w/o rformat去掉 rformat	w/o rans去掉 rans	w/o r3D去掉 r3D
MindCube-Tiny overallMindCube-Tiny 总体	42.0	74.3	74.1	74.8	64.2	68.3	75.2

The 3D loss. Removing VGGT from Stage 1, so that only the cross-entropy trains the tokens around the special tokens, drops MindCube-Tiny from 62.7 to 54.1 (Table 16). That is still slightly above CoT SFT (53.4): the special tokens alone add some representational room, but the geometry comes from the teacher.

3D 损失。 在阶段 1 中去掉 VGGT、只用交叉熵训练特殊 token 之外的 token 时，MindCube-Tiny 从 62.7 降到 54.1（表 16）。这仍略高于 CoT SFT（53.4）：特殊 token 本身带来了一些表示上的余地，但几何信息来自教师模型。

Table 16The 3D alignment loss in Stage 1, Qwen2.5-VL-3B, MindCube-Tiny accuracy %, from the 3DThinker paper’s supplementary material.表 16阶段 1 中的 3D 对齐损失，Qwen2.5-VL-3B，MindCube-Tiny 准确率（%），取自 3DThinker 论文补充材料。

Stage 1 loss阶段 1 损失	Rotation旋转	Among穿行	Around环绕	Overall总体
w/o L3D (CE only)去掉 L3D（仅 CE）	35.0	55.1	66.8	54.1
Full: Ltext + L3D完整：Ltext + L3D	44.0	64.8	72.4	62.7

What 4D imagery adds4D 想象带来了什么

Compared with other ways to train与其他训练方式的比较

The ablations use Qwen2.5-VL-3B on DSR-Bench. With supervised training alone, fine-tuning on the raw questions (25.1) or on text-only reasoning (26.8) barely moves the base model (24.6), while DIFT reaches 31.1, 4.3 points above CoT SFT (Table 17). With reinforcement learning, plain GRPO gives 27.1 and CoT SFT + GRPO 29.7, against 34.2 for DIFT + 4DRL. The paper’s reading: reasoning in discrete text alone hits an expressiveness ceiling that the latent imagery lifts.

消融实验使用 Qwen2.5-VL-3B 和 DSR-Bench。仅用监督训练时，在原始问答上微调（25.1）或在纯文本推理上微调（26.8）几乎没有改变基座模型（24.6），而 DIFT 达到 31.1，比 CoT SFT 高 4.3 分（表 17）。加入强化学习后，直接做 GRPO 得到 27.1，CoT SFT + GRPO 得到 29.7，DIFT + 4DRL 则达到 34.2。论文的解读是：只用离散文本推理会遇到表达能力的上限，潜在想象打破了这个上限。

The imagery in action想象过程示例

In the paper’s examples the model first states its strategy, then writes a few latent blocks as its imagined frames, and then derives the answer. In Figure 12, a DSR-Bench question, it splits a guinea pig’s motion into two phases, the distance first growing and then staying constant, where Gemini-3 and the base Qwen2.5-VL-3B both answer wrongly. In Figure 13, from Dyn-Bench, it follows a player diagonally across the whole court, while both baselines conclude from local frames that he stays in one half.

在论文给出的例子中，模型先说明推理策略，再写出几段潜在 token 作为想象中的画面，然后得出答案。图 12 是一道 DSR-Bench 题，模型把一只豚鼠的运动分成两个阶段：距离先变大、后保持不变，而 Gemini-3 和基座 Qwen2.5-VL-3B 都答错了。图 13 来自 Dyn-Bench，模型跟踪一名球员沿对角线穿过整个球场，而两个基线都根据局部画面认为他一直待在半场内。

Two more examples: posture versus zoom, and a driving scene (Figures 14, 15)另外两个例子：姿态变化与镜头缩放、驾驶场景（图 14、图 15）

Limitations局限

3DThinker

The 3DThinker paper shows a failure case (Figure 16). From two indoor views, 3DThinker misplaces the object at the back, and the decoded point cloud shows why: the wall with two photographs sits on the right of the window view when it should be on the left. Two views constrain the scene too little, the reconstruction goes wrong and the reasoning follows it. With extremely sparse views, the authors suggest bringing in other information as future work.

3DThinker 论文给出了一个失败案例（图 16）。根据两张室内视角图，3DThinker 判断错了后方物体的位置，解码出的点云揭示了原因：挂着两幅照片的那面墙被放在了窗户视角的右侧，而它本应在左侧。两个视角对场景的约束太少，重建出错，推理也随之出错。对于极度稀疏的视角，作者建议今后引入其他信息来补充。

• Training depends on VGGT. Stage 1 needs its features as targets, and the reward r_3D uses them in Stage 2; inference does not.
• The Stage 1 traces are written by GPT-4.1, so “annotation-free” means free of manual geometric annotation, not free of a stronger model.
• The latent block is fragile in two ways: placed inside the reasoning it breaks the text, and 32 or more tokens collapse the output.
• Travel-time questions in Ego3D-Bench do not benefit from Stage 2: on six of the eight models in Table 4 it leaves them at or slightly below Stage 1.

• 训练依赖 VGGT。阶段 1 需要它的特征作为目标，阶段 2 的 r_3D 奖励也要用到它；推理时则不需要。
• 阶段 1 的推理链由 GPT-4.1 生成，因此“无需标注”指的是不需要人工几何标注，而不是不依赖更强的模型。
• 潜在 token 段在两方面比较脆弱：放进推理文本中间会破坏文本，数量达到 32 个及以上时输出会崩溃。
• Ego3D-Bench 中的行程时间题无法从阶段 2 获益：在表 4 的 8 个模型中有 6 个，阶段 2 在这一项上与阶段 1 持平或略低。

4DThinker

The 4DThinker paper names two limitations, and its setup implies a third.

4DThinker 论文指出了两点局限，其设置还隐含第三点。

• The data pipeline relies on off-the-shelf estimators such as MegaSaM, whose errors can propagate into the training data; 4DRL partly mitigates this, and more robust geometric priors could improve the data.
• Evaluation covers multiple-choice benchmarks only; open-ended generation such as embodied planning remains open.
• The questions, landmark choices and reasoning traces come from Gemini-3-pro and the masks from SAM3, so here too “annotation-free” means free of human labels, not of stronger models.

• 数据流程依赖 MegaSaM 等现成的估计器，它们的误差可能传入训练数据；4DRL 能部分缓解这一问题，更可靠的几何先验可以进一步提升数据质量。
• 评测只覆盖选择题基准；具身规划等开放式生成任务仍待解决。
• 问题、地标选择和推理链来自 Gemini-3-pro，掩码来自 SAM3，因此这里的“无需标注”同样指不需要人工标注，而不是不依赖更强的模型。

Open-source resources开源资源

Both repositories are Apache-2.0. 3DThinker’s released model is the Stage 1 checkpoint of Qwen2.5-VL-3B trained on MindCube’s training data and tested on MindCube-Tiny (Apache-2.0); its README notes that the models in Table 5 were trained on different data. The 3DThinker-10K data is CC BY-NC 4.0, and so are the VGGT-1B weights that its Stage 1 and r_3D reward rely on. 4DThinker releases Qwen2.5-VL-3B checkpoints after DIFT and after 4DRL and its training data, both marked Apache-2.0 on Hugging Face; the data is built on SpatialVID, whose own licence is CC BY-NC-SA 4.0, and on DSR_Suite-Data.

两个代码仓库都采用 Apache-2.0。3DThinker 发布的模型是在 MindCube 训练数据上训练、在 MindCube-Tiny 上测试的 Qwen2.5-VL-3B 阶段 1 检查点（Apache-2.0）；其 README 说明表 5 中的模型使用了不同的训练数据。3DThinker-10K 数据采用 CC BY-NC 4.0，其阶段 1 和 r_3D 奖励所依赖的 VGGT-1B 权重同样采用 CC BY-NC 4.0。4DThinker 发布了 Qwen2.5-VL-3B 在 DIFT 与 4DRL 之后的检查点及训练数据，在 Hugging Face 上均标注为 Apache-2.0；这些数据基于 SpatialVID（其许可为 CC BY-NC-SA 4.0）和 DSR_Suite-Data 构建。

git clone https://github.com/zhangquanchen/3DThinker.git
cd 3DThinker

# Stage 1
conda create -n 3DThinker-stage1 python=3.10 -y && conda activate 3DThinker-stage1
pip install -r envs/requirements_stage1.txt

# Stage 2 (then swap in the Stage 1 transformers)
conda create -n 3DThinker-stage2 python=3.10 -y && conda activate 3DThinker-stage2
bash 3dthinker/stage2/setup.sh

# SFT: LLaMA-Factory, for weight merging
conda create -n SFT python=3.10 -y && conda activate SFT
cd SFT/env
pip install -e ".[torch,metrics]" --no-build-isolation

cd eval
sh eval_mindcube.sh   # inference: scripts/run_inference_clean.py --model-type qwen2.5vl ...
sh get_result.sh      # scoring:   scripts/run_evaluation.py -t basic ...

Citation引用

@inproceedings{chen2026think,
  title     = {Think with 3D: Geometric Imagination Grounded Spatial Reasoning
               from Limited Views},
  author    = {Chen, Zhangquan and Zhang, Manyuan and Yu, Xinlei and Luo, Xufang and
               Sun, Mingze and Pan, Zihao and An, Xiang and Feng, Yan and Pei, Peng and
               Cai, Xunliang and Huang, Ruqi},
  booktitle = {CVPR},
  year      = {2026}
}
@article{chen20264dthinker,
  title     = {4DThinker: Thinking with 4D Imagery for Dynamic Spatial Understanding},
  author    = {Chen, Zhangquan and Zhang, Manyuan and Yu, Xinlei and An, Xiang and
               Li, Bo and Xie, Xin and Wang, ZiDong and Sun, Mingze and Chen, Shuang and
               Li, Hongyu and Hu, Xiaobin and Huang, Ruqi},
  journal   = {arXiv preprint arXiv:2605.05997},
  year      = {2026}
}