Partial FC: Training 10 Million Identities

Decoupling classifier cost from the number of identities让分类层开销与身份数解耦

The softmax classifier of a face-recognition model stores one center per identity, so its weights and logits grow linearly with the number of identities and dominate GPU memory at the million scale. Partial FC first shards the class centers across GPUs and computes the exact softmax normalizer with scalar collectives, so no communication depends on the class count. It then keeps every positive class and samples only a fraction $r$ of the negative classes on each GPU, which shrinks per-GPU logits, classifier FLOPs and classifier updates to a fraction $r$ . With $r=0.2\text{–}0.3$ , Partial FC matches or exceeds full-softmax accuracy on WebFace4M/12M/42M, is markedly more robust to label noise and long-tailed identities, and trains 10M identities on a single machine.

人脸识别模型的 Softmax 分类层为每个身份保存一个类别中心，其权重与 logits 都随身份数线性增长，在百万级规模下成为显存的主要开销。Partial FC 首先把类别中心切分到多张 GPU，并用标量集合通信精确计算 Softmax 归一化项，使通信量与类别数无关；随后在每张 GPU 上保留全部正类、只采样比例为 $r$ 的负类，使每卡 logits、分类层 FLOPs 与参数更新都缩小为原来的 $r$ 。当 $r=0.2\text{–}0.3$ 时，Partial FC 在 WebFace4M/12M/42M 上达到或超过全量 Softmax 的精度，对标签噪声和长尾身份明显更鲁棒，并可在单机上训练 1000 万身份。

Why full softmax stops scaling为什么全量 Softmax 无法继续扩展

Margin-based softmax losses such as ArcFace train a classifier $W\in\mathbb R^{d\times C}$ with one learned center per identity, and every sample is compared with every center. Both the classifier and the $B\times C$ logit matrix therefore grow linearly with the number of classes $C$ . On noisy, long-tailed web data, this full inter-class interaction is not only expensive but can also be harmful:

ArcFace 等带间隔 Softmax 损失训练一个分类层 $W\in\mathbb R^{d\times C}$ ，每个身份对应一个可学习的中心，每个样本都要与所有中心比较。因此分类层与 $B\times C$ 的 logits 矩阵都随类别数 $C$ 线性增长。在带噪声、长尾的网络数据上，这种全量类间交互不仅昂贵，还可能有害：

1. Inter-class conflict1. 类间冲突

The same person may be split across labels. Full softmax then pushes visually identical identities apart.

同一个人可能被拆成多个标签，全量 Softmax 会错误地把这些近似身份推远。

2. Passive tail updates2. 尾类被动更新

Rare classes seldom appear as positives but are repeatedly moved as negatives, so their centers can drift.

稀有类别很少作为正类出现，却不断作为负类被更新，类别中心容易漂移。

3. Logit explosion3. Logits 膨胀

Sharding W helps store centers, but the global batch still produces a massive B × C logit tensor.

即使分片存储 W，全局 batch 仍会产生巨大的 B × C logits 张量。

This page follows the expanded manuscript, which makes three contributions:

本页内容基于扩展稿，其贡献有三点：

Exact class sharding. A model-parallel softmax whose communication, $\Theta(Nkd)+\Theta(Nk)$ , is independent of $C$ .精确的类别切分。一种模型并行 Softmax，其通信量 $\Theta(Nkd)+\Theta(Nk)$ 与 $C$ 无关。
Positive-preserving sampling. Each GPU keeps all of its positive classes and samples a fraction $r$ of its negatives, shrinking per-GPU logits and classifier FLOPs to a fraction $r$ .保留正类的采样。每张 GPU 保留全部正类、只采样比例为 $r$ 的负类，把每卡 logits 与分类层 FLOPs 缩小为原来的 $r$ 。
Evidence. Accuracy on par with full softmax, large gains under label noise, and training with 10M identities on one machine.实验验证。精度与全量 Softmax 相当，在标签噪声下大幅提升，并可在单机上训练 1000 万身份。

Exact model-parallel softmax精确的模型并行 Softmax

Consider $k$ GPUs, each holding a local batch of $N$ embeddings of dimension $d$ . After an AllGather, every GPU holds the global batch $X\in\mathbb R^{Nk\times d}$ . GPU $i$ owns the classes $\mathcal C_i$ , with $|\mathcal C_i|=C_i\approx C/k$ , through a shard $W_i\in\mathbb R^{d\times C_i}$ , and computes only its block of logits $Z_i=XW_i$ . Let $z_{n,c}$ denote the logit of sample $n$ for class $c$ ; it is available only on the GPU that owns $c$ . The softmax normalizer needs just two statistics per sample, and both are reductions over the class partition:

设共有 $k$ 张 GPU，每张持有 $N$ 个 $d$ 维特征。AllGather 之后，每张 GPU 都拥有全局 batch $X\in\mathbb R^{Nk\times d}$ 。GPU $i$ 通过分片 $W_i\in\mathbb R^{d\times C_i}$ 持有类别集合 $\mathcal C_i$ （ $|\mathcal C_i|=C_i\approx C/k$ ），并只计算自己的 logits 块 $Z_i=XW_i$ 。记 $z_{n,c}$ 为样本 $n$ 在类别 $c$ 上的 logit，它只存在于拥有类别 $c$ 的 GPU 上。Softmax 归一化项对每个样本只需要两个统计量，二者都是在类别划分上的规约：

\displaystyle m_n=\operatorname{AllReduce}_{\max}\Bigl(\max_{c\in\mathcal C_i} z_{n,c}\Bigr)

(1)

\displaystyle s_n=\operatorname{AllReduce}_{\mathrm{sum}}\Bigl(\sum_{c\in\mathcal C_i} e^{\,z_{n,c}-m_n}\Bigr)

(2)

\displaystyle p_{n,y_n}=\frac{e^{\,z_{n,y_n}-m_n}}{s_n},\qquad \mathcal L=-\frac{1}{Nk}\sum_{n=1}^{Nk}\log p_{n,y_n}

(3)

No approximation is made: the loss and its gradients equal those of single-device softmax, the full $Nk\times C$ logit matrix is never assembled, and each collective moves one scalar per sample (Figure 1).

这一过程没有任何近似：损失及其梯度与单卡 Softmax 完全相同，完整的 $Nk\times C$ logits 矩阵从不拼出，每次集合通信每个样本只搬运一个标量（图 1）。

Why Eqs. (1)–(3) are exact为什么式 (1)–(3) 是精确的

Eq. (1). $e^{z}$ overflows fp16 near $z\approx 11$ . One device subtracts the row maximum; with a split row, each GPU reduces its local maximum and all GPUs obtain the same $m_n$ .式 (1)。fp16 下 $z\approx 11$ 时 $e^{z}$ 就会溢出。单卡上减去行最大值即可；行被切开后，各 GPU 先求本地最大值再规约，得到相同的 $m_n$ 。
Eq. (2). The classes form a disjoint partition, so the global denominator is exactly the sum of the per-GPU partial sums; no GPU needs another GPU's logits.式 (2)。类别构成不相交的划分，全局分母恰好等于各 GPU 局部和之和；任何 GPU 都不需要别的 GPU 的 logits。
Eq. (3). Cross-entropy reads one entry per row. Only the GPU that owns $y_n$ holds $z_{n,y_n}$ , so an AllReduce over a vector that is zero elsewhere gathers the $Nk$ target probabilities.式 (3)。交叉熵每行只读取一个元素。只有拥有 $y_n$ 的 GPU 持有 $z_{n,y_n}$ ，因此对一个在其他位置为 0 的向量做 AllReduce，即可收集 $Nk$ 个目标概率。
Cost. Three collectives of $Nk$ scalars each, none of which depends on $C$ ; the gradients are value-for-value those of single-device softmax.开销。三次集合通信，各搬运 $Nk$ 个标量，均与 $C$ 无关；梯度与单卡 Softmax 逐值相同。

(a) Full softmax全量 Softmax

one device holds all $C$ classes单卡保存全部 $C$ 个类别

X

Nk\times d

W

d\times C

Z=XW

Nk\times C

m_n=\max_c Z_{nc}

s_n=\sum_c e^{Z_{nc}-m_n}

P=e^{Z-m}/s

Nk\times C

both $W$ and $Z$ grow with $C$ $W$ 与 $Z$ 都随 $C$ 增长

(b) Class-sharded forward类别切分：前向

exact global softmax, logits never gathered精确的全局 Softmax，logits 从不聚合

GPU 1

W_1

⋯GPU

k

W_k

X_1

N\times d

X_k

N\times d

\mathrm{AllGather}

X

:

Nk\times d

values

X

：

Nk\times d

个值

Z_1=XW_1

Nk\times C/k

⋯

Z_k=XW_k

Nk\times C/k

m^{(1)}=\max Z_1

m^{(k)}=\max Z_k

\mathrm{AllReduce}_{\max}\to m_n

Nk

scalars

Nk

个标量

s^{(1)}=\sum e^{Z_1-m}

s^{(k)}=\sum e^{Z_k-m}

\mathrm{AllReduce}_{\mathrm{sum}}\to s_n

Nk

scalars

Nk

个标量

P_1=e^{Z_1-m}/s

P_k=e^{Z_k-m}/s

\mathrm{AllReduce}_{\mathrm{sum}}\to p_{n,y_n}

Nk

scalars, Eq. (3)

Nk

个标量，式 (3)

no collective depends on $C$ 所有集合通信都与 $C$ 无关

(c) Class-sharded backward类别切分：反向

local classifier update, summed feature gradients分类层本地更新，特征梯度求和

GPU 1

W_1

⋯GPU

k

W_k

G_1=\tfrac{1}{Nk}(P_1-Y_1)

G_k=\tfrac{1}{Nk}(P_k-Y_k)

\nabla W_1=X^{\top}G_1

applied in place原地更新

\nabla X^{(1)}=G_1W_1^{\top}

⋯

\nabla W_k=X^{\top}G_k

applied in place原地更新

\nabla X^{(k)}=G_kW_k^{\top}

\mathrm{ReduceScatter}_{\mathrm{sum}}

Nk\times d

gradients

Nk\times d

个梯度

\nabla X_1

N\times d

\nabla X_k

N\times d

$\nabla W_i$ never leaves its GPU $\nabla W_i$ 从不离开所属 GPU

features and their gradients特征与特征梯度 resident classifier shard常驻分类层分片 logits, probabilities, logit gradientslogits、概率与 logit 梯度 collective communication集合通信

Figure 1. Class-sharded softmax, redrawn from the manuscript. (a) One device stores all of

W

and all

Nk\times C

logits, so both grow with

C

. (b) GPU

i

keeps its shard

W_i

resident and scores the gathered batch only against it; AllReduces of the local maxima and of the exponentiated sums give the exact global normalization (Eqs. 1–2), and a third gathers the target probabilities (Eq. 3). (c) Classifier gradients are applied where they are computed; ReduceScatter sums the feature-gradient contributions and returns each GPU its

N\times d

rows. No collective moves more than

Nk\times d

values.

图 1. 类别切分的 Softmax（据论文原图重绘）。(a) 单卡保存全部

W

与

Nk\times C

logits，两者都随

C

增长。(b) GPU

i

常驻分片

W_i

，只用它为聚合后的 batch 计算 logits；对局部最大值和指数和各做一次 AllReduce 即得到精确的全局归一化（式 1–2），第三次 AllReduce 收集目标概率（式 3）。(c) 分类层梯度在计算处原地更新；ReduceScatter 汇总特征梯度，并把各 GPU 自己的

N\times d

行发回。任何集合通信都不超过

Nk\times d

个值。

Limitation. Every GPU still scores the whole global batch (Nk rows) against its C/k centers, so its logit block is Nk × C/k = N × C, independent of k. More GPUs shrink the weights but not the logits: at N = 64 and d = 512, logits take 40% of per-GPU classifier memory at 8 GPUs and 87% at 80.

局限：每张 GPU 仍要用整个全局 batch（Nk 行）与本卡的 C/k 个中心计算，logits 块大小为 Nk × C/k = N × C，与 k 无关。加卡只能缩小权重，缩不小 logits：在 N = 64、d = 512 时，8 卡下 logits 占每卡分类层显存的 40%，80 卡时升至 87%。

Figure 2. Memory analysis of exact model parallelism图 2. 精确模型并行的显存分析

✓

M_W = 4dC/k \ \text{bytes}

Shrinks as 1/k. Per-GPU classifier weights — the part sharding does fix. The class dimension C is split k ways, so each rank stores a k-th of it and every GPU you add makes this term smaller.随 1/k 缩小。每卡分类器权重 —— 这是切分确实解决掉的部分。类别维度 C 被切成 k 份，每个 rank 只存其中一份，每加一张卡这一项就小一分。

✗

M_{\mathrm{logit}} = 4NC \ \text{bytes}

Independent of k. Per-GPU logits — the part sharding does not touch. Every rank scores the whole global batch Nk against its own shard C/k, so the k cancels and adding GPUs does nothing at all.与 k 无关。每卡 logits —— 这是切分完全没碰到的部分。每个 rank 都要拿整个全局 batch Nk 去打自己那份 C/k 的分，k 被约掉，加卡毫无作用。

Σ

M_{\mathrm{logit}}\big/M_W = Nk\big/d

The verdict. The ratio therefore grows linearly in k. At N=64, d=512 it is already 1:1 at 8 GPUs and 10:1 at 80 — past that point you are buying classifier capacity you no longer need, in order to feed logits you cannot shrink. Figure 2 plots the totals as M_FC ≈ 3M_W + 2M_logit: momentum SGD keeps three copies of W, the margin softmax two of the logits.结论。于是比值随 k 线性增长。在 N=64、d=512 下，8 卡时已是 1:1，80 卡时变成 10:1 —— 越过这一点后，你是在买已经不再需要的分类器容量，去供养一个根本压不下去的 logits。图 2 按 M_FC ≈ 3M_W + 2M_logit 画总量：动量 SGD 保存三份 W，带间隔 Softmax 保存两份 logits。

Figure 2. Why exact class-sharded model parallelism still hits a memory wall (N=64, d=512, 125K classes per GPU, so C grows with k). (a) Elements moved per rank in one step, on a log axis: only embeddings, softmax statistics, target probabilities and feature gradients are communicated, while the Nk×C logits and the d×C classifier never leave their owner. (b) Per-GPU FC memory splits into a constant weight term 3M_W and a logit term 2M_logit=8NC that grows linearly with k, rising from 1.3 GB at 8 GPUs to 5.9 GB at 80 GPUs. (c) The logit share of M_FC therefore climbs from 40% to 87%. Adding ranks buys classifier capacity, not logit capacity.

图 2. 为什么精确的类别切分模型并行仍会撞上显存墙（N=64、d=512、每卡 12.5 万类，因此 C 随 k 增长）。(a) 单步每 rank 搬运的元素数（对数坐标）：只通信特征、Softmax 统计量、目标概率和特征梯度，Nk×C logits 与 d×C 分类器始终不离开各自的 owner。(b) 每卡 FC 显存分成恒定的权重项 3M_W 和随 k 线性增长的 logits 项 2M_logit=8NC，从 8 卡的 1.3 GB 升到 80 卡的 5.9 GB。(c) 于是 logits 在 M_FC 中的占比从 40% 升至 87%：加卡只买到分类器容量，买不到 logits 容量。

Positive-preserving partial softmax保留正类的部分 Softmax

Partial FC keeps the exact class sharding above and changes only which classes enter each step. GPU $i$ builds an active set from its positive classes $P_i=\{y_n\}\cap\mathcal C_i$ and negatives $N_i$ sampled uniformly without replacement from $\mathcal C_i\setminus P_i$ :

Partial FC 沿用上文的精确类别切分，只改变每一步参与计算的类别。GPU $i$ 由其正类 $P_i=\{y_n\}\cap\mathcal C_i$ 与从 $\mathcal C_i\setminus P_i$ 中无放回均匀采样的负类 $N_i$ 构成激活集合：

\displaystyle S_i=P_i\cup N_i,\qquad |S_i|=\max\bigl(|P_i|,\ \lfloor rC_i\rfloor\bigr)

(4)

\displaystyle \mathcal L_{\mathrm{PFC}}=-\frac{1}{Nk}\sum_{n=1}^{Nk}\log\frac{e^{\,z_{n,y_n}}}{\sum_{c\in S}e^{\,z_{n,c}}},\qquad S=\bigcup_{i=1}^{k}S_i

(5)

Eq. (5) is evaluated with the same collectives as Eqs. (1)–(3), restricted to $S$ , and the ArcFace or CosFace margin is applied to the target logit $z_{n,y_n}$ . Because every target class belongs to $S$ , no sample loses its positive term; only the denominator is approximated. Per-GPU logits, classifier FLOPs and classifier updates shrink to a fraction $r$ , while communication is unchanged (Table 1).

式 (5) 使用与式 (1)–(3) 相同的集合通信，只是限定在 $S$ 上计算；ArcFace 或 CosFace 间隔作用于目标 logit $z_{n,y_n}$ 。由于每个目标类别都属于 $S$ ，没有样本会丢失正类项，被近似的只有分母。每卡 logits、分类层 FLOPs 与参数更新都缩小为原来的 $r$ ，通信量不变（表 1）。

Table 1. Per-GPU cost of exact model parallelism and Partial FC (local batch $N$ , $k$ GPUs, sampling ratio $r$ ).表 1. 精确模型并行与 Partial FC 的每卡开销（本地 batch $N$ ， $k$ 张 GPU，采样率 $r$ ）。

Per GPU每张 GPU	Exact model parallel精确模型并行	Partial FC
Active class centers激活的类别中心	C/k	rC/k
Logits	NC	rNC
Classifier FLOPs分类层 FLOPs	2NdC	2NdrC
Communication通信量	Θ(Nkd)+Θ(Nk)	unchanged不变
Rows with loss gradients获得损失梯度的行	all C/k全部 C/k	sampled rC/k采样到的 rC/k

Features X (one tint per GPU)特征 X（每卡一种色阶）

Class centers W类别中心 W

Positives P_i正类 P_i

Sampled negatives N_i采样负类 N_i

Target logits目标 logits

Collective集合通信

Figure 3. One Partial FC training step (schematic: four GPUs on desktop, two on phones).图 3. Partial FC 的一个训练步（示意：桌面端四卡，手机端两卡）。 (1) Gather. Each GPU embeds its local batch; one AllGather gives every GPU the global batch

X

(

Nk\times d

values).(1) 聚合特征。每张 GPU 计算本地 batch 的特征，一次 AllGather 后每张 GPU 都得到全局 batch

X

（

Nk\times d

个值）。

Figure 4. Extension: hierarchical class storage on CPU / NVMe图 4. 扩展：基于 CPU / NVMe 的分级类别存储

Each class proxy has one permanent owner, but only the active set $S_i$ enters the GPU classifier buffer. In the expanded manuscript's optional hierarchy, the complete class bank and optimizer state can live in pinned CPU memory or an NVMe-backed cache, reducing GPU-resident classifier storage from $\Theta(C_i d)$ to $\Theta(rC_i d)$ . The released PartialFC_V2 path instead keeps each rank's complete owned shard on its GPU.

每个类别代理只有一个长期 owner，但只有激活集合 $S_i$ 会进入 GPU 分类器缓冲区。在扩展稿提出的可选分级方案中，完整类别库与优化器状态可以位于锁页 CPU 内存或 NVMe 缓存中，从而把 GPU 常驻分类器存储从 $\Theta(C_i d)$ 降至 $\Theta(rC_i d)$ 。已发布的 PartialFC_V2 路径则仍把每个 rank 拥有的完整分片放在 GPU 上。

The only subtlety is overlap between consecutive active sets: cold rows may arrive early, but repeated rows must carry the result of optimizer $t$ .

真正需要处理的是相邻激活集合的重叠：冷行可以提前到达，但重复行必须携带 optimizer $t$ 的最新结果。

Figure 4. Overlap-safe hierarchical class storage. B becomes readable only after both the cold-row prefetch and the post-optimizer overlap patch complete; A becomes reusable only after its D2D readers and D2H write-back complete. Row versions validate host state, and CUDA events enforce these dependencies.图 4. 避免旧版本行的分级类别存储。冷行预取与 optimizer 后的重叠行补丁都完成，B 才可读取；D2D 读取与 D2H 写回都完成，A 才可复用。行版本号用于校验主存状态，CUDA event 用于强制这些依赖。

Overlap condition. The cold-row H2D copy and the overlap-row handoff must both finish before the next PFC read, otherwise the ready gate stalls. Host or NVMe staging therefore pays off only when transfers overlap computation.

重叠条件。冷行 H2D 拷贝与重叠行交接都必须在下一次 PFC 读取前完成，否则就绪门会等待。因此只有传输能与计算重叠时，主存或 NVMe 分级存储才有收益。

Sampling 10–30% of classes keeps full-softmax accuracy只采样 10–30% 的类别，精度不输全量 Softmax

Trained on WebFace4M/12M/42M and tested on the hard MFR benchmark, Partial FC stays within 0.5 points of full softmax at r = 0.1 and matches or beats it at r = 0.2–0.3. It is also far more robust when labels are noisy.

在 WebFace4M/12M/42M 上训练、在更困难的 MFR 上评测：r = 0.1 时 Partial FC 与全量 Softmax 相差不到 0.5 个点，r = 0.2–0.3 时追平或超过全量 Softmax；标签有噪声时也明显更鲁棒。

r = 0.2–0.3matches or beats full FC on WebFace4M/12M/42M在 WebFace4M/12M/42M 上追平或超过全量 FC

+36.33MFR points under 40% label flips40% 错标下 MFR 提升点数

≈10×less memory traffic per GPU than full model parallel每卡访存量比全量模型并行少约 10 倍

100Msynthetic classes scaled on 64 GPUs64 张 GPU 扩展的合成类别数

Robustness: conflict, label flips and long tails鲁棒性：类别冲突、错标与长尾

Reducing negative interaction is most valuable when labels are imperfect. PFC* additionally filters abnormal negative centers whose cosine similarity exceeds 0.4.

当标签不完美时，减少负类交互的价值最明显。PFC* 还会过滤余弦相似度高于 0.4 的异常负类中心。

Table 2. MFR-All (TAR@FAR = 1e−6) under inter-class conflict, label flips and long-tailed identities.表 2. 类间冲突、标签翻转与长尾身份下的 MFR-All（TAR@FAR = 1e−6）。

Stress test压力测试	FC	PartialFC	PFC*	Best gain最佳提升
Inter-class conflict类间冲突	79.93	91.20	91.68	+11.75
40% label flips40% 标签翻转	43.87	78.53	80.20	+36.33
Long-tail identities长尾身份分布	87.44	91.96	-	+4.52

MFR-All from paper Tables 3–5. Long-tail PartialFC uses r=0.2; conflict PFC/PFC* entries use r=0.1 and r=0.2 respectively.

MFR-All 取自论文表 3–5。长尾实验使用 r=0.2；冲突实验的 PFC / PFC* 分别取 r=0.1 / 0.2。

Table 3. Sampling-ratio ablation (ResNet50)表 3. 采样率消融（ResNet50）

Complete ResNet50 sampling-rate ablation. MFR columns use TAR@FAR=1e-6; Mask uses FAR=1e-4. Extremely low rates fail, while 0.1-0.3 generally recovers full FC.

完整 ResNet50 采样率消融。MFR 各列使用 TAR@FAR=1e-6，Mask 使用 FAR=1e-4。极低采样率明显退化，0.1-0.3 通常可恢复全量 FC。

Configuration配置	All	African	Caucasian	South Asian	East Asian	Mask
WF4M + FC-1.0	86.25	83.35	91.11	88.14	65.79	72.05
WF4M + PFC-0.04	74.11	71.43	81.79	76.25	52.24	54.21
WF4M + PFC-0.1	85.76	83.82	91.00	87.90	66.04	71.13
WF4M + PFC-0.3	86.85	84.86	91.57	88.57	67.52	72.28
WF12M + FC-1.0	91.70	90.72	94.94	93.44	75.10	80.47
WF12M + PFC-0.013	87.85	87.07	92.32	90.70	68.28	72.98
WF12M + PFC-0.1	91.24	90.80	94.67	93.18	74.97	79.73
WF12M + PFC-0.2	91.78	91.09	95.00	93.53	75.90	79.92
WF12M + PFC-0.3	91.82	91.14	95.00	93.61	75.55	80.08
WF42M + FC-1.0	93.86	93.33	96.20	95.24	79.46	83.90
WF42M + PFC-0.008	91.27	90.34	95.16	93.04	76.93	81.24
WF42M + PFC-0.1	93.95	93.48	96.37	95.51	80.03	83.79
WF42M + PFC-0.2	94.04	93.67	96.38	95.49	80.07	84.32
WF42M + PFC-0.3	94.03	93.68	96.38	95.52	79.76	84.46

Source: paper Table 1. The optimum is broad rather than a single fragile setting.

来源：论文表 1。最优区间较宽，并非依赖单一脆弱超参数。

Table 4. Eight training sets with and without PFC表 4. 八个训练集上有无 PFC 的对比

Partial FC remains close to full FC across datasets, but does not improve every metric. MFR-All is TAR@FAR=1e-6; IJB-C is TAR@FAR=1e-4; the other columns are verification accuracy.

Partial FC 在不同数据集上总体接近全量 FC，但并非每个指标都提升。MFR-All 为 TAR@FAR=1e-6，IJB-C 为 TAR@FAR=1e-4，其余为验证准确率。

Training set训练集	MFR-All	AgeDB	CFP-FP	LFW	IJB-C
CASIA	36.79	99.45	95.21	94.90	87.22
CASIA + PFC	37.11	99.37	95.43	94.60	84.97
VGGFace	38.58	99.55	97.41	95.08	91.22
VGGFace + PFC	40.67	99.68	98.53	95.40	92.49
GlintAsian	62.66	99.58	93.19	95.40	91.50
GlintAsian + PFC	63.15	99.65	93.03	95.23	91.14
MS1MV2	77.70	99.83	98.08	98.08	96.14
MS1MV2 + PFC	77.74	99.78	98.07	98.02	96.08
MegaFaceMS1M	78.37	99.75	97.56	97.40	95.35
MegaFaceMS1M + PFC	78.77	99.80	97.87	97.73	95.40
MS1MV3	82.52	99.80	98.53	98.27	96.58
MS1MV3 + PFC	81.68	99.80	98.44	98.17	96.43
Glint360K	86.79	99.82	99.14	98.45	97.13
Glint360K + PFC	87.08	99.82	99.14	98.45	97.02
WebFace12M	90.57	99.80	99.20	98.10	97.12
WebFace12M + PFC	89.95	99.82	99.14	98.12	97.01

Source: active benchmark table in the expanded manuscript.

来源：扩展稿当前启用的基准表。

Table 5. Datasets and training settings表 5. 数据集与训练配置

Training covers CASIA, MS1MV2, Glint360K (17M images, 360K identities), and WebFace4M/12M/42M (200K/600K/2M identities). Evaluation uses LFW, CFP-FP, AgeDB-30, CPLFW, CALFW, MegaFace, IJB-B, IJB-C and ICCV21-MFR. WebFace42M is strongly long-tailed: 44.57% of identities have fewer than ten images.

训练数据包括 CASIA、MS1MV2、Glint360K（1700 万图像、36 万身份），以及 WebFace4M/12M/42M（20 万/60 万/200 万身份）。评测覆盖 LFW、CFP-FP、AgeDB-30、CPLFW、CALFW、MegaFace、IJB-B、IJB-C 与 ICCV21-MFR。WebFace42M 长尾显著：44.57% 的身份少于 10 张图像。

Runs实验	GPU	Batch批量	Optimizer / LR优化器 / LR	Schedule训练周期
CASIA / MS1MV2 / Glint360K	8× RTX2080Ti	512 total	SGD / 0.1	32K / 180K / 600K iters
WebFace42M CNN	32	128 / GPU	SGD / 0.4	20 epochs, 2 warm-up
WebFace42M ViT	-	384 / GPU	AdamW / 10^-3	40 epochs, 4 warm-up

Normalized margins: feature scale s=64; CosFace margin 0.4; ArcFace angular margin 0.5.

归一化间隔：特征缩放 s=64；CosFace margin 0.4；ArcFace 角度 margin 0.5。

Figure 5. Efficiency analysis: GEMM shapes and memory traffic图 5. 效率分析：GEMM 形状与访存量

Model-parallel Partial FC communicates $\Theta(Nkd)+\Theta(Nk)$ values per step, independent of $C$ ; sampled data parallelism must synchronize $\Theta(rCd)$ replicated proxy gradients. Even at identical FLOPs, the two also differ in how much memory each GPU has to stream:

模型并行 Partial FC 每步通信 $\Theta(Nkd)+\Theta(Nk)$ 个值，与 $C$ 无关；采样数据并行则必须同步 $\Theta(rCd)$ 个复制代理的梯度。即使 FLOPs 完全相同，两者每张 GPU 需要搬运的数据量也不同：

Per-rank GEMM shapes每个 rank 的 GEMM 形状

First-order traffic per rank每个 rank 的一阶访存量

Full MP

192.5M

Sampled DP

64.1M

MP-PFC

19.7M

050M100M150M192.5M

feature readweight readlogit write

Why identical FLOPs do not run at identical speed为什么相同的 FLOPs 跑不出相同的速度

Sampled DP and MP-PFC issue the same arithmetic, $F=2NdrC$ . Counting elements is not yet an answer either, because a kernel runs at whichever of compute or bandwidth saturates first. The quantity that decides which one is the ratio between them: how much arithmetic the GEMM performs per byte it is forced to pull from memory. For the classifier read, that ratio is set entirely by the shape.

采样数据并行与 MP-PFC 发射的算术量完全相同，都是 $F=2NdrC$ 。而只数元素同样不算回答问题，因为 kernel 的速度取决于算力和带宽哪一个先饱和。决定这一点的是两者之比：GEMM 每从显存拉取一个字节，能做多少次算术。对分类器的读取而言，这个比值完全由形状决定。

Every classifier weight that a rank loads participates in exactly one multiply-accumulate per batch row it is multiplied against. So the reuse each loaded center earns is the number of rows in the GEMM. Sampled DP multiplies against the local batch and gets $N$ ; MP-PFC multiplies against the gathered batch and gets $Nk$ . Same weights, same flops, $k\times$ the reuse.

一个 rank 载入的每个分类器权重，与多少个 batch 行相乘，就参与多少次乘加。因此每个被载入的中心所获得的复用次数就等于 GEMM 的行数。采样数据并行只与本地 batch 相乘，得到 $N$ ；MP-PFC 与聚合后的全局 batch 相乘，得到 $Nk$ 。同样的权重、同样的浮点运算，复用高出 $k$ 倍。

Per-rank GEMM每 rank 的 GEMM	Centers held持有的中心数	Rows multiplied参与相乘的行数	Reuse per center每中心复用	Classifier traffic分类器访存
Full MP	$C/k$	$Nk$	$Nk$	$Cd/k$
Sampled DP	$rC$	$N$	$N$	$rCd$
MP-PFC	$rC/k$	$Nk$	$Nk$	$rCd/k$

This is why the short, wide block is the bad one. A GEMM whose row count is small has few rows to amortize each loaded weight over, so it spends most of its time streaming the classifier past the tensor cores rather than multiplying with it — a bandwidth-bound kernel, no matter that its flop count is small. The tall, narrow block performs the same multiplications while touching a fraction of the weights, and amortizes each one over the full gathered batch, which is what moves it back into the compute-bound regime.

这正是"又矮又宽"那一块糟糕的原因。行数很小的 GEMM 没有足够的行去摊薄每个载入的权重，于是大部分时间都花在把分类器流过 tensor core，而不是拿它做乘法 —— 这是一个带宽受限的 kernel，哪怕它的浮点运算量本身很小。而"又高又窄"那一块，在只触碰一小部分权重的前提下完成了同样多的乘法，并把每个权重摊薄到整个全局 batch 上，这才把它推回算力受限的区间。

Shrinking the shard has a second, discontinuous effect. A GEMM tiles its output, and the weight tiles get revisited as the kernel walks down the rows; whether those revisits are served by cache or by HBM depends on whether the whole shard fits in L2. Sharding and sampling shrink the resident classifier by $rk$ , which is what can carry it across that threshold — and crossing it changes the cost of reuse by an order of magnitude, not a few percent.

缩小分片还有第二个、且不连续的收益。GEMM 会对输出做分块，kernel 沿行方向推进时会反复访问权重分块；这些重访是由缓存还是由 HBM 承担，取决于整个分片能否放进 L2。分片与采样把常驻分类器缩小了 $rk$ 倍，这个倍数足以让它跨过那道门槛 —— 而跨过与否，带来的是数量级的差异，不是百分之几。

One scoping note: all of the above concerns the forward GEMM alone. The logit block is where full model parallelism actually hurts, because it is written by the GEMM, read back for the softmax normalization and read once more in backward. Sampling shrinks that block by $1/r$ , and it is the only term in the whole pipeline that scales with the product of batch and class count.

一点范围说明：以上讨论都只涉及前向 GEMM。全量模型并行真正吃亏的地方在 logit 矩阵 —— 它由 GEMM 写出、为 Softmax 归一化读回、反向时再读一次。采样把这一块缩小了 $1/r$ ，而它是整条流水线中唯一随 batch 与类别数之积增长的项。

Figure 5. Per-rank GEMM shapes and first-order memory traffic, re-drawn from the manuscript analysis. Sampled DP and MP-PFC have the same leading FLOPs, 2NdrC, and produce NrC logits per rank; MP-PFC wins through data movement, because sharding and sampling reduce classifier traffic from Cd to rCd/k while the gathered batch reuses each active center.图 5. 每个 rank 的 GEMM 形状与一阶访存量，根据论文分析重新绘制。采样数据并行与 MP-PFC 的主导 FLOPs 都是 2NdrC，每个 rank 都产生 NrC 个 logits；MP-PFC 的优势来自数据移动：分片与采样把分类器访存从 Cd 降到 rCd/k，同时全局 batch 复用每个激活中心。

T_DP ≈ Nd + rCd + NrC T_MP ≈ Nkd + rCd/k + NrC F_DP = F_MP = 2NdrC

Minimal training loop最小训练循环

The implementation needs one global feature gather, local class sampling, distributed softmax statistics, and sparse optimizer updates.

实现只需要全局特征聚合、本地类别采样、分布式 softmax 统计，以及稀疏的优化器更新。

Listing 1. One model-parallel Partial FC training step (PyTorch-style pseudocode).代码 1. 模型并行 Partial FC 的一个训练步（PyTorch 风格伪代码）。

# Model-parallel Partial FC training step
X, y = differentiable_all_gather(local_X, local_y)
S_i = local_positives(y) | sample_local_negatives(r)
Z_i = normalize(X) @ normalize(W_i[S_i]).T
loss = distributed_sampled_margin_softmax(Z_i, remap(y, S_i))
loss.backward()
# sampled classifier rows update locally; feature gradients reduce-scatter
optimizer.step(sampled_rows=S_i)

Conclusion and limitations结论与局限

Partial FC decouples the cost of the classifier from the number of identities in two steps: exact class sharding makes communication independent of $C$ , and positive-preserving sampling shrinks per-GPU logits and classifier computation to a fraction $r$ . Moderate ratios keep full-softmax accuracy while improving robustness to noisy and long-tailed labels.

Partial FC 分两步把分类层的开销与身份数解耦：精确的类别切分使通信与 $C$ 无关，保留正类的采样把每卡 logits 与分类层计算缩小为原来的 $r$ 。适中的采样率在保持全量 Softmax 精度的同时，提升了对噪声与长尾标签的鲁棒性。

Limitations. Renormalizing over the active set makes the sampled gradient a biased estimate of the full-softmax gradient, and very low ratios (r ≤ 0.04 in Table 3) clearly lose accuracy. Host or NVMe staging of the class bank pays off only when transfers overlap computation.

局限。在激活集合上重新归一化，使采样梯度成为全量 Softmax 梯度的有偏估计；采样率过低（表 3 中 r ≤ 0.04）会明显掉点。类别库的主存或 NVMe 分级存储只有在传输能与计算重叠时才有收益。

References参考文献

[1]

Killing Two Birds with One Stone: Efficient and Robust Training of Face Recognition CNNs by Partial FC An, Deng, Guo, Feng, Zhu, Yang, Liu · CVPR 2022 · arXiv:2203.15565

@inproceedings{an2022partialfc,
  title     = {Killing Two Birds with One Stone: Efficient and Robust Training
               of Face Recognition CNNs by Partial FC},
  author    = {An, Xiang and Deng, Jiankang and Guo, Jia and Feng, Ziyong and
               Zhu, XuHan and Yang, Jing and Liu, Tongliang},
  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and
               Pattern Recognition (CVPR)},
  pages     = {4042--4051},
  year      = {2022}
}

[2]

Partial FC: Training 10 Million Identities on a Single Machine An, Zhu, Gao, Xiao, Zhao, Feng, Wu, Qin, Zhang, Zhang, Fu · ICCV Workshops 2021 · arXiv:2010.05222

@inproceedings{an2021partialfc,
  title     = {Partial FC: Training 10 Million Identities on a Single Machine},
  author    = {An, Xiang and Zhu, Xuhan and Gao, Yuan and Xiao, Yang and
               Zhao, Yongle and Feng, Ziyong and Wu, Lan and Qin, Bin and
               Zhang, Ming and Zhang, Debing and Fu, Ying},
  booktitle = {Proceedings of the IEEE/CVF International Conference on
               Computer Vision (ICCV) Workshops},
  pages     = {1445--1449},
  year      = {2021}
}