DeepSeek-V3 Technical Report

Reading guide

Problem and motivation

Frontier large language models (LLMs) continue to close the gap toward AGI, but scaling open-source models typically demands prohibitive compute budgets and infrastructure. Furthermore, scaling Mixture-of-Experts (MoE) architectures introduces specific engineering challenges:

1. Auxiliary Loss Degradation: Traditional auxiliary load-balancing losses heavily restrict routing flexibility and harm language modeling capacity.
2. Communication Bottlenecks: Fine-grained cross-node routing causes severe all-to-all communication overheads, often reaching a computation-to-communication ratio of 1:1.
3. Numerical Precision & Memory: Standard 16-bit training is memory-intensive, while naive FP8 training suffers numerical instability and underflow due to activation outliers and limited Tensor Core accumulation precision.

DeepSeek-V3 addresses these challenges through co-designed algorithmic strategies and hardware-level infrastructure optimizations to achieve frontier performance at minimal training cost.

Key ideas

• Architecture Efficiency: Combines low-rank Key-Value compression via Multi-head Latent Attention (MLA) with fine-grained routed experts via DeepSeekMoE (1 shared + 256 routed experts, 8 activated).
• Auxiliary-Loss-Free Load Balancing: Replaces performance-degrading auxiliary losses with dynamically updated expert bias terms, enabling distinct domain specialization without routing collapse.
• Multi-Token Prediction (MTP): Predicts multiple future tokens sequentially at each position along a preserved causal chain to enrich training signals and support speculative decoding (1.8× TPS).
• DualPipe Parallelism: A bidirectional pipeline schedule that fully overlaps forward and backward computation with cross-node MoE all-to-all and PP communications.
• FP8 Mixed-Precision Training: Uses E4M3 across all tensors, applying tile-wise ($1\times 128$) activation and block-wise ($128\times 128$) weight scaling, accompanied by accumulation offloaded to CUDA Core registers in FP32 every $N_c=128$ steps.
• Reasoning Distillation: Distills verification and reflection long-CoT reasoning patterns from the DeepSeek-R1 series directly into SFT and RL post-training stages.

Method

1. Multi-head Latent Attention (MLA)

For token hidden state ${\mathbf{h}}_t$, MLA compresses KV cache into a shared latent vector ${\mathbf{c}}_t^{KV}\in {\mathbb{R}}^{d_c}$ and query states into ${\mathbf{c}}_t^Q\in {\mathbb{R}}^{d_c^{\prime }}$:

$${\mathbf{c}}_t^{KV}=W^{DKV}{\mathbf{h}}_t,{\mathbf{k}}_t^C=W^{UK}{\mathbf{c}}_t^{KV},{\mathbf{v}}_t^C=W^{UV}{\mathbf{c}}_t^{KV}$$ $${\mathbf{k}}_t^R=\mathrm{RoPE}(W^{KR}{\mathbf{h}}_t),{\mathbf{k}}_{t,i}=[{\mathbf{k}}_{t,i}^C;{\mathbf{k}}_t^R]$$

During inference, caching is restricted to only ${\mathbf{c}}_t^{KV}$ and ${\mathbf{k}}_t^R$, dramatically lowering memory footprints.

2. DeepSeekMoE with Auxiliary-Loss-Free Balancing

Feed-forward network outputs combine $N_s$ shared experts and $N_r$ routed experts:

$${\mathbf{h}}_t^{\prime }={\mathbf{u}}_t+\sum _{i=1}^{N_s}{\mathrm{FFN}}_i^{(s)}({\mathbf{u}}_t)+\sum _{i=1}^{N_r}{g}_{i,t}{\mathrm{FFN}}_i^{(r)}({\mathbf{u}}_t)$$

Expert affinities are calculated as $s_{i,t}=\mathrm{Sigmoid}({\mathbf{u}}_t^T{\mathbf{e}}_i)$. Gating values $g_{i,t}$ are computed by normalizing $g_{i,t}^{\prime }$, where routing decisions incorporate a dynamic bias term $b_i$:

$$g_{i,t}^{\prime }=\{\begin{array}{ll}{s}_{i,t},& s_{i,t}+b_i\in \mathrm{Topk}(\{s_{j,t}+b_j\mid 1\le j\le N_r\},K_r)\\ 0,& \text{otherwise}\end{array}$$

If expert $i$ is overloaded at step end, $b_i\leftarrow b_i-\gamma$; if underloaded, $b_i\leftarrow b_i+\gamma$ (with $\gamma =0.001$). A tiny sequence-wise balance loss ($\alpha =0.0001$) is used as a complement.

3. Multi-Token Prediction (MTP)

For prediction depth $k\in [1,D]$ (with $D=1$), representation ${\mathbf{h}}_i^{k-1}$ is combined with the shifted token embedding:

$${\mathbf{h}}_i^{\prime k}=M_k[\mathrm{RMSNorm}({\mathbf{h}}_i^{k-1});\mathrm{RMSNorm}(\mathrm{Emb}(t_{i+k}))]$$ $${\mathbf{h}}_{1:T-k}^k={\mathrm{TRM}}_k({\mathbf{h}}_{1:T-k}^{\prime k}),P_{i+k+1}^k=\mathrm{OutHead}({\mathbf{h}}_i^k)$$

The total training loss adds ${\mathcal{L}}_{\text{MTP}}=\frac{\lambda }{D}{\sum }_{k=1}^D{\mathcal{L}}_{\text{MTP}}^k$.

Results

• Pre-Training Budget: Pre-trained on 14.8T tokens for only 2.664M H800 GPU hours (3.7 days per 1T tokens on 2048 GPUs). Full training consumed 2.788M GPU hours (~$5.576M at $2/hour).
• Stability: Zero rollbacks and zero irrecoverable loss spikes across all 14.8T tokens.
• Base Model Benchmarks:
  • Outperforms LLaMA-3.1 405B Base on MMLU (87.1 vs. 84.4), BBH (87.5 vs. 82.9), HumanEval (65.2 vs. 54.9), and MATH (61.6 vs. 49.0).
  • Outperforms Qwen2.5 72B Base on C-Eval (90.1 vs. 89.2) and GSM8K (89.3 vs. 88.3).
• Chat Model Benchmarks:
  • Math & Code: Sets state-of-the-art non-o1 scores: MATH-500 (90.2), AIME 2024 (39.2), LiveCodeBench CoT (40.5).
  • Broad Knowledge: MMLU (88.5), MMLU-Pro (75.9), GPQA-Diamond (59.1).
  • Open-Ended: Exceeds 85% win rate on Arena-Hard (85.5%) and achieves 70.0% on length-controlled AlpacaEval 2.0.

Limitations and open questions

• Hardware Inference Minimums: Efficient serving requires large deployment units (32 GPUs for prefilling unit, 320 GPUs for decoding unit), presenting infrastructure barriers for small organizations.
• Cross-Node SM Allocation: 20 SMs per GPU were dedicated strictly to managing cross-node all-to-all communication routines rather than matrix math.
• Factual Discrepancies: Lags closed-source frontier models (e.g., GPT-4o, Claude-3.5-Sonnet) on English factual tasks like SimpleQA (24.9% vs. 38.2%), despite leading on Chinese SimpleQA (64.8%).