DeepSeek Explained 5: DeepSeek-V3-Base

Innovations in pre-training strategies of DeepSeek-V3.

This is the fifth article in our DeepSeek series, and also the first one focusing on the training procedure of DeepSeek-V3 [1, 2].

As we shown in the figure below, DeepSeek-V3 is trained with multiple stages, including

• A pre-training stage which produces DeepSeek-V3-Base.
• Starting from DeepSeek-V3-Base, DeepSeek-R1-Zero and DeepSeek-R1 are trained by exploring large-scale Reinforcement Learning with and without Supervised Finetuning as cold-start.
• DeepSeek-R1 is then used to generate reasoning data in the Supervised Finetuning stage of DeepSeek-V3, followed by a RL stage that is not depicted in the figure.

Figure 1. DeepSeek-V3 training workflow. Image by author.

In particular, this article will focus on the pre-training stage that produces DeepSeek-V3-Base, explaining the key techniques involved in this stage to make the pre-training both effective and efficient.

Later, we will move on to other topics including Grouped Relative Policy Optimization (GRPO) [7], how DeepSeek-R1-Zero and DeepSeek-R1 are trained, and finally return to the post-training stage of DeepSeek-V3, i.e., supervised finetuning stage and the RL stage.

Table of contents for this article:

• Background: explain related techniques in the pre-training phase of DeepSeek-V3, including document packing, Fill-in-Middle and long context extension.
• Pre-training: explain how the pre-training data is constructed, highlight some key training strategies, and review the evaluation results.
• Summary.
• References.

If you are interested in exploring more in the DeepSeek series — where we break down the architectural innovations and training strategies that drive DeepSeek’s success — check out these articles:

• Part 1: Multi-head Latent Attention
• Part 2: DeepSeekMoE
• Part 3: Auxiliary-Loss-Free Load Balancing
• Part 4: Multi-Token Prediction
• Part 6: Grouped Relative Policy Optimization
• Part 7: Advancing LLM Reasoning with Reinforcement Learning
• Part 8: Post-Training of DeepSeek-V3

Background

In this section we introduce several techniques used in pre-training DeepSeek-V3, including document packing, Fill-in-the-Middle (FIM), and long context extension with YaRN.

Document Packing

To understand why we need document packing, let’s revisit how Transformer models construct their input sequence tokens.

Transformer models by default require a fix sized of token sequence as input, however the input texts in the same batch are often with different lengths. To adapt to that, the texts input often needs to be pre-processed with the following steps:

• Tokenize each raw text input into a sequence of tokens.
• Truncate or pad to a predefined fixed length (max_seq_len): if raw sequence is too long, truncate it, otherwise pad it with a special [PAD] token.
• Generate mask ids so that the model can ignore the padding tokens during training.

To show this more clearly, below is an example where we use GPT-2 [10] tokenizer to process two sentences:

from transformers import AutoTokenizer

# Load a tokenizer (GPT-2 example)
tokenizer = AutoTokenizer.from_pretrained("gpt2", padding_side="right")

# Manually set a padding token (GPT-2 does not have one by default)
tokenizer.pad_token = tokenizer.eos_token

# Example input sequences
prompt = ["The cat sat on a mat",
          "I love machine learning but do not have time to read papers"
          ]

# Tokenize and pad to max length of 10
tokenized = tokenizer(prompt, return_tensors="pt", padding="max_length", truncation=True, max_length=10)

# Print tokenized input
print("Input IDs:\n", tokenized["input_ids"])
print("Attention Mask:\n", tokenized["attention_mask"])

After running the above script, we get the following output, where

• The first sentence is padded with 4 extra padding tokens, which can be seen in both the input_ids and mask_ids;
• The second sentence gets truncated, so no padding tokens are added.

Figure 2. An example of padding. Image created by author.

The above truncation and padding method enables the model to handle input with varied lengths, but also causes issues when input sequence lengths varied too much (which is pretty common in LLM training):

• For overlong sequence, useful information might be lost due to truncation;
• For short sequence, padding with too much extra tokens is a waste of computation resources.

For that reason, LLM training commonly apply document packing techniques to handle input sequences.

More specifically, given several documents with different lengths, we first split them into smaller chunks, as shown in the figure below where each document is represented by a different color:

Figure 3. Document segmentation. Image edited from [3].

Then, we concatenate chunks from different documents to avoid truncation of the long documents and padding to the short ones:

Figure 4. Conventional concatenation. Image edited from [3].

In the above example:

• First input contains tokens from document 1 only.
• Second input is a concatenation of tokens from document 1 and 2.
• Third input is a concatenation of tokens from document 2 and 3.
• Fourth input is a concatenation of tokens from document 3, 4, and 5.

While this approach removes the need for padding and truncation to some degree, it simply concatenate chunks from different documents according to their relative order in the data, and hence cannot control how the final input sequences will be constructed.

For example, document 3 (in purple) is truncated into two parts while its length can actually fit into the max_seq_len, causing unnecessary truncations.

To fix that, [3] proposes a Best-fit Packing technique that can completely removes unnecessary truncations with two steps, as shown in the figure below:

• Step 1: split each document into smaller chunks.
• Step 2: group chunks into training sequences in a smart way that results in the smallest number of sequences, without further splitting any chunks.

Figure 5. Best-fit packing. Image edited from [3].

Fill-in-the-Middle (FIM)

In traditional autoregressive generation, models are trained only on a left-to-right manner, meaning that they can only predict the next token given previous tokens. However, in many real-world applications, a model might need to generate missing content in the middle of a given context.

This is particularly useful in code generation, since we often prompt the LLM with input/output and some of the code snippets, and ask it to fill the logic in the middle, as shown in the following example:

def calculate_area(radius):
    // calculate circle area given radius

    return area

radius = 5
print(calculate_area(radius))

To adapt to such demand, [4] proposes a straightforward yet effective approach called fill-in-the-middle, by randomly splitting documents into three pieces called prefix, middle and suffix, and then move the middle piece to the end:

Since the data will be organized as Prefix-Suffix-Middle, this is often referred to as the PSM framework. This is commonly implemented by adding a set of special tokens to mark the boundry of each component:

where

• <|fim_begin|> and <|fim_hole|> marks the prefix.
• <|fim_hole|> and <|fim_end|> marks the suffix.
• <|fim_end|> and <|eos_token|> marks the middle.

Taking the following input as example:

def calculate_area(radius):
    area = 3.14 * radius ** 2
    return area

radius = 5
print(calculate_area(radius))

If we want the model to predict the second line, we can split that line as the middle part and construct the FIM input as

Figure 6. An illustration of the PSM framework. Image created by author.

And expected output from model should be:

    area = 3.14 * radius ** 2

Long Context Extension with YaRN

Modern LLMs are often required to process extremely long prompts such as the entire code repo, but pre-training with long context windows such as 128K is impractical.

Instead, a common strategy used by many LLMs is to firstly pre-train the model on smaller context windows, and then progressively extend to significantly longer context windows with multiple stages, which can significantly reduce the training efforts.

For example in DeepSeek-V3, the model is firstly pre-trained using context window 4K, and then extended to 128K with 2 stages:

• Extending from 4K to 32K with 1000 steps.
• Extending from 32K to 128K with another 1000 steps.

One thing to mention is that this cannot be achieved by simply configuring the context window to a larger value, instead we need to apply some modifications to the positional encoding using a technique called Yet another RoPE extensioN (YaRN) building upon Rotary Position Encoding (RoPE).

For a more detailed introduction of RoPE, please refer to our previous article DeepSeek-V3 Explained 1: Multi-head Latent Attention.

RoPE is a relative position encoding method, and its core idea is to modify the Query and Key using complex rotation embeddings so that their inner product becomes dependent on their relative positions:

However, with a fixed θ, a model pre-trained with 1K tokens may confuse when testing with positions far beyond the context window in pre-training such as 5K or 10K, since cosine and sine functions are periodic and the inner product between (pos_i, pos_j) might look similar to that of (pos_i, pos_k).

This also leads to decayed attention scores for position pairs with cosine close to zero, making the model struggle to maintain long-range coherence.

This is shown in the figure below, where the model pre-trained with 32K context window shows dramatically increased Perplexity when tested beyond that window.

Figure 7. Perplexity vs. context window. Image edited from [6].

So how does YaRN address that challenge?

As extrapolating doesn’t work well, YaRN takes an alternative approach by interpolating the frequency.

Let’s assume we have a model trained on 4 tokens and want to extend it to 8 tokens, and the base frequency θ is 0.5.

With vanilla RoPE, we simply rotate the Query and Key with cos(θ × pos) and sin(θ × pos).

However with YaRN:

• we first calculate a scale factor using the extended context length diving the original length, which in our case would be 2.
• Then, we get a new frequency represented as θ’ = θ / 2 = 0.25.
• We then rotate the Query and Key with this new frequency, i.e., with cos(θ’ × pos) and sin(θ’ × pos).

The figure below illustrates the cosine and sine values under RoPE and YaRN, respectively.

Figure 8. An illustration of how YaRN works. Image created by author.

According to this figure:

• In RoPE, the cosine and sine values oscillate rapidly as position index increases, causing issues in scaling to longer context.
• In YaRN, we observe a much smoother transition as the original cosine and sine functions are interpolated to the extended context length (see the area highlighted in blue) with a scaled frequency, allowing the model to handle longer sequences more effectively.

The figure below shows the evaluation results conducted on the “Needle In A Haystack” (NIAH) tests, demonstrating DeepSeek-V3’s performance across all context window lengths up to 128K.

Figure 9. Needle In A Haystack evaluation for DeepSeek-V3. Image from [2].

Pre-training

In this section, we cover how DeepSeek-V3-Base is trained. In particular, we will focus on data construction and highlight some of the key strategies in pre-training.

Data Construction

Data scale and quality are crucial for LLM training. In DeepSeek-V3, the pre-training corpus is built by continuously optimizing their data corpus using insights gained from previous models:

• In DeepSeek 67B [8], the training corpus is curated using a deduplication-filtering-remixing strategy, where aggressive deduplication is applied to the Common Crawl corpus first, followed by filtering using a robust criteria for document quality assessment, and finally a data remixing phase focusing on addressing data imbalances.
• In DeepSeek-V2 [9], training corpus is enlarged by 1) adding more Chinese data and high-quality data from various sources, and 2) recovering a large amount of data that was previously deleted in [8] by optimizing data cleaning process. Data quality is also enhanced by improving the quality-based filtering algorithm.
• In DeepSeek-V3 [2], the pre-training corpus is further enriched with more mathematical and programming samples and multilingual samples beyond English and Chinese.

The collected pre-training corpus is then pre-processed with the FIM strategy using the Prefix-Suffix-Middle (PSM) framework introduced previously, combined with the document-packing technique.

Training

The original paper [2] presents a detailed description on the pre-training parameters, here we only want to highlight a few things:

• Long context extension: pre-training with 4K context window on 14.8T tokens first, and then extend to 32K with 1000 steps, finally extend to 128K with another 1000 steps.
• Multi-token Prediction: As explained in our previous article Multi-token Prediction, DeepSeek-V3 applied an optimized version of multi-token prediction, allowing the model to decode multiple tokens simultaneously, to accelerate the decoding process in training.
• FP8 training: DeepSeek-V3 employs a mixed-precision arithmetic to improve computational efficiency, by using lower-precision formats (such as 8-bit floating point numbers) for certain calculations, reducing memory usage and speeding up computations without significantly compromising accuracy.
• Learning rate scheduling: learning rate is linearly increased from 0 to 2.2e–4 during the first 2K steps, and is kept constant during training on 10T tokens. Then the learning rate is decreased to 2.2e-5 in 4.3T tokens following a cosine curve. In training with the final 500B tokens, the learning rate is kept constant during the training of the first 333B tokens and then further decreased to 7.3e-6 in the remaining 167B tokens.
• Batch size scheduling: batch size is increased from 3072 to 15360 during the training of the first 469B tokens, and then kept constant in the remaining training.

Evaluation

The table below compares DeepSeek-V3 with several other open-source base models on different tasks, where DeepSeek-V3 achieves the best performance on most datasets, especially on tasks related to math and coding.

Note that the strong performance of DeepSeek-V3 is achieved with extremely high training efficiency, due to all the innovations we’ve covered in this series. More specifically, training DeepSeek-V3 on each trillion tokens requires only 180K H800 GPU hours, much cheaper than training 72B or 405B dense models.

Table 3 in [2].

The original paper [2] also conducts comprehensive ablation studies to validate key innovations such as auxiliary-loss-free load balancing and multi-token prediction. However, as we have already covered these topics in previous articles, we will not repeat them here.

Summary

This article explores key innovations in DeepSeek-V3’s pre-training strategies to enhance efficiency, scalability and performance. The resulting DeepSeek-V3-Base model serves as the foundation for more advanced reasoning models like DeepSeek-R1-Zero and DeepSeek-R1, which, in turn, help improve DeepSeek-V3 by knowledge distillation.

Beyond previously discussed architectural innovations — Multi-head Latent Attention, DeepSeekMoE, auxiliary-loss-free load balancing and Multi-token Prediction — this article introduce several techniques including document packing, Fill-in-the-Middle (FIM) and long context extension with YaRN.

Together, these techniques push the boundaries of LLM efficiency and scalability, setting a new benchmark for high-performance AI models.

References

• [1] DeepSeek
• [2] DeepSeek-V3 Technical Report
• [3] Fewer Truncations Improve Language Modeling
• [4] Efficient Training of Language Models to Fill in the Middle
• [4] DeepSeek-Coder: When the Large Language Model Meets Programming — The Rise of Code Intelligence
• [5] DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence
• [6] YaRN: Efficient Context Window Extension of Large Language Models
• [7] DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
• [8] DeepSeek LLM: Scaling Open-Source Language Models with Longtermism
• [9] DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model
• [10] Language Models are Unsupervised Multitask Learners