Brief Review — DeepSeek LLM: Scaling Open-Source Language Models with Longtermism

A Brief Review for DeepSeek-v1

**DeepSeek LLM: Scaling Open-Source Language Models with Longtermism DeepSeek-v1, by DeepSeek-AI 2024 arXiv v1, Over 400 Citations **(Sik-Ho Tsang @ Medium)

Large Langauge Model (LLM) 2020 … 2023 [GPT-4] [LLaMA] [Koala] [BloombergGPT] [GLM-130B] [UL2] [PaLM 2] [Llama 2] [MultiMedQA, HealthSearchQA, Med-PaLM] [Med-PaLM 2] [Flan 2022, Flan-T5] [AlphaCode 2] [Mistral 7B] [Alpaca] [Inflection-1] 2024 [Nemotron-4 15B] ==== My Other Paper Readings Are Also Over Here ====

• DeepSeek has been introduced in 2024 already although DeepSeek became widely popular in 2025. So I would like to dive into this early DeepSeek paper to see what techs make it popular.
• In this paper, authors trained DeepSeek LLM 7B and 67B, which are two prevalent used opensource configurations, guided by the scaling laws.
• To support the pre-training phase, a dataset is developed that currently consists of 2 trillion tokens and is continuously expanding.
• Supervised fine-tuning (SFT) and direct preference optimization (DPO) are supervised on DeepSeek LLM Base models, resulting in the creation of DeepSeek Chat models.

Outline

1. DeepSeek Data
2. DeepSeek Model Architecture
3. DeepSeek Model Scaling
4. DeepSeek Alignment
5. Results

1. DeepSeek Data

DeepSeek Data

Three essential stages are used: deduplication, filtering, and remixing.

• The deduplication and remixing stages ensure a diverse representation of the data by sampling unique instances.
• The filtering stage enhances the density of information, thereby enabling more efficient and effective model training.
• In the remixing phase, DeepSeek team addresses data imbalances, focusing on increasing the presence of underrepresented domains.
• The Byte-level Byte-Pair Encoding (BBPE) algorithm based on the tokenizers library (Huggingface Team, 2019) is used. The number of conventional tokens in the vocabulary is set at 100000. The tokenizer was trained on a multilingual corpus of approximately 24 GB, and the final vocabulary is augmented with 15 special tokens, bringing the total size to 100015.

2. DeepSeek Model Architecture

2.1. Model

DeepSeek Model

DeepSeek mainly follows *LLaMA and LLaMA 2*, adopting a Pre-Norm structure with RMSNorm, and using SwiGLU for FFN, with an intermediate layer dimension of 8/3×(d_model).

• It also incorporates Rotary Embedding for positional encoding. To optimize inference cost, the 67B model uses Grouped-Query Attention (GQA) instead of the traditional Multi-Head Attention (MHA).

Specifically, DeepSeek LLM 7B is a 30-layer network, while DeepSeek LLM 67B has 95 layers.

2.2. Pretraining

• A multi-step learning rate scheduler is usedfor pretraining instead of instead of the typical cosine scheduler.
• Specifically, the learning rate of the model reaches its maximum value after 2000 warmup steps, and then decreases to 31.6% of the maximum value after processing 80% of the training tokens. It further reduces to 10% of the maximum value after 90% of the tokens.
• The gradient clipping during the training phase is set to 1.0.

2.3. Infrastructure

• To train a LLM, it is not easier and it is expensive.
• Data parallelism, tensor parallelism, sequence parallelism, and 1F1B pipeline parallelism are integrated into this framework similar to Megatron.
• The model is trained in bf16 precision but gradients are accumulated in fp32 precision. In-place cross-entropy is performed to reduce GPU memory consumption, i.e.: bf16 logits are converted to fp32 precision on the fly in the cross-entropy CUDA kernel.

3. DeepSeek Model Scaling

DeepSeek Model Scaling

Using small-scale experiments can accurately predict the performance of models with 1000× compute budget.

Scaling Laws with Different Data

Current in-house data has higher data quality than early in-house data.

Furthermore, the quality of OpenWebText2 even surpasses the current in-house data.

4. DeepSeek Alignment

Around 1.5 million instruction data instances in English and Chinese are collected, covering a wide range of helpfulness and harmlessness topics.

• The helpful data contains 1.2 million instances, with a distribution of 31.2% for general language tasks, 46.6% for mathematical problems, and 22.2% for coding exercises.
• The safety data consists of 300K instances, covering various sensitive topics.
• The alignment pipeline contains two stages:

1. Supervised Fine-Tuning: 7B model is fine-tuned with 4 epochs, but only 2 epochs for the 67B model.
2. Direct preference optimization (DPO) algorithm (Rafailov et al., 2023), is further used to enhance the model’s ability, which is proven to be a simple but effective method for LLM alignment. An epoch is used for DPO training.

5. Results

5.1. Base Model

Base Model

Despite DeepSeek models are pre-trained on 2T bilingual corpus, they show comparable performance on English language understanding benchmarks with LLaMA 2 models, which also consume 2T tokens but focus on English.

Furthermore, DeepSeek 67B achieves considerably better performance on MATH, GSM8K, HumanEval, MBPP, BBH, and Chinese benchmarks compared to LLaMA 2 70B.

• Some task performance is boosted as model scaling, such as GSM8K and BBH.
• An interesting observation is that the advantage of DeepSeek 67B over LLaMA 2 70B is larger than that of DeepSeek 7B over LLaMA 2 7B.

5.2. Chat Model

Chat Model

The results of the DeepSeek Chat models, showcasing overall improvements in most tasks following tuning.

5.3. Chinese Open-Ended Evaluation

AlignBench

• AlignBench includes a total of 8 primary categories, 36 secondary categories, and encompasses 683 questions.

DeepSeek 67B Chat model surpasses ChatGPT and other baseline models, and is only after the two versions of GPT-4.

5.4 English Open-Ended Evaluation

MT-Bench

• DeepSeek LLM 67B Chat outperforms other open-source models such as LLaMA-2-Chat Touvron et al. (2023b) 70B, Xwin 70b v0.1, and TÜLU 2+DPO 70B (Ivison et al., 2023), and achieves 8.35 score comparable with GPT-3.5-turbo.
• Besides, after the DPO stage, DeepSeek LLM 67B Chat DPO further improves the average score to 8.76, which is only behind GPT-4.

5.5. Safety Evaluation

• The expert team constructed dozens of high-quality test cases for each safety subcategory manually.

DeepSeek exhibits good security performance across numerous safety test categories.

• DeepSeek 67B Chat model has demonstrated notable performance, achieving a score of 97.8, which is higher than both ChatGPT and GPT-4.

5.6. System prompt

• System prompt is proposed:

You are DeepSeek Chat, a helpful, respectful and honest AI assistant developed by DeepSeek. The knowledge cut-off date for your training data is up to May 2023. Always answer as helpfully as possible, while being safe. Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature. If a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don’t know the answer to a question, please don’t share false information.

• When utilizing a 67B LLM, the addition of a prompt leads to significantly improved results.
• (There are a lot of details, please read the paper directly if interested.)