[Hands-On] Building GPT-OSS from Scratch — Series Introduction

(You can find the Korean version of the post at this link.)

Series Introduction

This is the first post in the [Hands-On] Building GPT-OSS from Scratch series.

In the previous Mastering GPT-OSS series, we explored the core technologies behind GPT-OSS one by one. We learned why Attention Sink matters, how MoE enables large models with less computation, and how YaRN extends RoPE. In short, we took a deep dive into each building block needed to understand GPT-OSS.

Now it’s time to put all these pieces together and build a complete GPT-OSS ourselves.

Why Build GPT-OSS from Scratch?

“Why bother when I can just use from_pretrained on HuggingFace?”

What you gain from building it yourself is more than just “working code.”

First, true understanding. Concepts that seemed clear when reading papers often reveal gaps when you try to code them. A single line like “compute the dot product of Query and Key” actually involves tensor dimension handling, batch processing, dtype management, and countless other details. You also gain a deeper understanding of how each component connects to the others.

Second, debugging skills. When a model behaves strangely, you need to understand its internals to find the problem. If you treat it as a black box, you get stuck at “Why isn’t this working?” But if you’ve built it yourself, you can think, “The Attention output looks wrong — let me check the RoPE implementation.”

Third, practical flexibility. When you want to add a custom Attention mechanism, apply a new quantization technique, or implement a special decoding strategy, knowing the basic structure lets you modify it with confidence. Many variant architectures (QWEN, DeepSeek, ExaOne, etc.) are emerging these days. A deep understanding of GPT-OSS makes it easy to grasp these variants too.

Series Feature: Bottom-Up Build

The key feature of this series is the step-by-step build approach. All hands-on work is done through Colab notebooks. We start from the smallest unit — token embedding — and add one new component per notebook. Code from previous notebooks is reused in later ones. By the final notebook, you’ll have a fully working GPT-OSS.

Notebook 01: Implementing CustomTokenEmbedding  
    ↓  
Notebook 02: + Implementing CustomRotaryEmbedding and CustomCausalMask  
    ↓  
Notebook 03: + Implementing CustomSelfAttention  
    ↓  
Notebook 04: + Implementing CustomRMSNorm, CustomRouter, CustomExperts, and CustomDecoderLayer  
    ↓  
Notebook 05: + Implementing CustomTwoLayerModel → Complete!

Each notebook runs independently. Code from earlier notebooks is included with # REUSED FROM TUTORIAL 0X comments, so you can open any notebook and run it right away.

Series Outline

This series consists of 5 posts and hands-on notebooks. Each notebook can be run directly in Google Colab.

1. Token Embedding in GPT-OSS 20B
2. RoPE and Attention Mask in GPT-OSS 20B
3. Self-Attention Mechanism in GPT-OSS 20B
4. Complete Decoder Layer in GPT-OSS 20B
5. Two-Layer Transformer Model in GPT-OSS 20B

This hands-on series works best alongside the Mastering GPT-OSS theory series. If the theory taught you “why,” this series shows you “how” through implementation.

I hope this series helps those who want to learn GPT-OSS deeply.