LunarLander with Deep Q-Networks — From Scratch, No Libraries

What Are We Building?

The first four projects in this series used tabular methods — Q-tables where every state got its own row. That works fine for small discrete environments like grid worlds and card games. But LunarLander has a continuous 8-dimensional state space. You cannot build a table for that.

This is where Deep Q-Networks come in. Instead of a table, I use a neural network to approximate the Q-function. The network takes a state vector as input and outputs Q-values for every action. Everything else — the Bellman update, epsilon-greedy exploration — stays the same.

I implement DQN from scratch using PyTorch. No Stable-Baselines3, no pre-built agents. Every component — the network, the replay buffer, the target network, the training loop — is written by hand so you can see exactly what is happening.

By the end you will have:

• A Q-network implemented in PyTorch
• An experience replay buffer from scratch
• A target network with periodic syncing
• Four visualizations including Q-value estimates vs actual returns

Project Structure

project-05-lunarlander-dqn/
├── replay_buffer.py    # Experience replay
├── agent.py            # QNetwork + DQNAgent
├── train.py            # Training loop + all plots
└── requirements.txt

pip install -r requirements.txt
python train.py

Note: gymnasium[box2d] is required — the square bracket installs the Box2D physics engine that LunarLander runs on.

The Environment

LunarLander-v3 is a continuous state, discrete action environment.

env = gym.make("LunarLander-v3")
# observation_space: Box(8,)   →  8 continuous floats
# action_space:      Discrete(4)  →  nothing, left engine, main engine, right engine

The 8-element state vector contains: x position, y position, x velocity, y velocity, angle, angular velocity, and two boolean flags for whether each leg is touching the ground.

Reward structure: landing safely gives +100 to +140, crashing gives -100, each engine fire costs a small fuel penalty. An episode is considered solved when the agent averages +200 reward over 100 consecutive episodes.

Why Tabular Q-Learning Breaks Here

In CliffWalking, the state space had 48 discrete positions. I built a Q-table of shape (48, 4) and updated individual cells.

LunarLander’s state is 8 continuous floats. Position alone can take infinitely many values. A table is impossible. I need a function that generalises — given a state it has never seen before, it should output reasonable Q-value estimates based on nearby states it has seen.

A neural network does exactly this. It learns a smooth mapping from state space to Q-values, so it can estimate Q(s, a) for states it has never visited.

The Q-Network — agent.py

class QNetwork(nn.Module):
    def __init__(self, state_dim, action_dim, hidden=128):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(state_dim, hidden),   # 8 → 128
            nn.ReLU(),
            nn.Linear(hidden, hidden),      # 128 → 128
            nn.ReLU(),
            nn.Linear(hidden, action_dim),  # 128 → 4
        )

def forward(self, x):
    return self.net(x)

Input: 8-dimensional state vector. Output: 4 Q-values, one per action. Two hidden layers of 128 neurons with ReLU activations. Simple and effective for this environment.

The architecture.png visualization shows this as a node diagram — state inputs on the left, Q-value outputs on the right, two hidden layers in between.

Experience Replay — replay_buffer.py

This is the first major innovation in DQN over vanilla Q-Learning.

class ReplayBuffer:
    def __init__(self, capacity=50_000):
        self.buffer = deque(maxlen=capacity)

def push(self, state, action, reward, next_state, done):
        self.buffer.append((state, action, reward, next_state, done))
    def sample(self, batch_size):
        batch = random.sample(self.buffer, batch_size)
        states, actions, rewards, next_states, dones = zip(*batch)
        return (
            np.array(states,      dtype=np.float32),
            np.array(actions,     dtype=np.int64),
            np.array(rewards,     dtype=np.float32),
            np.array(next_states, dtype=np.float32),
            np.array(dones,       dtype=np.float32),
        )

Every step, the transition (state, action, reward, next_state, done) gets stored in the buffer. During training, I sample a random mini-batch of 64 transitions from the buffer rather than training on the most recent experience.

Why? Two reasons. First, neural networks trained on sequential data develop correlations between consecutive updates — consecutive steps in an episode are highly similar, which causes the network to overfit to recent experience and forget earlier lessons. Random sampling breaks this correlation. Second, each transition gets used multiple times across different batches, which is more sample-efficient than discarding experiences after one use.

The Target Network

This is the second major DQN innovation.

# Online network — trained every step
self.online_net = QNetwork(state_dim, action_dim).to(self.device)

# Target network - frozen, copied every target_update episodes
self.target_net = QNetwork(state_dim, action_dim).to(self.device)
self.target_net.load_state_dict(self.online_net.state_dict())
self.target_net.eval()

During the Bellman update, I need a target value to train toward:

with torch.no_grad():
    max_next_q = self.target_net(next_states).max(1)[0]
    td_target  = rewards + self.gamma * max_next_q * (1 - dones)

The target is computed using the target network, not the online network. Every 10 episodes, the target network gets a copy of the online network’s weights:

def sync_target(self):
    self.target_net.load_state_dict(self.online_net.state_dict())

Without the target network, training is unstable. The online network is being trained to match targets that are themselves produced by the online network — a moving target problem. Every update changes both the prediction and the target simultaneously, which causes oscillations and divergence. Freezing the target network for 10 episodes gives the online network a stable reference point to train toward.

The target_network_effect.png chart marks every sync point as a vertical line on the reward curve — you can see how learning tends to make progress in the windows between syncs.

The Training Step

def train_step(self):
    if len(self.buffer) < self.batch_size:
        return None

states, actions, rewards, next_states, dones = self.buffer.sample(self.batch_size)
    # Convert to tensors
    states      = torch.FloatTensor(states).to(self.device)
    actions     = torch.LongTensor(actions).to(self.device)
    rewards     = torch.FloatTensor(rewards).to(self.device)
    next_states = torch.FloatTensor(next_states).to(self.device)
    dones       = torch.FloatTensor(dones).to(self.device)
    # Q-values for actions that were taken
    q_values = self.online_net(states).gather(1, actions.unsqueeze(1)).squeeze(1)
    # Bellman target using frozen target network
    with torch.no_grad():
        max_next_q = self.target_net(next_states).max(1)[0]
        td_target  = rewards + self.gamma * max_next_q * (1 - dones)
    loss = F.mse_loss(q_values, td_target)
    self.optimizer.zero_grad()
    loss.backward()
    torch.nn.utils.clip_grad_norm_(self.online_net.parameters(), 1.0)
    self.optimizer.step()

The .gather(1, actions.unsqueeze(1)) line is worth understanding. The network outputs Q-values for all 4 actions. I only want the Q-value for the action that was actually taken. gather selects exactly those values from the output tensor — one per sample in the batch.

Gradient clipping (clip_grad_norm_ with max norm 1.0) prevents individual large gradients from destabilizing training. Without it, a single bad transition with a large TD error can produce a massive gradient update that corrupts the network weights.

Training Loop

for ep in range(1, n_episodes + 1):
    state, _ = env.reset()
    done = False

while not done:
        action = agent.select_action(state)
        next_state, reward, terminated, truncated, _ = env.step(action)
        done = terminated or truncated
        agent.store(state, action, reward, next_state, done)
        agent.train_step()   # one gradient update per environment step
        state = next_state
    agent.decay_epsilon()
    if ep % agent.target_update == 0:
        agent.sync_target()

One gradient update per environment step. The buffer must have at least batch_size transitions before training starts, so the first few episodes are pure exploration with no learning.

Plot 1 — Training Curves

training_curves.png shows three charts side by side.

Reward — starts deep negative as the agent crashes repeatedly, then climbs toward 200 as it learns to land. The raw curve is noisy but the smoothed line shows clear upward progress. A dashed green line at 200 marks the solved threshold.

Loss — starts high and decays exponentially as the network converges. High early loss is expected — the network has random weights and Q-value estimates are far from the true values. Loss decreasing confirms learning is happening.

Epsilon — smooth exponential decay from 1.0 to 0.01 over 600 episodes. You can see the exploration-exploitation transition visually — early episodes are almost entirely random, later episodes are almost entirely greedy.

Plot 2 — Q-Value Estimates vs Actual Returns

q_vs_actual.png is the most analytically interesting visualization from this project.

After training, I run 50 greedy episodes and record two things per episode: the average Q-value the network predicted, and the actual return achieved. The left chart plots both over evaluation episodes. The right chart plots them against each other as a scatter, with a diagonal line showing where perfect estimation would sit.

The network overestimates Q-values — a known and well-documented property of DQN. The estimated Q-values are consistently higher than the actual returns.

This overestimation comes from the max operation in the Bellman target. When I compute max Q(s', a') over noisy Q-value estimates, I systematically pick the action whose Q-value is noise-inflated upward. Over many updates, this bias accumulates. Double DQN, which came after the original DQN paper, fixes this by separating action selection from action evaluation — but that is a project for another day.

The scatter plot makes the overestimation bias explicit — most points sit above the diagonal, meaning the network thinks states are worth more than they actually are.

Plot 3 — Network Architecture

architecture.png shows the full forward pass as a node diagram: 8 input neurons, two hidden layers of 128 neurons with ReLU activations, 4 output Q-values.

This replaces the Q-table entirely. In Project 4, Q[state, action] was a lookup. Here, online_net(state)[action] is a forward pass through three linear layers. The weights of those layers are what get updated by backpropagation — the equivalent of updating a Q-table entry, but generalising across nearby states.

Key Takeaways

The Q-table becomes a neural network. Everything else about DQN is recognisable from tabular Q-Learning — Bellman updates, epsilon-greedy, off-policy learning. The network is a drop-in replacement for the table that allows generalisation across continuous state spaces.

Replay buffer breaks temporal correlation. Sequential experience creates correlated training batches that destabilise neural network training. Random sampling from a large buffer fixes this and improves sample efficiency.

The target network provides a stable learning signal. Without it, you are chasing a moving target — every gradient update changes both the prediction and the thing you are predicting toward. Periodic hard copies give the online network a fixed reference for 10 episodes at a time.

DQN overestimates Q-values. The max operator in the Bellman target introduces an upward bias that compounds over training. It does not prevent learning but means Q-values are not accurate predictions of actual returns — they are optimistic upper bounds.

Gradient clipping is not optional. LunarLander has variable episode lengths and reward scales. Without clipping, a single crash with a large TD error can produce a gradient that corrupts weeks of learned weights in one step.