From Q-Tables to Neural Networks: An Introduction to Deep Q-Networks (DQN)

Transition from Q-Tables to Deep Q-Networks (DQN) ). Learn how “Experience Replay” and “Target Networks” stabilize AI training in complex environments like Space Invaders.

In our last articles, we built our first “smart” agent. We learned about Q-Learning and used a Q-table to store a “cheat sheet” of Q-values. This worked wonderfully for simple problems like ‘FrozenLake’ and (with a bit of a hack) ‘CartPole’. In those worlds, our agent could look up the best action for any given state.

But what happens when the world isn’t simple? What happens when our environment becomes so complex that a cheat sheet is no longer possible?

Not a Medium member? Click here to read the full article

Visualizing the evolution of Reinforcement Learning: Why Q-Tables fail in complex environments like Space Invaders and how Deep Q-Networks (DQN) solve the problem of infinite state spaces.

This is the “wall” that researchers in AI hit for years. Our simple Q-table, as intuitive as it is, has a critical, fatal flaw. This article will explain that flaw and introduce the revolutionary idea that finally broke through it: the Deep Q-Network (DQN).

In case you are not familiar Q-Learning and Q-tables read the following:

[embed]Understanding Q-Learning: The Original ‘Smart’ Algorithm A guide to the Q-table, the Bellman Equation, and the balance between exploration and exploitation.python.plainenglish.io

This is the conceptual leap that takes us from classic Reinforcement Learning to Deep Reinforcement Learning — the field that powers everything from game-playing AI to robotic control.

💡 What You’ll Learn in This Article

• ✔️ The “Curse of Dimensionality”: The critical, fatal flaw of Q-tables that stops them from scaling to complex, real-world problems.
• ✔️ The Core Idea of DQN: The revolutionary concept of switching from storing Q-values in a table to predicting them with a neural network.
• ✔️ How a DQN Works: The high-level architecture of how a state is fed into a network to get action values.
• ✔️ The Two Tricks That Make It Stable: A clear explanation of Experience Replay (the “memory” buffer) and Target Networks (the “stable” copy) that are essential for successful training.
• ✔️ The Big Picture: How this one idea bridges the gap from classic “toy” problems to the powerful field of Deep Reinforcement Learning that can master complex games from raw pixels.

The “Curse of Dimensionality”: Why Q-Tables Fail

A Q-table is just a lookup table. Its rows are all possible states, and its columns are all possible actions. Mathematically: Q[state, action]

Lunat Lander by Gymnasium

This works perfectly when you have a small, manageable number of states.

• FrozenLake (4x4): 16 states. No problem.
• A Chessboard: An estimated 10⁴⁷ states. Our table would be larger than all the hard drives on Earth.
• LunarLander (8 continuous values): The number of states is infinite. How can you have infinite rows in a table?

This exponential explosion of states is called the Curse of Dimensionality.

As we saw, we can try to “hack” our way around continuous states using discretization (binning). For CartPole (4 state variables), we could split each into 10 bins, giving us 10⁴ (10,000) states. This is manageable.

But for LunarLander (8 state variables), 10 bins each would mean 10⁸ (100 million) states. And for an Atari game like SpaceInvaders, the "state" is the screen pixels (e.g., an 84x84 image). The number of possible pixel combinations is practically infinite.

We are forced to conclude: A Q-table is not scalable.

The Revolutionary Idea: From Storing to Predicting

This is where the breakthrough happens. We need to stop thinking about storing the Q-value for every single state. What if, instead, we could predict the Q-value on the fly for any state we encounter?

What tool do we know that is incredibly good at learning to approximate complex functions?

A Neural Network.

This is the entire core idea of a Deep Q-Network. We are replacing the Q-table (a data structure) with a neural network (a function approximator).

• Q-Table (Old Way): A big table that stores Q-values. You find the value by looking up a row.
• Q-Network (New Way): A neural network that predicts Q-values. You find the value by feeding the state into the network and getting a prediction.

This simple switch solves both of our problems:

1. Handles Infinite States: The network takes the continuous state vector (e.g., 8 numbers from LunarLander) as input and just outputs the Q-values. It doesn't care that it's never seen that exact combination of numbers before. It generalizes from past experiences.
2. Handles Complex States: For a game like SpaceInvaders, we can use a Convolutional Neural Network (CNN). The CNN takes the raw screen pixels as its input and learns to identify patterns (the player, the aliens, the bullets) all on its own, outputting the Q-values for "left," "right," and "fire."

How a DQN Works

The data flow is simple. The agent’s decision-making process is still the same as before, just with a new tool.

1. The Environment provides the current State (e.g., an 8-number vector for LunarLander).
2. The Agent feeds this state into its Neural Network.
3. The Network outputs a vector of Q-values, one for each possible action (Q(Do Nothing), Q(Fire Left), Q(Fire Main), Q(Fire Right)]).
4. The Agent uses its Epsilon-Greedy policy:

• Exploit: Choose the action with the highest Q-value ( np.argmax(q_values)).
• Explore: Choose a random action.

This is how the agent acts. But how does it learn?

The Two Tricks That Make It Stable

Training a neural network while it’s actively collecting data in a feedback loop is notoriously unstable. It’s like trying to build a moving train.

In 2015, the researchers at DeepMind who first successfully trained an agent to play Atari games introduced two key techniques to make this process stable.

1. Trick: Experience Replay

• The Problem: If the agent learns only from its experiences as they happen, the data is highly correlated. If it’s in a bad spot, it might spend 100 steps in a row learning “this is bad.” This isn’t efficient and can lead the network astray.
• The Solution: We give the agent a “memory” in the form of a replay buffer (a deque in Python). As the agent plays, it stores all its experiences (state, action, reward, next_state, done) in this buffer.
• The Learning: To train the network, we don’t just use the last experience. Instead, we sample a random mini-batch (e.g., 64 experiences) from the memory buffer.
• Why it Works: This breaks the correlation. The agent learns from a diverse set of its past experiences — some good, some bad, some recent, some old. This is far more stable and efficient.

2. Trick: The Target Network

• The Problem: We update the network’s weights using the Bellman equation, which relies on its own predictions for the next state’s value. This is called “bootstrapping.” The problem is that our network is constantly changing. We’re trying to hit a “target Q-value” that is also moving at the same time. This is like trying to measure your height with a ruler that is also growing.
• The Solution: We use two neural networks (DDQN).

1. Policy Network: This is the main network that the agent uses to pick actions. Its weights are updated at every training step. This is the “live” network.
2. Target Network: This is an exact clone of the policy network. Its weights are frozen. We use this stable, unchanging network to calculate the target Q-values for our learning updates.

• The Learning: Periodically (e.g., every 1,000 steps), we copy the weights from the policy network over to the target network. This “shifts” the stable target, but it keeps it steady between updates.

This provides a stable, consistent target for our policy network to learn towards, solving the “moving target” problem.

How they work together in steps:

Understanding DQN through analogy: The Policy Network (Student) learns from the Replay Buffer (Experience) while being graded by a stable Target Network (Answer Key).

1. Student Tries an Action: The Student (Policy Network) looks at the game screen and decides to do something (e.g., press the ‘jump’ button).

2. Experience & Memory (Replay Buffer): The game reacts: the Student gets a score (reward), and the screen changes (next state). The Student remembers this whole “experience” (what it saw, what it did, what score it got, and what happened next) and saves it in its “memory” (the Replay Buffer). It keeps a lot of past experiences in this memory.

3. Student Studies: Compares Guess to Answer Key: To learn, the Student pulls out some random old experiences from its memory. For each experience, it tries to predict what the “score” should have been for the action it took. This is its guess.

4. Answer Key Helps Grade: At the same time, the Answer Key (Target Network) also looks at what happened after the Student’s action (the “next state”). The Answer Key uses its stable knowledge to calculate what the correct ideal score should have been for that situation. This is like the actual answer from a textbook.

5. Student Learns & Updates Brain (Policy Network Updates): The Student compares its own guess to the Answer Key’s correct ideal score. If its guess was wrong, the Student adjusts its brain (its “weights” or strategy) to make its guesses better next time. It’s like a student learning from mistakes on a test.

6. Answer Key Periodically Updated: After the Student has learned and updated its brain many, many times, the Answer Key eventually copies the new, improved brain of the Student. This is like giving the student a new, updated answer key after they’ve learned a lot, but it happens slowly so the Answer Key remains a stable source of truth.

By having these two separate “brains,” the AI learns much more smoothly and effectively, just like a student benefits from a stable answer key rather than one that changes its answers every time the student makes a guess!

Conclusion

We’ve now bridged the gap from classical RL to Deep RL. We saw that the Q-table, while intuitive, fails completely in the face of complex or continuous environments (the Curse of Dimensionality).

The solution was the Deep Q-Network (DQN), which replaces the Q-table with a neural network that predicts Q-values instead of storing them.

This revolutionary idea, combined with the stabilizing tricks of Experience Replay and a Target Network, is what allowed an agent to learn to play Atari games, like SpaceInvaders, from raw pixels, often achieving superhuman performance.

This breakthrough laid the foundation for the entire field of Deep Reinforcement Learning and the incredible achievements (like AlphaGo) that followed. Now you’re ready to tackle the code that makes it all happen!

In the next Episode we will train Space Invaders with a DQN. Stay tuned and follow me to get notified when we launch Space Invader A.I. 🤖🤖🤖

[embed]From Theory to Code: Let’s Build a Deep Q-Network (DQN) An Implementation Walkthrough for the Atari Classic 'Space Invaders'christianbernecker.medium.com

Want to Connect?

• If you enjoyed the article, give me a few claps below 👏👏👏…
• Follow me to learn more about AI 🤖🤖🤖 …
• Find me on LinkedIn

As always, if you have any questions, ideas, recommendations don’t hesitate to ask in the comments.