Creating a Machine-Learning Model to Play Pong

AI dominates most of the news these days, but how do they actually work? I wanted to find that out, and to do that, I created my own.

The game of Pong is one of the oldest and simplest games on Earth. If you don’t know, Pong is a multiplayer game where two players each control a paddle on opposite sides of the screen. There’s also a ball that bounces around the screen, and the goal is to move your paddle so the ball bounces off of it when it reaches your side. If the ball gets past your paddle, the other players gains a point.

Now, even though the game is simple, I have a question: When playing this, what is the optimal strategy?

Well, it’s (somewhat obviously) to move your paddle to follow where the ball is. If you’re always moving your paddle to block the ball, you will never lose.

OK, but here’s another question: How did you find out that the optimal strategy was to follow the ball?

This question is important because in order to teach a machine how to play this game, we need to know how anything can learn in the first place.

The Data

Normally, machine-learning models work by taking in data and using it to make predictions. In our case, we want to make predictions (we want to predict the best possible move), but we don’t have any existing data of pong games being played. Instead, the model will create its own data by playing pong games over and over against itself. Each piece of data will contain the initial state of the game, what action the model took, the reward that the action resulted in, and the state of the game after the action was taken.

The Model

There are many methods to teach a machine how to play a game, but the one I used is known as Deep Q-Learning (DQN), a form of reinforcement learning. How it learns is similar to how a child learns. First, it takes in information about the world. Then, it does a bunch of random actions and looks at how much it is rewarded for doing each action. Then, by looking at which actions gave them more rewards and which actions gave them less rewards, it changes its behavior to optimize for more rewards.

Here is the basic loop for the reinforcement learning model that I used:

1. Obtain all the information (e.g. location of your own paddle, location of your opponent’s paddle, speed of the ball)
2. Feed this information through the model. The model will give an output of a list how much reward each possible action (moving left, moving right, staying still) will result in. This list is known as Q-Values
3. Perform the action with the highest reward that the model gave out.
4. After the action is done, obtain the reward that the model gets based off of the results of the action (e.g. having the ball bounce off your paddle will give a +0.1 reward but losing a point will give a -1 reward). The reward should ideally be in between -1 and 1.
5. After finding out the reward, Modify the model slightly based off of the difference between the predicted and the actual reward.
6. Repeat the process every frame.

This is the core of DQN. Everything else is optimizations to make the model work better.

Optimizations

There are a few problems with the training loop I mentioned. Here are the problems and how each of them are solved:

Problem 1: When you’re training on every single frame, consecutive frames are going to be very similar. For example, if you’re moving a paddle to the right, there’s not going to be much difference in your action when you just started moving and two frames after you just started moving. This means that if you train a model on this, it will get trained hundreds of times on essentially the exact same data, meaning it will overfit to this exact scenario.

Solution: Instead of always training on the previous frame, store the previous frame in a list along with every other frame that has previously happened. Then, when you want to train the model, randomly pick a few frames to train on. This means that the model will train on a variety of scenarios once instead of training on a single scenario hundreds of times. This is known as a Replay Buffer.

Problem 2: Some actions don’t have immediate rewards. For example, when moving to the right to intercept the ball, the model only gets rewarded on the same frame that the ball hits the paddle even though the action of moving to the right started a long time ago. This means that the model won’t know the actual cause behind it getting the reward other than “move right immediately before the ball reaches the paddle”.

Solution: Give rewards not only to the action on the same frame as the reward but also to actions before it. Each reward is multiplied by a value known as “Gamma” before getting added to each previous action to incentivize actions with immediate effects, but previous actions also giving rewards lets the model know what it did that helped contribute to getting a reward. This is known as the Bellman Equation.

Problem 3: All the model knows is how to optimize for rewards. This means that it might lose track of the end goal. For example, if I penalize my model too much, the model might purposefully lose to that it won’t get penalized as much since the net reward would still be more than if it kept playing.

Solution: Tweak the method of giving rewards. Ensure that winning gives a lot higher reward than anything else and that losing subtracts a greater reward than anything else.

Problem 4: Here’s a hypothetical scenario. There’s a button that sometimes gives -1 point but usually gives 10 points. The model is in its initial, untrained state and decides to press the button. It gets -1 point. The model learns that the button is bad. It gives pressing the button an expected reward of around -1 and proceeds to never press it in favor of other decisions that it thinks will give it more points. Even though the expected value of pressing the button is more than 0, the model never learns that the button is good.

Solution: The model, when training, should always have a random chance of performing a completely random action. The chance of a completely random action being outputted is called the Epsilon. Initially, the Epsilon should be high because the model’s predicted reward for each action is inaccurate and it should explore as many different options as possible. The Epsilon should gradually decrease over time because as the model improves and becomes more accurate, it should perform the action it thinks is best more often. The rate at which Epsilon decreases is called the Epsilon Decay.

The Results

Here is a short clip of my model at work. Both the player at the top and the player at the bottom are the same model. As you can see the paddle is very shaky. This is likely because the model is trying very hard to keep the paddle’s x position close to the ball’s, since I reward it for doing this. Overall, the model worked well and could survive a long time.

Conclusion

The lesson I learned the most from doing this was that even if you coded everything correctly, reinforcement learning models might not work as intended because of bad settings. For example, I initially set the model’s Epsilon Decay value to make the Epsilon decrease way too slowly, resulting in it taking an extremely long time to learn. This was different to my usual coding experience where I’ll usually see some error or at least notice some glitch happening if my code is not working as I wanted it to. Overall though, this was a great experience and I think I know how Q-learning and reinforcement learning in general work really well now.