From Random Moves to Intelligent Behavior: Understanding Q-Learning

A technical walkthrough of reinforcement learning ; the framework that turns trial, error, and a scalar number into something that looks a lot like judgment.

In 2013, a small London startup called DeepMind published a paper describing an algorithm that learned to play Atari video games from raw pixel input. It was given the screen, the score, and nothing else ; no rules, no game manuals, no hints about what the joystick was supposed to do. It learned to play by playing. After enough practice, it exceeded human-level performance on 29 of 49 games tested.

The technique that made this possible was not a new idea. Q-Learning had been formalized by Chris Watkins in his 1989 PhD thesis. What DeepMind did was combine it with a deep neural network and some engineering ingenuity and suddenly a 35-year-old algorithm was beating professional video game players.

This article is about what reinforcement learning actually is, why it is structurally different from supervised learning, and how Q-Learning works from first principles , including the mathematics that drive it.

Why supervised learning is not enough

Most machine learning taught in courses and deployed in industry is supervised learning. We have a dataset, each example is labelled with the correct answer, we train a model to predict that label, and we ship it. This works extraordinarily well for image classification, speech recognition, fraud detection, and many other problems but it has dependency. Someone, or some process, has to know the right answer in advance.

Now consider a different class of problems. How do we label the correct move in a chess position? Even grandmasters disagree, and the right move depends on the opponent, the clock, and the broader game state. How do we label the correct action for a robot arm trying to pick up an irregular object? How do we label the correct bid in an online auction where the optimal bid depends on every other participant’s private valuation?

These are sequential decision-making problems under uncertainty. The right action is not known in advance ,it has to be discovered through interaction with the environment. Supervised learning has no mechanism for this. Reinforcement learning was built for exactly this class of problem.

What is Reinforcement Learning?

Reinforcement learning is a framework for learning through interaction, through trial and error over a period of time. The canonical analogy is teaching a child to ride a bicycle. We do not hand them a physics textbook and explain the mechanics of balance. We put them on the bicycle, let them try, and respond to their progress with encouragement when they stay upright longer, catching them when they fall. Over many attempts, they internalize a control strategy that no explicit instruction could have transmitted. Reinforcement learning is the computational formalization of this process.

Reinforcement learning is formally defined within the Markov Decision Process (MDP) framework ,a mathematical structure for modeling sequential decision-making with five components.

Agent and environment :- The agent is the learner and decision-maker. The environment is everything the agent interacts with. The boundary between them matters: the agent can observe parts of the environment and act upon it, but does not have direct access to the environment’s internal mechanics. The agent and environment interact in a loop ,at each discrete time step; the agent observes the current state, selects an action, receives a reward, and the environment transitions to a new state. This loop continues until a terminal condition (the end of an episode) is reached.

State :- The state s is a representation of the environment at a given time step. In theory, the full state contains all information relevant to predicting future states and rewards. This is the Markov property: the future depends only on the present state, not on the history of how you got there. In practice, defining state is one of the most consequential decisions in any RL project because too little information means the agent cannot make good decisions but on the other hand, too much information makes the state space intractably large.

Action :- The action space A defines the set of choices available to the agent at each time step. It can be discrete (UP, DOWN, LEFT, RIGHT) or continuous (a real-valued vector, like the torque applied to each joint of a robot arm). The algorithms appropriate for each case differ substantially.

Reward :- The reward R is a scalar signal received after each action. It is the only feedback the agent receives about the quality of its decisions. The agent's objective is to maximize the cumulative discounted reward over an episode:

G(t) = r(t) + γ·r(t+1) + γ²·r(t+2) + γ³·r(t+3) + …

Equation 1 — Discounted return from time step t

Policy :- The policy π is what the agent learns: a mapping from states to actions (or to probability distributions over actions). The goal of RL is to find the optimal policy π* that maximizes expected cumulative reward.

Q-values: what are they, really?

To find the optimal policy, we need a way to evaluate how good it is to take a specific action in a specific state. This is the Q-function, also called the action-value function:

Q(s, a) = E[ G(t) | S(t) = s, A(t) = a ]

Equation 2 — Expected cumulative return from state s, taking action a

Q(s, a) answers a concrete question: if I am in state s, take action a right now, and then follow the optimal policy from that point forward, what total reward can I expect? If you know Q(s, a) for every state-action pair, the optimal policy is trivial (always take the action with the highest Q-value):

π(s) = argmax[a] Q(s, a)

Equation 3 — Optimal policy from optimal Q-function

The entire challenge of Q-Learning is computing/ approximating these Q-values from experience.

The Bellman Equation

The Bellman equation is the recursive relationship that makes Q-Learning possible. Richard Bellman derived it in the 1950s as part of his work on dynamic programming, long before reinforcement learning existed as a field. It says: the value of a state-action pair today equals the immediate reward received plus the discounted value of the best action available in the next state.

Q(s, a) = E[ r + γ · max[a’] Q(s’, a’) ]

Equation 4 — Bellman optimality equation

This is elegant because it defines Q* in terms of itself. Each time you take action a in state s, receive reward r, and land in state s’, you have a sample of what Q(s, a) should be:

Target = r + γ · max[a’] Q(s’, a’)

Equation 5 — Q-learning update target

The difference between your current estimate and this target is the temporal difference error δ (delta). The larger this error, the more your estimate was wrong, and the more it should shift.

Notice that Q-Learning is off-policy: the target uses max Q(s’, a’) , the greedy best action ,regardless of what the agent actually does in s'. This means the agent can explore randomly during training while still learning about the optimal greedy policy. It is one of Q-Learning's most useful properties.

Q — Table : Storing what we’ve learnt

When the state and action spaces are small enough — say, a few hundred states and four possible actions — you can store Q(s, a) explicitly as a table. Each row is a state, each column is an action, and each cell holds the current estimate of that state-action value. At initialization, every cell is zero because the agent knows nothing. As training proceeds and the update rule fires after each step, the table gradually fills with learned values. The table below shows a partially trained Q-table for a Snake agent using an 8-bit binary state (four danger direction bits + four food direction bits).

Exploration V/S Exploitation

In machine learning, exploration is the action of allowing an agent to discover new features about the environment, while exploitation is making the agent stick to the existing knowledge gained. If the agent continuously exploits past experiences, it likely gets stuck. On the other hand, if it continues to explore, it might never find a good policy, which results in exploration-exploitation dilemma.

Epsilon-greedy :- The most widely used approach is epsilon-greedy. At each step, with probability ε (epsilon) the agent selects a random action. With probability 1 − ε, it selects the greedy best action from the Q-table. ε is initialized at 1.0 (pure exploration) and decayed over training toward a small floor like 0.05:

ε(t) = max( ε_min , ε₀ · decay_rate^t )

Equation 7 — Epsilon decay schedule

The decay schedule matters. Too fast and the agent commits before it has explored enough. Too slow and it wastes training time on random actions after it already has a good policy. The chart below shows how epsilon evolves across episodes for different decay rates and what fraction of time the agent spends exploring vs. exploiting at each point in training.

Reward Engineering

The reward function is the learning objective. It is the only channel through which you communicate what you want the agent to achieve. And it is much harder to get right than it looks.

Sparse rewards are the baseline problem. If the only reward signal is +1 for winning a game that takes 200 moves, the agent receives almost no useful feedback for most of its training time. It effectively operates blind until it accidentally discovers the reward by chance.

Reward shaping is the practice of adding intermediate reward signals to guide learning faster. The reward function is a formal specification of what you want. If your specification has gaps, the agent will find them. This is why reward engineering is considered one of the hardest open problems in RL.

Why RL is genuinely hard ?

Four structural difficulties make RL harder than supervised learning:

Credit assignment :- when an agent loses after 200 moves, which moves were responsible? The reward arrives at the end; the consequential decisions may have been made 50 steps earlier.

Sample efficiency :- RL agents require enormous numbers of environment interactions to converge. The DQN that mastered Atari trained on hundreds of hours of gameplay per game.

Non-stationarity :- as the agent’s policy changes, the distribution of states it visits changes too. The environment, from the agent’s perspective, is constantly shifting.

Hyperparameter sensitivity :- learning rate, discount factor, epsilon decay schedule: small changes in any of these can be the difference between convergence and divergence.

The Ceiling of Tabular Methods

Q-tables work well when the state space is small. An 8-bit binary state gives 256 possible states; the Q-table has 1,024 entries and converges quickly. But encode the full 20×20 Snake board as a binary grid and the state space is 2⁴⁰⁰ (larger than the number of atoms in the observable universe). We cannot build a table with 2⁴⁰⁰ rows.

This is the curse of dimensionality applied to RL. Beyond a few hundred states, you need function approximation(a model that generalizes from states it has seen to states it hasn’t).

DQN (DeepMind, 2013) replaces the Q-table with a neural network. Two innovations made it work: experience replay (storing transitions in a buffer and sampling random mini-batches to break temporal correlation) and a target network (a separate, periodically-updated copy of the network to stabilize the training target). PPO takes a different approach entirely; rather than learning Q-values, it directly optimizes the policy itself, using a clipping mechanism to prevent updates so large they destroy previously learned behavior.

Conclusion

RL is a rigorous mathematical framework for solving sequential decision-making problems through experience. The ingredients are simple: agent, environment, state, action, reward, update rule. The difficulty is in the engineering like designing state representations that are rich enough without being intractably large, writing reward functions without loopholes, and knowing when to move from a Q-table to a neural network.

Q-Learning is where most people’s journey into reinforcement learning begins ,and for good reason. The math is approachable, the results are interpretable, and the core idea scales all the way from a toy Snake game to the systems that beat world champions at Go and StarCraft. Understanding it properly is understanding the foundation that everything else in RL is built on.

References: Watkins, C.J.C.H. & Dayan, P. (1992). Q-Learning. Machine Learning 8, 279–292. Mnih, V. et al. (2015). Human-level control through deep reinforcement learning. Nature 518, 529–533. Schulman, J. et al. (2017). Proximal Policy Optimization Algorithms. arXiv:1707.06347. Sutton, R.S. & Barto, A.G. (2018). Reinforcement Learning: An Introduction (2nd ed.). MIT Press. Bellman, R. (1957). Dynamic Programming. Princeton University Press.