Temporal Difference (TD) Learning is Beautiful

To master reinforcement learning (RL), you have to transition from planning with a known model (like Dynamic Programming) to learning purely from raw experience.

At the absolute core of this transition are Temporal Difference (TD) learning and Q-learning. Let’s break down how they work, how they connect, and the math that drives them.

1. The Core Paradigm: Agent-Environment Loop

In any reinforcement learning setup, an agent interacts with an environment in a continuous feedback loop:

2. Temporal Difference (TD) Learning: “Learning on the Fly”

Before TD, the main way to learn from experience was Monte Carlo (MC) methods. Under MC, an agent has to play out an entire episode to the very end before it can update its estimate of how good a state is.

Temporal Difference (TD) learning changes this by updating estimates based on other learned estimates, a concept known as bootstrapping. Instead of waiting for final outcomes, TD updates its belief about a state’s value after just a single step.

Here is a minimalist Python representation of a single step of a tabular Q-learning update to illustrate the math in code:

import numpy as np

def q_learning_update(Q_table, state, action, reward, next_state, alpha, gamma):
    # 1. Find the estimate of the optimal action in the next state
    best_next_action_value = np.max(Q_table[next_state])

    # 2. Calculate the TD Target
    td_target = reward + gamma * best_next_action_value

    # 3. Calculate the TD Error
    td_error = td_target - Q_table[state, action]

    # 4. Update the Q-value
    Q_table[state, action] += alpha * td_error

    return Q_table