Importance Sampling in Reinforcement Learning

Importance Sampling (IS) is a statistical technique used to estimate expectation under one distribution using samples from another.

In reinforcement learning, it is use to handle the mismatch when:

• You collected trajectories using one policy (e.g. π_old)
• But you want to evaluate or imporve another policy

Why do we need it in RL:

In each episode, we collect trajectories data under a certain policy, say π_old, and then use this data to improve the policy. Once the policy is improved to π_new, the trajectory data collected under π_old may seem outdated. If we want to continue improve π_new, it will take a significant time to collect a new set of trajectory data under π_new. Therefore, it would be highly beneficial if we could still leverage the trajectories collected under π_old instead of discarding them. That’s where importance sampling comes into play.

Suppose you’ve already collected data (state, action, reward) using your old policy π_old, but now you want to improve your new policy π_new.

But you can’t re-collect data every time your policy changes (too slow or expensive). So you re-use old data, but with a correction — importance weights.

How It Works

For example, if you want to compute the expectation of a function under distribution p(x):

In practice, you often estimate this by sampling x_i from p(x), and approximating:

Note: sum has limits: {i=1} to {N}

But what if you can’t sample from p(x) (too expensive, unknown, etc.)?

You do have samples x_i ~ q(x) from a different distribution q(x), called the proposal distribution.

To fix the mismatch, you rewrite the integral:

Now you’re integrating over q(x), which you can sample from, and just re-weight each sample using the ratio:

So now you can approximate:

Note: sum has limits: {i=1} to {N}

In Reinforcement Learning

In RL, this is often used when:

• You have data collected under old policy π_old
• But you want to evaluate or optimize a new policy π_new.

You rewrite the expectation:

Now you can use data from π_old — by reweighting each sample with the ratio:

This ratio is called the importance sampling weight.

Example in PPO

PPO uses the ratio:

This corrects for the fact that actions were sampled using the old policy, but the gradient is trying to update the new policy.

The Problem with Plain Importance Sampling

If there are regions q(x) and p(x) are very different, that weight ratio blows up. In the example from Dr. Hung-yi Lee’s lecture, the samples (green dots) are drawn from q(x), but not from the high-probability region of p(x), where the weight becomes very large. This can lead to biased or wildly inaccurate estimates — like predicting a positive expectation when the actual value is negative — unless we have a huge number of samples.

An example from Dr. Hung-yi Lee’s lecture

Fixes to this issue: e.g. PPO

Proximal Policy Optimization (PPO) avoids this pitfall by clipping the importance ratio. During policy updates, PPO only allows r_t to stay within a small trust region: [1 — epsilon, 1 + epsilon]

This simple trick:

• Caps the variance, even if the policy drifts a bit
• Keeps training stable
• Encourages small, safe updates instead of wild leaps

That’s the essence of what makes PPO both effective and practical in real-world RL tasks.

✦ ✦ ✦Thanks for reading! I hope this breakdown made the topic easier to understand.✦ ✦ ✦

📘 Also check out my related articles about Reinforcement Learning:

• [Policy Gradients Explained Intuitively]
• [Potential-Based Reward Shaping in Reinforcement Learning]
• [Advantage Function — A Core Concept in Actor-Critic Architecture]
• [Rollout in On-Policy Reinforcement Learning: How it’s Structured and Utilized]
• [Trajectory in Deep Reinforcement Learning]
• [What is Bootstrapping in Reinforcement Learning?]
• [Three Ways to Solve Bellman Equations: Intuition Behind Policy Iteration, Value Iteration, and Q-Learning]
• [Game Theory: From Fundamentals to Strategic Depth]
• [Importance Sampling in Reinforcement Learning]