Monte Carlo Reinforcement Learning

Let’s briefly explore a reinforcement learning method for estimating value. I bet (pun intended) you’ve heard the term Monte Carlo method before.

“Monte Carlo methods are a broad class of computational algorithms that rely on repeated random sampling to obtain numerical results.” – [Wikipedia](https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fen.wikipedia.org%2Fwiki%2FMonte_Carlo_method&data=02%7C01%7C%7C35b98717d6674556103508d622111c7d%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636733854222505758&sdata=uONNmmg1rsvc7WiXeNTalhv81c6pqOAZBC4kvISaqoM%3D&reserved=0)****

This policy is all about using randomness for solving real-world problems in mathematics, physics, engineering, finance and of course artificial intelligence.

This project comes from Siraj Raval’s Move37 week 3 assignment. I decided to use the Frozen Lake environment from OpenAI Gym .

*“The agent controls the movement of a character in a grid world. Some tiles of the grid are walkable, and others lead to the agent falling into the water. Additionally, the movement direction is uncertain and only partially depends on the chosen direction. The agent is rewarded for finding a walkable path to a goal tile.” -Frozen Lake*

The grid consist of following tiles:

• S starting point, safe
• F frozen tile, safe
• H hole, not safe at all
• G goal

It will look like this:

Monte Carlo RL algorithm

Let’s go over the Monte Carlo Algorithms. You can view my full Note Book Here..

• On-policy algorithm is improving the policy it’s working on. The opposite is the off-policy.
• first-visit algorithm averages returns for visits following the first visit to given state.The opposite is every-visit.
• MC control or Monte Carlo control. This type of algorithm first estimates values for states, then builds greedy (or near greedy) policy on them just to use the new policy to create another estimate. It loops forever (or rather until reaching a certain amount of episodes).

Function parameters:

• **env** – gym environment, I decided on convention to pass it as a parameter to every function needing one.
• **episodes** – the number of episodes we will base our estimates on, the episode is generated play through data we can learn on.
• **policy** – we can pass policy to improve an existing one instead of generating new random one.
• **epsilon** – exploration rate. A chance to take random action instead of the greedy one.

Here is a snippet of the Monte Carlo Policy portion of the notebook in python:

[embed]

In Conclusion:

What I learned in implementing a Monte Carlo Method in a game environment is the unique aspect of randomness. I also had to grapple with a few math terms like epsilon. All and all this week’s journey through Reinforcement Learning was very fulfilling and it gave me a new way of managing randomness.