Explain Markov Decision Process (MDP).

Markov Decision Process (MDP)

A Markov Decision Process (MDP) is a mathematical framework used in reinforcement learning to model decision-making problems where outcomes depend on both the current state and the chosen action.

Components of an MDP

1. States (S)

• Different situations the agent can be in.
• Example: Positions of a robot in a maze.

2. Actions (A)

• Choices available to the agent in each state.
• Example: Move up, down, left, or right.

3. Transition Probability (P)

• Probability of moving from one state to another after taking an action.
• Example: A robot may move in the intended direction with 90% probability.

4. Reward Function (R)

• Immediate feedback received after taking an action.
• Example: +10 for reaching the goal, −1 for each step.

5. Discount Factor (γ)

• Determines the importance of future rewards.
• Range: 0 ≤ γ ≤ 1.
• Higher γ values give more importance to future rewards.

Markov Property

• The future depends only on the current state and action.
• Past states do not directly affect future outcomes.

Objective

• Find an optimal policy that maximizes the expected cumulative reward over time.

Applications

• Robotics
• Game playing
• Autonomous vehicles
• Resource allocation
• Recommendation systems

Key Points

• Foundation of **reinforcement learning.**
• Defines states, actions, rewards, and transitions.
• Uses the Markov property for decision-making.
• Goal is to learn the best action in each state to maximize long-term rewards.