What is SARSA? Understanding the On-Policy Learning Algorithm

Discover the Fundamentals of SARSA, the On-Policy Reinforcement Learning Algorithm Used to Learn Optimal Actions

Reinforcement Learning (RL) is a branch of machine learning in which an agent learns to make decisions by interacting with an environment. Unlike supervised learning, where models learn from labeled examples, reinforcement learning relies on trial and error. The agent receives rewards or penalties based on its actions and gradually learns a strategy that maximizes long-term rewards.

Among the many reinforcement learning algorithms, SARSA stands out as one of the most fundamental and intuitive methods. SARSA is a Temporal Difference (TD) learning algorithm that follows an on-policy learning approach, meaning it learns the value of the policy it is currently using to make decisions.

Although newer algorithms such as Deep Q-Networks (DQN), Proximal Policy Optimization (PPO), and Soft Actor-Critic (SAC) have gained popularity, SARSA remains an essential concept for understanding how reinforcement learning works.

In this article, we will explore what SARSA is, how it works, its advantages and limitations, and when it should be used.

What is SARSA?

SARSA is a model-free reinforcement learning algorithm used to learn an optimal policy for decision-making tasks.

The name SARSA comes from the sequence of elements involved in its update process:

• S = Current State
• A = Current Action
• R = Reward received
• S’ = Next State
• A’ = Next Action

The algorithm updates its knowledge based on the transition:

(State, Action, Reward, Next State, Next Action)

This sequence gives SARSA its name.

As a model-free algorithm, SARSA does not require knowledge of the environment’s transition probabilities or dynamics. Instead, it learns directly from experience.

Understanding On-Policy Learning

To understand SARSA, we first need to understand what a policy is.

A policy is a strategy that tells an agent which action to take in a given state. It is often represented as:

π(a|s)

which denotes the probability of taking action (a) in state (s).

What Does On-Policy Mean?

An on-policy algorithm learns from the same policy it uses to make decisions.

In other words:

• The agent follows a policy.
• The agent learns from actions generated by that policy.
• The learned value function reflects the behavior of the current policy.

SARSA updates its estimates based on the actual action selected in the next state, even if that action was chosen randomly for exploration.

This characteristic makes SARSA sensitive to the effects of exploration during learning.

SARSA vs Q-Learning

SARSA is often compared to Q-Learning because both algorithms learn action-value functions.

However, there is a fundamental difference:

Q-Learning assumes the agent will always take the best possible action in the future.

SARSA assumes the agent will continue following its current policy, including exploratory actions.

As a result, SARSA often learns safer strategies in environments where exploration can be risky.

How SARSA Works

The SARSA learning cycle follows these steps:

1. Observe the current state.
2. Select an action using the current policy.
3. Execute the action.
4. Receive a reward.
5. Observe the next state.
6. Select the next action according to the same policy.
7. Update the Q-value.
8. Repeat until the episode ends.

The critical distinction is that SARSA chooses the next action before updating the Q-value.

The SARSA Update Equation

The core of SARSA is its update rule:

Q(S, A) ← Q(S, A) + α [R + γQ(S’, A’) − Q(S, A)]

Where:

• (Q(S,A)) = current estimate of the action value
• (α) = learning rate
• (R) = reward received
• (γ) = discount factor
• (Q(S’,A’)) = estimated value of the next state-action pair

The term inside the brackets is known as the Temporal Difference (TD) Error.

Breaking Down the Equation

Current Estimate

The value:

Q(S,A)

represents the agent’s current belief about how good an action is in a particular state. Initially, these values may be random or zero.

Immediate Reward

The reward (R) provides feedback from the environment.

• Positive rewards encourage actions.
• Negative rewards discourage actions.

Future Value Estimate

The term:

Q(S’,A’)

represents the expected future reward after moving to the next state and taking the next action. SARSA directly uses the next action actually chosen by the policy.

Learning Rate (α)

The learning rate determines how much new information influences the current estimate.

Typical values include:

• 0.1
• 0.01
• 0.5

A large learning rate leads to faster adaptation but can make learning unstable. A small learning rate provides smoother convergence.

Discount Factor (γ)

The discount factor determines the importance of future rewards.

• (γ = 0): only immediate rewards matter.
• (γ = 1): future rewards are fully considered.

Common values range between 0.90 and 0.99.

SARSA Algorithm Pseudocode

Initialize Q(s,a) arbitrarily
For each episode:
    Initialize state S
    Choose action A using policy
    Repeat:
        Take action A
        Observe reward R
        Observe next state S'
        Choose next action A'
        Q(S,A) = Q(S,A) +
                 α[R + γQ(S',A') - Q(S,A)]
        S = S'
        A = A'
    Until terminal state

This simple procedure allows the agent to gradually improve its policy through repeated interactions.

Example: SARSA in a Grid World

Imagine a robot navigating a grid.

The objective is to reach a goal location while avoiding dangerous cells.

Rewards

• Goal: +10
• Normal move: -1
• Hazard: -100

Initially, the robot moves randomly.

As it experiences rewards and penalties, SARSA updates its Q-values.

Eventually, the robot learns paths that maximize cumulative rewards.

Because SARSA incorporates exploration into its updates, it often learns routes that are safer rather than simply shortest.

Why SARSA Learns Safer Policies

One famous example is the Cliff Walking Problem.

In this environment:

• The shortest route passes near a cliff.
• Falling into the cliff incurs a large penalty.

Q-Learning tends to learn the shortest route because it assumes optimal future behavior.

SARSA learns a safer route because it accounts for the possibility of exploratory actions causing the agent to fall.

This makes SARSA particularly useful in situations where mistakes are costly.

Examples include:

• Autonomous vehicles
• Robotics
• Medical decision systems
• Industrial control systems

Advantages of SARSA

Photo by Igor Omilaev on Unsplash

1. Considers Exploration During Learning

SARSA learns from actual actions taken. This often results in more realistic value estimates.

2. Safer Decision-Making

The algorithm naturally learns conservative policies. This is beneficial when risky actions can lead to severe consequences.

3. Simple to Implement

SARSA is one of the easiest reinforcement learning algorithms to understand and code. It is commonly taught in introductory RL courses.

4. Stable Learning Behavior

Because it follows the current policy consistently, SARSA often exhibits stable convergence properties.

5. Strong Educational Value

Understanding SARSA provides insight into:

• Temporal Difference Learning
• Policy Learning
• Value Functions
• Exploration Strategies

These concepts form the foundation of more advanced reinforcement learning methods.

Limitations of SARSA

Photo by Google DeepMind on Unsplash

Despite its strengths, SARSA has several limitations.

1. Slower Learning

Because it incorporates exploration into updates, SARSA can converge more slowly than Q-Learning.

2. Exploration Dependency

Performance is highly influenced by the exploration strategy. Poor exploration can lead to suboptimal policies.

3. Scalability Issues

Traditional SARSA uses a Q-table. Large state spaces can make the table impractically large.

Examples include:

• Video games
• Robotics
• Autonomous driving

4. Hyperparameter Sensitivity

Choosing appropriate values for:

• Learning rate
• Discount factor
• Exploration rate

can significantly impact performance.

Variants of SARSA

Researchers have developed several improved versions of SARSA.

1. SARSA(λ)

SARSA(λ) introduces eligibility traces.

Benefits include:

• Faster learning
• Better credit assignment
• Improved convergence

2. Expected SARSA

Expected SARSA replaces the sampled next action with the expected value over all possible actions.

Advantages include:

• Lower variance
• More stable updates

3. Deep SARSA

Deep SARSA combines SARSA with neural networks. Instead of using a Q-table, it approximates Q-values using deep learning models. This enables learning in high-dimensional environments.

Real-World Applications of SARSA

Photo by Andy Kelly on Unsplash

SARSA has been applied to numerous practical problems.

1. Robotics

• Path planning
• Obstacle avoidance
• Autonomous navigation

2. Traffic Signal Control

• Adaptive traffic light timing
• Congestion reduction

3. Resource Allocation

• Cloud computing
• Network optimization

4. Energy Management

• Smart grid control
• Battery optimization

5. Industrial Automation

• Process optimization
• Production scheduling

When Should You Use SARSA?

SARSA is particularly useful when:

• Exploration can be dangerous.
• Safety is more important than absolute performance.
• You want stable learning behavior.
• The environment is relatively small and discrete.
• You are learning reinforcement learning fundamentals.

SARSA may not be ideal when:

• The state space is extremely large.
• Fast convergence is required.
• Computational efficiency is the primary concern.

Conclusion

SARSA is one of the most important algorithms in reinforcement learning because it introduces the concept of on-policy learning. By learning from the same policy it follows, SARSA naturally incorporates exploration into its value estimates, often leading to safer and more conservative behavior than off-policy methods such as Q-Learning.

Although newer algorithms dominate modern reinforcement learning research, SARSA remains highly relevant for educational purposes and for applications where safety and stability matter. Understanding SARSA provides a strong foundation for studying more advanced techniques, including Expected SARSA, SARSA(λ), Deep Q-Networks, and modern policy optimization methods.

For anyone beginning their reinforcement learning journey, mastering SARSA is an excellent step toward understanding how intelligent agents learn from experience and improve their decision-making over time.