Building a Regime-Aware Deep RL Trading Agent

Four-layer architecture

Lessons from the Trenches How I built an algorithmic forex trading system using Deep Reinforcement Learning, survived two catastrophic failures, and learned why risk constraints matter more than reward shaping.

TL;DR

Built a Deep RL trading agent for EUR/USD using regime classification and off-policy learning

• Failure 1: PPO agent learned to WAIT forever (100% inaction, zero trades)
• Failure 2: DQN agent learned to overtrade (5,000+ trades, 85% drawdown)
• Final fix: Exponential drawdown penalties plus hard episode termination at 15% DD
• Key insight: In trading RL, risk constraints are not preferences. They are physics.

The Original Idea

Can Deep Reinforcement Learning learn when to trade rather than what the market will do next?

Instead of predicting prices, the goal was to learn selective participation based on market regimes.

The system architecture was simple in theory:

1. A Random Forest classifier labels each market state
2. An RL agent decides: WAIT, LONG, or SHORT
3. Regime gating blocks trades in choppy conditions.
4. A risk engine enforces stop-loss and take-profit

Elegant. Modular. And completely broken on the first attempt.

Act I: The 100% WAIT Agent

After 250,000 PPO training steps, the metrics looked like this:

WAIT%: 100.0%
Trades: 0
ep_rew_max: -inf
Worst DD: 0.00%

The agent never placed a single trade. The agent never placed a single trade.

What Went Wrong?

The reward function had three components:

1. Trade P&L with a convex utility
2. A linear drawdown penalty
3. A time penalty for holding positions

The PPO agent found the optimal solution immediately:

Never trade. Never lose. Never pay time penalties.

This wasn’t a bug. The agent was correct. PPO simply did exactly what the environment incentivized.

The Real Problem: Sparse Rewards

In 17,000 hourly EUR/USD bars, profitable trades are rare. PPO, being on-policy, never experienced enough positive outcomes to learn that trading could be beneficial at all.

The agent wasn’t lazy. It was blind.

Bootstrapping the Agent Back to Reality

To fix the sparsity issue, I switched to off-policy learning and seeded the replay buffer.

The process:

1. Run two episodes with random actions
2. Collect ~6,000 transitions
3. Include both wins and losses (36 profitable trades)
4. Seed DQN’s replay buffer
5. Let the agent replay rare successes repeatedly

This isn’t a hack. In sparse RL environments, it’s standard practice. Markets are sparse. I had simply underestimated how sparse.

Act II: The 85% Drawdown Catastrophe

With DQN trained on bootstrapped experience, the agent finally traded.

The results:

WAIT%: 19.8%
Trades: 5,136
ep_rew_max: +0.68
Worst DD: 84.91%

The Paradox

• The agent proved profitability exists
• And simultaneously blew up the account

Action analysis showed a 64% SHORT bias. The agent found a pattern, exploited it aggressively, and refused to stop when conditions changed.

Why Risk Failed

The drawdown penalty was linear:

dd_penalty = max_drawdown_pct / 10

At 85% drawdown, the penalty was only 8.5. Trade rewards could exceed that, so the agent rationally chose to keep trading. Again, the agent wasn’t wrong.

The environment’s physics were.

Act III: Fixing the Physics

Two failures, opposite behaviors, same root cause.

• PPO learned to avoid penalties by never trading
• DQN learned to chase rewards by ignoring risk

Neither learned the actual objective:

Trade selectively, survive drawdowns, stay in the game.

The Final Fix

def _dd_penalty(self, current_dd):
    if current_dd < 0.05:
        return 0.0
    if current_dd < 0.15:
        normalized = (current_dd - 0.05) / 0.10
        return 10.0 * (normalized ** 2)
    return 100.0

def _check_done(self):
    return self._calculate_drawdown() > 0.15

Key changes:

• No penalty below 5% drawdown
• Quadratic penalties between 5% and 15%
• Hard episode termination at 15% DD
• Removed time penalties entirely

This was not reward shaping. This was defining what is allowed.

Regime Classification

Markets are continuous, but behavior is not. I used a Random Forest with the following features:

• ATR (absolute and percentile-based)
• Bollinger Band width
• Rolling volatility
• Moving average spread and slope
• Directional consistency
• Volume ratio
• Trading session

Regimes:

• TREND: Directional bias, allowed to trade
• VOLATILE: Breakout risk, cautious trading
• RANGE: Mean-reverting noise, WAIT
• DEAD: Low volatility, WAIT

Regime gating prevents trading in bad conditions, but it does not prevent overtrading in good ones. That distinction mattered.

Reward Design

The reward function was intentionally asymmetric:

if R > 0:
    reward = sqrt(R)
    if regime == VOLATILE:
        reward *= 0.7
else:
    reward = -2 * abs(R)

Small consistent wins were preferred. Losses hurt more than wins helped. Gambling behavior was discouraged.

Why DQN Worked Where PPO Failed

• Markets produce rare positive feedback
• PPO cannot reuse rare experiences
• DQN replays them thousands of times

In sparse domains, off-policy learning isn’t optional.

Current Status

Training with exponential drawdown penalties is ongoing.

Expected behavior:

• WAIT%: 65–80%
• Trades per episode: 5–20
• Worst drawdown: capped at 15%
• Returns: secondary to survivability

In trading, staying alive comes first.

What This Project Taught Me

1. Risk constraints are physics

You don’t ask an agent to respect drawdown. You force it.

2. Sparse rewards change everything

Markets don’t reward often. Your algorithm must be able to remember rare success.

3. Two failures reveal more than one success

PPO exposed incentive flaws. DQN exposed missing constraints. Together, they showed the truth.

4. Regime awareness helps, but isn’t enough

Filtering bad markets doesn’t prevent self-destruction in good ones.

5. Behavior metrics beat returns

I track:

• WAIT%
• Trades per episode
• Worst drawdown
• Max episodic reward

Returns are lagging indicators. Behavior tells the story early.

Code and Reproducibility

Full implementation: GitHub: *Deep_Reinforcement_Learning_Agent*

Stack:

• Stable-Baselines3
• Gymnasium
• scikit-learn
• EUR/USD 1H data (2020–2023)

Quick start:

git clone https://github.com/Mrugendra7911/Deep_Reinforcement_Learning_Agent.git
cd Deep_Reinforcement_Learning_Agent
pip install -r Requirements.tx

Final Thought

RL agents don’t lie.

If an agent refuses to trade, your incentives are broken. If it trades itself into ruin, your constraints are missing.

The art of RL trading isn’t reward engineering. It’s defining the rules of reality.

Get the physics wrong, and no hyperparameter will save you.

If this was useful, follow for more deep dives into applied RL, algorithmic trading, and the uncomfortable truths of production ML.