Teaching an RL Agent to Play Hangman: Practical Tips for training RL&DL systems

Hangman looks simple, but training a reinforcement learning agent to play it well is anything but. This toy problem quickly exposed debugging challenges and failure modes that apply to real-world RL systems.

This post shares practical lessons I learned while building a PPO-based agent from scratch to play Hangman. The goal isn’t to present a “solved” system, but to document debugging insights and what actually moved the needle — lessons that only surfaced after many iterations.

Brief PPO Background

Before diving in, a quick refresher on why I chose PPO (Proximal Policy Optimization).

Reinforcement learning algorithms generally fall into a few categories:

On-policy vs Off-policy: On-policy methods (like PPO) must learn from data generated by the current version of the policy — every time the policy updates, old experience becomes stale and must be discarded. Off-policy methods (like DQN) can reuse past experiences from a replay buffer, even if those actions were taken by an older version of the policy. This makes off-policy methods more sample-efficient but harder to stabilize.

Value-based vs Policy gradient: Value-based methods (like Q-learning) learn a value function that estimates “how good is this state/action?” and derive a policy by choosing actions with the highest value. Policy gradient methods directly learn a probability distribution over actions and adjust it based on which actions led to better outcomes. The key difference: value methods learn to evaluate, policy methods learn to act.

PPO is an on-policy, policy gradient method that balances learning stability with sample efficiency. It uses a clipping mechanism to prevent destructively large policy updates, making it more stable than vanilla policy gradients while remaining simpler than methods like TRPO.

For readers interested in PPO theory, this article gives an excellent breakdown.

Problem Setup

Data: Large dictionary with variable word lengths. Train/test split ensures the agent sees held-out words during evaluation. Importantly, no data apart from the dictionary can be used for the problem, which also means no pretrained embeddings or language models.

Environment: Each episode, the agent guesses letters until the word is revealed or maximum incorrect guesses are reached. Simple intuitive reward scheme to start:

• +1 correct guess, -1 incorrect/repeated guess
• +5 win, -5 loss

Model: At each step, the agent observes (1) the partially revealed word with blanks, and (2) which letters remain available to guess. It outputs a probability distribution over available letters; unavailable letters are zeroed out, and an action is stochastically chosen. For a start, a simple 2-layer CNN is used to capture local n-gram features.

Practical Tips for training RL&DL systems

1. Step-Through Debugging Is Your Best Friend

The first critical lesson came from using VSCode’s debug mode to step through each tensor operation. This revealed broadcasting mistakes that weren’t raising errors but were silently breaking training. These bugs would have been nearly impossible to catch from loss curves alone.

When training stalls or behaves oddly, step through your calculations line-by-line before touching hyperparameters.

2. Comprehensive Logging Infrastructure Is Non-Negotiable

Early on, I was aimlessly trying different architectures and parameters on a trial-and-error basis, watching only the win rate. No amount of tweaking improved performance beyond 15%.

Setting up TensorBoard properly changed everything. Beyond just loss curves and win rates, I started tracking:

• Advantage distributions
• Policy entropy over time
• KL divergence between updates
• Clip fractions
• Return vs value prediction histograms

This revealed a severe skew in my advantage distribution. Debugging led me to a bug in my truncation logic and normalization, which caused the last few samples to dominate updates while earlier steps received negative weight. Fixing this, combined with moving away from purely sparse rewards, produced the first real improvement.

Dense reward structure (fraction of unique letters revealed) provided a continuous learning signal, jumping performance from 15% to 20%.

Here’s what the advantage distribution looked like before and after the fix:

Before (Left) : Heavy negative skew, with late-game steps dominating updates, After (Right) : Centered distribution with reasonable variance

3. Entropy Penalties Need Context-Aware Scaling

Despite improvement, something still looked wrong. By rendering games, I noticed something odd: the agent made reasonable guesses early but then random, bizarre choices near the end of games.

This led me to realize that entropy penalties should scale with the remaining action space. As the number of valid letters shrank, exploration pressure increased instead of collapsing. I scaled the entropy loss by the fraction of remaining letters.

Win rate jumped to 30% with this single change.

4. Warm-Up Routines Stabilize Training, Not Performance Ceilings

I introduced a supervised warm-up phase where the model learned to predict a single masked letter in a word. While this significantly stabilized early training and reduced variance, gains saturated at the same 30% plateau.

This suggested the bottleneck had shifted from training dynamics to representation or information efficiency.

5. Value Clipping Matters More Than I Expected

I attempted to boost performance by switching from CNNs to a Transformer encoder. Despite having fewer parameters, training became highly unstable.

Adding KL divergence tracking to my debugging dashboard revealed consistently high values during updates — higher than the recommended 0.03. Two key insights emerged:

Transformer sensitivity: Transformers are far more sensitive to learning rates than CNNs due to their softmax attention mechanisms. The default settings (Adam with lr=3e-4) worked best. Critically, higher learning rates to “compensate” for slow learning consistently made things worse, not better.

Value clipping is essential: I added value function clipping (similar to policy clipping in PPO). While value loss should naturally increase as the agent explores new states, large spikes indicate instability rather than healthy exploration. Value clipping prevented these explosions, which were creating an illusion of policy improvement while actually destabilizing training. This parallels gradient clipping in deep learning: both prevent large, destabilizing updates that look like rapid progress but actually break training.

However, even with stable training and proper hyperparameters, performance remained at 30%. The plateau was real.

Key Takeaways

Most gains came from fixing training dynamics, not from adding representational power.

PPO Diagnostics That Actually Matter

The following metrics consistently predicted failure before win rate dropped:

• KL divergence spikes → imminent collapse
• Clip fraction ≈ 0 → policy not updating
• Early entropy collapse → overconfidence from sparse signal
• Exploding value loss → illusion of policy improvement

Without tracking these, learning curves are misleading.

What Did Not Work

To save you time, here’s what I tried that failed:

• Increasing model size to “push through” plateaus
• Aggressive entropy bonuses without action-space awareness
• Assuming CNN → Transformer would automatically improve reasoning

What’s Next

Without language priors, there’s a hard ceiling. The agent can’t infer letters with no visible pattern in the revealed word — this is an information limit, not a modeling one. A human with no knowledge of English would hit the same wall.

Two promising directions remain unexplored:

1. Explicit belief-state modeling through auxiliary losses that predict letter distributions rather than just actions
2. Prior-based initialization using letter frequency distributions from the corpus

Both target information efficiency rather than raw model capacity, which seems to be where the real bottleneck lies.

Closing Thoughts

This project reinforced a lesson that applies broadly to reinforcement learning: most performance gains come from understanding failure modes, not from adding layers.

Hangman wasn’t solved here, but the process revealed where RL systems quietly fail and how PPO breaks in practice. The simplicity of the game made these failures visible — complex environments often hide the same issues under confounding variables.

If nothing else, this journey taught me to distrust smooth learning curves and to debug rewards, advantages, and training dynamics before touching architectures. Deep learning is less about building the fanciest model and more about building the right debugging infrastructure to understand why your current model is failing.

Deep learning is less about building the fanciest model and more about building the right debugging infrastructure to understand why your current model is failing.