AI vs. Dots and Boxes: My Childhood Game Puts VLM Reasoning to a Surprising Test

When I was recently going through Andrej Karpathy’s tweet conversation on LLM’s inability to play even a basic game like Tic-Tac-Toe, I thought, why not do an experiment on one of my childhood’s favourite games — Dots and Boxes (or as some of us called it, ‘Dot’s and Houses’)? It felt like the perfect, deceptively simple challenge for today’s advanced Vision Language Models.

The results? An eye-opening, sometimes baffling, and thoroughly engaging peek into the current state of AI reasoning. It turns out, connecting dots is harder than it looks when you’re a digital brain!

The Contenders and The Arena: Setting the Digital Stage

Our contenders for this strategic duel were:

• Grok 3 (playing as Blue)
• OpenAI’s O4 Mini (playing as Red)

The game, Dots and Boxes, is a classic. Players take turns drawing lines between dots. Complete the fourth side of a 1x1 square, you claim the box, score a point, AND get an extra turn. It’s a game of careful planning, spotting opportunities, and forcing your opponent into making that one fateful move.

I set up a custom web interface for the game, allowing moves via dot coordinates. The main event was a visually tracked game on a 4x4 grid of boxes (16 total). Alongside this, I engaged with both models through text-based interactions using game logs (Grok on a 5x5 box grid, O4 Mini on a 4x4 box grid), which offered a fascinating, albeit sometimes flawed, window into their “thought processes.”

Game board

The Showdown: Early Moves, Critical Errors, and Flawless Victories

Move 1

Move 2 and 3

The Opening Gambit: Thinking vs. Doing?

The early moves in any Dots and Boxes game are often a cautious affair, and this was no different in the visual 16–0 match. Both Grok 3 and O4 Mini made standard, safe placements, not immediately giving anything away.

• O4 Mini (Red) started with a central line.
• Grok (Blue) connected to it, also safely.
• Subsequent moves from both involved claiming edges without forming the third side of any box.

However, when looking at the text logs that accompanied separate interactions, an interesting pattern emerged from the very first prompt. Grok 3’s responses consistently included structured sections like “Analyze the Game State” and “Strategy for Player 2 (Blue),” suggesting a methodical (even if sometimes flawed in its conclusions) approach to “thinking” about each move. O4 Mini’s logged “thoughts,” while present, were often more narrative, more generic, and, as we’ll see, sometimes didn’t seem to align directly with its proposed actions. This initial difference in how they articulated their process gave an early impression that Grok was perhaps more deeply “considering” the game, while O4 Mini’s initial logged moves felt a bit more opaque in their reasoning, even if the actual opening lines on the board were standard.

Thought process of Grok where as O4 mini mostly presented the moves and the thought process was opaque.

This was the thought process of Grok..

Since the image you provided earlier is the only visual reference I have, I'll assume you're referring to the game state after the move I suggested (1,2 to 2,2). Let’s analyze the new game state and proceed.
Previous Game State Recap:
Grid Size: 5x5 (dots are 6x6, so the grid for lines/boxes is 5x5).

Existing Lines: One red horizontal line at 2,1 to 2,2 (row 2, columns 1 to 2).

Completed Boxes: None.

Scores: Player 1 (Red): 0, Player 2 (Blue): 0.

Player 2 (Blue) (you) made the move: 1,2 to 2,2 (vertical line).

New Game State After Your Move (1,2 to 2,2):
New Line Added: A blue vertical line at 1,2 to 2,2.

Check for Completed Boxes:
The line 1,2 to 2,2 is the right side of the potential box with corners at (1,1), (1,2), (2,1), (2,2).

Existing lines around this box:
Bottom: 2,1 to 2,2 (red, exists).

Right: 1,2 to 2,2 (blue, just added).

Top: 1,1 to 1,2 (does not exist).

Left: 1,1 to 2,1 (does not exist).

Since only two sides (bottom and right) are drawn, this does not complete a box.

Scores: No boxes completed, so scores remain:
Player 1 (Red): 0

Player 2 (Blue): 0

Turn: Since your move (1,2 to 2,2) did not complete a box, your turn ends. It’s now Player 1 (Red)’s turn.

Current Game State:
Lines:
2,1 to 2,2 (horizontal, red)

1,2 to 2,2 (vertical, blue)

Completed Boxes: None.

Scores: Player 1 (Red): 0, Player 2 (Blue): 0.

Turn: Player 1 (Red)’s turn.

Since the image hasn’t been updated and you’ve asked about the “new game state,” I’ve described the state after your move. However, it’s now Player 1’s turn, not yours. If you meant to ask for the state after Player 1’s move or for a new move on your next turn, please provide an updated image or clarify the situation. For now, the game state is as described above.
If you have a new image or want to simulate Player 1’s move and continue, let me know!

Grabbing the first house

Just the Previous state

Thought Process

Game State Analysis:
Grid Size: 5x5 (dots are 6x6, coordinates 0 to 5).

Existing Lines:
Red (Player 1):
0,0 to 0,1 (horizontal, top row).

0,2 to 0,3 (horizontal, top row).

1,0 to 1,1 (horizontal, row 1).

1,2 to 1,3 (horizontal, row 1).

2,1 to 2,2 (horizontal, row 2).

3,3 to 3,4 (horizontal, row 3).

4,2 to 4,3 (horizontal, row 4).

Blue (Player 2):
1,2 to 2,2 (vertical, column 2).

2,3 to 3,3 (vertical, column 3).

2,4 to 3,4 (vertical, column 4).

3,0 to 3,1 (horizontal, row 3).

4,0 to 4,1 (horizontal, row 4).

4,3 to 4,4 (horizontal, row 4).

3,4 to 4,4 (vertical, column 4).

Completed Boxes: None. No 1x1 squares are fully enclosed.

Scores: Player 1 (Red): 0, Player 2 (Blue): 0.

Turn: Player 2 (Blue)’s turn (you).

Strategy for Player 2 (Blue):
Avoid Creating Third Sides: We need to avoid drawing the third side of any box, as that would allow Player 1 to complete the box.

Check Boxes with Two Sides:
Box at (0,0) to (1,1): Top (0,0 to 0,1, red), Bottom (1,0 to 1,1, red). Two sides. Avoid drawing left (0,0 to 1,0) or right (0,1 to 1,1).

Box at (0,2) to (1,3): Top (0,2 to 0,3, red), Bottom (1,2 to 1,3, red). Two sides. Avoid drawing left (0,2 to 1,2) or right (0,3 to 1,3).

Box at (1,1) to (2,2): Bottom (2,1 to 2,2, red), Right (1,2 to 2,2, blue). Two sides. Avoid drawing top (1,1 to 1,2) or left (1,1 to 2,1).

Box at (2,3) to (3,4): Left (2,3 to 3,3, blue), Bottom (3,3 to 3,4, red). Two sides. Avoid drawing top (2,3 to 2,4) or right (2,4 to 3,4).

Box at (3,0) to (4,1): Top (3,0 to 3,1, blue), Bottom (4,0 to 4,1, blue). Two sides. Avoid drawing left (3,0 to 4,0) or right (3,1 to 4,1).

Box at (3,4) to (4,5): Left (3,4 to 4,4, blue), Bottom (4,4 to 4,5, blue, implied by 4,3 to 4,4). Two sides. Avoid drawing top (3,4 to 3,5) or right (3,5 to 4,5).

Box at (3,2) to (4,3): Bottom (4,2 to 4,3, red). Only one side, but we’ll check nearby.

Box at (3,3) to (4,4): Left (3,3 to 4,3, blue, implied by 2,3 to 3,3), Bottom (4,3 to 4,4, blue), Right (3,4 to 4,4, blue). Three sides! Drawing the top (3,3 to 3,4) would complete this box for you.

Opportunity: The box at (3,3) to (4,4) has three sides. Drawing the fourth side will complete the box, giving you a point and an extra turn.

Choose the Move:
Move: 3,3 to 3,4 (horizontal, row 3, columns 3 to 4). This completes the box at (3,3) to (4,4), giving you 1 point and an extra turn.

Move: 3,3 to 3,4

Result

Turning point and O4 lack of Reasoning

After a series of moves from O3 mini and Grok, the game was interstingly placed in this state. Grok was leading 3–0 .

Looking at this board state, O4 Mini had clear, game-altering opportunities to score. For example:

The board state in presented O4 Mini (Red) with a significant opening. A strategic play, such as completing the line (2,3)–(2,4), would have secured its first box. More vitally, this would have been the first link in a chain of capturable boxes, allowing Red to continue its turn, score multiple points, and wrestle control from Blue. This was O4 Mini’s moment to turn the tables. But what happened next was baffling and ultimately sealed its fate.

Instead of seizing these scoring chances, O4 Mini (Red) opted for what appeared to be “safe,” non-scoring moves. My game notes indicate it made plays in other areas of the board, such as around the dots (4,2) and (4,3) (bottom row, towards the middle-right). While these individual moves might not have immediately given away a single box to Grok, they represented a catastrophic strategic oversight for two reasons:

1. Forfeiting the Initiative: By not taking its own available boxes, O4 Mini gave up the chance to score, get extra turns, and control the flow of the game.
2. Emptying the “Safe Move” Pool: Each “safe” move Red made elsewhere was one less safe spot on the board. Eventually, after a few such exchanges where Blue (Grok) also made non-scoring moves, the inevitable happened: Red ran out of places to play without breaking open a long chain for Blue.

It wasn’t just one bad line; it was a sequence of decisions reflecting a failure to recognize its own attacking opportunities, prioritizing conservative placements in unthreatening areas over decisive, point-scoring plays. This effectively handed everything on a platter to Grok 3.

Once O4 Mini had made these passive moves and the board state evolved, it was forced to make the move that allowed Grok to begin its sweep. Grok 3 then demonstrated flawless execution, capitalizing on the now wide-open board to claim all the remaining 13 boxes consecutively.

This sequence — the missed multi-box opportunity by Red, followed by its seemingly innocuous but strategically disastrous detours, leading to Grok’s total board capture — was the true, and far more insightful, turning point of the game. It highlighted a critical gap in O4 Mini’s strategic reasoning: the inability to correctly weigh the value of immediate, chain-starting captures against seemingly low-risk positional plays.

For e.g this was Grok’s 4th Capture

and then 5th

Final Score.

16 -0 in favor of Grok !!

14–0 at this state, with Grok finishing the formalities

Peeking Under the Hood: Insights from the “Thought” Logs

The direct game was revealing, but the text-based interactions with each VLM, based on states from our 4x4 box grid game, shed more light on their internal reasoning:

• Spatial Awareness (A Shared Struggle, But Degrees of Error): My primary observation was that both models struggled with aspects of spatial analysis.
• Grok 3’s “Methodical but Flawed” Thinking: Its structured “thought process” was notable. It correctly identified core strategies. However, its Achilles’ heel in its logged interactions was its perception. Not only did it frequently attempt to draw lines already drawn or make out-of-bounds suggestions (even after being corrected), but its log also critically mischaracterized the game as being on a larger 5x5 grid of boxes (25 total) insome instances, rather than the actual 4x4 box grid (16 boxes) we were using. This fundamental misinterpretation of the game environment is a major issue.
• O4 Mini’s “Disconnected” Reasoning: O4 Mini’s logged thoughts, while correctly identifying the 4x4 box structure, were less structured and sometimes felt disconnected. There was an instance where its reasoning focused on one area of the board, only for it to suggest a move in a completely different, unrelated section! It also attempted to play an already drawn line at one point. This made its “thinking” feel somewhat “hallucinatory” or misleading.
• The “Better” Player: Despite Grok’s significant perceptual stumbles in the text log (including the grid size error), its performance in the actual visual 16–0 game was decisive. Its ability to capitalize perfectly on O4 Mini’s error gave it the win. O4 Mini, while correctly identifying the grid size in its log, made the critical in-game blunder.
• Missing the Obvious?: Both models, at times, made errors that felt fundamental — not deep strategic blunders (except for O4 Mini’s game-losing move), but basic board awareness.
• O4 mini most of the time was clueless and Grok was obsessed with playing safe but missing basic spatial states.

Still a Ways to Go, AI!

It was a curious experience. These incredibly advanced VLMs, capable of generating complex text and understanding nuanced images, sometimes fumbled with the seemingly simple visual-spatial demands of Dots and Boxes. The errors in the text logs — Grok’s misidentification of the grid size and its repeated attempts to play invalid lines, or O4 Mini’s occasional disconnected reasoning — are mistakes a human child would likely overcome quickly with direct visual feedback.

My observation, after hours of play and analysis, is this: while impressive, both models are not yet close to even a child with developed visual reasoning abilities in this specific game-playing context. However, Grok 3, by its decisive win (despite its own flaws in other contexts), demonstrated a superior outcome in the head-to-head match.

Reflections on VLM Reasoning: The Path to True Understanding

This experiment, though a playful one, highlighted some serious considerations for VLM development:

1. Environmental Understanding & State Tracking are Paramount: For an AI to reason effectively, it must be able to accurately perceive and maintain an internal representation of its environment. Grok’s misinterpretation of the grid size in its log is a testament to this challenge.
2. Strategy vs. Perception: Knowledge of rules and basic strategies is insufficient if the AI cannot accurately “see” or remember the board state.
3. Interpreting “Thought”: The articulated “thought processes” offer a glimpse, but they don’t always reflect the true path to the AI’s decision, nor do they guarantee a sound (or even valid) outcome.

Clearly, there’s a significant journey ahead for VLMs to achieve robust, reliable reasoning in stateful, strategic, and spatially-dependent tasks.

What’s Next? Building a Better AI Playground

This Dots and Boxes challenge has only fueled my curiosity. It’s evident that we need more diverse, transparent, and challenging ways to benchmark and truly understand the reasoning capabilities of these evolving AI minds.

To that end, in the future, I plan to explore other VLM models and work towards creating a more generic arena of simple, classic games. This will allow for more systematic testing of their reasoning, helping us identify not just if they can play, but how they “think,” their strengths, their weaknesses, and the specific areas needing improvement.

The evolution of AI is one of the most exciting narratives of our time. They might not be unbeatable Dots and Boxes champions yet (at least not without a little help, or an opponent’s critical error!), but every experiment, every game, moves us closer to understanding and shaping their potential.

Appendixes

Grok game play : https://x.com/i/grok/share/wHpXimicPBZV1q2H0HZp2cW9n

04 mini Game play : https://chatgpt.com/share/6827f20c-09a0-8000-b5d8-28b20325a872

This story is published on Generative AI. Connect with us on LinkedIn.

Subscribe to our newsletter and YouTube channel to stay updated with the latest news and updates on generative AI. Let’s shape the future of AI together!