I Spent Months Building Tests For My AI Agent. The Bug Was In The First Hour Of Actually Using It.

The embarrassing discovery that changed how I think about testing forever.

— -

I had over 1,000 unit tests.

All green.

Every commit.

I had stress harnesses firing 1,700 prompts in loops.

I had SWE-bench running at 70% file-match.

I had a 52-case behavioral suite.

I had an eval harness for end-to-end scenarios.

And the biggest bug in my codebase — a file-edit loop that burned 5–9 minutes per occurrence — had been driving users crazy for months.

None of my tests caught it.

Then I opened my tool in a real terminal.

Used it for one hour.

The way a normal user would.

I found four distinct bug classes in the first two minutes.

— -

## What I Was Doing Instead Of Using My Product

Someone asked me directly:

”What have you been doing this whole time to test this?”

I had to admit the truth.

I was running stress harnesses.

Gauntlet runs.

SWE-bench batch evals.

Classifier-dispatcher loops.

Autonomous review crons.

I was building automation around the product.

I was not opening the product and using it.

— -

## What My Tests Were Actually Measuring

Let me be specific about what failed:

Pytest unit tests (1,000+): Green on every commit. Caught zero of the bugs below.

Pexpect scripts: Caught some bugs early. Then became pexpect-debugging-pexpect.

Stress harness (1,700 prompts): Measured “messages arrived.” Not “correct code produced.” Ran at 1.5% skip rate while a 15-minute interactive session surfaced 11 NameErrors.

SWE-bench (70%): Looked good. Real users still hit bugs because SWE-bench is one-shot patches. My tool is interactive, multi-turn, full of edge cases benchmarks never trigger.

52-case behavioral suite (30–40%): Failures looked like model-capability problems. Many were actually harness problems hiding under the model’s struggle. Couldn’t tell them apart.

Each layer was a sincere attempt to find bugs.

They produced numbers.

They didn’t surface the 9-minute loop.

— -

## The Pivot That Took 30 Seconds

tmux has send-keys and capture-pane:

tmux new-session -d -s dd ‘cd /tmp/repro && drydock’

tmux send-keys -t dd “fix the bug in broken.py” Enter

sleep 30

tmux capture-pane -t dd -p

That’s it.

Open a real TUI in a real terminal.

Type a real prompt.

Read the output.

Same as the user does.

Not pexpect.

Not a custom harness.

Not a programmatic-mode flag.

The actual tool with the actual widgets, actual renderer, actual middleware, actual autocomplete.

All of it live.

— -

## What I Found In The First Two Minutes

I seeded a 12-line file with two intentional bugs.

Sent the prompt.

Watched the TUI.

Four distinct bug classes my test suite had never seen.

The headliner:

write_file with an empty path returned a result object the dispatcher treated as success.

The TUI rendered ✓ Created (missing).

A literal green checkmark for a tool call that didn’t write a file.

The model saw success.

Emitted the same empty-path call again.

Looped for nine minutes.

I’d written a “validation-scrub” feature six weeks earlier specifically to break this loop.

It only fired on status == “failure”.

The empty-path branch returned a success-typed result.

The scrub never ran.

The bug had quietly evaded my fix for six weeks.

While my 28 unit tests for the scrub helper passed every commit.

— -

## What Was Already In The Logs

A few hours in, I had a clean baseline.

Then the user pasted their TUI excerpt with errors I’d never seen.

Different errors. Different sessions.

So I did the obvious thing I should have done weeks earlier:

grep “Traceback” ~/.drydock/logs/drydock.log

Seven more unique crashes.

Every one already in the log.

Every one in production.

• Pydantic schema mismatch crashing mid-tool-call

• Widget lifecycle race

• Python 3.14 subprocess return-value change

• OSError from 500-character filenames

Every one had been hitting users for weeks.

Every one was visible if anyone had looked.

I hadn’t looked.

I was too busy maintaining test layers.

— -

## The Scale That Followed

Once tmux-driven TUI sessions became the methodology, I parallelized.

Four LLM hosts. Weighted round-robin.

A cron entry keeps four parallel workers alive at any time.

Each drives a real coding task through a real TUI.

~80 scenarios per hour.

~1,900 per day.

Every code path the user exercises gets exercised by the pool.

When something fails, a curl to a Telegram bot pings immediately.

Per-scenario forensics land on disk.

— -

## The Meta-Lesson

Stress harnesses measure liveness.

Eval harnesses measure scores.

Unit tests measure helper functions in isolation.

None of them measure what your user is actually doing.

Opening the tool.

Typing a prompt.

Watching the output.

Reacting.

Automation feels productive.

You can show a green CI badge.

You can put a benchmark number in your README.

You can write a PRD and run 50 benchmark passes and produce a postmortem doc.

None of it finds the bug where the TUI renders a green checkmark for a tool call that didn’t do anything.

— -

## Where The Testing Pyramid Still Applies

The whole testing pyramid still has its place.

Pyright catches NameErrors before they ship.

Pytest locks in regressions for every fix you actually ship.

The harness pool surfaces new failure modes at scale.

But the foundation — the layer that turns “code compiles, tests pass” into “this product is actually usable” — is hands-on driving.

Same input device.

Same output stream.

Same lifecycle as the user.

— -

## The Permission I’m Giving You

If you have thousands of tests and users are still hitting bugs:

Your tests aren’t testing what users experience.

It’s okay to admit that.

I had to.

The first hour of actually using my product — after weeks of avoiding it — paid for every other layer of testing I’d built around it.

It also, finally, made the bug stop.

— -

If you build a developer tool and you don’t have a hands-on testing loop where you sit and operate it on real tasks:

Your tests are lying to you.

That’s the whole post.

AI Disclosure: To enhance the writing process and streamline technical explanations, generative AI was utilized to assist with drafting, code optimization, or formatting parts of this article. All content has been reviewed, and refined by the author to ensure accuracy and quality.