I Spent One Week Building an AI Agent That Fixes GitHub Issues. Here’s Everything I Learned.

A week ago I decided to build something I’d been curious about for a while: an AI agent that doesn’t just chat about code, but actually finds a real bug, understands the codebase well enough to fix it, writes a test proving the fix works, and only then asks a human to approve it before anything gets submitted.

I called it Bounty. This is the story of building it, what actually worked, and the mistakes that taught me more than any tutorial could.

*(Click here to read for free -> )*

Why I Built This

Most AI coding demos show a model writing a function from scratch in an empty file. That’s not what real engineering looks like. Real engineering is opening someone else’s codebase, understanding how pieces connect, finding the one place a bug actually lives, and making a change small enough that it doesn’t break anything else.

I wanted to see if I could build a system that does that honestly, end to end, with a human still in control of what actually gets shipped.

How a Human Would Do This

Before writing any code, I thought through how I’d actually solve a GitHub issue myself.

I’d browse the open issues and skip most of them immediately. I’d read the ones that looked doable and ask myself if I really understood what was broken. I’d clone the repo, poke around until I found where the problem lived, understand that part of the code, figure out what needed to change, write the fix, write a test proving it worked, run everything to make sure nothing else broke, and only then open a pull request.

That list became the entire architecture. Every part of Bounty maps to one of those steps.

Deciding What’s Actually Solvable

The first real piece of the system reads an issue and decides whether it’s something an agent can genuinely fix. This turned out to be the most important decision in the whole project, and I built it to be deliberately pessimistic.

Saying no to an issue costs almost nothing. Saying yes to something that can’t actually be fixed wastes a repo clone, an embedding run, several model calls, and sandbox execution time for nothing. So the system is biased hard toward rejecting anything vague, anything without clear reproduction steps, anything that’s really a feature request or a design discussion disguised as a bug report.

In practice this means it rejects far more issues than it accepts, and that’s exactly the point.

Understanding a Codebase Without Reading All of It

A real repository can have hundreds of files. A model can only hold a small fraction of that in context at once. The naive approach is to split every file into fixed size chunks and search over them, but that destroys meaning. You end up with a chunk that ends halfway through a function and starts halfway through an unrelated one.

I used tree sitter instead, a tool that actually parses code into a real syntax tree, so I could split files along function and class boundaries. Each chunk is a complete, meaningful unit instead of an arbitrary slice of text.

That solved half the problem. The other half was trickier. Semantic search finds files that talk about the right topic, but sometimes the actual bug lives in a file that never mentions the topic at all because it’s two imports away from the file that does. So I built an import graph, a map of which file imports which, and after search finds candidates, the system walks one hop outward along that graph to catch what search alone would miss.

Getting the import graph right took longer than I expected. Python’s relative imports, the ones that start with a dot, are structured completely differently in a parse tree than absolute imports, and my first attempt silently failed to resolve any of them. I only caught it by printing out exactly what the parser found and noticing every relative import was being treated as garbage text.

Planning Before Coding

Once the system knows which files matter, it doesn’t jump straight to writing code. It writes a plan first, in plain English, explaining what needs to change and why, with no code at all.

I split this into two separate steps on purpose. A bad plan produces bad code, and it’s much cheaper to catch a bad plan in a few sentences than to catch it after a hundred lines of generated code already exist.

The Mistake That Taught Me the Most

My first version had the coding agent generate an entire modified file and hand back the complete content. It worked fine on small files. Then I tested it against a larger file in a real repository and the model hit its own output length limit halfway through generating the response. The structured output parser couldn’t even read the result, since it was cut off mid file.

I rebuilt this part to have the model propose small, surgical changes instead: an exact block of existing code to find, and the exact replacement for it, rather than reproducing the whole file. This was smaller, safer, and much less likely to fail, and it’s honestly closer to how real patch tools work anyway.

The Check That Actually Matters

Here’s the part I’m most proud of.

Whenever the system writes a fix, it also writes a test. But a test is worthless if it passes whether or not the fix is actually applied. So every test has to pass two checks before it counts as real:

1. Apply the fix and run the test. It should pass.
2. Revert the fix and run the exact same test. It should fail.

If the test passes both times, it gets thrown out, because it isn’t testing anything at all. This one check is what separates a fix that actually works from one that just looks plausible, and it’s a step most AI coding tools skip entirely.

Running Code You Don’t Fully Trust

The agent is writing code on its own, and I couldn’t just run that directly on my machine. It needed to run somewhere isolated, somewhere a mistake or an infinite loop couldn’t do any real damage.

I originally planned to use Docker for this. I lost most of a day fighting a Windows and WSL2 issue that turned out, after hours of digging through log files, to just be a nearly full hard drive silently breaking every install attempt. Once I understood that, I made the call to switch to a cloud based sandbox instead, which turned out to be a perfectly legitimate choice on its own, not just a workaround. Real production coding agents increasingly run code this way rather than managing their own local containers.

Where Things Stand Right Now

As of today, everything runs end to end. The plan gets written, the code gets generated as small precise changes, the test gets written and validated in the sandbox, and if anything fails, the actual error gets fed back into the next attempt instead of just blindly retrying.

The most recent run passed every check. The test failed on the original broken code exactly as it should have, passed once the fix was applied, and the rest of the existing test suite still passed afterward.

I haven’t opened a real pull request yet. Before I do, a human always reviews everything, and that’s not a limitation I plan to remove. Open source maintainers are already dealing with a wave of low effort AI generated pull requests, and the last thing I want to do is add to that pile. If this ever gets submitted anywhere, it will say clearly that it was AI assisted and human reviewed, and I’d rather get two genuinely good pull requests merged than fifty rejected ones.

What I’d Tell Someone Starting Something Like This

Build the smallest piece first and prove it works before adding the next layer on top. Every one of the four bugs I hit this week was something I could only see once I had real output to look at, not something I could have predicted by thinking harder in advance.

And write down your failures while they’re still fresh. The moment I understood why the WSL2 issue was actually happening, or why my test was silently passing when it shouldn’t have, taught me more than anything that worked on the first try.

I’ll be sharing the first real pull request soon, along with the real numbers once I’ve run this against more issues. If you’re building something similar or you’ve hit any of the same walls, I’d genuinely like to hear about it.