The AI PR reviewer my team didn’t mute — ReviewIQ

I build tools when there’s a real reason to. Not because the tech is cool, but because something in the day-to-day is bleeding hours and nobody’s calling it out.

PR reviews were that thing for my team.

The honest math

We follow a 2-review rule on every PR — one peer, one senior. On paper it’s a healthy practice. In reality, here’s what it looked like:

• An average PR takes around an hour to review properly.
• A “small” PR is rarely small. You’ll happily open one and find 10–15 files changed, mixed concerns, a config tweak that snuck in, a refactor nobody asked for.
• Reviews are never one-shot. There’s always a second round, sometimes a third. Each round burns context-switch tax for both the reviewer and the author.
• Every inline comment is manual work too. A good comment isn’t a one-liner like “fix this” — it has to be understandable on its own: what’s wrong, why it matters, and what to do about it. Writing 20 of those across a 15-file PR is its own kind of fatigue, and the quality of comments quietly drops as the reviewer gets tired.
• And even after doing all of this, humans still miss things. A subtle missing idempotency check, a regex that breaks on a corner case, a logging line that leaks PII — these slip through not because the reviewer wasn’t good, but because by file 12 of 15, attention is finite. The bugs that hurt most in production are usually the ones that looked fine on review.
• Multiply that across teams and a lot of engineering time was sitting inside the GitHub PR view instead of building things.

The quality and intent quietly degrade PR by PR. The volume is the real problem underneath.

Why I didn’t just bolt on an AI bot

There are plenty of “AI PR reviewers” floating around. Most of them do the same thing: read the diff, leave generic comments about naming and null checks. And even when the suggestions are decent, the tool just reviews — it doesn’t actually post anything inline on the PR. You’re left copy-pasting feedback line by line into GitHub yourself, which kills the whole point of automating it. After two weeks the team mutes the bot — or stops opening the tab altogether — and we’re back to square one.

I wanted something that fits how we actually review and fix:

• It needs domain awareness. A review on a payment retry handler should know about idempotency and exponential backoff, not just suggest a better variable name.
• It needs state. A real review has a Round 1, a Round 2, and someone tracking what got fixed and what didn’t. A stateless bot starts from scratch every time and that’s exhausting.
• It needs to act, not just talk. If something’s a clear fix, just fix it, run the tests, push and done.

That’s what ReviewIQ became.

What it does, in plain terms

ReviewIQ is a CLI + Claude Code agent that reviews PRs the way a thoughtful senior would, and remembers what it said last time.

There are four commands and they map to how a review actually unfolds:

• reviewiq-pr — opens the PR, picks the right skill modules based on what’s in the diff, posts inline suggestion blocks on the exact lines, drops a summary report. (Modes: — interactive and — full)
• reviewiq-recheck — runs after the author pushes new commits. It auto-resolves findings that got fixed, keeps the ones still open and flags anything new. No re-litigating closed comments.
• reviewiq-resolve — for all findings, it checks out the branch, applies the fixes, runs tests, commits, pushes and approves.
• reviewiq-test — detects the test framework and runs targeted tests for the changed files.

Under the hood — how it actually works

This is the part most “AI reviewer” tools get wrong, so it’s worth a closer look.

The whole design rests on one decision: state lives on the PR itself, not in the tool. That single choice is what makes the full loop possible — review → recheck → resolve → test → approve — because both the author and the reviewer are reading and writing to the same shared state, right there on the PR, from whichever side of the workflow they're on.

In practice, that means:

• A reviewer runs reviewiq-pr and posts findings inline. The state is now on the PR.
• The author opens the same PR, runs reviewiq-resolve, and the agent picks up the exact same findings the reviewer just posted — applies fixes, runs tests, commits, pushes.
• Either side can run reviewiq-recheck to see what got fixed, what's still open, and what regressed.
• The reviewer comes back, sees a clean state, and approves — or runs one more recheck round if they want to verify.

No DM-ing the bot, no “which version of the findings are you looking at?”, no separate dashboard. The PR is the source of truth, and both sides operate on it.

State lives on the PR itself

There’s no database, no local file, no separate dashboard. Every round of review writes a hidden comment to the PR with two parts inside it:

1. A collapsed <details> summary table — id / severity / status / file:line / title — that's the fast path. Most decisions during a recheck only need this.
2. A fenced JSON block with the full state — findings, status history, suggested fixes, the SHA we last reviewed against. Human-readable, diffable in GitHub’s UI, no base64. The whole thing is bracketed by HTML markers (<!-- REVIEWIQ_STATE_COMMENT --> etc.) so the agent can find and parse it deterministically.

Why this design: state goes wherever the PR goes. Switch machines, run from CI, hand it to a teammate — the state is right there. Plain JSON also means I can read it with my eyes when something looks off, which I couldn’t do with base64.

Every round is a new comment, never an overwrite

Each invocation creates a new state comment with an incremented round marker. Nothing gets overwritten. The PR ends up with a clean timeline:

ReviewIQ State (Round 1) — 10 open, 0 resolved
ReviewIQ State (Round 2) — 3 open, 7 resolved
ReviewIQ State (Round 3) — 0 open, 10 resolved

The agent always picks the highest round when loading, but the older rounds are preserved as audit trail — you can scroll back and see what was originally flagged, what flipped, when and why.

This also handles a sneaky problem: if CI auto-rechecks at the same time a human runs recheck locally, you can get phantom rounds. Round numbering is max(all existing rounds) + 1, so gaps are tolerated and nothing collides.

Finding statuses (and why wontfix matters)

Every finding has one of these statuses:

• open — found, not fixed yet
• resolved — fix confirmed in code
• wontfix — developer intentionally skipped it
• retracted — the finding was wrong, agent walks it back
• partially_fixed — addressed but not fully

The interesting one is wontfix. AI reviewers love to be persistent — flag the same thing every round, forever. That's the fastest way to lose trust. Both recheck and resolve start with a skip prompt:

Mark any as wontfix? Type `W <N>` or `W 1,3,5`, or Enter to keep all open:

Maybe the suggestion doesn’t fit the codebase context. Maybe it’s a stylistic preference you don’t share. Maybe it’s correct in theory but irrelevant for this particular hotfix. You mark it W 3,5 and the agent records who skipped it, when, and why in the finding's status_history. From that point on, those findings are carried forward in every future round with zero re-reads — the agent will never lecture you about them again.

That single design choice is what made the team actually keep the tool on instead of muting it.

Inline comments, the right way

GitHub has two comment APIs and most tools use the wrong one. ReviewIQ posts via the Reviews API (/pulls/{N}/reviews) which batches every inline comment into a single review event — one notification, one timeline entry, not 12 separate emails to the reviewer.

Before posting, every finding’s (file, line) is validated against the actual diff hunks. GitHub will reject any inline comment that doesn't sit inside a changed hunk with a 422 Line could not be resolved. So findings that fall outside the diff get demoted — they still appear in the markdown summary report at the bottom, just not as inline anchors. The developer still sees them, the API call still succeeds.

Each inline comment carries an actual suggestion block, meaning the developer can hit "Commit suggestion" in the GitHub UI and apply the fix in one click. No copy-paste.

Recheck — read as little as possible

The whole point of recheck is to not re-do the work. So it’s surgical:

1. Pull the latest commits, diff against last_reviewed_sha from the previous state.
2. Get touched_files — files actually changed since last round.
3. For every finding from the last round, route by status × was-the-file-touched:

• wontfix or retracted → carry forward, no read.
• open or resolved in an untouched file → carry forward, no read.
• open in a touched file → read ~30 lines around the finding, decide: fixed → resolved, still broken → stays open, code moved → needs-review.
• resolved in a touched file → check for regression. If the previous fix got reverted, flip back to open with a Regressed in <sha> note.

4. Then scan the new commits for new issues — only the additions in each hunk, never whole files.

In practice, recheck on a 30-finding PR with 2 changed files might do 4–5 small reads instead of re-reviewing everything. That keeps tokens cheap and rounds fast.

There’s also a “foreign state” rule: if the previous state comment was written by a different actor (say a CI bot), its resolved statuses aren't trusted blindly. Each one gets re-verified against the actual code before being accepted. Otherwise you can game the bot by pre-stuffing a state comment, and that's not a system anyone should rely on.

Resolve — the part that actually edits code

reviewiq-resolve is the one that takes liberties, so it's deliberately careful. The flow:

1. Skip prompt first. Same W 1,3,5 mechanism. Anything you don't want auto-fixed, mark it now.
2. Read each remaining open finding’s file, locate the line, apply the suggested_fix from the state JSON. If the suggestion is ambiguous, the agent uses judgment — but it never invents a fix without grounding it in the file's current contents.
3. Run tests. Detect the framework (pytest, jest, go test, maven, rspec, etc.), run targeted tests for the changed files first, then a broader suite. If there are no tests, fall back to linter, then to syntax checks.
4. Commit and push. Without the push, the PR still has broken code and approval would be meaningless.
5. Save state, marking everything as resolved (or wontfix for what you skipped).
6. Auto-approve via gh pr review --approve — but only if at least one fix was applied and zero findings remain open. If everything got skipped via wontfix, no approval happens. If a fix was rejected mid-flow, the agent posts a partial-resolution report and bails out without approving.

That last guardrail matters. An agent that approves PRs it didn’t actually fix anything in is a liability. The auto-approve only fires when the agent has earned it.

The round-end summary

After every round, a markdown report is posted as a regular PR comment. It calls out status flips prominently — resolved → open (regressed), open → wontfix (skipped by author), resolved (foreign) → open (re-verified). Status changes are higher signal than vanilla open findings, so they get their own visual treatment.

You end up with a PR that reads like a chronological log: what was found, what was fixed, what was knowingly skipped, what regressed, and finally an APPROVED event. New reviewers joining late can scroll the timeline and reconstruct the entire conversation without asking anyone.

The “skills” idea is the actual differentiator

Generic AI review fails because it treats every diff like a CS101 assignment. ReviewIQ loads a different mix of expert skills based on what changed:

• A Django/Gin view in a payments file pulls in the payments, fintech, and security skills.
• A Kafka consumer pulls in messaging, stability, and scalability.
• An Airflow DAG pulls in the Airflow skill — schedules, XComs, idempotent tasks.
• A Terraform file or Dockerfile pulls in DevOps.

There are 6 always-on skills (security, scalability, stability, maintainability, performance, plus a “commandments” file of universal rules) and the rest are auto-detected from filenames and content. Token cost stays sane because we only load the relevant skill sections, not every word of every file.

For an India-context payments shop like ours, this mattered a lot. The skill set understands UPI, NACH, RBI guidelines, credit bureau patterns, IFSC validation — things a generic reviewer would never flag.

Where the adoption surprised me

I expected ReviewIQ to be used for peer reviews. That happened.

What I didn’t expect was how heavily the team started using it on their own PRs, before anyone else even saw them. The flow went from “raise PR → wait for reviewer” to “raise PR → run reviewiq-pr on yourself → run reviewiq-resolve to fix what you agree with → mark the rest as wontfix with a reason → then ping a human".

Engineers were essentially reviewing and resolving their own PRs end-to-end — catching the mechanical issues, applying the obvious fixes, pushing the cleanup commits, and only then requesting a human reviewer. By the time the senior engineer opens the PR, half the noise is already gone, the easy stuff is fixed, and the timeline shows what was caught and what was knowingly skipped.

The human review that follows is shorter and sharper — focused on architecture, intent, trade-offs, and the judgment calls that actually need a human in the loop. The stuff humans are genuinely good at.

That shift — AI catches and fixes the mechanical layer, humans focus on judgment — is the version of “AI in dev workflow” I actually wanted.

What I’d tell anyone building something similar

A few things I’d repeat:

1. Solve a workflow, not a task. The first instinct is “AI reviews a PR.” The real workflow is “AI reviews → author pushes fixes → AI rechecks → AI resolves what it can → human looks only at what’s left.” Build for the loop, not the moment.
2. State is non-negotiable. Without round numbers and finding statuses, you’re back to a noisy bot.
3. Hide the infra. No tokens to manage locally, no separate database, no setup steps beyond gh auth login and one install script. If a teammate has to read docs, adoption dies.
4. Domain context is everything. Skill modules are the difference between “useful tool” and “another bot we mute.”

What’s next

Two things on the roadmap, both about closing loops.

Auto-review on reviewer assignment. Right now ReviewIQ triggers on PR open and on every push. The next step: the moment a human reviewer is added to the PR, ReviewIQ kicks off automatically. The idea is the reviewer should never walk into a cold PR — by the time they open it, the mechanical findings are already posted, the obvious fixes are already suggested, and they can spend their time on the things only humans should be deciding. Makes the “you’ve been requested as a reviewer” notification feel less like a tax.

Marrying it with Traceback. Traceback is the RCA orchestrator I built earlier — a multi-phase Claude Code workflow that takes an incident and walks through init → analyze → solutions → impact → implement. ReviewIQ and Traceback are solving the same problem from opposite ends of the timeline: ReviewIQ catches issues before they ship, Traceback diagnoses them after they cause an incident.

The interesting bit is what happens when you connect them:

• A Traceback RCA almost always ends with “we should have caught this in review.” Today that lesson stays in a Slack thread. Tomorrow, it should automatically become a new skill rule in ReviewIQ — so the next PR that looks similar gets flagged.
• And going the other way: when ReviewIQ flags a risky change pre-merge, Traceback’s impact-analysis phase can be invoked on that diff to surface what could break downstream — services, consumers, contracts, dashboards.

Net effect: every incident makes future reviews smarter, and every review is informed by the system’s actual failure history. That feedback loop between prevention and post-mortem is the part I’m most excited to build.

Try it

It’s open source — github.com/Sanmanchekar/reviewiq. Built in Go, runs against any repo, works with Claude Code (no API key needed) or as a CLI. Install is one line.

If you’re a tech lead watching your team’s calendar fill up with PR review blocks, give it a try. And if you build a skill module for a stack I haven’t covered, send a PR — I’ll review it. With ReviewIQ, obviously.