Bank reconciliation: when confident AI scoring fails

The case for interpretable design over opaque rankings

When an AI ranking system can’t show its reasoning, you can’t tell correct answers from wrong ones. Both look equally confident.

In a previous piece, I described reconciling a bank statement entry with an adjustment code that carried almost no information of its own. The code told you something happened. It didn’t tell you what, or why, or which of the entries already sitting in the ledger it was meant to correspond to. In financial messaging standards like ISO 20022, these adjustment codes are part of the structured data, but they carry minimal semantic weight. They’re placeholders, not explanations. The scoring pipeline behind it did most of the real work: narrowing a large pool of possible entries down to a shortlist using amount, date, and counterparty, each one a concrete, checkable fact rather than a guess. Often, that shortlist collapses to one clear answer, one candidate that fits so well and cleanly that no further judgment is needed.

This piece is about what happens when it doesn’t. When the shortlist survives the scoring pass with two or three candidates still standing, none of them a clean winner, each one right in a different way and wrong in a different way. That’s a distinct problem from the one the earlier piece solved, and it calls for a distinct kind of reasoning to go with it, exactly the type of problem LLMs excel at.

Not all signals are the same

Not every signal that feeds a match decision is doing the same kind of work. Most of them measure how well a candidate fits, more of one thing, less of another. A smaller number decide whether a candidate belongs in the running at all, independent of how well it scores anywhere else. A few common examples, looked at on their own, make the difference easier to see before watching them collide.

• Amount matched after FX conversion. A candidate’s amount lines up with what’s expected, but only once a currency conversion is applied. The arithmetic itself is trivial, any script can multiply by a rate. What isn’t trivial is judging whether that rate assumption was reasonable for that particular date. That judgment call determines how much weight a match built on an assumption deserves compared to one that needs no assumption at all. That judgment call, not the multiplication behind it, is the part worth handing to something capable of reasoning about it rather than just computing it.
• Settlement-cycle-adjusted timing. Raw date proximity, on its own, doesn’t mean much. Two days between an expected date and a candidate’s actual date could be a red flag or a total non-issue, and the difference depends entirely on what kind of transfer is being reconciled. A rules engine needs someone to spell out every transfer type and its expected settlement window in advance. An LLM can figure it out from context, recognizing the instrument type and inferring the settlement expectation on the fly. No hard-coding required.
• Fuzzy counterparty match. The name on the candidate doesn’t match the name on file, not exactly. The bank statement shows a trading name, but the ledger holds the legal entity name. Or the statement uses an abbreviation while the ledger has the full form. Or the transliteration differs across systems. A person would recognize these as the same counterparty instantly. This is the clearest case in the set where you need language judgment, not just a lookup. An LLM handles it; string-matching does not.
• Sequence check. This one isn’t a measurement candidates have more or less of. It’s a gate. A reversal cannot precede what it reverses. That’s a structural rule, not a judgment. No reasoning required here, just dates. It’s a date check, straightforward. If you let stronger signals override it, a structurally impossible candidate can still rank highest because its other signals align well.
• Narrative specificity. A remittance line or adjustment note can be specific enough to point at one candidate, or generic enough to describe several of them equally well. A note that reads “adjustment per prior correspondence” doesn’t favor any single candidate over another, because it would be equally true of all of them. A generic note reads true of any candidate. Recognizing it as generic requires reading for meaning, not just checking if the field is populated. That’s where keyword search fails and reading for meaning succeeds.

Four of the above examples behave like measurements, each one a place where judgment about degree and plausibility genuinely helps. The sequence check behaves like a rule, and rules like it should run before any weighing starts, the same way every time.

What happens when the measurements themselves point in different directions is where this piece goes next.

When shortlists deadlock

A scoring function is good at producing a ranked list. It is not automatically good at knowing when that ranking means anything. Two or three candidates with close scores are not equally likely. Treating them as interchangeable just because numbers are near is where these pipelines, built to rank-order everything, start to fail. This isn’t a rare edge case either. When signals vary in reliability, near-ties are expected. An FX-adjusted amount match rests entirely on an assumed rate you can’t verify. A fuzzy counterparty match is genuine but soft. A person recognizes the trading name versus legal entity name, but it cannot be independently confirmed. Both can score the same, despite resting on fundamentally different foundations.

Take a case where three candidates survive the initial filter for the same statement entry:

• Candidate A relies on the FX-adjusted amount signal described above. Nothing about the reference or narrative supports it directly. The match exists because the arithmetic works out under an assumed rate, not because anything independently points to it. A reviewer looking at this candidate in isolation would have no way of confirming the rate assumption without pulling the actual conversion used on that value date, something the statement entry itself may not carry.
• Candidate B carries the missing reference field problem. Amount and counterparty both match perfectly, but there’s no invoice number, transaction ID, or memo attached. Everything checks out numerically and by counterparty, but you can’t trace it to anything to verify what it actually corresponds to. Possible, but unconfirmed.
• Candidate C carries the fuzzy counterparty match: a trading name, not the legal entity name on file, close enough that a person scanning the list would recognize it, but different enough that an exact string match would miss it entirely. A rules engine would miss this completely, not rank it low. Exact-match checks don’t catch fuzzy counterparty names.

None of these candidates is decisive on its own, and none of them is weak in the same way as the others. Candidate A depends entirely on an assumption you can’t verify. Reasonable-looking, perhaps, but still an assumption. Candidate B looks strong because amount and counterparty both match perfectly. But that apparent strength rests entirely on matching numbers. Without the supporting reference field, you have no way to verify what it actually represents. You only know it fits numerically. Candidate C’s evidence is intuitive to a person but invisible to algorithms. It requires recognizing similarity rather than verifying exact matches.

A single score for each candidate would flatten all of this into three numbers that look directly comparable. They are not comparable. Each fails for a different reason: unverifiable assumption, missing supporting detail, invisible pattern that only humans can surface. Ranking them by score answers “which number is highest.” A score doesn’t answer what really matters. Which type of uncertainty can a reviewer actually live with? That’s the actual question.

What weighted sums hide

A tempting next step is to assign each signal a weight, add them up, and let the highest score win. This is tempting because one number is simpler than a rationale. It’s easy to sort, easy to log, easy to defend in the moment. It is also the wrong move here, because the number doesn’t tell you why it won, and in a case like this one, the why is the entire question.

A weighted sum reduces “amount matched, conditional on an assumed FX rate” using the same calculation it applies to an exact match with no assumption. Both become numbers on the amount axis. You can’t tell them apart. But they are not the same kind of evidence, one assumes something that can’t be independently confirmed, the other doesn’t assume anything. A score built on one signal that depends on an unverifiable assumption is fundamentally different from a score built on multiple weak but verifiable signals. The sum can’t tell the two apart, and once it’s collapsed into one figure, that difference is gone.

This is what interpretability by design actually means in practice: producing the reasoning as part of the output itself, rather than trying to reconstruct an explanation for a black-box score after the fact.

That loss has a real cost. Someone eventually reviews this match, whether to approve it, question it, or catch an error. They need to know which parts of the case are solid, which are assumptions, and which assumptions the outcome depends on. A weighted sum throws all of this away. These are completely different situations. They deserve different amounts of scrutiny, yet a single number can’t distinguish between them no matter how the weights are chosen.

This is where AI-powered reasoning differs. It keeps the evidence separate instead of flattening it into a single score. It can hold onto the distinction a weighted sum throws away, and say so, in a form a person can actually use.

Interpretability by design

Ranking these three candidates isn’t a matter of picking the one with the most going for it. It’s weighing what each one’s evidence actually costs to accept.

Candidate A’s case rests on an assumption that cannot be verified from the entry itself, no matter how plausible that assumption is. The FX rate looks reasonable, but you can’t check it independently against the actual conversion used that day.

Candidate B’s numbers align perfectly, both amount and counterparty matching, but there’s nothing underneath to explain what the adjustment represents. Everything checks out on the surface. Nothing clarifies what it actually is.

Candidate C’s counterparty match is genuine but soft. A person would recognize the trading name versus legal entity name instantly. But that recognition exists only for a reader capable of seeing similarity. Algorithms won’t surface it at all.

A useful rationale for this shortlist might read: Candidate A relies on an assumed FX rate not present in the entry data. Candidate B’s numbers align but carry no supporting reference or memo. Candidate C’s counterparty name roughly matches with the record on file.

What makes this a rationale rather than a verdict is that every claim is checkable. The FX assumption, the missing reference field, the counterparty name match are all verifiable. A reviewer can assess them independently. That distinction is what separates this step from the scoring step it follows. It also makes each candidate’s weakness visible rather than folded into a smoothed-over total.

Confidence earned, not asserted

With three candidates like these, a confidence number isn’t what the rationale earns. What it earns is clarity about why no confident answer exists.

Candidate A’s assumption cannot be verified from the entry alone. Candidate B carries no supporting detail to explain what it represents. Candidate C’s match exists only as pattern recognition, invisible to rule-based systems. Each relies on different assumptions and carries different risk.

A high-confidence guess here would be dishonest. It would look like a decision while hiding the fact that the evidence doesn’t support one candidate over the others. The rationale lays that out instead. This is what genuine uncertainty looks like when laid out plainly. No number needed.

That’s different from a low-confidence score like ‘38% confidence, Candidate A,’ which still designates one candidate as the answer even though confidence is low. A rationale that declines to rank says something different: nothing here supports picking one over the others.

This is what honest reasoning looks like: refusing to rank when evidence is mixed, saying ‘nothing here clears the bar’ instead of forcing an answer. No number needed.

Decline and be honest

A pipeline that always produces a ranked answer, even when evidence genuinely doesn’t support one, isn’t more capable than one that declines. It’s just better at hiding when it shouldn’t have decided at all.

Force Candidate A and you’ve posted an adjustment built on an FX assumption you couldn’t verify. If the rate is wrong, the adjustment is wrong, and someone catches it later during audit or reconciliation. Now there’s a correction to make, one that could have been avoided by flagging it upfront.

Force Candidate B and you’ve reconciled to an entry with no supporting documentation explaining what it represents. Later, when someone audits the adjustment or tries to match it to source documents, there’s nothing to check against. You’ve created a gap in the trail that costs time to investigate.

Force Candidate C and you’ve trusted an algorithm’s fuzzy match strongly enough to mark the reconciliation complete. But if the pattern match was slightly off, you’ve reconciled to the wrong counterparty entirely. The actual unreconciled item stays open, and the false match masks a real problem.

Declining to decide is not a failure. It’s refusing to introduce preventable downstream costs in exchange for a confident-looking output that isn’t actually confident.

AI’s core contribution

Scoring does the structural work. It builds the shortlist, applies hard rules, narrows the search space. What natural-language reasoning adds is judgment: weighing evidence that differs in kind, not just magnitude, and being honest when the evidence won’t support a ranked answer.

That’s a smaller claim than ‘AI resolves ambiguous reconciliation entries.’ It’s also a more useful one. But that’s just the floor. The real work starts when you’re weighing not five kinds of evidence, but dozens — each with its own uncertainty, its own context.