Mitigating AI Safety Risks: The High Cost of Being Wrong with AI and Automations

Photo by Joe Dudeck on Unsplash

AI is being implemented in nearly every layer of business workflows. This includes business-critical tasks like order fulfillment, financial transactions, and structural design, and back-office functions like onboarding, accounting, and communications. A frequent question across business units is “Can AI do this task?”

Is this the right question to ask? AI can do a lot, after all. It has unlocked advanced SQL queries for no- and low-code business analysts, sped up marketing campaigns for marketing managers, and analyzed volumes of case law for law firms. There are many tasks it can attempt, but what happens when AI is wrong?

This is an important question both in everyday business and in more risk-heavy fields such as medicine, law, and finance. AI is not infallible, and the decisions it makes and the outputs it generates have a direct impact on people. That gap between what AI can attempt and what it should be trusted to do unsupervised is where AI safety risk lives.

Before we ask if AI can do something, we should always ask what happens if it does it wrong. For a CTO, product leader, or tech lead deciding what belongs on the automation roadmap next, that question matters more than almost any other. We must know how frequently AI may be wrong and the costs associated with a given task because mitigating AI safety risk starts with knowing where to actually look.

Understanding AI safety risk

The question of “what if AI is wrong” has two components:

1. How frequently will it be wrong?
2. How expensive is the fix for being wrong?

Figure 1. AI + automations application safety matrix

Plotting a task on these two axes tells you almost everything you need to know about whether and how an AI application should be automated, reviewed, or avoided.

So why is AI wrong in the first place? Most errors trace back to a mismatch between the model and the problem. AI needs domain-specific data to train on, and if that data is corrupted, biased, or simply not representative of your domain, you get bad results. A model trained mostly on general text and images doesn’t automatically know how to read a lung scan or price a piece of commercial real estate. Those patterns were never in what it learned. The less overlap there is between what a model was trained on and what you’re asking it to do, the more often it will be wrong.

It’s important to understand where your business case falls on these two axes before deciding how much automation and human oversight it deserves. Let’s walk through real-world examples for each of the four quadrants.

AI errors that are rare and cheap

This is the best quadrant for businesses looking to add AI or automations to their workflows. These errors:

• Happen occasionally
• Are easy to correct
• Have low business impact
• Are usually reversible

Everything about these errors is temporary and easily corrected, often without users ever knowing an error occurred. LLMs are trained primarily on text, so they’re very good at generating text. Some of the best AI applications involve writing copy or summarizing documents.

Good applications that fall in this quadrant include:

• About Us pages
• Internal documentation
• Quarterly newsletters
• Annual reports
• Employee announcements

You can feed in notes, previous versions of the document, and pointers to internal sources to have an AI model generate a first draft. That draft may not be perfect. There may be spelling errors, misformatted content, or incorrect numbers or references. However, with one or two reviews you can correct the issues and have a workable document.

AI errors that are rare and expensive

Some errors rarely occur, but when they do, they can cause lasting and sometimes irreversible damage to customers and the business. Areas where these errors occur include:

• Cancer diagnosis
• Aircraft maintenance
• Mergers and acquisitions (M&A) due diligence
• Prescription recommendations

Applications in this quadrant tend to share a few traits:

• They involve complex, often novel situations
• They require deep domain context that doesn’t fit in a prompt
• A single miss can have consequences that are difficult or impossible to reverse

All the items listed above are complex operations that often involve novel datasets and require immense amounts of context. They rely not just on the primary data given to an LLM, but on all the relevant supporting data as well.

Take medical imaging. Most popular LLMs are trained primarily on text, not images, so spotting a tumor on a scan is a poor fit for a general-purpose model unless it has been specifically and rigorously trained for that task. A model trained mostly on pictures of cats, for instance, has no meaningful basis for reading a lung scan and would miss abnormalities far more often than a purpose-built diagnostic model. That’s what makes this quadrant dangerous: the errors are rare precisely because they only surface when the model meets a case it wasn’t built for, but each one lands hard. M&A due diligence carries a version of the same problem. It requires vetting company financials, asset valuations, and talent and leadership assets that are hard to quantify and don’t reduce cleanly to a prompt.

Any of these applications needs more context than a sentence or a list of conditions, and the cost of getting it wrong can be staggering. A missed cancer diagnosis can lead to missed or unnecessary treatment. Incorrect M&A due diligence can leave shareholders with lower returns and expose customers to loss of value or data.

All of these applications can lead to regulatory or legal action and a loss of public trust. We trust experts with complex medical and business domains because no one person can know everything, and because the impact on people’s lives is so high.

The best way to mitigate errors here is strong human review from knowledgeable reviewers. This practice is generally known as human in the loop (HITL): keeping a qualified person in the workflow to review and approve AI output before it reaches a decision point. AI should never be the autonomous decision-maker in these applications; it should act in an assistant role.

AI errors that are frequent and cheap

Some AI errors happen frequently, but have such a small business impact that heavy human investment isn’t necessary. Errors are expected in these applications, which makes them a great opportunity for automation.

Applications in this quadrant tend to share a few traits:

• The output is a draft or an input to something else, not the final deliverable
• Mistakes are easy to spot and cheap to fix
• Volume is high enough that manual review doesn’t scale anyway

Examples of these applications include:

• Tweets
• Marketing emails
• Blog outlines
• Grammar checking
• Meeting summaries
• Internal chat responses

All of these applications are focused on communications that support the creation of a final product, rather than being the final deliverable themselves. Automating the work of a note-taker, summary generator, or draft writer frees up valuable time for teams and because the output always gets a look before it goes out the door, an occasional bad draft costs a few minutes, not a few million dollars.

If a summary has an incorrect section, or a draft has bad spelling, it can be corrected with another automation tool tailored to that error, or with light human oversight. This is the one quadrant where it’s usually fine to let AI run and fix problems after the fact, rather than gating every output before it ships.

AI errors that are frequent and expensive

AI can be applied to almost any problem, but at some point the cost and frequency of review can outweigh the upside of the application. Frequency is relative to the volume of work being sent to AI and the impact of each error. Ten errors out of a thousand jobs may sound low, but when each one costs a million dollars, those ten occurrences carry a lot of weight.

Characteristics of this quadrant include:

• High volume of items being evaluated
• Every mistake matters
• A small error rate still adds up to a large business problem

If your restaurant cooks thousands of dinners a month, it doesn’t matter that there’s only been one fire. That single fire outweighs an otherwise perfect record of safe meals delivered.

Business applications with high-frequency exposure to expensive errors include:

• Financial fraud detection
• Loan approvals
• Insurance claims
• Malware detection
• Medical triage
• Autonomous driving
• Manufacturing quality inspection

It only takes the approval of one bad invoice to sink a company. It only takes one malicious network request to expose user data, and it only takes a handful of bad brake pads to trigger a full product-line recall. Being right most of the time isn’t good enough in these domains.

How do you fix this? First, understand that these tasks will never be perfect. Even with human oversight and controls, there will still be incidents, recalls, and bogus claims. The fact that errors already occur in the current system doesn’t mean we can automate these tasks without caution for the new kinds of errors AI introduces.

Reducing risk here takes a blended approach. You need confidence thresholds: minimum reliability scores, set through testing, that a given action must clear before it’s allowed to proceed on its own. You need monitoring, sampling, and auditing to keep a live read on how healthy the process is. And, as in the rare-and-expensive quadrant, you need HITL: a human reviewing the cases the system flags as uncertain, and spot-checking the ones it doesn’t.

How to add AI and automations safely

Calls for AI and automation are everywhere. Every business says they’re using AI, and if you search long enough there’s always a SaaS company willing to solve your specific problem. But not every business application is the same. Some are a perfect fit for AI and automation because the errors are temporary and low-impact. Others could seriously harm your company or the public with only a handful of mistakes.

When you’re looking to integrate AI and automation into your business, don’t stop at asking whether AI can perform the task. Ask instead:

1. How frequently might AI be wrong?
2. How expensive is it if AI is wrong?
3. What controls and processes can reduce or prevent those errors?

Mitigating AI safety risk isn’t about avoiding automation. It’s about matching your level of human oversight to the quadrant a task actually falls in, and putting your first automation bets on the tasks that are most forgiving of error. The most successful automations and AI applications aren’t the ones that do the most work. They’re the ones that provide the most value relative to their error costs, error frequency, and the human labor needed to correct them. Keep asking how AI can safely and effectively benefit your customers, and you’ll deliver service that’s cheaper, faster, and higher quality.