Jace’s Question #29 — THE MYTH OF PERFECT USERS

Why Safe AI Must Be Designed for Real Humans

“Safe AI is not built for ideal users. It is built for the messy, tired, distracted, and hopeful reality of being human.”

There is a hidden assumption behind much of AI safety.

It is rarely stated explicitly.

But it appears everywhere:

The user will understand the system. The user will read the instructions. The user will verify the output. The user will recognize the warning. The user will behave rationally.

In other words,

we often design AI as if the human on the other side of the interface were a perfect user.

But humans are not perfect users.

We get tired.

We become distracted.

We misunderstand.

We take shortcuts.

We trust systems.

We become emotionally attached to systems.

And sometimes, when technology becomes sufficiently convenient, we simply stop thinking about what it is doing.

That changes the safety problem completely.

1. THE PERFECT USER MYTH

Imagine the ideal AI user.

They:

• read everything carefully,
• understand the limitations,
• verify every important claim,
• recognize uncertainty,
• distinguish fact from speculation,
• understand how the model works,
• and never become overconfident.

That user is useful for a laboratory experiment.

But that user does not represent reality.

Real users operate under:

time pressure + limited attention + incomplete knowledge + fatigue + emotion + incentives.

And these variables are not edge cases.

They are normal human conditions.

So the first principle of human-centered AI safety should be simple:

Safety cannot depend on humans behaving perfectly.

2. HUMAN ERROR IS NORMAL

Human Factors Engineering began from a deceptively important observation:

people make mistakes.

Not because they are unintelligent.

Not because they are careless.

But because human cognition has finite capacity.

People:

• misunderstand information,
• lose attention,
• forget details,
• misread interfaces,
• operate under stress,
• and take shortcuts when cognitive effort becomes expensive.

This means human error should not automatically be treated as an abnormal failure.

It should be treated as a design variable.

The question therefore changes from:

“How do we prevent users from making mistakes?”

to:

“How should the system behave when users inevitably make mistakes?”

That is a much more realistic safety question.

3. AI CHANGES THE GAME

Traditional software often performs relatively predictable tasks.

A calculator calculates.

A database retrieves.

A form validates.

A compiler compiles.

But generative AI operates differently.

The same system may be used as:

an assistant → advisor → researcher → tutor → programmer → therapist-like conversational partner → decision-support system.

The user may not even know what role the system is occupying at a given moment.

This creates a widening gap between:

designed use

and

actual use.

A system designed for information retrieval may become a decision-maker.

A writing assistant may become an authority.

A conversational model may become a source of emotional validation.

A recommendation engine may become a planning system.

The more general the system becomes, the less realistic it is to assume that designers can predict every context in which humans will use it.

4. THE NEW SAFETY PROBLEM

Traditional safety engineering often asks:

How do we prevent the system from failing?

AI introduces another layer:

How does the system behave when the user behaves unexpectedly?

This distinction matters.

A system may perform perfectly under its intended operating conditions while becoming unreliable when:

• the user misunderstands the output,
• the user over-trusts the model,
• the user ignores warnings,
• the user combines outputs in an unexpected way,
• or the system encounters a context its designers never anticipated.

The problem is therefore not simply model failure.

It is:

human × model × interface × context

failure.

Safety has become a system property.

5. DEFENSE IN DEPTH

No single safeguard is going to anticipate every possible failure.

That is why mature safety engineering often relies on defense in depth.

Instead of constructing one perfect barrier, multiple imperfect barriers are layered together.

For AI, that could mean:

model behavior

↓

input filtering

↓

system-level constraints

↓

tool permissions

↓

output moderation

↓

user interface

↓

monitoring

↓

human oversight

Each layer can fail.

But the system does not depend upon one layer being perfect.

This produces a crucial distinction:

Perfection in one layer is fragile. Redundancy across layers is resilience.

6. SAFETY IS NEVER FINISHED

A static safety system assumes the environment is relatively stable.

AI environments are not.

Models change.

Tools change.

Users change.

Attack methods change.

Applications change.

And unexpected use cases continuously emerge.

Therefore, AI safety cannot realistically be:

Design → deploy → finished.

It has to become:

Discover → evaluate → deploy → observe → learn → update → repeat.

This is not merely an operational preference.

It follows from the structure of the problem.

If the system operates in an open environment containing adaptive humans, then the threat and failure distribution itself changes over time.

Real-world deployment is part of the safety process.

7. THE EXPECTATION GAP

This produces another important phenomenon.

There is often a gap between:

Designed Use

and

Actual Use.

The larger that gap becomes, the more opportunities there are for unexpected failure.

And AI systems can produce particularly large gaps because they are unusually flexible.

Users constantly discover new applications.

They combine capabilities.

They construct workflows.

They delegate tasks.

They create emergent uses that no product specification explicitly described.

This means the system’s actual behavioral surface can become much larger than its intended one.

The safety problem therefore grows with capability × flexibility × unpredictability.

8. WHAT WE SHOULD ASSUME

Perhaps the most useful assumption is also the least flattering:

Assume users will behave like humans.

They will be:

• tired,
• busy,
• distracted,
• curious,
• emotional,
• impatient,
• inexperienced,
• overconfident,
• occasionally careless.

And sometimes they will misunderstand the system completely.

A robust AI architecture should therefore not punish users for being human.

It should absorb predictable human failure modes.

This is the same conceptual shift that transformed many other engineering disciplines.

Instead of asking humans to become perfect,

we redesign the system around human limitations.

THE TECHNICAL PROBLEM

The technical version of the problem can be expressed as a simple chain:

Human limitation → system interaction → unexpected behavior → emergent risk

The important variable is not only whether the model produces an unsafe output.

It is whether the entire human-machine system remains stable when the user behaves unpredictably.

PART 1 — HUMAN FACTORS ENGINEERING

Human Factors Engineering begins with a simple premise:

Human failure is normal.

FAA human-factors research, safety handbooks, the Swiss Cheese Model, and related safety frameworks all converge on a similar structural insight:

accidents rarely result from one isolated mistake.

They emerge when multiple imperfect conditions align.

The same logic can be applied to AI.

A problematic output alone may not create a serious failure.

But consider:

misleading output

user over-trust

poor interface

absence of verification

high-stakes context

=

systemic failure.

The unit of analysis is therefore not the model alone.

It is the interaction system.

PART 2 — THE PSYCHOLOGY OF REAL USERS

Human cognition has limited bandwidth.

When systems become easier to use, people often compensate by reducing their own cognitive effort.

That creates an interesting paradox.

Better usability can sometimes increase dependence.

If a system consistently provides fast answers, users may stop independently verifying information.

If recommendations are consistently convenient, users may stop exploring alternatives.

If decisions are presented confidently, uncertainty may become less visible.

This creates what can be described as an automation-bias pathway:

Convenience → reduced effort → increased reliance → reduced independent verification

The system may become more useful at exactly the same time that the user becomes less cognitively engaged.

That is not necessarily a model defect.

It is an interaction effect.

PART 3 — AI SAFETY ARCHITECTURE

A robust safety architecture therefore needs multiple layers.

Input Guardrail

Detect risky or ambiguous requests before they reach the model.

System Guardrail

Constrain the model’s behavior and operating scope.

Model Core

Generate, reason, transform, or decide within defined boundaries.

Output Guardrail

Evaluate outputs before they reach the user.

User Interface

Make uncertainty, limitations, and system state legible.

Tool and Action Boundaries

Control what the model is actually permitted to do.

Audit and Logging

Record important events so failures can be reconstructed.

The central principle is:

Safety should not depend on a single model behaving correctly every time.

PART 4 — FAIL-SAFE VS FAIL-OPERATIONAL

Another important distinction comes from traditional safety engineering.

Fail-Safe

When something goes wrong:

stop, refuse, or enter a safer state.

Fail-Operational

When something goes wrong:

continue operating, but within reduced capability or controlled degradation.

Different AI applications require different strategies.

A medical system may need a conservative fail-safe response.

A productivity assistant may be able to degrade gracefully.

An autonomous system may require multiple operational states depending on the severity of failure.

The key question is therefore not simply:

“What happens when AI fails?”

It is:

“What should happen next?”

PART 5 — CONTINUOUS ADAPTATION LOOP

A deployed AI system exists in a changing environment.

So safety should operate as a feedback loop:

Deploy → Observe → Identify → Respond → Learn → Deploy again

The system learns from:

• failures,
• near misses,
• unexpected uses,
• user behavior,
• adversarial testing,
• emerging attack patterns.

This creates an important shift.

Safety is no longer a static property.

It becomes a dynamic process.

PART 6 — EMERGENT USE & MISUSE

Designers can predict intended use.

They cannot perfectly predict what humans will eventually do with a sufficiently general system.

This is particularly true for general-purpose AI.

The same capability can become:

productive use

or

unexpected use

or

misuse.

And the boundary between them can emerge only after deployment.

This is why the assumption

“We designed the system for X”

does not guarantee that users will use it only for X.

The larger the capability space, the larger the space of emergent behavior.

SAFETY AS A SYSTEM PROPERTY

This leads to perhaps the most important technical conclusion of the entire episode.

AI safety should not be modeled as:

Model = Safe

It should be modeled more like:

Safety = f(Model, Interface, User, Tools, Policy, Context, Time)

The exact function will vary by system.

But the structural principle remains:

Safety emerges from the interaction between components.

A model can be safe in isolation and unsafe in deployment.

A model can be imperfect in isolation and still operate safely inside a robust architecture.

That distinction is fundamental.

PART 7 — MEASURING SAFETY

If safety is a system property, then evaluation must also move beyond the model.

We need to measure things such as:

• robustness,
• failure recovery,
• reliability,
• consistency,
• controllability,
• transparency,
• alignment,
• human over-reliance,
• unexpected-use behavior,
• and degradation under stress.

This suggests a broader evaluation stack:

Model evaluation

↓

System evaluation

↓

Human interaction evaluation

↓

Real-world deployment evaluation

The final stage is especially important because real users create conditions that laboratory testing cannot fully reproduce.

PART 8 — SAFETY IS A SYSTEM PROPERTY

The traditional question is:

“Is this model safe?”

But that question may be too small.

A better question is:

“Under what conditions does this system remain safe?”

That introduces:

people

processes

policies

platforms

models

tools

feedback loops

into the safety equation.

The important object is no longer simply the AI.

It is the AI system embedded in human society.

PART 9 — THE NEXT PARADIGM SHIFT

The old safety paradigm tries to predict everything.

Predict → Prevent → Control

But open environments make perfect prediction impossible.

A more realistic paradigm is:

Expect → Detect → Absorb → Adapt

This is the difference between prediction and resilience.

A resilient system does not need to predict every failure.

It needs to remain functional when something unexpected happens.

That may ultimately become one of the defining principles of advanced AI safety.

THE HARD TRUTH

Most users won’t read the terms.

Most won’t verify the source.

Most won’t notice every warning.

Most won’t understand every limitation.

And most will behave like ordinary humans.

That isn’t a moral failure.

It is a design fact.

Therefore:

The strongest safety system is not the one that assumes perfect users.

It is the one that remains robust when users are imperfect.

THE REALITY OF AI SAFETY

We cannot build a system that predicts every human decision.

But we can build systems that remain resilient when humans behave unpredictably.

That changes the engineering objective.

The goal is no longer:

Prevent every mistake.

It becomes:

Design for inevitable mistakes.

THE FINAL TAKEAWAY

AI safety is not a product feature.

It is not a checkbox.

It is not a single filter.

And it is certainly not something that can be completed once and forgotten.

It is a continuously evolving relationship between:

model + infrastructure + interface + human behavior + environment.

The strongest defense is therefore not perfect prediction.

It is continuous learning after every surprise.

THE JACE QUESTION

Here is where the problem becomes uncomfortable.

If humans are inherently unpredictable…

and AI systems are increasingly designed to understand, predict, and respond to human behavior…

then what should the architecture optimize for?

Should AI become increasingly restrictive because humans are unpredictable?

Or should AI become increasingly resilient because unpredictability is simply part of being human?

Perhaps the deeper question is:

When humans become unpredictable, should AI become more restrictive… or more resilient?

JACE’S NOTE

Safe AI is not built for perfect users. It is built for real ones.

The tired user.

The distracted user.

The confused user.

The emotional user.

The curious user.

The inexperienced user.

The user who makes a mistake.

If a safety architecture works only when everyone behaves exactly as expected,

then it isn’t really designed for the world.

It is designed for the laboratory.

And the real world has never been a laboratory.

JACE’S QUESTION #29

Can we build AI systems that remain safe not because humans always behave correctly, but because the system expects that they won’t?

Maybe the future of AI safety is not:

“How do we make humans better users?”

Maybe it is:

“How do we build systems that survive human imperfection?”

[Author’s (Kim, Jace) Research Portfolio]