The Myth of Perfect Users and Perfect Systems

A Cross-Disciplinary Framework for Human-Centered AI Safety and Resilience

Author

Jace (Jeong Hyeon) Kim, Independent Researcher,

k73066720@nate.com, https://x.com/Jace_blog

September 07, 2026

Abstract

AI safety is often discussed in terms of model capabilities, alignment, robustness, and adversarial behavior. Yet failures in complex sociotechnical systems rarely originate from a single component. They can emerge through interactions among human behavior, system design, organizational practices, information environments, and changing operational conditions.

This paper examines a related problem: the tendency to design AI systems around implicit assumptions of both a perfect user and a perfect system. Human users are variable, adaptive, and context-dependent, while complex systems operate under conditions that cannot always be fully anticipated or controlled. As AI becomes increasingly embedded in information retrieval, decision support, automation, and organizational workflows, these assumptions may create vulnerabilities that cannot be adequately addressed through model-level safeguards alone.

Drawing on perspectives from human factors engineering, safety science, cognitive psychology, sociology, organizational research, and AI security, this paper develops a compact cross-disciplinary framework for understanding these interactions. The framework connects five elements: design assumptions, human adaptation, system behavior, environmental change, and feedback. It further considers how trust, information integrity, and organizational context can mediate the propagation of failure across system boundaries.

Rather than proposing another model-specific safety mechanism, the paper argues for a shift from perfection-oriented design toward assumption-aware resilience. The central proposition is that AI safety should be evaluated not only by whether a model behaves correctly under expected conditions, but also by whether the surrounding sociotechnical system can detect, absorb, and learn from violations of its underlying assumptions.

The paper concludes that resilient AI design may require treating imperfect users, imperfect information, and imperfect systems not as exceptional failure conditions, but as normal properties of the environments in which AI operates.

Author’s Note

This paper began with a relatively simple observation: safety is never determined entirely by the controls we design in advance.

The paper examines AI safety from a cross-disciplinary perspective, bringing together human factors, safety engineering, cognitive psychology, organizational research, AI security, and resilience engineering. Rather than proposing a new safety theory, it attempts to connect several established observations around a common problem: systems are designed according to assumptions, while the humans, information, organizations, and environments within those systems continue to change.

The concepts of the Perfect User, the Perfect System, and Cognitive Hijack are therefore not presented as entirely new theoretical constructs. Much of the underlying knowledge already exists across established research traditions. The intended contribution is more modest: to place these perspectives within a shared framework and ask what becomes visible when they are considered as interacting layers of the same sociotechnical system.

A reviewer may reasonably argue that the framework synthesizes concepts that are already well known. That criticism would be fair. This paper does not claim to have discovered human error, system failure, trust calibration, organizational adaptation, or resilience. Nor does it claim that every proposed relationship has been empirically demonstrated as a unified mechanism.

The purpose is different.

Even when individual failure modes are well understood, their interactions may remain difficult to anticipate. A system can incorporate extensive safeguards and still encounter conditions that were not represented in its original assumptions. A user can behave differently from expected. Information can change. Organizations can adapt procedures in unintended ways. Trust can develop in ways designers did not anticipate. And several individually manageable deviations can interact to produce a condition that no single control was designed to address.

This is why I believe safety and uncertainty can never be emphasized too many times.

No matter how carefully a system is designed, there will remain domains that cannot be completely predicted in advance. The appropriate response is not to abandon rigorous prevention, but to recognize the limits of prediction and design systems that can detect, recover from, and learn from unexpected conditions.

There is an old East Asian idea that even a young child can teach an adult something. The point is not that every idea is equally valuable, but that knowledge does not always arrive from the person we expect to teach us.

I approach this paper in much the same spirit.

I am an independent researcher, not an institutional safety engineer or a member of a large research laboratory. This paper should therefore be read with appropriate skepticism. Its framework may prove incomplete, its synthesis may be challenged, and some of its proposed connections may require empirical investigation before they can be treated as established relationships.

That is acceptable.

If even a modest argument can cause a researcher, engineer, designer, or policymaker to pause for a moment and reconsider an assumption that had previously gone unquestioned, then the paper has served its purpose.

Even an imperfect observation can be useful if it helps us notice a variable we had previously overlooked.

And if a seemingly insignificant piece of writing can give one reader a new question to investigate, I will consider that contribution sufficient.

Kim, J. H. (2026). The Myth of Perfect Users/Systems: Human-Centered AI Safety Zenodo. https://doi.org/10.5281/zenodo.22559555

[embed]The Myth of Perfect Users and Perfect Systems: A Cross-Disciplinary Framework for Human-Centered AI… Abstract AI safety is often discussed in terms of model capabilities, alignment, robustness, and adversarial behavior…doi.org