WP4.1 — System Failure as a Problem of Formation

A growing number of discussions around artificial intelligence are converging on a shared observation: what is being encountered today are not isolated model failures, but failures that appear at the level of systems.

This shift matters. It redirects attention away from individual components toward the broader conditions in which AI operates where models interact with human judgment, institutional structures, and evolving environments. The problem no longer appears contained within the model; it extends across the system that surrounds it.

Yet even this reframing remains incomplete.

Because once the analysis begins from the idea of “systems,” it already accepts a premise that is rarely examined: that the system itself provides a valid and stable starting point. The system is taken as given, and the question becomes how to improve, regulate, or align it.

What remains unasked is whether this starting point is sufficient.

In practice, a recurring pattern is difficult to ignore.

AI systems are constructed, refined, and optimized. Failures are identified, analyzed, and addressed through improved training, stronger controls, and more elaborate governance structures. Yet similar forms of difficulty continue to reappear across contexts:

• outputs that diverge from human expectations
• behaviors that shift unpredictably under changing conditions
• decisions that amplify unintended consequences when scaled

These patterns are typically approached as operational problems. They are treated as issues that emerge during execution and therefore can be corrected through better design, tuning, or oversight.

But their persistence suggests another possibility.

What appears during operation may not originate there.

What is being observed as system behavior may already reflect something that precedes the system itself.

To approach this possibility, it becomes necessary to examine how systems are formed.

Most contemporary approaches begin by separating key dimensions of the problem:

• technical structures — models, data, and computational processes
• human factors — perception, judgment, incentives, and behavior
• institutional constraints — rules, governance, and accountability

Each of these dimensions is developed according to its own internal logic. Technical systems are optimized for performance, human behavior is modeled or guided through incentives, and institutional structures are designed to regulate and constrain outcomes.

Only at a later stage are these dimensions brought together and treated as a unified system.

This separation is rarely questioned. It is regarded as a practical necessity, a way to manage complexity by dividing it into manageable domains.

When these independently formed components are combined, however, a persistent tension becomes visible.

The resulting system does not settle into coherence. Instead, it requires continuous adjustment:

• to reconcile model outputs with human interpretation
• to constrain behavior within institutional rules

This condition is often described as complexity.

But complexity does not fully capture what is occurring.

Because the tension does not diminish as systems improve. It persists and, in many cases, intensifies, despite increasing sophistication in each component.

What appears is not simply a system that is difficult to manage, but a structure in which alignment must be constantly produced and reproduced without ever becoming inherent.

This suggests that the difficulty does not lie primarily in how systems operate, but in how they are formed.

When fundamental dimensions are separated at the outset, their later integration does not generate coherence. It produces interaction without unity.

In such a structure, misalignment is not an anomaly to be corrected. It is a condition that arises from the way the system is brought into existence.

From this perspective, what is described as “system failure” does not indicate the breakdown of an otherwise stable entity.

It is the visible expression of an underlying inconsistency.

Efforts to address these failures at the level of operation through optimization, regulation, or controlling can mitigate specific outcomes. They can improve performance within defined parameters. But they do not alter the conditions from which those outcomes arise.

The system continues to function. Yet the same patterns of divergence reappear, not as isolated incidents, but as recurring features of its operation.

This leads to a more fundamental question.

Can systems constructed from independently defined layers ever sustain alignment at scale?

Or will they inevitably reproduce the same patterns of divergence, regardless of how refined their components become?

If the latter is the case, then the challenge cannot be reduced to improving existing systems.

It requires reconsidering the point at which systems are conceived.

A system expected to remain coherent under increasing complexity cannot be assembled from elements that were never aligned in their formation.

In this sense, the issue is not simply that systems fail.

It is that what is called a system may not yet have been formed in a way that can sustain coherence to begin with.

Until this level is addressed, “AI system failure” will continue to be approached as a problem to solve.

What appears as failure will remain a recurring condition, it is not because systems are insufficiently engineered, but because the conditions of their formation remain unchanged.