Why PINNs Fail?

When we express physical systems through partial differential equations, we gain a remarkably powerful advantage: the equation itself already defines which universe we live in. We don’t have to separately worry about how energy is calculated, how information propagates, which speed limits are meaningful, which quantities are conserved, or where to expect continuity. Because all of these are intrinsic to the physical theory we’ve chosen and its mathematical expression. Of course, there are exceptions like weak forms and retarded solutions, but the general case holds. Maxwell, Navier-Stokes, Schrödinger, elasticity equations and many others are not merely mathematical formulations; these equations are also a recipe for a spacetime fabric. What we call a solution is an object that can exist within that fabric.

If you are an AI researcher with a theoretical physics background, you have internalized the absoluteness of the spacetime fabric. For this reason, it is quite natural for you to fall into the following misconception: “I integrated the equations defining the physical system into the loss function, therefore the model will learn the correct physical system.”

It’s actually a pretty reasonable perspective. After all, you’re teaching physics equations, the model is learning physics. If the equations are correct, the solution should be correct too. But no, unfortunately there is no such guarantee for PINNs.

At first glance this may sound speculative, but PINNs (and really all neural networks) belong to a different universe. We live in a physical universe defined by the laws of physics. The structure of our universe is expressed by the four-dimensional spacetime manifold. But PINNs, like all neural networks, exist in a parameter space. The structure of this space is a high-dimensional manifold shaped by the loss landscape.

In the physical universe, matter and energy curve spacetime to produce what we perceive as gravitational force. What produces the curvatures in a neural network’s parameter space, we could say, is the training data and the chosen model architecture. The dataset creates wells in the parameter space similar to gravitational force, that is, basins of attraction. Much like the event horizons of black holes in our universe, these regions have inescapable centers of attraction. But the singularity here does not mean physical annihilation as at the center of a black hole; these are convergence points where information is reduced to its purest form. Gradient vectors, similar to gravitational force, pull the parameters toward these points.

Of course this analogy is not perfect. It will provide intuitive grasp for those unfamiliar with the subject, but I should note that it may have its shortcomings.

Clarifying this distinction matters because in the spacetime manifold, the laws of physics define the solution. In the loss landscape, there is generally only one law: move in the opposite direction of the gradient. During this progression, there is no guarantee about what will be conserved. A neural network during training only cares about reducing the loss values. It does not care whether the solution it produces is consistent with the laws of physics of our universe.

When we are talking about an LLM generating text or a CNN generating images, this is not a problem; it doesn’t matter whether these models obey principles like conservation of energy or causality. But when it comes to PINNs, we cannot ignore the model’s indifference to the laws of physics. Because that is exactly what PINNs were developed for: a neural network learning the laws of physics. How can we teach the laws of our universe to neural networks that belong to a different universe?

The Nature of Time in Parameter Space

One of the critical differences between a neural network’s parameter space and our universe is the nature of time. In the physical universe, time is the fourth dimension that constitutes the spacetime fabric. Nothing can escape it. The loss landscape, however, is inherently timeless. You can think of the landscape of parameter space as a static map, or to put it poetically, a frozen infinity.

This difference in the nature of time manifests itself as the causality problem in PINNs. When we model an electromagnetic environment in our universe, we don’t bother with externally integrating causality. For example, Maxwell’s equations are hyperbolic, and the domain of dependence structure of hyperbolic equations already limits the speed at which information propagates. By choosing retarded solutions, we also determine the direction of information flow to be from the past toward the future. All of this manifests as causality in our everyday life.

In the loss landscape, causality is not a necessity. The answer to the question of which information affects what at which step is determined by the integral structure of the loss. We can confirm this directly through mathematical expressions. Let’s look at the loss function of a typical PINN model:

The first part on the right-hand side is the residual loss, I mean the expression with λᵣ in front of it. The expression with λᵢc in front of it is called the initial condition loss. Depending on the physical problem chosen, a boundary condition loss can also be added here, but the structure is roughly like this. Nᵣ and Nᵢc represent the collocation points, that is, the spatial points where the equations are enforced.

We know that the common input to gradient-based optimization in neural networks is expressed as ∇θ L. For convenience, let’s consider only the residual loss part of the equation above, that is, let’s assume the PINN model contains only residual loss. In that case, the gradient of the loss with respect to the parameters becomes:

As can be seen from the equation, the signal that updates the parameters during training is determined by the sum of contributions from all collocation points (and therefore all times t_i). I want you to pay particular attention to the presence of t_i, that is, time, in the equation. Gradient-based optimization incorporates time into the equation just like a spatial dimension. In short, the past and the future are thrown into the same optimization pool.

What is happening here is not just a violation of causality. The situation is even worse because this simply means the model has no concept of past or future. So much so that a parameter update at t=1.0 can be made because of an error at t=2.0. In science fiction jargon, we could say the model is trying to go back to the past to change events in order to fix the future.

The bad news is, if you are evaluating the model only with the accuracy metric, you generally won’t notice this problem. Because even if causality is violated, the model’s accuracy values can come out quite high, it may look like it has achieved near-perfect success. But if you check the relative error values, that’s when the real picture emerges. You see that the model has learned a solution space that has nothing to do with physical reality.

Personal anecdote: I experienced this firsthand while developing my PINN work. After four months of work I completed the project and the accuracy values looked quite successful. Then this success made me suspicious, and I decided to run the energy metrics and relative error analysis. The result was painful: 100% relative error. In short, what my model had learned had not the slightest connection to physical reality. In fact, that moment of great collapse was actually the moment the research truly began for me. To reduce the relative energy error to 0.02%, I had to spend the next eight months working on PINN.

To solve the problem, we can integrate the laws of physics into the model as regularizers, and this is mostly what is done in the literature. But which laws of physics do we integrate? Since PINNs are specifically developed for well-posed equations, there is unfortunately no single definitive answer. Maxwell’s equations require different interventions, the nonlinear Navier-Stokes equations require different interventions. Sometimes even different problem types of a particular equation may require different interventions. For example, a regularizer that improves results in the low-frequency regime in electromagnetism can make results worse at high frequencies.

We should also not forget that the regularizers included in the loss equation affect each other. For instance, we can use the Poynting theorem as a regularizer to dictate the law of conservation of energy to the model. However, if the weight of the Poynting loss is high, it generally suppresses the PDE loss, which causes the field accuracy to decrease.

Building a structure that mimics causality in parameter space also has various costs. One of the most commonly used approaches in the literature is dividing time into windows, but this approach generally creates discontinuity problems at window transitions. Curriculum learning and causal training approaches have their own disadvantages as well.

How do we get out of this mess? As I just said, there is no single answer. There is no single regularizer that will bring all PINN models into line. And we don’t need one either. A researcher who has internalized the fact that PINNs belong to a different universe will begin to sense at which point the neural networks will fail, the regions with high potential for causing trouble. Because even if the equations change, what we are actually doing doesn’t change: we are trying to translate the laws of physics of this universe for neural networks that live in another universe.

P.S.: You can access my PINN work via the URL below. Regards!

*https://arxiv.org/abs/2512.23396v2*