Paper Insights: Sobolev Training

I was recently reading a Medium article on Physics-Informed Machine Learning (PIML), and Sobolev Training was mentioned as a research frontier in PIML. Thus, I decided to read the original NeurIPS 2017 paper that introduces Sobolev Training for Neural Networks from DeepMind UK. However, I still wanted to explore how the Sobolev Training is connected with physics-informed neural networks and decided to read Sobolev Training for Physics-Informed Neural Networks.

Sobolev Training:

In many Machine Learning problems, it is common to have input-output ground-truth pairs; however, some problems may involve derivatives of the target outputs with respect to the inputs. For instance, in distillation tasks, a differentiable target output would be a neural network. The authors believe there can be data-efficiency advantages when these differentiable targets are used. In these scenarios, authors introduce Sobolev Training for Neural Networks, in which target derivatives and target values are incorporated into the training.

The authors mention Hornik's work(Approximation Capabilities of Multilayer Feedforward Networks) numerous times, where it is shown that neural networks satisfy the universal approximation theorem in Sobolev spaces. Hornik shows that neural networks with non-constant, unbounded, continuous activations and continuous derivatives up to order K are universal approximators in the Sobolev spaces of order K.

Here is a normal objective of a loss function where f is some function parameterized by a neural network m with weights theta, and l is some loss function:

Normal training objective

When learning in Sobolov spaces, this loss function is transformed into:

Sobolov training objective

where the first part is the same as the normal training objective, and the second part is the loss calculation between the sum of 1…K neural network j-th order derivative and the j-th order of the target. If the loss involves calculating a high-order K-th derivative, this optimization process can be expensive. Instead of calculating a full Jacobian/Hessian matrix, we can compute its projection onto a random vector. Thus, a random variable v is sampled from the unit sphere, and these random projections are matched:

Sobolev Training for Solving PDEs:

The authors of Sobolev Training for Physics-Informed Neural Networks introduce a novel loss function that accelerates the training of a standard PINN, making it more efficient. PINNs have achieved great success in solving Partial Differential Equations (PDEs), but they still suffer from slow convergence rates. PINNs add PDE residuals to the loss function; however, with Sobolev-PINNs, the derivatives of these residuals are also placed into the loss function.

The paper provides mathematical proofs that the proposed loss functions provide an upper bound on the approximation error in Sobolev spaces for several types of equations, including the 1-D Heat Equation, the Viscous Burgers’ Equation, and the Kinetic Fokker-Planck Equation. These proofs guarantee that as the Sobolev loss goes to zero, the neural network solution converges to the true analytic solution, not just in value, but also in its derivative profiles.

The authors conducted simulations across various PDEs (Heat, Burgers, Fokker-Planck, and high-dimensional Poisson equations). They found that Sobolev-PINNs achieved target accuracy levels in significantly fewer epochs (often a 3x to 10x speedup in training iterations). The training was less sensitive to random weight initialization, resulting in more stable, reproducible results.

Sobolev Training and Physics-Informed Machine Learning are related because both try to teach a neural network not just values, but the underlying differential structure of a physical system. Since Physics is usually expressed through differential equations, physical correctness is about the behavior of derivatives, and Sobolev training supervises derivatives. Both are mechanisms to inject inductive biases about derivatives into training. Overall, by reading both these papers, I got a better understanding of Sobolev Training and why it is being considered as a related method to physics-informed machine learning.