From NeurIPS Rejection to ICLR 2025 Oral: Why KAN is More Than Just a Hype

The “Yes we KAN” craze that took over the AI world in the first half of 2024 had a bit of a rollercoaster ride. After reportedly getting a cold shoulder from NeurIPS, Kolmogorov-Arnold Networks (KAN) didn’t just make a comeback at ICLR 2025 — it secured an Oral presentation.

At its core, this paper is an appeal for interpretability. In an era of opaque black boxes, there is a growing hunger for models that reveal the underlying patterns they’ve learned. KAN’s mission is to replace the traditional Multi-Layer Perceptron (MLP) with a structure that looks less like a matrix of random weights and more like a human-readable formula.

From Matrices to Functions

The KAN architecture is built on the Kolmogorov–Arnold representation theorem, which posits that any multivariate continuous function can be decomposed into a sum of univariate functions. While a standard MLP performs a linear transformation followed by a fixed activation function (where weights are simple scalars), KAN moves the non-linearity onto the “edges” between nodes.

In a KAN, each edge is no longer a single number; it is a learnable, one-dimensional function curve. The nodes themselves do nothing more than sum up the outputs of these incoming edges. These univariate functions are represented using B-splines, where the input is a scalar activation from the previous layer and the output is the spline’s value at that point. Essentially, a KAN layer is a matrix of functions, and the entire network is a composition of these matrices — reconstructing high-dimensional mappings through a series of 1D non-linearities and additions.

The Two-Pronged Strategy: Interpretability and Precision

To make this both practical and transparent, the authors introduced a clever training workflow:

1. For Interpretability: They start with an intentionally over-wide and deep KAN, applying sparsity regularization to force most edges toward zero. After pruning the “dead” edges, they take the remaining splines and attempt to match them against a library of symbolic functions (like $sin$, $exp$, $log$, or polynomials). If a learned spline fits a symbolic function well, they swap it out for the explicit formula. This effectively turns parts of the neural network back into a readable mathematical equation.
2. For Numerical Precision: Splines rely on a grid. If the range of activations shifts during training but the grid remains static, expressivity is lost. To fix this, KAN dynamically updates the grid range based on the minimum and maximum activations observed. Furthermore, once a model is partially trained, you can perform “grid refinement” — increasing the density of the spline knots and projecting the old spline onto the new, finer grid using least squares. This allows for a resolution boost without restarting training from scratch.

Proving the Point

The paper demonstrates these strengths through several compelling examples. By comparing shallow and deep KANs on a four-variable synthetic function, the authors show that deeper KANs learn structured, hierarchical compositions of functions, whereas shallow versions tend to oscillate wildly on certain edges as they struggle to compensate for the lack of depth.

In image fitting tasks (specifically a turbulence profile from PDEBench), KAN treats the $(x, y)$ coordinates as inputs and the field values as outputs. When it comes to fitting these complex physical fields, KAN outperformed standard baselines in terms of PSNR (Peak Signal-to-Noise Ratio).

It seems the “KAN-do” attitude paid off. By bridging the gap between symbolic mathematics and deep learning, KAN offers a glimpse into a future where we don’t just ask what a model predicts, but why it does so in a language we can actually understand.

| Find papers faster on arXivSub with AI summary (CVPR/ICCV/ICML/ICLR/NeurIPS/AAAI/MICCAI)