Kolmogorov-Arnold Networks

Kolmogorov–Arnold Networks (KANs) are a type of artificial neural network architecture inspired by the Kolmogorov–Arnold representation theorem, also known as the superposition theorem. Unlike traditional multilayer perceptrons (MLPs), which rely on fixed activation functions and linear weights, KANs replace each weight with a learnable univariate function, often represented using splines.^[1][2][3]

History

KANs (Kolmogorov-Arnold Networks) were proposed by Liu et al. (2024)^[4] as a generalization of the Kolmogorov–Arnold representation theorem (KART), aiming to outperform MLPs in small-scale AI and scientific tasks. Before KANs, numerous studies explored KART’s connections to neural networks or used it as a basis for designing new network architectures.

In the 1980s and 1990s, early research applied KART to neural network design. Kůrková et al. (1992)^[5], Hecht-Nielsen (1987)^[6], and Nees (1994)^[7] established theoretical foundations for multilayer networks based on KART. Igelnik et al. (2003)^[8] introduced the Kolmogorov Spline Network using cubic splines to model complex functions. Sprecher (1996, 1997)^[9][10] developed numerical methods for constructing network layers, while Nakamura et al. (1993)^[11] designed activation functions with guaranteed approximation accuracy. These efforts bridged KART’s theoretical potential with practical neural network implementation.

KART has also been applied in other computational and theoretical domains. Coppejans (2004)^[12] developed nonparametric regression estimators using B-splines, Bryant (2008)^[13] applied it to high-dimensional image tasks, Liu (2015)^[14] explored theoretical applications in optimal transport and image encryption, and more recently, Polar and Poluektov (2021)^[15] used Urysohn operators for efficient KART construction, while Fakhoury et al. (2022)^[16] introduced ExSpliNet, integrating KART with probabilistic trees and multivariate B-splines for enhanced function approximation.

Architecture

KANs are based on the Kolmogorov–Arnold representation theorem, which was linked to the 13th Hilbert problem.^[17][18][19]

Given ${\displaystyle x=(x_{1},x_{2},\dots ,x_{n})}$ consisting of n variables, a multivariate continuous function ${\displaystyle f(x)}$ can be represented as:

${\displaystyle f(x)=f(x_{1},\dots ,x_{n})=\sum _{q=1}^{2n+1}\Phi _{q}\left(\sum _{p=1}^{n}\varphi _{q,p}(x_{p})\right)}$   (1)

This formulation contains two nested summations: an outer and an inner sum. The outer sum ${\displaystyle \sum _{q=1}^{2n+1}}$ aggregates ${\displaystyle 2n+1}$ terms, each involving a function ${\displaystyle \Phi _{q}:\mathbb {R} \to \mathbb {R} }$. The inner sum ${\displaystyle \sum _{p=1}^{n}}$ computes n terms for each q, where each term ${\displaystyle \varphi _{q,p}:[0,1]\to \mathbb {R} }$ is a continuous function of the single variable ${\displaystyle x_{p}}$. The inner continuous functions ${\displaystyle \varphi _{q,p}}$ are universal, independent of ${\displaystyle f}$, while the outer functions ${\displaystyle \Phi _{q}}$ depend on the specific function ${\displaystyle f}$ being represented. The representation (1) holds for all multivariate functions ${\displaystyle f}$. If ${\displaystyle f}$ is continuous, then the outer functions ${\displaystyle \Phi _{q}}$ are continuous; if ${\displaystyle f}$ is discontinuous, then the corresponding ${\displaystyle \Phi _{q}}$ are generally discontinuous, while the inner functions ${\displaystyle \varphi _{q,p}}$ remain the same universal functions.^[19]

Liu et al.^[1] proposed the name KAN. A general KAN network consisting of L layers takes x to generate the output as:

${\displaystyle \mathrm {KAN} (x)=(\Phi ^{L-1}\circ \Phi ^{L-2}\circ \cdots \circ \Phi ^{1}\circ \Phi ^{0})x}$   (3)

Here, ${\displaystyle \Phi ^{l}}$ is the function matrix of the l-th KAN layer or a set of pre-activations.

Let i denote the neuron of the l-th layer and j the neuron of the (l+1)-th layer. The activation function ${\displaystyle \varphi _{j,i}^{l}}$ connects (l, i) to (l+1, j):

${\displaystyle \varphi _{j,i}^{l},\quad l=0,\dots ,L-1,\;i=1,\dots ,n_{l},\;j=1,\dots ,n_{l+1}}$   (4)

where n_l is the number of nodes of the l-th layer.

Thus, the function matrix ${\displaystyle \Phi ^{l}}$ can be represented as an ${\displaystyle n_{l+1}\times n_{l}}$ matrix of activations:

${\displaystyle x^{l+1}={\begin{pmatrix}\varphi _{1,1}^{l}(\cdot )&\varphi _{1,2}^{l}(\cdot )&\cdots &\varphi _{1,n_{l}}^{l}(\cdot )\\\varphi _{2,1}^{l}(\cdot )&\varphi _{2,2}^{l}(\cdot )&\cdots &\varphi _{2,n_{l}}^{l}(\cdot )\\\vdots &\vdots &\ddots &\vdots \\\varphi _{n_{l+1},1}^{l}(\cdot )&\varphi _{n_{l+1},2}^{l}(\cdot )&\cdots &\varphi _{n_{l+1},n_{l}}^{l}(\cdot )\end{pmatrix}}}$

Functions used in KAN

The choice of functional basis strongly influences the performance of KANs. Common function families include:

• B-splines: Provide locality, smoothness, and interpretability; they are the most widely used in current implementations.^[3]

• RBFs: Capture localized features in data and are effective in approximating functions with non-linear or clustered structures.^[3][20]

• Chebyshev polynomials: Offer efficient approximation with minimized error in the maximum norm, making them useful for stable function representation.^[3][21]

• Rational functions: Useful for approximating functions with singularities or sharp variations, as they can model asymptotic behavior better than polynomials.^[3][22]

• Fourier series: Capture periodic patterns effectively and are particularly useful in domains such as physics-informed machine learning.^[3][23]

• Wavelet functions (DoG, Mexican hat, Morlet, Shannon): Used for feature extraction as they can capture both high-frequency and low-frequency data components.^[3][24][25]

• Piecewise linear functions: Provide efficient approximation for multivariate functions in KANs.^[26][27]

Usage

KANs are usually employed as drop-in replacements for MLP layers in modern neural architectures such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), and Transformers. While KANs are designed for general purposes, scientists have developed and applied them to a variety of tasks:

• Science Machine Learning (SciML): Function fitting^[1], partial differential equations (PDEs)^[1][28][29] and physical/mathematical laws.^[2]

• Continual learning: KANs better preserve previously learned information during incremental updates, avoiding catastrophic forgetting due to the locality of spline adjustments.^[2][30]

• Graph neural networks: Extensions such as Kolmogorov–Arnold Graph Neural Networks (KA-GNNs) integrate KAN modules into message-passing architectures, showing improvements in molecular property prediction tasks.^[3][31][32]

Drawbacks of KAN

KANs can be computationally intensive and require a large number of parameters due to their use of polynomial functions to capture data.^[33][34]

See also

• Kolmogorov–Arnold representation theorem
• Universal approximation theorem