Neural differential equation

Neural differential equations are a class of models in machine learning that combine neural networks with the mathematical framework of differential equations.^[1] These models provide an alternative approach to neural network design, particularly for systems that evolve over time or through continuous transformations.

The most common type, a neural ordinary differential equation (neural ODE), defines the evolution of a system's state using an ordinary differential equation whose dynamics are governed by a neural network: $${\displaystyle {\frac {\mathrm {d} \mathbf {h} (t)}{\mathrm {d} t}}=f_{\theta }(\mathbf {h} (t),t).}$$In this formulation, the neural network parameters θ determine how the state changes at each point in time.^[1] This approach contrasts with conventional neural networks, where information flows through discrete layers indexed by natural numbers. Neural ODEs instead use continuous layers indexed by positive real numbers, where the function ${\displaystyle h:\mathbb {R} _{\geq 0}\to \mathbb {R} }$ represents the network's state at any given layer depth t.

Neural ODEs can be understood as continuous-time control systems, where their ability to interpolate data can be interpreted in terms of controllability.^[2] They have found applications in time series analysis, generative modeling, and the study of complex dynamical systems.

Connection with residual neural networks

Neural ODEs can be interpreted as a residual neural network with a continuum of layers rather than a discrete number of layers.^[3] Applying the Euler method with a unit time step to a neural ODE yields the forward propagation equation of a residual neural network:

$${\displaystyle \mathbf {h} _{\ell +1}=f_{\theta }(\mathbf {h} _{\ell },\ell )+\mathbf {h} _{\ell },}$$

with ℓ being the ℓ-th layer of this residual neural network. While the forward propagation of a residual neural network is done by applying a sequence of transformations starting at the input layer, the forward propagation computation of a neural ODE is done by solving a differential equation. More precisely, the output ${\displaystyle \mathbf {h} _{\text{out}}}$ associated to the input ${\displaystyle \mathbf {h} _{\text{in}}}$ of the neural ODE is obtained by solving the initial value problem

$${\displaystyle {\frac {\mathrm {d} \mathbf {h} (t)}{\mathrm {d} t}}=f_{\theta }(\mathbf {h} (t),t),\quad \mathbf {h} (0)=\mathbf {h} _{\text{in}},}$$

and assigning the value ${\displaystyle \mathbf {h} (T)}$ to ${\displaystyle \mathbf {h} _{\text{out}}}$.

Universal differential equations

This section is about combinations of common and neural ordinary differential equations. For differential algebraic equations which can approximate any continuous function, see Universal differential equation.

In physics-informed contexts where additional information is known, neural ODEs can be combined with an existing first-principles model to build a physics-informed neural network model called universal differential equations (UDE).^[4][5][6][7] For instance, an UDE version of the Lotka-Volterra model can be written as^[8]

$${\displaystyle {\begin{aligned}{\frac {dx}{dt}}&=\alpha x-\beta xy+f_{\theta }(x(t),y(t)),\\{\frac {dy}{dt}}&=-\gamma y+\delta xy+g_{\theta }(x(t),y(t)),\end{aligned}}}$$

where the terms ${\displaystyle f_{\theta }}$ and ${\displaystyle g_{\theta }}$ are correction terms parametrized by neural networks.