Pruning (artificial neural network)

For other uses, see Pruning (disambiguation).

In deep learning, pruning is the practice of removing parameters from an existing artificial neural network.^[1] The goal of this process is to reduce the size (parameter count) of the neural network (and therefore the computational resources required to run it) whilst maintaining accuracy. This can be compared to the biological process of synaptic pruning which takes place in mammalian brains during development.^[2]

Node (neuron) pruning

A basic algorithm for pruning is as follows:^[3][4]

1. Evaluate the importance of each neuron.
2. Rank the neurons according to their importance (assuming there is a clearly defined measure for "importance").
3. Remove the least important neuron.
4. Check a termination condition (to be determined by the user) to see whether to continue pruning.

Edge (weight) pruning

Most work on neural network pruning focuses on removing weights, namely, setting their values to zero. Early work suggested to also change the values of non-pruned weights.^[5]

See also

• Knowledge distillation
• Neural Darwinism