In deep learning, a multilayer perceptron (MLP) is a kind of modern feedforward neural network consisting of fully connected neurons with nonlinear activation functions, organized in layers, notable for being able to distinguish data that is not linearly separable.Cybenko, G. 1989. Approximation by superpositions of a sigmoidal function Mathematics of Control, Signals, and Systems, 2(4), 303–314.
Modern neural networks are trained using backpropagationRosenblatt, Frank. x. Principles of Neurodynamics: Perceptrons and the Theory of Brain Mechanisms. Spartan Books, Washington DC, 1961Rumelhart, David E., Geoffrey E. Hinton, and R. J. Williams. "Learning Internal Representations by Error Propagation". David E. Rumelhart, James L. McClelland, and the PDP research group. (editors), Parallel distributed processing: Explorations in the microstructure of cognition, Volume 1: Foundation. MIT Press, 1986. and are colloquially referred to as "vanilla" networks.Hastie, Trevor. Tibshirani, Robert. Friedman, Jerome. The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer, New York, NY, 2009. MLPs grew out of an effort to improve on single-layer perceptrons, which could only be applied to linearly separable data. A perceptron traditionally used a Heaviside step function as its nonlinear activation function. However, the backpropagation algorithm requires that modern MLPs use continuous activation functions such as sigmoid or ReLU.
Multilayer perceptrons form the basis of deep learning, and are applicable across a vast set of diverse domains.
Terminology
The name "multilayer perceptron" is a misnomer, as the network relies on neurons with continuous nonlinear activation functions (such as sigmoid, tanh, or rectified linear unit (ReLU)) rather than the Heaviside step function used in the original perceptron. Nevertheless, the name persists to describe feedforward neural networks with at least one hidden layer of fully connected neurons.
In deep learning terminology, MLPs are frequently referred to as deep feedforward networks, feedforward neural networks (FNNs), or simply dense networks. They are also colloquially called "vanilla" neural networks to distinguish them from more specialized architectures. The term "fully connected" distinguishes this architecture, where every neuron in one layer connects to every neuron in the next, from architectures with sparse connectivity such as convolutional neural networks. While the general term "neural network" encompasses all varied architectures (including recurrent and convolutional types), "MLP" specifically denotes the class of feedforward networks with dense connections and no cycles.
Timeline
In 1943, Warren McCulloch and Walter Pitts proposed the binary artificial neuron as a logical model of biological neural networks. In 1958, Frank Rosenblatt proposed the multilayered perceptron model, consisting of an input layer, a hidden layer with randomized weights that did not learn, and an output layer with learnable connections. In 1962, Rosenblatt published many variants and experiments on perceptrons in his book Principles of Neurodynamics, including up to 2 trainable layers by "back-propagating errors". However, it was not the backpropagation algorithm, and he did not have a general method for training multiple layers.
In 1965, Alexey Grigorevich Ivakhnenko and Valentin Lapa published Group Method of Data Handling. It was one of the first deep learning methods, used to train an eight-layer neural net in 1971.
In 1967, Shun'ichi Amari reported the first multilayered neural network trained by stochastic gradient descent, was able to classify non-linearily separable pattern classes. Amari's student Saito conducted the computer experiments, using a five-layered feedforward network with two learning layers.
Backpropagation was independently developed multiple times in early 1970s. The earliest published instance was Seppo Linnainmaa's master thesis (1970). Paul Werbos developed it independently in 1971, but had difficulty publishing it until 1982.
In 1986, David E. Rumelhart et al. popularized backpropagation.
In 2003, interest in backpropagation networks returned due to the successes of deep learning being applied to language modelling by Yoshua Bengio with co-authors.
In 2021, a very simple NN architecture combining two deep MLPs with skip connections and layer normalizations was designed and called MLP-Mixer; its realizations featuring 19 to 431 millions of parameters were shown to be comparable to vision transformers of similar size on ImageNet and similar image classification tasks.
Architecture
A multilayer perceptron is a feedforward neural network, meaning information flows in one direction from input to output with no cycles or loops. Unlike recurrent neural networks, which maintain internal state through feedback connections, or convolutional neural networks, which use sparse local connectivity, MLPs are characterized by full connectivity: every neuron in one layer connects to every neuron in the next layer. This architecture is also called a fully connected network or dense network.
Artificial neuron
Each artificial neuron in an MLP receives multiple inputs, processes them through a weighted sum, and applies a nonlinear activation function to produce an output. The neuron model consists of three main parts:
Input connections: Each input is associated with a synaptic weight that represents the strength of that connection. These weights are the learnable parameters of the network. Linear combiner: The neuron computes a weighted sum of its inputs, typically including a bias term :
Activation function: The weighted sum is passed through a nonlinear function to produce the neuron's output:
The bias term can be viewed as an adjustable threshold for neuron activation.
Layer structure
The MLP consists of three or more layers: an input layer, one or more hidden layers, and an output layer. The input layer receives the initial data signal (such as pixels of an image or features of a dataset) and passes it to the first hidden layer. It does not perform any computation.
The hidden layers are the computational core of the MLP. Each hidden layer consists of neurons that receive weighted inputs from the previous layer, apply a nonlinear activation function, and pass the result to the next layer. These layers are called "hidden" because their values are not directly observed in the training data, unlike inputs or outputs. Deep networks (those with many hidden layers) can learn hierarchical representations of data: lower layers detect simple features while higher layers represent more abstract concepts.
The output layer produces the final prediction. Its structure depends on the specific task: For regression (predicting a continuous value), it typically has a single neuron with a linear activation function. For binary classification (predicting yes/no), it uses a single neuron with a sigmoid activation. For multi-class classification, it usually has one neuron per class (e.g., 10 neurons for digit recognition), often using the softmax activation to produce a probability distribution.
Connectivity patterns
MLPs are defined by their fully connected (dense) topology: each neuron in layer connects to every neuron in layer . This architecture provides maximum representational flexibility, as the network can learn arbitrary input-output mappings. Using matrix notation, the output of a layer with input is computed as:
where is the weight matrix, is the bias vector, and is the activation function applied element-wise.
The number of parameters in a fully connected layer equals the product of the input and output dimensions, plus the bias terms. A layer connecting inputs to outputs contains learnable parameters. This quadratic scaling means MLPs require substantially more parameters than architectures with sparse connectivity, making them less practical for high-dimensional inputs such as images.
Activation functions
If a multilayer perceptron has a linear activation function in all neurons, that is, a linear function that maps the weighted inputs to the output of each neuron, then linear algebra shows that any number of layers can be reduced to a two-layer input-output model. In MLPs some neurons must use a nonlinear activation function to learn non-trivial relationships. Early nonlinear activation functions were developed to model the firing frequency of biological neurons.
Common activation functions include:
Sigmoid, : Historically common, it maps inputs to the range . It is often used in the output layer for binary classification but is less commonly used in hidden layers due to the vanishing gradient problem.
Hyperbolic Tangent, : Similar to the sigmoid but maps inputs to . Its zero-centered output often leads to faster convergence than the sigmoid during training.
Rectified linear unit (ReLU), : A common choice for hidden layers. Because its gradient is exactly 1 for positive inputs (and 0 otherwise), ReLU avoids the saturation problem and enables training of deeper networks. The related softplus function provides a smooth approximation.
Softmax: Typically found in the output layer for multi-class classification. It generalizes the logistic function to multiple dimensions, normalizing a vector of real values into a probability distribution where all values sum to 1.
Mathematical foundations Activation function If a multilayer perceptron has a linear activation function in all neurons, that is, a linear function that maps the weighted inputs to the output of each neuron, then linear algebra shows that any number of layers can be reduced to a two-layer input-output model. In MLPs some neurons use a nonlinear activation function that was developed to model the frequency of action potentials, or firing, of biological neurons.
The two historically common activation functions are both sigmoids, and are described by
.
The first is a hyperbolic tangent that ranges from −1 to 1, while the other is the logistic function, which is similar in shape but ranges from 0 to 1. Here is the output of the th node (neuron) and is the weighted sum of the input connections. Alternative activation functions have been proposed, including the rectifier and softplus functions. More specialized activation functions include radial basis functions (used in radial basis networks, another class of supervised neural network models).
In recent developments of deep learning the rectified linear unit (ReLU) is more frequently used as one of the possible ways to overcome the numerical problems related to the sigmoids.
Layers
The MLP consists of three or more layers (an input and an output layer with one or more hidden layers) of nonlinearly-activating nodes. Since MLPs are fully connected, each node in one layer connects with a certain weight to every node in the following layer.
Learning Learning occurs in the perceptron by changing connection weights after each piece of data is processed, based on the amount of error in the output compared to the expected result. This is an example of supervised learning, and is carried out through backpropagation, a generalization of the least mean squares algorithm in the linear perceptron.
We can represent the degree of error in an output node in the th data point (training example) by , where is the desired target value for th data point at node , and is the value produced by the perceptron at node when the th data point is given as an input.
The node weights can then be adjusted based on corrections that minimize the error in the entire output for the th data point, given by
.
Using gradient descent, the change in each weight is