In machine learning, a neural network (NN) or artificial neural network (ANN) is a computational model inspired by the structure and functions of biological neural networks.

A neural network consists of connected units or nodes called artificial neurons, which loosely model the neurons in the brain. These are connected by edges, which model the synapses in the brain. Each artificial neuron receives signals from connected neurons, then processes them and sends a signal to other connected neurons. The "signal" is a real number, and the output of each neuron is computed by some non-linear function of the totality of its inputs, called the activation function. The strength of the signal at each connection is determined by a weight, which adjusts as part of the training process.

Groups of neurons are aggregated into layers. Each layer performs a transformation on its inputs. Signals travel from the first layer (the input layer) to the last layer (the output layer), typically passing through multiple intermediate layers (hidden layers). A network is typically called a deep neural network if it has at least two hidden layers. Deep neural networks are capable of learning sophisticated hierarchical representations.

Training neural networks is a compute-intensive process, accelerated by the use of graphics processing units (GPUs), and large datasets.

In reality, such textures and outlines would not be represented by single nodes, but rather by associated weight patterns of multiple nodes.

Architectural innovations such as convolutional neural networks (CNNs) significantly improved performance in computer vision tasks, while recurrent neural networks (RNNs) enabled modeling of sequential data such as speech and time-series information. Transformer architectures introduced attention mechanisms that allow neural networks to model long-range dependencies in data and have been the basis of large language models.

Artificial neural networks are used for myriad tasks including chatbots, large-scale text, image, and video generation, and robotics.

History

Mathematical foundations Deep neural networks are based on statistics developed over 200 years ago. The simplest kind of feedforward neural network (FNN) is a linear network, which consists of a single layer of output nodes with linear activation functions; the inputs are fed directly to the outputs via weights. The sum of the products of the weights and the inputs is calculated at each node. The mean squared errors between these calculated outputs and the given target values are minimized by adjusting to the weights. This technique is the method of least squares or linear regression. It was used to find a rough linear fit to a set of points by Legendre (1805) and Gauss (1795) for the prediction of planetary movement.

Perceptrons Computers are based on John von Neumann's model. They execute explicit lists of instructions with access to memory to record their changing state. Neural networks instead originated from efforts to model information processing in biological systems via connectionism. Unlike the von Neumann model, connectionist computing does not separate memory and processing.

Warren McCulloch and Walter Pitts (1943) considered a non-learning computational model for neural networks. This model paved the way for research to split into one branch focused on biological processes and another focused on artificial intelligence. McCulloch and Pitts also developed mathematical models of artificial neurons capable of representing logical functions.

In the late 1940s, D. O. Hebb proposed a learning hypothesis based on neural plasticity that became known as Hebbian learning. It was used in many early neural network experiments, such as Rosenblatt's perceptron and the Hopfield network. Farley and Clark (1954) used computational machines to simulate a Hebbian network. Other neural networks computational machines were created by Rochester, Holland, Habit and Duda (1956).

In 1958, psychologist Frank Rosenblatt described the perceptron, one of the first implemented neural networks,Haykin (2008) Neural Networks and Learning Machines, 3rd edition funded by the United States Office of Naval Research. R. D. Joseph (1960) mentioned an earlier perceptron-like device by B. G. Farley and W. A. Clark of the MIT Lincoln Laboratory; however, according to Joseph, "they dropped the subject."

The first perceptrons did not have adaptive hidden units. However, Joseph (1960) discussed multilayer that did. Rosenblatt (1962) cited and adopted these ideas, crediting work by H. D. Block and B. W. Knight. However, these early efforts did not lead to a working learning algorithm for hidden units, i.e., deep learning.

The perceptron raised public excitement in neural networks, causing the US government to drastically increase funding. This contributed to "the Golden Age of AI", fueled by the optimistic claims made by computer scientists regarding the ability of perceptrons to emulate human intelligence.

Historical foundations and the Dartmouth proposal Artificial neural networks were identified as a promising direction for artificial intelligence research in the 1955 proposal for the Dartmouth Summer Research Project on Artificial Intelligence. Neural network models initially faced major limitations. Hardware constraints limited network size and training efficiency, while theoretical understanding of learning algorithms remained incomplete. Many models used single-layer perceptrons, which were restricted to solving linearly separable problems. These limitations were highlighted in the book Perceptrons by Marvin Minsky and Seymour Papert, which deflated interest during the late 1960s and 1970s.

1960s and 1970s Fundamental research was conducted on NNs in the 1960s and 1970s. The first working deep learning algorithm was the group method of data handling, a method to train arbitrarily deep neural networks, published by Alexey Ivakhnenko and Valentin Lapa in the Soviet Union (1965). They regarded it as a form of polynomial regression, generalizing Rosenblatt's perceptron. A 1971 paper described a deep network with eight layers trained by this method, training layer by layer via regression analysis. Superfluous hidden units were pruned using a separate validation set. The activation functions of the nodes were Kolmogorov-Gabor polynomials, the first deep networks with multiplicative units or "gates".

The first deep learning multilayer perceptron (MLP) trained by stochastic gradient descent was published in 1967 by Shun'ichi Amari. In computer experiments conducted by Amari's student S. Saito, a five layer MLP with two modifiable layers learned internal representations to classify non-linearily separable pattern classes. Subsequent developments in hardware and hyperparameter tuning made end-to-end stochastic gradient descent the dominant technique for reducing loss (error).

In 1969, Kunihiko Fukushima introduced the ReLU (rectified linear unit) activation function. ReLU is the most common activation function.

Nevertheless, research stagnated in the United States after Minsky and Papert (1969), who emphasized that perceptrons were incapable of processing the exclusive-or circuit. However, this insight was irrelevant for the deep networks of Ivakhnenko (1965) and Amari (1967).

In 1976, transfer learning was introduced.

Backpropagation Interest in neural networks revived during the 1980s because of the novel backpropagation algorithm, which allowed multi-layer neural networks to be trained efficiently by propagating error gradients backward (from output back to input) through network layers. Backpropagation is an efficient application of the chain rule derived by Gottfried Wilhelm Leibniz in 1673 to networks of differentiable nodes. The terminology "back-propagating errors" was introduced in 1962 by Rosenblatt, but he did not describe how to implement this. Henry J. Kelley developed a precursor of backpropagation in 1960 in the context of control theory. In 1970, Seppo Linnainmaa published the modern form of backpropagation in his master's thesis (1970). G.M. Ostrovski et al. republished it in 1971.Ostrovski, G.M., Volin, Y.M., and Boris, W.W. (1971). On the computation of derivatives. Wiss. Z. Tech. Hochschule for Chemistry, 13:382–384. Paul Werbos applied backpropagation to neural networks in 1982 (his 1974 PhD thesis, reprinted in a 1994 book, did not describe the algorithm). In 1986, David E. Rumelhart, et al., popularized backpropagation but did not cite the original work.

Convolutional neural networks Deep learning architectures for convolutional neural networks (CNNs) with convolutional layers and downsampling layers and weight replication began with the neocognitron introduced by Kunihiko Fukushima in 1979. Fukushima's CNN architecture also introduced max pooling, a popular downsampling procedure for CNNs. CNNs have become an essential tool for computer vision.

The time delay neural network (TDNN) was introduced in 1987 by Alex Waibel to apply CNNs to phoneme recognition. It used convolutions, weight sharing, and backpropagation.Alexander Waibel et al., Phoneme Recognition Using Time-Delay Neural Networks IEEE Transactions on Acoustics, Speech, and Signal Processing, Volume 37, No. 3, pp. 328. – 339 March 1989. In 1988, Wei Zhang applied a backpropagation-trained CNN to recognizing individual letters. In 1989, Yann LeCun et al. created a CNN called LeNet for recognizing handwritten ZIP codes on mail. Training required 3 days. In 1990, Wei Zhang implemented a CNN on optical computing hardware. In 1991, a CNN was applied to medical image object segmentation and breast cancer detection in mammograms. LeNet-5 (1998), a 7-level CNN by Yann LeCun et al. that classifies hand-written digits, was applied by banks to recognize numbers on checks digitized in 32×32 pixel images.

From 1988 onward,Qian, Ning, and Terrence J. Sejnowski. "Predicting the secondary structure of globular proteins using neural network models." Journal of molecular biology 202, no. 4 (1988): 865–884.Bohr, Henrik, Jakob Bohr, Søren Brunak, Rodney MJ Cotterill, Benny Lautrup, Leif Nørskov, Ole H. Olsen, and Steffen B. Petersen. "Protein secondary structure and homology by neural networks The α-helices in rhodopsin." FEBS letters 241, (1988): 223–228 the use of neural networks transformed the field of protein structure prediction, in particular when the first cascading networks were trained on profiles (matrices) produced by multiple sequence alignments.Rost, Burkhard, and Chris Sander: "Prediction of protein secondary structure at better than 70% accuracy." Journal of molecular biology 232, no. 2 (1993): 584–599.

Recurrent neural networks One source of RNN was statistical mechanics. In 1972, Shun'ichi Amari proposed to modify the weights of an Ising model by Hebbian learning rule as a model of associative memory, adding in learning. This was popularized as the Hopfield network by John Hopfield (1982). Another origin of RNN was neuroscience. The word "recurrent" is used to describe loop-like structures in anatomy. In 1901, Cajal observed "recurrent semicircles" in the cerebellar cortex. Hebb considered "reverberating circuit" as an explanation for short-term memory. The McCulloch and Pitts paper (1943) considered neural networks that contain cycles, and noted that the current activity of such networks can be affected by activity indefinitely far in the past.