Artificial neural networks (ANNs) are models created using machine learning to perform a number of tasks. While the computational implementations of ANNs relate to earlier discoveries in mathematics, their creation was inspired by biological neural circuitry. The first implementation of ANNs was the perceptron by Frank Rosenblatt. Despite the frequent claim that little research was conducted on ANNs in the 1970s and 1980s—the AAAI calling this period an "AI winter"—there was ongoing development of ANNs, especially connected to the development of specialized computers for simulating these networks.

Later, advances in hardware and the development of the backpropagation algorithm, as well as recurrent neural networks and convolutional neural networks, renewed interest in ANNs. The 2010s saw the development of a deep neural network (i.e., one with many layers) called AlexNet. It greatly outperformed other image recognition models, and is thought to have launched the ongoing AI spring. The transformer architecture was first described in 2017 as a method to teach ANNs grammatical dependencies in language, and is the predominant architecture used by large language models such as GPT-4. Diffusion models were first described in 2015, and became the basis of image generation models such as DALL-E in the 2020s.

Mathematical foundations Jürgen Schmidhuber suggests that the first neural network was the method of linear regression by least squares, first published by Adrien-Marie Legendre in 1805Merriman, Mansfield. A List of Writings Relating to the Method of Least Squares: With Historical and Critical Notes. Vol. 4. Academy, 1877. and independently developed by Friedrich Gauss (who claimed use since 1795) and Robert Adrain (1808), as it is mathematically equivalent to a two layer neural network without activation functions. The chain rule, developed by Gottfried Wilhelm Leibniz in 1676, and gradient descent, independently proposed by Augustin-Louis Cauchy in 1847 and Jacques Hadamard in 1907, are also central to the development of neural networks, as they form the basis of techniques for updating weights within complex networks.

Biological and computational models

Artificial neural network development of the 20th century was primarily based on research into the function of biological neurons. The neuron's place as the primary functional unit of the nervous system was first recognised in the late 19th century through the work of Santiago Ramón y Cajal, notably through his 1888 paper presenting staining of axons in the cerebellum of birds. Alexander Bain's Mind and Body (1873) proposed that thoughts and bodily activity result from neuronal processes, with each thought corresponding to a distinct neural grouping. William James's The Principles of Psychology (1890) advanced two principles on a quasi-neurological basis: first, that when two brain processes are active together, one tends to propagate excitement into the other, and second, that activity at any brain point is the sum of tendencies from all other points discharging into it.

Warren McCulloch and Walter Pitts's 1943 paper "A Logical Calculus of the Ideas Immanent in Nervous Activity" studied several abstract models for neural networks, using the symbolic logic of Rudolf Carnap and Principia Mathematica. The paper argued that several abstract models of neural networks (some learning, some not) have the same computational power as Turing machines. This model paved the way for research to split into two approaches: one focused on biological processes, while the other focused on the application of neural networks to artificial intelligence. This also led to work on nerve networks and their link to finite automata. Some consider McCulloch and Pitts to be the founders of connectionism, a theory of mind in opposition to classical computationalism.

In his 1948 report "Intelligent Machinery", published posthumously in 1969, Alan Turing proposed randomly connected networks of neuron-like nodes trainable through "education", defining A-type machines with random networks of NAND gates and B-type machines with modifiable connections.

In 1949, the psychologist Donald O. Hebb published The Organization of Behavior, which proposed a learning hypothesis based on the mechanism of neural plasticity which became known as Hebbian learning, summarized as "neurons that fire together, wire together". Similar observations were made by Jerzy Konorski in 1948. The concept was used in many early neural networks, such as Rosenblatt's perceptron and the Hopfield network. This evolved into models for long-term potentiation.

Belmont Farley and Wesley A. Clark (1954) were the first to use computational machines to simulate a Hebbian network. Other neural network computational machines were simulated by Nathaniel Rochester, John Holland, Lois Haibt and William Duda (1956).

In 1959, a biological model was proposed by David H. Hubel and Torsten Wiesel based on their discovery of two types of cells in the primary visual cortex: simple cells and complex cells.

Perceptrons and other early neural networks

The perceptron was created by Frank Rosenblatt in 1957 while working at the Cornell Aeronautical Laboratory, publishing the details the following year. The perceptron was designed to classify objects into two categories, updating based on error feedback. He initially simulated the perceptron on an IBM 704, later designing the Mark I Perceptron, the first hardware neural net. In 1958, Rosenblatt proposed the multilayer perceptron (MLP) model, consisting of an input layer, a hidden non-learning layer with randomised weights, and an output layer with learnable connections. He published the book Principles of Neurodynamics in 1962, which also introduced variants and computer experiments, including a version (developed alongside Henry David Block and Bruce Knight) with four-layer perceptrons where the last two layers have learned weights.Block, H. D., Knight, B. W., Jr., Rosenblatt, F. (1962) Analysis of a four-layer series-coupled perceptron. II. Rev. Mod. Phys. 34: 135

Bernard Widrow and his doctoral student Marcian Hoff developed ADALINE (Adaptive Linear Neuron) in 1960. Unlike Rosenblatt's perceptron, ADALINE adjusted weights based on their least mean squares (LMS) algorithm before applying the threshold function. MADALINE, the multilayer extension, was used to eliminate echo on phone lines, likely the first artificial neural network applied to a real‑world engineering problem.

Multi-layer learning Group method of data handling, a method to train arbitrarily deep neural networks, was published by Alexey Ivakhnenko and Valentin Lapa in 1965; they regarded it as a form of polynomial regression or a generalisation of Rosenblatt's perceptron. A 1971 paper described a deep network with the equivalent of eight layers trained by this method.

The first deep learning multilayer perceptron trained by stochastic gradient descent was published in 1967 by Shun'ichi Amari. According to Amari, in computer experiments conducted by his student Saito, a five layer MLP with two modifiable layers learned internal representations to classify non-linearly separable pattern classes. Subsequent developments in hardware and hyperparameter tunings have made end-to-end stochastic gradient descent the currently dominant training technique.

Backlash and AI winter

In 1969, Marvin Minsky and Seymour Papert published Perceptrons with the goal of showing the limitations of neural network systems. They proved single-layer perceptrons cannot compute non-linearly separable functions such as XOR. They also demonstrated that certain problems (e.g. determining parity) are impossible for single-layer networks under restriction of "conjunctive locality", and the maximum number of connections required to compute others (e.g. connectedness) grows arbitrarily large with input size.

Despite these limitations not applying to multi-layered perceptrons, there was a subsequent decrease in funding from American and British agencies given to neural network projects in favour of symbolic AI, and a reduction in the number of computer scientists working in the field.

Backpropagation

Up until the 1970s, neural networks were limited by their capacity to learn and update their neurons. The "learning rule" used by Rosenblatt for the perceptron only allowed for training a single layer of a neural network. The terminology "back-propagating errors" was introduced by Rosenblatt in 1962 to describe a (hypothetical) multilayer generalisation of his perceptron learning algorithm. The aforementioned least mean squares (LMS) algorithm, also known as the Widrow–Hoff learning rule or the delta rule, was more general but still limited to single layers.

Backpropagation is an efficient application of the chain rule derived by Gottfried Wilhelm Leibniz in 1673 to networks of differentiable nodes. Henry J. Kelley had a continuous precursor of backpropagation in 1960 in the context of control theory, also discovered by Arthur E. Bryson independently around the same time. He presented a form of gradient descent to solve problems where neurons have continuous output, as opposed to the discrete (binary) output of existing neural networks.

The modern form of backpropagation was developed multiple times in early 1970s. The earliest published instance was Seppo Linnainmaa's 1970 master thesis. His FORTRAN code efficiently computed the derivatives of nested, differentiable functions by caching intermediate steps, used to calculate arithmetic rounding errors for the results of complex expressions. He published some of his results in English in 1976. Paul Werbos developed it independently in 1971 or 1972, published in his PhD thesis in 1974. In 1982, he became the first person to apply backpropagation to neural networks. In 1986, David E. Rumelhart et al. popularised backpropagation.

Recurrent network architectures

One origin of the recurrent neural network (RNN) was statistical mechanics. The Ising model was developed by Wilhelm Lenz and Ernst Ising in the 1920s as a simple statistical mechanical model of magnets at equilibrium. Glauber in 1963 studied the Ising model evolving in time, as a process towards equilibrium (Glauber dynamics), adding in the component of time. Shun'ichi Amari in 1972 proposed to modify the weights of an Ising model by Hebbian learning rule as a model of associative memory, adding in the component of learning. This was popularised as the Hopfield network (1982).

Another origin of RNN was neuroscience. The word "recurrent" is used to describe loop-like structures in anatomy. In 1901, Santiago Ramón y Cajal observed "recurrent semicircles" in the cerebellar cortex. In 1933, Rafael Lorente de Nó discovered "recurrent, reciprocal connections" by Golgi's method, and proposed that excitatory loops explain certain aspects of the vestibulo-ocular reflex. Hebb considered "reverberating circuits" as an explanation for short-term memory. considered neural networks that contains cycles, and noted that the current activity of such networks can be affected by activity indefinitely far in the past.

Two early influential works were the Jordan network (1986) and the Elman network (1990), which applied RNN to study cognitive psychology. In 1993, a "neural history compressor" system by Schmidhuber solved a task that required more than 1000 subsequent layers in an RNN unfolded in time. Page 150 ff demonstrates credit assignment across the equivalent of 1,200 layers in an unfolded RNN.

Long short-term memory