Outline of deep learning

The following outline is provided as an overview of, and topical guide to, deep learning:

Deep learning is a subfield of machine learning and artificial intelligence based on artificial neural networks with multiple processing layers. It emphasizes representation learning and is widely used in areas such as computer vision, natural language processing, speech recognition, recommender systems, robotics, and generative artificial intelligence.

Ways to categorize deep learning A field of study A branch of artificial intelligence A subfield of machine learning A subfield of computer science A form of representation learning A class of methods based on artificial neural networks An approach used in computational statistics

History Precursors Cybernetics Perceptron Connectionism Neocognitron Backpropagation

Milestones LeNet Long short-term memory Deep belief network AlexNet Sequence to sequence learning Generative adversarial network Residual neural network Transformer BERT Generative pre-trained transformer Diffusion model

Related histories History of artificial intelligence History of machine learning Timeline of machine learning

Core concepts

Artificial neural network Representation learning Feature learning Gradient descent Backpropagation Loss function Optimization Training, validation, and test data sets Generalization Overfitting Underfitting Hyperparameter Hyperparameter optimization Foundation model Large language model

Learning settings

Supervised learning Unsupervised learning Self-supervised learning Semi-supervised learning Reinforcement learning Transfer learning Multitask learning Multimodal learning Online machine learning Continual learning

Common tasks Image classification Object detection Image segmentation Automatic speech recognition Neural machine translation Question answering Automatic summarization Text-to-image model Protein structure prediction

Architectures Feedforward and convolutional architectures Feedforward neural network Multilayer perceptron Convolutional neural network Radial basis function network Residual neural network U-Net

Recurrent and sequence architectures Recurrent neural network Long short-term memory Gated recurrent unit Sequence to sequence learning Recursive neural network

Representation-learning architectures Autoencoder Denoising autoencoder Sparse autoencoder Variational autoencoder Restricted Boltzmann machine Deep belief network

Attention and transformer architectures Attention (machine learning) Transformer BERT Generative pre-trained transformer Vision transformer

Generative and probabilistic architectures Autoregressive model Diffusion model Energy-based model Generative adversarial network Mixture of experts

Graph and memory architectures Graph neural network Graph convolutional network Siamese network Neural Turing machine Memory network Echo state network Capsule neural network

Neural network components and techniques

Artificial neuron Activation function Rectified linear unit Sigmoid function Softmax function Embedding Convolution Pooling layer Attention Batch normalization Layer normalization Residual connections

Training and optimization Backpropagation Gradient descent Stochastic gradient descent Adam optimization Learning rate Loss function Cross-entropy Mean squared error Regularization Dropout Early stopping Batch normalization Data augmentation Transfer learning Knowledge distillation Ensemble learning Curriculum learning

Datasets and benchmarks CIFAR-10 ImageNet MNIST database Common Objects in Context (COCO) General Language Understanding Evaluation (GLUE) benchmark LibriSpeech SQuAD

Applications

Computer vision Computer vision Facial recognition system Image classification Image segmentation Medical imaging Object detection Optical character recognition

Natural language processing Automatic summarization Chatbot Information retrieval Large language model Natural language processing Neural machine translation Question answering Sentiment analysis

Speech and audio

Automatic speech recognition Music information retrieval Speaker recognition Speech synthesis

Science and medicine Bioinformatics Computational biology Drug discovery Medical diagnosis Protein structure prediction

Robotics and control Autonomous car Computer game bot Control theory Robotics

Recommendation, search, and forecasting Anomaly detection Forecasting Fraud detection Recommender system Search engine

Generative artificial intelligence Deepfake Generative artificial intelligence Large language model Speech synthesis Text-to-image model

Computer graphics and video games Deep Learning Anti-Aliasing (DLAA) Deep Learning Super Sampling (DLSS)

Hardware AMD Instinct AMD XDNA Application-specific integrated circuit Deep learning processor, Neural processing unit (NPU), or Neural Engine Field-programmable gate array General-purpose computing on graphics processing units (GPGPU) Graphics processing unit NVIDIA Deep Learning Accelerator (NVDLA) Tensor processing unit Vision processing unit Wafer-scale integration

Supporting software platforms CUDA Metal ROCm

Software

Open-source frameworks and libraries

Apache MXNet Apache Singa BigDL Caffe Chainer Deeplearning4j DeepSpeed Dlib Fastai Flux JAX Horovod Keras Microsoft Cognitive Toolkit MindSpore mlpack PyTorch PyTorch Lightning TensorFlow Theano Torch

Neural network software

EDLUT Emergent Encog JOONE Neuroph NeuroSolutions OpenNN Peltarion Synapse SNNS

Platforms, tools, and deployment Amazon SageMaker Google Colab Hugging Face Kaggle Kubeflow MLflow ONNX OpenVINO TensorFlow Hub

Algorithms for deep learning and neural networks

Backpropagation Conjugate gradient method Generalized Hebbian algorithm Gradient descent Levenberg–Marquardt algorithm Perceptron Quasi-Newton method Wake-sleep algorithm

Methods and related topics Representation and metric learning Contrastive learning Embedding Feature learning Manifold learning Metric learning

Generative modeling Autoregressive model Diffusion model Generative adversarial network Generative model Variational inference

Efficient and scalable deep learning Knowledge distillation Low-rank approximation Mixture of experts Quantization Sparsity

Reliability, safety, and interpretability Adversarial machine learning AI alignment Algorithmic bias Catastrophic forgetting Differential privacy Explainable artificial intelligence Federated learning Hallucination (artificial intelligence)

Conferences and workshops Annual Meeting of the Association for Computational Linguistics Conference on Computer Vision and Pattern Recognition Conference on Neural Information Processing Systems International Conference on Computer Vision International Conference on Learning Representations International Conference on Machine Learning

Organizations

Research laboratories and institutions Allen Institute for AI Alberta Machine Intelligence Institute European Laboratory for Learning and Intelligent Systems Google DeepMind Meta AI Mila Microsoft Research Vector Institute

Companies Anthropic Cerebras Cohere DeepSeek Mistral AI OpenAI Stability AI xAI

Publications

Books Deep Learning – Ian Goodfellow and Yoshua Bengio Neural Networks and Deep Learning – Michael Nielsen Perceptrons – Marvin Minsky and Seymour Papert

Journals

IEEE Transactions on Neural Networks and Learning Systems Neural Networks Neural Computation

Influential persons

Alex Graves Alex Krizhevsky Andrew Ng Andrej Karpathy Ashish Vaswani Christopher Bishop Demis Hassabis Fei-Fei Li Geoffrey Hinton Ian Goodfellow Ilya Sutskever John Hopfield Jürgen Schmidhuber Noam Shazeer Oriol Vinyals Paul Werbos Quoc V. Le Ruslan Salakhutdinov Sepp Hochreiter Seppo Linnainmaa Terry Sejnowski Yann LeCun Yoshua Bengio

Outline of deep learning

Bron en licentie