Machine learning in biology is the application of machine learning algorithms to biology, especially in computational biology and bioinformatics, including genomics, proteomics, microarrays, systems biology, evolution, and text mining.
Prior to the emergence of machine learning, bioinformatics algorithms had to be programmed by hand; for problems such as protein structure prediction, this proved difficult. Machine learning techniques such as deep learning can learn features of data sets rather than requiring the programmer to define them individually. The algorithm can further learn how to combine low-level features into more abstract features, and so on. This multi-layered approach allows such systems to make sophisticated predictions when appropriately trained. These methods contrast with other computational biology approaches which, while exploiting existing datasets, do not allow the data to be interpreted and analyzed in unanticipated ways.
Tasks Machine learning algorithms in bioinformatics can be used for prediction, classification, and feature detection and selection. Classification or recognition algorithms output a categorical class, while prediction algorithms output a numerical valued feature.
Algorithms differ in the type of process used to build the predictive models from data using analogies, rules, neural networks, probabilities, and/or statistics.
Advances in information technologies, artificial intelligence, and data mining, together with the growing availability of open data sets, have enabled the development of methods that can identify patterns and learn from data.
Approaches
Artificial neural networks Artificial neural networks in bioinformatics have been used for:
Comparing and aligning RNA, protein, and DNA sequences. Identification of promoters and finding genes from sequences related to DNA. Interpreting the expression-gene and micro-array data. Identifying the network (regulatory) of genes. Learning evolutionary relationships by constructing phylogenetic trees. Classifying and predicting protein structure. Molecular design and docking
Feature engineering The way that features, often vectors in a many-dimensional space, are extracted from the domain data is an important component of learning systems. In genomics, a typical representation of a sequence is a vector of k-mers frequencies, which is a vector of dimension whose entries count the appearance of each subsequence of length in a given sequence. Since for a value as small as the dimensionality of these vectors is huge (e.g. in this case the dimension is ), techniques such as principal component analysis are used to project the data to a lower dimensional space, thus selecting a smaller set of features from the sequences.
Classification
In this type of machine learning task, the output is a discrete variable. One example of this type of task in bioinformatics is labeling new genomic data (such as genomes of unculturable bacteria) based on a model of already labeled data.
Hidden Markov models Hidden Markov models (HMMs) are a class of statistical models for sequential data (often related to systems evolving over time). An HMM is composed of two mathematical objects: an observed state‐dependent process , and an unobserved (hidden) state process . In an HMM, the state process is not directly observed – it is a 'hidden' (or 'latent') variable – but observations are made of a state‐dependent process (or observation process) that is driven by the underlying state process (and which can thus be regarded as a noisy measurement of the system states of interest). HMMs can be formulated in continuous time.
HMMs can be used to profile and convert a multiple sequence alignment into a position-specific scoring system suitable for searching databases for homologous sequences remotely. Additionally, ecological phenomena can be described by HMMs.
Convolutional neural networks Convolutional neural networks (CNN) are a class of deep neural network whose architecture is based on shared weights of convolution kernels or filters that slide along input features, providing translation-equivariant responses known as feature maps. CNNs take advantage of the hierarchical pattern in data and assemble patterns of increasing complexity using smaller and simpler patterns discovered via their filters.
Convolutional networks were inspired by biological processes in that the connectivity pattern between neurons resembles the organization of the animal visual cortex. Individual cortical neurons respond to stimuli only in a restricted region of the visual field known as the receptive field. The receptive fields of different neurons partially overlap such that they cover the entire visual field.
CNN uses relatively little pre-processing compared to other image classification algorithms. This means that the network learns to optimize the filters (or kernels) through automated learning, whereas in traditional algorithms these filters are hand-engineered. This reduced reliance on prior knowledge of the analyst and on human intervention in manual feature extraction makes CNNs a desirable model.
A phylogenetic convolutional neural network (Ph-CNN) is a convolutional neural network architecture proposed by Fioranti et al. in 2018 to classify metagenomics data. In this approach, phylogenetic data is endowed with patristic distance (the sum of the lengths of all branches connecting two operational taxonomic units [OTU]) to select k-neighborhoods for each OTU, and each OTU and its neighbors are processed with convolutional filters.
Self-supervised learning (Attention and Transformer Models)
Unlike supervised methods, self-supervised learning methods learn representations without relying on annotated data. That is well-suited for genomics, where high throughput sequencing techniques can create potentially large amounts of unlabeled data. Some examples of self-supervised learning methods applied on genomics include DNABERT and Self-GenomeNet.
Because of their parallelism and the ability to extract correlation across the whole sequences, transformer-based models achieve state-of-the-art performance in a variety of important tasks such as machine translation and question answering. The vanilla transformer model can be divided into two parts: encoder and decoder, which have similar basic architectures composed of a stack of identical blocks. Each block consists of two kinds of sub-layers: the multi-head attention sub-layer and the position-wise feed-forward sub-layer. Both kinds of sublayers are followed by layer normalization. A residual connection around every sub-layer will be applied in each block to speed up the training process.
Attention modules
The key innovation in the Transformer architecture is the multi-head self-attention layer, which can relate all relevant tokens to better encode every word or residue in the input sequence. The self-attention layer takes a sequence of tokens as input (tokens equivalent to words in a language or amino acids/nucleotides in a sequence) and learns sequence-wide context information. Multi-head attention represents multiple simultaneous attention heads. Before calculating the attention function, each token embedding is transformed into three corresponding vectors: the , the , and the vectors. This transformation is achieved by multiplying the token embedding with three randomly initialized, learnable parameter matrices, , , and .
The core attention function is computed by three steps: Scoring: The attention head computes the dot products of the Query vector with all Key vectors. Scaling and Weighting: Each dot product is divided by (where is the dimension of the key vector) and a Softmax function is applied to obtain the weights on the Value vectors. Output: The output of the attention function is the weighted sum of these Value vectors, which contains information for the entire sequence. The weight assigned to each value is computed by a compatibility function of the Query with the corresponding Key.
In the parallel computation of the attention function, a set of query, key, and value vectors are packed into matrices , , and . The attention function is computed as follows:
When generalized to multi-head attention with heads, the results of the multiple heads (each assigned different parameters , , ) are concatenated and once again projected with a parameter matrix , resulting in the final output:
Position-wise feed-forward networks Except for the attention sub-layer, each block of the encoder and decoder contains a fully connected feed-forward network (FFN), which is applied identically and independently to each token (position-wise). This layer consists of two linear transformations with a Rectified Linear Unit (ReLU) activation function in the middle. The FFN is computed as follows: Here, and are the learnable parameters of the network. This layer is responsible for translating the output of the attention mechanism into a form suitable for the next layer.
Residual connection and layer normalization Each encoder and decoder block contains two residual connections and two layer normalization layers. These are applied to both the multi-head self-attention output and the feed-forward network (FFN) output. Layer normalization can accelerate the training process of the model by normalizing the output of the former layers to make it converge faster. The formulas are as follows: The variable represents the input of the multi-head self-attention or FFN. This original input is added to the sub-layer's output, forming a residual connection. For deep networks, the residual connection helps defend against vanishing and exploding gradients by keeping the original input signal, which stabilizes the training process.
Random forest
Random forests (RF) classify by constructing an ensemble of decision trees, and outputting the average prediction of the individual trees. This is a modification of bootstrap aggregating (which aggregates a large collection of decision trees) and can be used for classification or regression.
As random forests give an internal estimate of generalization error, cross-validation is unnecessary. In addition, they produce proximities, which can be used to impute missing values, and which enable novel data visualizations.
Computationally, random forests are appealing because they naturally handle both regression and (multiclass) classification, are relatively fast to train and to predict, depend only on one or two tuning parameters, have a built-in estimate of the generalization error, can be used directly for high-dimensional problems, and can easily be implemented in parallel. Statistically, random forests are appealing for additional features, such as measures of variable importance, differential class weighting, missing value imputation, visualization, outlier detection, and unsupervised learning.
Clustering Clustering - the partitioning of a data set into disjoint subsets, so that the data in each subset are as close as possible to each other and as distant as possible from data in any other subset, according to some defined distance or similarity function - is a common technique for statistical data analysis.
Clustering is central to much data-driven bioinformatics research and serves as a powerful computational method whereby means of hierarchical, centroid-based, distribution-based, density-based, and self-organizing maps classification, has long been studied and used in classical machine learning settings. Particularly, clustering helps to analyze unstructured and high-dimensional data in the form of sequences, expressions, texts, images, and so on. Clustering is also used to gain insights into biological processes at the genomic level, e.g. gene functions, cellular processes, subtypes of cells, gene regulation, and metabolic processes.
Clustering algorithms used in bioinformatics