A likelihood function (often simply called the likelihood) gives the relative merit of various statistical models for describing a data set. Often the models being compared are parameterized by a parameter, with the parameter often written as , or they are parameterized by multiple parameters given as the components of a vector. For a probability function (or probability density function) that gives the probability (or probability density) of data for a given model-specifying parameter , the likelihood is any function of equal to for some positive value .
In maximum likelihood estimation, the model parameter(s) or argument that maximizes the likelihood function serves as a point estimate for the unknown parameter, while the Fisher information (often approximated by the likelihood's Hessian matrix at the maximum) gives an indication of the estimate's precision.
In contrast, in Bayesian statistics, the estimate of interest is the converse of the likelihood, the so-called posterior probability of the parameter given the observed data, which is calculated via Bayes' rule.
Definition
The likelihood function, parameterized by a (possibly multivariate) parameter , is usually defined differently for discrete and continuous probability distributions (a more general definition is discussed below). Given a probability density or mass function
where is a realization of the random variable , the likelihood function is
often written
In other words, when is viewed as a function of with fixed, it is a probability density function, and when viewed as a function of with fixed, it is a likelihood function. In the frequentist paradigm, the notation is often avoided and instead or are used to indicate that is regarded as a fixed unknown quantity rather than as a random variable being conditioned on.
The likelihood function does not specify the probability that is the truth, given the observed sample . Such an interpretation is a common error, with potentially disastrous consequences (see prosecutor's fallacy).
Discrete probability distribution Let be a discrete random variable with probability mass function depending on a parameter . Then the function
considered as a function of , a possible value of the deterministic but unknown parameter , is the likelihood function, given the outcome of the random variable . Sometimes the probability of "the value of for the parameter value " is written as or . The likelihood is the probability that a particular outcome is observed when the true value of the parameter is , equivalent to the probability mass on ; it is not a probability density over the parameter . The likelihood, , should not be confused with , which is the posterior probability of given the data .
Example
Consider a simple statistical model of a coin flip: a single parameter that expresses the "fairness" of the coin. The parameter is the probability that a coin lands heads up ("H") when tossed. can take on any value within the range 0.0 to 1.0. For a perfectly fair coin, .
Imagine flipping a fair coin twice, and observing two heads in two tosses ("HH"). Assuming that each successive coin flip is i.i.d., then the probability of observing HH is
Equivalently, the likelihood of observing "HH" assuming is
This is not the same as saying that , a conclusion which could only be reached via Bayes' theorem given knowledge about the marginal probabilities and .
Now suppose that the coin is not a fair coin, but instead that . Then the probability of two heads on two flips is
Hence
More generally, for each value of , we can calculate the corresponding likelihood. The result of such calculations is displayed in Figure 1. The integral of over [0, 1] is 1/3; likelihoods need not integrate or sum to one over the parameter space.
Continuous probability distribution Let be a random variable following an absolutely continuous probability distribution with density function (a function of ) which depends on a parameter . Then the function
considered as a function of , is the likelihood function (of , given the outcome ). Again, is not a probability density or mass function over , despite being a function of given the observation .
Relationship between the likelihood and probability density functions
The use of the probability density in specifying the likelihood function above is justified as follows. Given an observation , the likelihood for the interval , where is a constant, is given by . Observe that
since is positive and constant. Because
where is the probability density function, it follows that
The first fundamental theorem of calculus provides that
Then
Therefore,
and so maximizing the probability density at amounts to maximizing the likelihood of the specific observation .
In general
In measure-theoretic probability theory, the density function is defined as the Radon–Nikodym derivative of the probability distribution relative to a common dominating measure. The likelihood function is this density interpreted as a function of the parameter, rather than the random variable. Thus, we can construct a likelihood function for any distribution, whether discrete, continuous, a mixture, or otherwise. (Likelihoods are comparable, e.g. for parameter estimation, only if they are Radon–Nikodym derivatives with respect to the same dominating measure.)
The above discussion of the likelihood for discrete random variables uses the counting measure, under which the probability density at any outcome equals the probability of that outcome.
Likelihoods for mixed continuous–discrete distributions The above can be extended in a simple way to allow consideration of distributions which contain both discrete and continuous components. Suppose that the distribution consists of a number of discrete probability masses and a density , where the sum of all the 's added to the integral of is always one. Assuming that it is possible to distinguish an observation corresponding to one of the discrete probability masses from one which corresponds to the density component, the likelihood function for an observation from the continuous component can be dealt with in the manner shown above. For an observation from the discrete component, the likelihood function for an observation from the discrete component is simply
where is the index of the discrete probability mass corresponding to observation , because maximizing the probability mass (or probability) at amounts to maximizing the likelihood of the specific observation.
The fact that the likelihood function can be defined in a way that includes contributions that are not commensurate (the density and the probability mass) arises from the way in which the likelihood function is defined up to a constant of proportionality, where this "constant" can change with the observation , but not with the parameter .
Regularity conditions In the context of parameter estimation, the likelihood function is usually assumed to obey certain conditions, known as regularity conditions. These conditions are in various proofs involving likelihood functions, and need to be verified in each particular application. For maximum likelihood estimation, the existence of a global maximum of the likelihood function is of the utmost importance. By the extreme value theorem, it suffices that the likelihood function is continuous on a compact parameter space for the maximum likelihood estimator to exist. While the continuity assumption is usually met, the compactness assumption about the parameter space is often not, as the bounds of the true parameter values might be unknown. In that case, concavity of the likelihood function plays a key role.
More specifically, if the likelihood function is twice continuously differentiable on the k-dimensional parameter space assumed to be an open connected subset of there exists a unique maximum if the matrix of second partials is negative definite for every at which the gradient vanishes, and if the likelihood function approaches a constant on the boundary of the parameter space, i.e.,
which may include the points at infinity if is unbounded. Mäkeläinen and co-authors prove this result using Morse theory while informally appealing to a mountain pass property. Mascarenhas restates their proof using the mountain pass theorem.
In the proofs of consistency and asymptotic normality of the maximum likelihood estimator, additional assumptions are made about the probability densities that form the basis of a particular likelihood function. These conditions were first established by Chanda. In particular, for almost all , and for all
exist for all in order to ensure the existence of a Taylor expansion. Second, for almost all and for every it must be that
where is such that This boundedness of the derivatives is needed to allow for differentiation under the integral sign. And lastly, it is assumed that the information matrix,
is positive definite and is finite. This ensures that the score has a finite variance.
The above conditions are sufficient, but not necessary. That is, a model that does not meet these regularity conditions may or may not have a maximum likelihood estimator of the properties mentioned above. Further, in case of non-independently or non-identically distributed observations additional properties may need to be assumed.
In Bayesian statistics, almost identical regularity conditions are imposed on the likelihood function in order to proof asymptotic normality of the posterior probability, and therefore to justify a Laplace approximation of the posterior in large samples.
Likelihood ratio and relative likelihood
Likelihood ratio
A likelihood ratio is the ratio of any two specified likelihoods, frequently written as:
The likelihood ratio is central to likelihoodist statistics: the law of likelihood states that the degree to which data (considered as evidence) supports one parameter value versus another is measured by the likelihood ratio.
In frequentist inference, the likelihood ratio is the basis for a test statistic, the so-called likelihood-ratio test. By the Neyman–Pearson lemma, this is the most powerful test for comparing two simple hypotheses at a given significance level. Numerous other tests can be viewed as likelihood-ratio tests or approximations thereof. The asymptotic distribution of the log-likelihood ratio, considered as a test statistic, is given by Wilks' theorem.
The likelihood ratio is also of central importance in Bayesian inference, where it is known as the Bayes factor, and is used in Bayes' rule. Stated in terms of odds, Bayes' rule states that the posterior odds of two alternatives, and , given an event , is the prior odds, times the likelihood ratio. As an equation:
The likelihood ratio is not directly used in AIC-based statistics. Instead, what is used is the relative likelihood of models (see below).
In evidence-based medicine, likelihood ratios are used in diagnostic testing to assess the value of performing a diagnostic test.
Relative likelihood function
Since the actual value of the likelihood function depends on the sample, it is often convenient to work with a standardized measure. Suppose that the maximum likelihood estimate for the parameter is . Relative plausibilities of other values may be found by comparing the likelihoods of those other values with the likelihood of . The relative likelihood of is defined to be (§9.3). (§1.4.2).Sprott, D. A. (2000), Statistical Inference in Science, Springer (chap. 2).Davison, A. C. (2008), Statistical Models, Cambridge University Press (§4.1.2). (§2.1).
Thus, the relative likelihood is the likelihood ratio (discussed above) with the fixed denominator . This corresponds to standardizing the likelihood to have a maximum of 1.
Likelihood region A likelihood region is the set of all values of whose relative likelihood is greater than or equal to a given threshold. In terms of percentages, a % likelihood region for is defined to be.