Coexistence theory is a framework to understand how competitor traits can maintain species diversity and stave-off competitive exclusion even among similar species living in ecologically similar environments. Coexistence theory explains the stable coexistence of species as an interaction between two opposing forces: fitness differences between species, which should drive the best-adapted species to exclude others within a particular ecological niche, and stabilizing mechanisms, which maintains diversity via niche differentiation. For many species to be stabilized in a community, population growth must be negative density-dependent, i.e. all participating species have a tendency to increase in density as their populations decline. In such communities, any species that becomes rare will experience positive growth, pushing its population to recover and making local extinction unlikely. As the population of one species declines, individuals of that species tend to compete predominantly with individuals of other species. Thus, the tendency of a population to recover as it declines in density reflects reduced intraspecific competition (within-species) relative to interspecific competition (between-species), the signature of niche differentiation (see Lotka-Volterra competition).

Types of coexistence mechanisms

Two qualitatively different processes can help species to coexist: a reduction in average fitness differences between species or an increase in niche differentiation between species. These two factors have been termed equalizing and stabilizing mechanisms, respectively. For species to coexist, any fitness differences that are not reduced by equalizing mechanisms must be overcome by stabilizing mechanisms.

Equalizing mechanisms Equalizing mechanisms reduce fitness differences between species. As its name implies, these processes act in a way that push the competitive abilities of multiple species closer together. Equalizing mechanisms affect interspecific competition (the competition between individuals of different species).

For example, when multiple species compete for the same resource, competitive ability is determined by the minimum level of resources a species needs to maintain itself (known as an R*, or equilibrium resource density). Thus, the species with the lowest R* is the best competitor and excludes all other species in the absence of any niche differentiation. Any factor that reduces R*s between species (like increased harvest of the dominant competitor) is classified as an equalizing mechanism.

Environmental variation (which is the focus of the Intermediate Disturbance Hypothesis) can be considered an equalizing mechanism. Since the fitness of a given species is intrinsically tied to a specific environment, when that environment is disturbed (e.g. through storms, fires, volcanic eruptions, etc.) some species may lose components of their competitive advantage which were useful in the previous version of the environment.

Stabilizing mechanisms Stabilizing mechanisms promote coexistence by concentrating intraspecific competition relative to interspecific competition. In other words, these mechanisms "encourage" an individual to compete more with other individuals of its own species, rather than with individuals of other species. Resource partitioning (a type of niche differentiation) is a stabilizing mechanism because interspecific competition is reduced when different species primarily compete for different resources. Similarly, if species are differently affected by environmental variation (e.g., soil type, rainfall timing, etc.), this can create a stabilizing mechanism (see the storage effect). Stabilizing mechanisms increase the low-density growth rate of all species.

Chesson's categories of stabilizing mechanisms In 1994, Chesson proposed that all stabilizing mechanisms could be categorized into four categories. These mechanisms are not mutually exclusive, and it is possible for all four to operate in any environment at a given time. Variation-independent mechanisms (also called fluctuation-independent mechanisms) are any stabilizing mechanism that functions within a local place and time. Resource partitioning, predator partitioning, and frequency-dependent predation are three classic examples of variation-independent mechanisms. When a species is at very low density, individuals gain an advantage, because they are less constrained by competition across the landscape. For example, under frequency-dependent predation, a species is less likely to be consumed by predators when they are very rare. The storage effect occurs when species are affected differently by environmental variation in space or time. For example, coral reef fishes have different reproductive rates in different years, plants grow differently in different soil types, and desert annual plants germinate at different rates in different years. When a species is at low density, individuals gain an advantage because they experience less competition in times or locations that they grow best. For example, if annual plants germinate in different years, then when it is a good year to germinate, species will be competing predominately with members of the same species. Thus, if a species becomes rare, individuals will experience little competition when they germinate whereas they would experience high competition if they were abundant. For the storage effect to function, species must be able to "store" the benefits of a productive time period or area, and use it to survive during less productive times or areas. This can occur, for example, if species have a long-lived adult stage, a seed bank or diapause stage, or if they are spread out over the environment. A fitness-density covariance occurs when species are spread out non-uniformly across the landscape. Most often, it occurs when species are found in different areas. For example, mosquitoes often lay eggs in different locations, and plants who partition habitat are often found predominately where they grow best. Species can gain two possible advantages by becoming very rare. First, because they are physically separated from other species, they mainly compete with members of the same species (and thus experience less competition when they become very rare). Second, species are often more able to concentrate in favorable habitat as their densities decline. For example, if individuals are territorial, then members of an abundant species may not have access to ideal habitat; however, when that species becomes very rare, then there may be enough ideal habitat for all of the few remaining individuals. The Janzen-Connell hypothesis is an excellent example of a stabilizing mechanism that operates (in part) through fitness-density covariance. Relative nonlinearity occurs when species benefit in different ways from variation in competitive factors. For example, two species might coexist if one can grow better when resources are rare, and the other grows better when resources are abundant. Species will be able to coexist if the species which benefits from variation in resources tends to reduce variation in resources. For example, a species which can rapidly consume excess resources tends to quickly reduce the level of excess resources favoring the other species, whereas a species which grows better when resources are rare is more likely to cause fluctuations in resource density favoring the other species.

Quantifying stabilizing mechanisms A general way of measuring the effect of stabilizing mechanisms is by calculating the growth rate of species i in a community as

where: is the long-term average growth rate of the species i when at low density. Because species are limited from growing indefinitely, viable populations have an average long-term growth rate of zero. Therefore, species at low-density can increase in abundance when their long-term average growth rate is positive. is a species-specific factor that reflects how quickly species i responds to a change in competition. For example, species with faster generation times may respond more quickly to a change in resource density than longer lived species. In an extreme scenario, if ants and elephants were to compete for the same resources, elephant population sizes would change much more slowly to changes in resource density than would ant populations. is the difference between the fitness of species i when compared to the average fitness of the community excluding species i. In the absence of any stabilizing mechanisms, species i will only have a positive growth rate if its fitness is above its average competitor, i.e. where this value is greater than zero. measures the effect of all stabilizing mechanisms acting within this community.

Example calculation: Species competing for resource

In 2008 Chesson and Kuang showed how to calculate fitness differences and stabilizing mechanisms when species compete for shared resources and competitors. Each species j captures resource type l at a species-specific rate, cjl. Each unit of resource captured contributes to species growth by value vl. Each consumer requires resources for the metabolic maintenance at rate μi.

In conjunction, consumer growth is decreased by attack from predators. Each predator species m attacks species j at rate ajm.

Given predation and resource capture, the density of species i, Ni, grows at rate

where l sums over resource types and m sums over all predator species. Each resource type exhibits logistic growth with intrinsic rate of increase, rRl, and carrying capacity, KRl = 1/αRl, such that growth rate of resource l is

Similarly, each predator species m exhibits logistic growth in the absence of the prey of interest with intrinsic growth rate rPm and carrying capacity KPm = 1/αPm. The growth rate of a predator species is also increased by consuming prey species where again the attack rate of predator species m on prey j is ajm. Each unit of prey has a value to predator growth rate of w. Given these two sources of predator growth, the density of predator m, Pm, has a per-capita growth rate

where the summation terms is contributions to growth from consumption over all j focal species. The system of equations describes a model of trophic interactions between three sets of species: focal species, their resources, and their predators.

Given this model, the average fitness of a species j is

where the sensitivity to competition and predation is

The average fitness of a species takes into account growth based on resource capture and predation as well as how much resource and predator densities change from interactions with the focal species.

The amount of niche overlap between two competitors i and j is

which represents the amount to which resource consumption and predator attack are linearly related between two competing species, i and j.

This model conditions for coexistence can be directly related to the general coexistence criterion: intraspecific competition, αjj, must be greater than interspecific competition, αij. The direct expressions for intraspecific and interspecific competition coefficients from the interaction between shared predators and resources are

and

Thus, when intraspecific competition is greater than interspecific competition,

which, for two species leads to the coexistence criteria

Notice that, in the absence of any niche differences (i.e. ρ = 1), species cannot coexist.

Empirical evidence A 2012 study reviewed different approaches which tested coexistence theory, and identified three main ways to separate the contributions of stabilizing and equalizing mechanisms within a community. These are:

Experimental manipulations, which involved determining the effect of relative fitness or stabilizing mechanisms by manipulating resources or competitive advantages.