An index of qualitative variation (IQV) is a measure of statistical dispersion in nominal distributions. Examples include the variation ratio or the information entropy.
Properties
There are several types of indices used for the analysis of nominal data. Several are standard statistics that are used elsewhere - range, standard deviation, variance, mean deviation, coefficient of variation, median absolute deviation, interquartile range and quartile deviation.
In addition to these several statistics have been developed with nominal data in mind. A number have been summarized and devised by Wilcox , , who requires the following standardization properties to be satisfied: Variation varies between 0 and 1. Variation is 0 if and only if all cases belong to a single category. Variation is 1 if and only if cases are evenly divided across all categories.This can only happen if the number of cases is a multiple of the number of categories.
In particular, the value of these standardized indices does not depend on the number of categories or number of samples.
For any index, the closer to uniform the distribution, the larger the variance, and the larger the differences in frequencies across categories, the smaller the variance.
Indices of qualitative variation are then analogous to information entropy, which is minimized when all cases belong to a single category and maximized in a uniform distribution. Indeed, information entropy can be used as an index of qualitative variation.
One characterization of a particular index of qualitative variation (IQV) is as a ratio of observed differences to maximum differences.
Wilcox's indexes
Wilcox gives a number of formulae for various indices of QV , the first, which he designates DM for "Deviation from the Mode", is a standardized form of the variation ratio, and is analogous to variance as deviation from the mean.
ModVR
The formula for the variation around the mode (ModVR) is derived as follows:
where fm is the modal frequency, K is the number of categories and fi is the frequency of the ith group.
This can be simplified to
where N is the total size of the sample.
Freeman's index (or variation ratio) isFreemen LC (1965) Elementary applied statistics. New York: John Wiley and Sons pp. 40–43
This is related to M as follows:
The ModVR is defined as
where v is Freeman's index.
Low values of ModVR correspond to small amount of variation and high values to larger amounts of variation.
When K is large, ModVR is approximately equal to Freeman's index v.
RanVR
This is based on the range around the mode. It is defined to be
where fm is the modal frequency and fl is the lowest frequency.
AvDev
This is an analog of the mean deviation. It is defined as the arithmetic mean of the absolute differences of each value from the mean.
MNDif
This is an analog of the mean difference - the average of the differences of all the possible pairs of variate values, taken regardless of sign. The mean difference differs from the mean and standard deviation because it is dependent on the spread of the variate values among themselves and not on the deviations from some central value.Kendal MC, Stuart A (1958) The advanced theory of statistics. Hafner Publishing Company p. 46
where fi and fj are the ith and jth frequencies respectively.
The MNDif is the Gini coefficient applied to qualitative data.
VarNC
This is an analog of the variance.
It is the same index as Mueller and Schussler's Index of Qualitative VariationMueller JE, Schuessler KP (1961) Statistical reasoning in sociology. Boston: Houghton Mifflin Company. pp. 177–179 and Gibbs' M2 index.
It is distributed as a chi square variable with K – 1 degrees of freedom.
StDev
Wilson has suggested two versions of this statistic.
The first is based on AvDev.
The second is based on MNDif
HRel
This index was originally developed by Claude Shannon for use in specifying the properties of communication channels.
where pi = fi / N.
This is equivalent to information entropy divided by the and is useful for comparing relative variation between frequency tables of multiple sizes.
B index
Wilcox adapted a proposal of KaiserKaiser HF (1968) "A measure of the population quality of legislative apportionment." The American Political Science Review 62 (1) 208 based on the geometric mean and created the B index. The B index is defined as
R packages
Several of these indices have been implemented in the R language.
Gibb's indices and related formulae
proposed six indexes.
M1
The unstandardized index (M1) is
where K is the number of categories and is the proportion of observations that fall in a given category i.
M1 can be interpreted as one minus the likelihood that a random pair of samples will belong to the same category, so this formula for IQV is a standardized likelihood of a random pair falling in the same category. This index has also referred to as the index of differentiation, the index of sustenance differentiation and the geographical differentiation index depending on the context it has been used in.
M2
A second index is the M2IQV at xycoon is:
where K is the number of categories and is the proportion of observations that fall in a given category i. The factor of is for standardization.
M1 and M2 can be interpreted in terms of variance of a multinomial distribution (there called an "expanded binomial model"). M1 is the variance of the multinomial distribution and M2 is the ratio of the variance of the multinomial distribution to the variance of a binomial distribution.
M4
The M4 index is
where m is the mean.
M6
The formula for M6 is
· where K is the number of categories, Xi is the number of data points in the ith category, N is the total number of data points, || is the absolute value (modulus) and
This formula can be simplified
where pi is the proportion of the sample in the ith category.
In practice M1 and M6 tend to be highly correlated which militates against their combined use.
Related indices
The sum
has also found application. This is known as the Simpson index in ecology and as the Herfindahl index or the Herfindahl-Hirschman index (HHI) in economics. A variant of this is known as the Hunter–Gaston index in microbiology
In linguistics and cryptanalysis this sum is known as the repeat rate. The incidence of coincidence (IC) is an unbiased estimator of this statisticFriedman WF (1925) The incidence of coincidence and its applications in cryptanalysis. Technical Paper. Office of the Chief Signal Officer. United States Government Printing Office.
where fi is the count of the ith grapheme in the text and n is the total number of graphemes in the text.
M1
The M1 statistic defined above has been proposed several times in a number of different settings under a variety of names. These include Gini's index of mutability,Gini CW (1912) Variability and mutability, contribution to the study of statistical distributions and relations. Studi Economico-Giuricici della R. Universita de Cagliari Simpson's measure of diversity, Bachi's index of linguistic homogeneity,Bachi R (1956) A statistical analysis of the revival of Hebrew in Israel. In: Bachi R (ed) Scripta Hierosolymitana, Vol III, Jerusalem: Magnus press pp 179–247 Mueller and Schuessler's index of qualitative variation,Mueller JH, Schuessler KF (1961) Statistical reasoning in sociology. Boston: Houghton Mifflin Gibbs and Martin's index of industry diversification, Lieberson's index. and Blau's index in sociology, psychology and management studies.Blau P (1977) Inequality and Heterogeneity. Free Press, New York The formulation of all these indices are identical.
Simpson's D is defined as
where n is the total sample size and ni is the number of items in the ith category.
For large n we have
Another statistic that has been proposed is the coefficient of unalikeability which ranges between 0 and 1.Perry M, Kader G (2005) Variation as unalikeability. Teaching Stats 27 (2) 58–60
where n is the sample size and c(x,y) = 1 if x and y are unalike and 0 otherwise.
For large n we have
where K is the number of categories.
Another related statistic is the quadratic entropy
which is itself related to the Gini index.
M2
Greenberg's monolingual non weighted index of linguistic diversity is the M2 statistic defined above.
M7
Another index – the M7 – was created based on the M4 index of Lautard EH (1978) PhD thesis.
where
and
where K is the number of categories, L is the number of subtypes, Oij and Eij are the number observed and expected respectively of subtype j in the ith category, ni is the number in the ith category and pj is the proportion of subtype j in the complete sample.
Note: This index was designed to measure women's participation in the work place: the two subtypes it was developed for were male and female.
Other single sample indices
These indices are summary statistics of the variation within the sample.
Berger–Parker index
The Berger–Parker index, named after Wolfgang H. Berger and Frances Lawrence Parker, equals the maximum value in the dataset, i.e. the proportional abundance of the most abundant type. This corresponds to the weighted generalized mean of the values when q approaches infinity, and hence equals the inverse of true diversity of order infinity (1/∞D).
Brillouin index of diversity
This index is strictly applicable only to entire populations rather than to finite samples. It is defined as
where N is total number of individuals in the population, ni is the number of individuals in the ith category and N! is the factorial of N. Brillouin's index of evenness is defined as
where IB(max) is the maximum value of IB.
Hill's diversity numbers
Hill suggested a family of diversity numbers
For given values of a, several of the other indices can be computed
a = 0: Na = species richness a = 1: Na = Shannon's index a = 2: Na = 1/Simpson's index (without the small sample correction) a = 3: Na = 1/Berger–Parker index
Hill also suggested a family of evenness measures
where a > b.
Hill's E4 is
Hill's E5 is
Margalef's index
where S is the number of data types in the sample and N is the total size of the sample.Margalef R (1958) Temporal succession and spatial heterogeneity in phytoplankton. In: Perspectives in marine biology. Buzzati-Traverso (ed) Univ Calif Press, Berkeley pp 323–347
Menhinick's index
where S is the number of data types in the sample and N is the total size of the sample.
In linguistics this index is the identical with the Kuraszkiewicz index (Guiard index) where S is the number of distinct words (types) and N is the total number of words (tokens) in the text being examined.Kuraszkiewicz W (1951) Nakladen Wroclawskiego Towarzystwa NaukowegoGuiraud P (1954) Les caractères statistiques du vocabulaire. Presses Universitaires de France, Paris This index can be derived as a special case of the Generalised Torquist function.Panas E (2001) The Generalized Torquist: Specification and estimation of a new vocabulary-text size function. J Quant Ling 8(3) 233–252
Q statistic
This is a statistic invented by Kempton and Taylor. and involves the quartiles of the sample. It is defined as
where R1 and R2 are the 25% and 75% quartiles respectively on the cumulative species curve, nj is the number of species in the jth category, nRi is the number of species in the class where Ri falls (i = 1 or 2).
Shannon–Wiener index
This is taken from information theory
where N is the total number in the sample and pi is the proportion in the ith category.
In ecology where this index is commonly used, H usually lies between 1.5 and 3.5 and only rarely exceeds 4.0.
An approximate formula for the standard deviation (SD) of H is
where pi is the proportion made up by the ith category and N is the total in the sample.