In information theory and statistics, negative entropy is used as a measure of distance to normality. It is also known as negentropy or syntropy. It is a counter-construct to entropy.

Etymology The concept and phrase "negative entropy" was introduced by Erwin Schrödinger in his 1944 book What is Life?.Schrödinger, Erwin, What is Life – the Physical Aspect of the Living Cell, Cambridge University Press, 1944 Later, the French physicist Léon Brillouin shortened the phrase to ().Brillouin, Leon: (1953) "Negentropy Principle of Information", J. of Applied Physics, v. 24(9), pp. 1152–1163Léon Brillouin, La science et la théorie de l'information, Masson, 1959 In 1974, Albert Szent-Györgyi proposed replacing the term negentropy with syntropy (Greek, 'together with change'). That term may have originated in the 1940s with the Italian mathematician Luigi Fantappiè, who tried to construct a unified theory of biology and physics. Buckminster Fuller tried to popularize this usage, but negentropy remains common.

In a note to What is Life?, Schrödinger explained his use of this phrase:

Information theory

In information theory and statistics, negentropy is used as a measure of distance to normality. Out of all probability distributions with a given mean and variance, the Gaussian or normal distribution is the one with the highest entropy. Negentropy measures the difference in entropy between a given distribution and the Gaussian distribution with the same mean and variance. Thus, negentropy is always nonnegative, is invariant by any linear invertible change of coordinates, and vanishes if and only if the signal is Gaussian.

Negentropy is defined as

where is the differential entropy of a normal distribution with the same mean and variance as , and is the differential entropy of , with as its probability density function:

Negentropy is used in statistics and signal processing. It is related to network entropy, which is used in independent component analysis.P. Comon, Independent Component Analysis – a new concept?, Signal Processing, 36 287–314, 1994.Didier G. Leibovici and Christian Beckmann, An introduction to Multiway Methods for Multi-Subject fMRI experiment, FMRIB Technical Report 2001, Oxford Centre for Functional Magnetic Resonance Imaging of the Brain (FMRIB), Department of Clinical Neurology, University of Oxford, John Radcliffe Hospital, Headley Way, Headington, Oxford, UK.

The negentropy of a distribution is equal to the Kullback–Leibler divergence between and a Gaussian distribution with the same mean and variance as (see for a proof):In particular, it is always nonnegative (unlike differential entropy, which can be negative).

Correlation between statistical negentropy and Gibbs free energy

There is a physical quantity closely linked to free energy (free enthalpy), with a unit of entropy and isomorphic to negentropy known in statistics and information theory. In 1873, Willard Gibbs created a diagram illustrating the concept of free energy corresponding to free enthalpy. On the diagram one can see the quantity called capacity for entropy. This quantity is the amount of entropy that may be increased without changing an internal energy or increasing its volume.Willard Gibbs, A Method of Geometrical Representation of the Thermodynamic Properties of Substances by Means of Surfaces, Transactions of the Connecticut Academy, 382–404 (1873) In other words, it is a difference between maximum possible, under assumed conditions, entropy and its actual entropy. It corresponds exactly to the definition of negentropy adopted in statistics and information theory. A similar physical quantity was introduced in 1869 by Massieu for the isothermal processMassieu, M. F. (1869a). Sur les fonctions caractéristiques des divers fluides. C. R. Acad. Sci. LXIX:858–862.Massieu, M. F. (1869b). Addition au precedent memoire sur les fonctions caractéristiques. C. R. Acad. Sci. LXIX:1057–1061.Massieu, M. F. (1869), Compt. Rend. 69 (858): 1057. (both quantities differs just with a figure sign) and by then Planck for the isothermal-isobaric process.Planck, M. (1945). Treatise on Thermodynamics. Dover, New York. More recently, the Massieu–Planck thermodynamic potential, known also as free entropy, has been shown to play a great role in the so-called entropic formulation of statistical mechanics,Antoni Planes, Eduard Vives, Entropic Formulation of Statistical Mechanics , Entropic variables and Massieu–Planck functions 2000-10-24 Universitat de Barcelona applied among the others in molecular biologyJohn A. Scheilman, Temperature, Stability, and the Hydrophobic Interaction , Biophysical Journal 73 (December 1997), 2960–2964, Institute of Molecular Biology, University of Oregon, Eugene, Oregon 97403 USA and thermodynamic non-equilibrium processes.Z. Hens and X. de Hemptinne, Non-equilibrium Thermodynamics approach to Transport Processes in Gas Mixtures, Department of Chemistry, Catholic University of Leuven, Celestijnenlaan 200 F, B-3001 Heverlee, Belgium

where: is entropy is negentropy (Gibbs "capacity for entropy") is the Massieu potential is the partition function the Boltzmann constant

In particular, mathematically the negentropy (the negative entropy function, in physics interpreted as free entropy) is the convex conjugate of LogSumExp (in physics interpreted as the free energy).

Brillouin's negentropy principle of information In 1953, Léon Brillouin derived a general equationLeon Brillouin, The negentropy principle of information, J. Applied Physics 24, 1152–1163 1953 stating that the changing of an information bit value requires at least energy. This is the same energy as the work Leó Szilárd's engine produces in the idealistic case. In his book,Leon Brillouin, Science and Information theory, Dover, 1956 he further explored this problem concluding that any cause of this bit value change (measurement, decision about a yes/no question, erasure, display, etc.) will require the same amount of energy.

See also Exergy Free entropy Entropy in thermodynamics and information theory

References