Statistics, when used in a misleading fashion, can trick the casual observer into believing something other than what the data shows. That is, a misuse of statistics occurs when a statistical argument asserts a falsehood. In some cases, the misuse may be accidental. In others, it is purposeful and for the gain of the perpetrator. When the statistical reason involved is false or misapplied, this constitutes a statistical fallacy.
The consequences of such misinterpretations can be quite severe. For example, in medical science, correcting a falsehood may take decades and cost lives; likewise, in democratic societies, misused statistics can distort public understanding, entrench misinformation, and enable governments to implement harmful policies without accountability.
Misuses can be easy to fall into. Professional scientists, mathematicians and even professional statisticians, can be fooled by even some simple methods, even if they are careful to check everything. Scientists have been known to fool themselves with statistics due to lack of knowledge of probability theory and lack of standardization of their tests.
Definition, limitations and context One usable definition is: "Misuse of Statistics: Using numbers in such a manner that – either by intent or through ignorance or carelessness – the conclusions are unjustified or incorrect." The "numbers" include misleading graphics discussed in other sources. The term is not commonly encountered in statistics texts and there is no single authoritative definition. It is a generalization of lying with statistics which was richly described by examples from statisticians 60 years ago.
The definition confronts some problems (some are addressed by the source):
Statistics usually produces probabilities; conclusions are provisional The provisional conclusions have errors and error rates. Commonly 5% of the provisional conclusions of significance testing are wrong Statisticians are not in complete agreement on ideal methods Statistical methods are based on assumptions which are seldom fully met Data gathering is usually limited by ethical, practical and financial constraints.
How to Lie with Statistics acknowledges that statistics can legitimately take many forms. Whether the statistics show that a product is "light and economical" or "flimsy and cheap" can be debated whatever the numbers. Some object to the substitution of statistical correctness for moral leadership (for example) as an objective. Assigning blame for misuses is often difficult because scientists, pollsters, statisticians and reporters are often employees or consultants.
An insidious misuse of statistics is completed by the listener, observer, audience, or juror. The supplier provides the "statistics" as numbers or graphics (or before/after photographs), allowing the consumer to draw conclusions that may be unjustified or incorrect. The poor state of public statistical literacy and the non-statistical nature of human intuition make it possible to mislead without explicitly producing faulty conclusion. The definition is weak on the responsibility of the consumer of statistics.
A historian listed over 100 fallacies in a dozen categories including those of generalization and those of causation. A few of the fallacies are explicitly or potentially statistical including sampling, statistical nonsense, statistical probability, false extrapolation, false interpolation and insidious generalization. All of the technical/mathematical problems of applied probability would fit in the single listed fallacy of statistical probability. Many of the fallacies could be coupled to statistical analysis, allowing the possibility of a false conclusion flowing from a statistically sound analysis.
An example use of statistics is in the analysis of medical research. The process includes In this article anything less than the best statistical practice is equated to the potential misuse of statistics. In a few pages 47 potential statistical errors are discussed; errors in study design, data analysis, documentation, presentation and interpretation. "[S]tatisticians should be involved early in study design, as mistakes at this point can have major repercussions, negatively affecting all subsequent stages of medical research." Contains a rich list of medical misuses of statistics of all types. experimental planning, the conduct of the experiment, data analysis, drawing the logical conclusions and presentation/reporting. The report is summarized by the popular press and by advertisers. Misuses of statistics can result from problems at any step in the process. The statistical standards ideally imposed on the scientific report are much different than those imposed on the popular press and advertisers; however, cases exist of advertising disguised as science, such as Australasian Journal of Bone & Joint Medicine. The definition of the misuse of statistics is weak on the required completeness of statistical reporting. The opinion is expressed that newspapers must provide at least the source for the statistics reported.
Simple causes Many misuses of statistics occur because The source is a subject matter expert, not a statistics expert. The source may incorrectly use a method or interpret a result. The source is a statistician, not a subject matter expert. An expert should know when the numbers being compared describe different things. Numbers change, as reality does not, when legal definitions or political boundaries change. The subject being studied is not well defined, or some of its aspects are easy to quantify while others hard to quantify or there is no known quantification method (see McNamara fallacy). For example: While IQ tests are available and numeric, it is difficult to define what they measure, as intelligence is an elusive concept. Publishing "impact" has the same problem. Scientific papers and scholarly journals are often rated by "impact", quantified as the number of citations by later publications. Mathematicians and statisticians conclude that impact (while relatively objective) is not a very meaningful measure. "The sole reliance on citation data provides at best an incomplete and often shallow understanding of an understanding that is valid only when reinforced by other judgments. Numbers are not inherently superior to sound judgments." A seemingly simple question about the number of words in the English language immediately encounters questions about archaic forms, accounting for prefixes and suffixes, multiple definitions of a word, variant spellings, dialects, fanciful creations (like ectoplastistics from ectoplasm and statistics), technical vocabulary, and so on. Data quality is poor. Apparel provides an example. People have a wide range of sizes and body shapes. It is obvious that apparel sizing must be multidimensional. Instead it is complex in unexpected ways. Some apparel is sold by size only (with no explicit consideration of body shape), sizes vary by country and manufacturer and some sizes are deliberately misleading. While sizes are numeric, only the crudest of statistical analyses is possible using the size numbers with care. The popular press has limited expertise and mixed motives. This Australian report on crime statistics provides numerous examples of interpreting and misinterpreting the data. "The increase in media access to information about crime has not been matched by an increase in the quality of media reporting on crime. The misuse of crime statistics by the media has impeded rational debate about law and order." Among the alleged media abuses: selective use of data, selective reporting of facts, misleading commentary, misrepresentation of facts and misleading headlines. Police and politicians also abused the statistics. If the facts are not "newsworthy" (which may require exaggeration) they may not be published. The motives of advertisers are even more mixed. "Politicians use statistics in the same way that a drunk uses lamp posts—for support rather than illumination" – Andrew Lang (WikiQuote) "What do we learn from these two ways of looking at the same numbers? We learn that a clever propagandist, right or left, can almost always find a way to present the data on economic growth that seems to support her case. And we therefore also learn to take any statistical analysis from a strongly political source with handfuls of salt." The term statistics originates from numbers generated for and utilized by the state. Good government may require accurate numbers, but popular government may require supportive numbers (not necessarily the same). "The use and misuse of statistics by governments is an ancient art."
Types of misuse
Discarding unfavorable observations
To promote a neutral (useless) product, a company must find or conduct, for example, 40 studies with a confidence level of 95%. If the product is useless, this would produce one study showing the product was beneficial, one study showing it was harmful, and thirty-eight inconclusive studies (38 is 95% of 40). This tactic becomes more effective when there are more studies available. Organizations that do not publish every study they carry out, such as tobacco companies denying a link between smoking and cancer, anti-smoking advocacy groups and media outlets trying to prove a link between smoking and various ailments, or miracle pill vendors, are likely to use this tactic.
Ronald Fisher considered this issue in his famous lady tasting tea example experiment (from his 1935 book, The Design of Experiments). Regarding repeated experiments, he said, "It would be illegitimate and would rob our calculation of its basis if unsuccessful results were not all brought into the account."
Another term related to this concept is cherry picking.
Ignoring important features Multivariable datasets have two or more features/dimensions. If too few of these features are chosen for analysis (for example, if just one feature is chosen and simple linear regression is performed instead of multiple linear regression), the results can be misleading. This leaves the analyst vulnerable to any of various statistical paradoxes, or in some (not all) cases false causality as below.
Loaded questions
The answers to surveys can often be manipulated by wording the question in such a way as to induce a prevalence towards a certain answer from the respondent. For example, in polling support for a war, the questions: Do you support the attempt by the US to bring freedom and democracy to other places in the world? Do you support the unprovoked military action by the USA? will likely result in data skewed in different directions, although they are both polling about the support for the war. A better way of wording the question could be "Do you support the current US military action abroad?" A still more nearly neutral way to put that question is "What is your view about the current US military action abroad?" The point should be that the person being asked has no way of guessing from the wording what the questioner might want to hear.
Another way to do this is to precede the question by information that supports the "desired" answer. For example, more people will likely answer "yes" to the question "Given the increasing burden of taxes on middle-class families, do you support cuts in income tax?" than to the question "Considering the rising federal budget deficit and the desperate need for more revenue, do you support cuts in income tax?"
The proper formulation of questions can be very subtle, but nonetheless can yield significant differences in results. Additionally, the responses to two questions can vary dramatically depending on the order in which they are asked. "A survey that asked about 'ownership of stock' found that most Texas ranchers owned stock, though probably not the kind traded on the New York Stock Exchange."
Overgeneralization