Differential item functioning (DIF) is a statistical property of a test item that indicates how likely it is for individuals from distinct groups, possessing similar abilities, to respond differently to the item. It manifests when individuals from different groups, with comparable skill levels, do not have an equal likelihood of answering a question correctly. There are two primary types of DIF: uniform DIF, where one group consistently has an advantage over the other, and nonuniform DIF, where the advantage varies based on the individual's ability level.
The presence of DIF requires review and judgment, but it doesn't always signify bias. DIF analysis provides an indication of unexpected behavior of items on a test. DIF characteristic of an item isn't solely determined by varying probabilities of selecting a specific response among individuals from different groups. Rather, DIF becomes pronounced when individuals from different groups, who possess the same underlying true ability, exhibit differing probabilities of giving a certain response. Even when uniform bias is present, test developers sometimes resort to assumptions such as DIF biases may offset each other due to the extensive work required to address it, compromising test ethics and perpetuating systemic biases.
Common procedures for assessing DIF are Mantel-Haenszel procedure, logistic regression, item response theory (IRT) based methods, and confirmatory factor analysis (CFA) based methods.
Description DIF refers to differences in the functioning of items across groups, oftentimes demographic, which are matched on the latent trait or more generally the attribute being measured by the items or test. When examining items for DIF, the groups must be matched on the measured attribute, otherwise this may result in inaccurate detection of DIF.
In order to create a general understanding of DIF or measurement bias, consider the following example offered by Osterlind and Everson (2009). In this case, refers to a response to a particular test item which is determined by the latent construct being measured. The latent construct of interest is referred to as theta () where is an indicator of which can be arranged in terms of the probability distribution of on by the expression . Therefore, response is conditional on the latent trait ().
Because DIF examines differences in the conditional probabilities of between groups, let us label the groups as the "reference" and "focal" groups. Although the designation does not matter, a typical practice in the literature is to designate the reference group as the group who is suspected to have an advantage while the focal group refers to the group anticipated to be disadvantaged by the test. Therefore, given the functional relationship and under the assumption that there are identical measurement error distributions for the reference and focal groups it can be concluded that under the null hypothesis:
with corresponding to the grouping variable, the reference group, and the focal group.
This equation represents an instance where DIF is not present. In this case, the absence of DIF is determined by the fact that the conditional probability distribution of is not dependent on group membership. To illustrate, consider an item with response options and , where indicates an incorrect response, and indicates a correct response. The probability of correctly responding to an item is the same for members of either group. This indicates that there is no DIF or item bias because members of the reference and focal group with the same underlying ability or attribute have the same probability of responding correctly. Therefore, there is no bias or disadvantage for one group over the other.
Consider the instance where the conditional probability of is not the same for the reference and focal groups. In other words, members of different groups with the same trait or ability level have unequal probability distributions on . Once controlling for , there is a clear dependency between group membership and performance on an item. For dichotomous items, this suggests that when the focal and reference groups are at the same location on , there is a different probability of getting a correct response or endorsing an item. Therefore, the group with the higher conditional probability of correctly responding to an item is the group advantaged by the test item. This suggests that the test item is biased and functions differently for the groups, therefore exhibits DIF.
It is important to draw the distinction between DIF or measurement bias and ordinary group differences. Whereas group differences indicate differing score distributions on , DIF explicitly involves conditioning on . For instance, consider the following equation:
This indicates that an examinee's score is conditional on grouping such that having information about group membership changes the probability of a correct response. Therefore, if the groups differ on , and performance depends on , then the above equation would suggest item bias even in the absence of DIF. For this reason, it is generally agreed upon in the measurement literature that differences on conditional on group membership alone is inadequate for establishing bias.
In fact, differences on or ability are common between groups and establish the basis for much research. Remember to establish bias or DIF, groups must be matched on and then demonstrate differential probabilities on as a function of group membership.
Forms Uniform DIF is the simplest type of DIF where the magnitude of conditional dependency is relatively invariant across the latent trait continuum (). The item of interest consistently gives one group an advantage across all levels of ability . Within an item response theory (IRT) framework this would be evidenced when both item characteristic curves (ICC) are equally discriminating yet exhibit differences in the difficulty parameters (i.e., and ) as depicted in Figure 1.
However, nonuniform DIF presents an interesting case. Rather than a consistent advantage being given to the reference group across the ability continuum, the conditional dependency moves and changes direction at different locations on the continuum. For instance, an item may give the reference group a minor advantage at the lower end of the continuum while a major advantage at the higher end. Also, unlike uniform DIF, an item can simultaneously vary in discrimination for the two groups while also varying in difficulty (i.e., and ).
Even more complex is "crossing" nonuniform DIF. As demonstrated in Figure 2, this occurs when an item gives an advantage to a reference group at one end of the continuum while favors the focal group at the other end. Differences in ICCs indicate that examinees from the two groups with identical ability levels have unequal probabilities of correctly responding to an item. When the curves are different but do not intersect, this is evidence of uniform DIF. However, if the ICCs cross at any point along the scale, there is evidence of nonuniform DIF.
Procedures for detecting DIF
Mantel-Haenszel A common procedure for detecting DIF is the Mantel-Haenszel (MH) approach. The MH procedure is a chi-squared contingency table based approach which examines differences between the reference and focal groups on all items of the test, one by one. The ability continuum, defined by total test scores, is divided into intervals which then serves as the basis for matching members of both groups. A 2 × 2 contingency table is used at each interval of comparing both groups on an individual item. The rows of the contingency table correspond to group membership (reference or focal) while the columns correspond to correct or incorrect responses. The following table presents the general form for a single item at the th ability interval.
+ Item of Interest Correct (1) Incorrect (0) Total Reference Focal Total
Note. , , and correspond to observed cell frequencies. Subscript corresponds to the ability interval.
Odds ratio The next step in the calculation of the MH statistic is to use data from the contingency table to obtain an odds ratio for the two groups on the item of interest at a particular interval. This is expressed in terms of and where represents the proportion correct and the proportion incorrect for both the reference () and focal () groups. For the MH procedure, the obtained odds ratio is represented by with possible value ranging from to . A value of 1.0 indicates an absence of DIF and thus similar performance by both groups. Values greater than suggest that the reference group outperformed or found the item less difficult than the focal group. On the other hand, if the obtained value is less than , this is an indication that the item was less difficult for the focal group.
Using variables from the contingency table above, the calculation is as follows:
The above computation pertains to an individual item at a single ability interval. The population estimate can be extended to reflect a common odds ratio across all ability intervals for a specific item. The common odds ratio estimator is denoted and can be computed by the following equation:
for all values of and where represents the total sample size at the th interval.
The obtained is often standardized through log transformation, centering the value around 0. The new transformed estimator is computed as follows:
Thus an obtained value of 0 would indicate no DIF. The minus sign changes the interpretation of values less than or greater than 0. Values less than 0 indicate a reference group advantage whereas values greater than 0 indicate an advantage for the focal group.
Item response theory Item response theory (IRT) is another widely used method for assessing DIF. IRT allows for a critical examination of responses to particular items from a test or measure. As noted earlier, DIF examines the probability of correctly responding to or endorsing an item conditioned on the latent trait or ability. Because IRT examines the monotonic relationship between responses and the latent trait or ability, it is a fitting approach for examining DIF.
Three major advantages of using IRT in DIF detection are: Compared to classical test theory, IRT parameter estimates are not as confounded by sample characteristics. Statistical properties of items can be expressed with greater precision which increases the interpretation accuracy of DIF between two groups. These statistical properties of items can be expressed graphically, improving interpretability and understanding of how items function differently between groups. In relation to DIF, item parameter estimates are computed and graphically examined via item characteristic curves (ICCs) also referred to as trace lines or item response functions (IRF). After examination of ICCs and subsequent suspicion of DIF, statistical procedures are implemented to test differences between parameter estimates.
ICCs represent mathematical functions of the relationship between positioning on the latent trait continuum and the probability of giving a particular response. Figure 3 illustrates this relationship as a logistic function. Individuals lower on the latent trait or with less ability have a lower probability of getting a correct response or endorsing an item, especially as difficulty increases. Thus, those higher on the latent trait or in ability have a greater chance of a correct response or endorsing an item. For instance, on a depression inventory, highly depressed individuals would have a greater probability of endorsing an item than individuals with lower depression. Similarly, individuals with higher math ability have a greater probability of getting a math item correct than those with lesser ability.