Anthony G. Greenwald, PhD

IMPLICIT ASSOCIATION TEST: VALIDITY DEBATES

Note posted 9 Jun 08:  Modifications made today include a new section on predictive validity, and addition of recently published article and in in-press article, both by Nosek & Hansen, under the "CULTURE VS. PERSON" heading, which replaces a previously listed unpublished ms. of theirs.  I continue to encourage all interested to send material that they are willing to be included on this page. Please also to let me know about errors, including faulty links.

Note posted 29 Dec 07:  An in press (JPSP) meta-analysis of predictive validity of the IAT is now available on the page listing unpublished manuscripts.  The current draft includes literature available thru January 2007.  The meta-analysis has some surprising findings about validity of IAT measures in predicting intergroup discrimination (i.e., prediction of discriminatory behavior from IAT measures of attitudes and stereotypes involving race, ethnicity, gender, and sexual orientation. This is the one domain in which IAT measures significantly outperform self-report measures (which have unexpectedly low predictive validity in this domain — aggregate effect size r of about .09).

I thank all of the authors who have provided permission to include either links to their papers or the PDFs that are downloadable from this page. Very special thanks to Brian Nosek, whose work in preparing the "age 7" chapter (cited below) brought together some of the sets of findings collected on this page.

PREDICTIVE VALIDITY

CONSTRUCT VALIDITY
salience asymmetry
self-esteem IAT
culture vs. person

INTERNAL VALIDITY

[return to home page]


Preface:  The IAT has received a remarkable level of attention to validity concerns in the relatively short period since its first publication in 1998, increasing both empirical and theoretical understanding of the IAT. This page was designed to survey the topics that have received attention, includng more than 50 papers, most of which are either published or in press, and most of which have been made downloadable with the permission of their authors.

The topics are divided into categories of predictive validity, construct validity, internal validity, and statistical conclusion validity. This is by no means a complete overview of research on the validity of IAT measures. Rather, it focuses only on topics about which validity-related questions have been raised.  This page makes no attempt to summarize the research on convergent validity of IAT measures. Overview of that literature is fortunately available in the meta-analysis by Hofmann et al. (2005), and in the recent overviews of IAT research by Nosek et al. (2007) and Lane et al. (2007), which are listed here:

Hofmann, W., Gawronski, B., Gschwendner, T., Le, H., & Schmitt, M. (2005). A meta-analysis on the correlation between the Implicit Association Test and explicit self-report measures. Personality and Social Psychology Bulletin, 31, 1369-1385.

Nosek, B. A., Greenwald, A. G., & Banaji, M. R. (2007). The Implicit Association Test at age 7: A methodological and conceptual review (pp. 265–292). In J. A. Bargh (Ed.), Automatic processes in social thinking and behavior. Psychology Press.

Lane, K. A., Banaji, M. R., Nosek, B. A., & Greenwald, A. G. (2007). Understanding and using the Implicit Association Test: IV. What we know (so far) (Pp. 59–102).  In B. Wittenbrink & N. S. Schwarz (Eds.). Implicit measures of attitudes: Procedures and controversies. New York: Guilford Press.


 Predictive Validity   Back to Top

Greenwald, A. G., Poehlman, T. A., Uhlmann, E., & Banaji, M. R. (JPSP, in press). Understanding and using the Implicit Association Test: III. Meta-analysis of predictive validity. (date of draft: 30 Oct 08)  

List of IAT predictive validity studies using real-world samples. The current draft is dated 3 November 2008 and contains 20 reports (19 published or in press).  If should be updated periodically. Please send additional studies for this list if you have them.


  Construct Validity    Back to Top
(Is the IAT measure properly interpreted as assessing variations in association strengths?)

  The three headings in this section focus on the interpretation that the IAT measures strengths of associations that they are nominally designed to assess. The critique for each of these topics was reasonable enough to justify collecting empirical data. It may be too soon to regard any of these critiques as a closed question, although interest in the salience asymmetry interpretation seems to have waned considerably.  It also seems apparent that these construct-validity criticisms have deterred few researchers from treating the IAT as a useful measure.  The next paragraph's quote from an in-press chapter by Nosek, Greenwald, and Banaji (cited just below the quote) indicates some of the other emphases in construct-validity studies of IAT measures.

 "The realization that the IAT and self-report are related introduces important questions about whether they measure distinct constructs. In a multitrait-multimethod (MTMM) investigation of the IAT and self-report across seven attitude domains, Nosek and Smyth (2004) found strong evidence for both convergent and discriminant validity - IAT attitude measures were related to their corresponding self-report measure and not measures of other traits. Further, using structural equation modeling, this MTMM investigation revealed that the best-fitting models represented the IAT and self-report as related, but distinct constructs, rather than as a single attitude construct, even after accounting for common method variance in both measures (Nosek & Smyth, 2004; see also Cunningham, Nezlek, & Banaji, 2004). This extends similar findings for individual constructs such as Greenwald and Farnham (2000) for self-esteem, and Cunningham et al. (2001) for racial attitudes. Finally, Nosek (2005) reported evidence that the relationship between the IAT and self-report is moderated by multiple interpersonal (self-presentation, perceived distinctiveness from the norm) and intrapersonal (e.g., evaluative strength) features of attitudes."

Nosek, B. A., Greenwald, A. G., & Banaji, M. R. (2007). The Implicit Association Test at age 7: A methodological and conceptual review (pp. 265–292). In J. A. Bargh (Ed.), Automatic processes in social thinking and behavior. Psychology Press.

  •   SALIENCE ASYMMETRY    Back to Top
      Rothermund and Wentura proposed that subjects responding to the IAT should find it easier to group categories by perceptual salience - highly salient categories are easily grouped together. Although this may well be true, especially when there are extreme variations of perceptual salience, it seems unlikely that this is a stronger influence on IAT measures than are association strengths, There is also some debate between Rothermund and Wentura and Greenwald et al. about the meaning of "association". Greenwald et al. suggest that the question about the meaning of association is non-central to interpretation of IAT findings.

    Rothermund, K., & Wentura, D. (2004). Underlying processes in the Implicit Association Test(IAT): Dissociating salience from associations. Journal of Experimental Psychology: General, 133, 139-165.

    Greenwald, A. G., Nosek, B. A., Banaji, M. R., & Klauer, K. C. (2005). Validity of the salience asymmetry interpretation of the IAT: Comment on Rothermund and Wentura (2004). Journal of Experimental Psychology: General, 134, 420-425. GNBK2005.pdf

    Rothermund, K., Wentura, D., & De Houwer, J. (2005). Validity of the salience asymmetry account of the Implicit Association Test: Reply to Greenwald, Nosek, Banaji, and Klauer (2005). Journal of Experimental Psychology: General, 134, 426-430.

  •   SELF-ESTEEM IAT (construct validity)    Back to Top
      Two types of questions have been raised in regard to uses of the IAT to measure implicit self-esteem. First, how to conceive what it measures - in other words, what is implicit self-esteem? This question was raised by Bosson, Swann, and Pennebaker (2000) in their study that showed weak correlations among several available measures of implicit self-esteem. Second, what is the most appropriate concept to contrast with self in an IAT measure of implicit self-esteem? This question was raised by Karpinski (2004), to which the Pinter and Greenwald (2005) article was a reply. Perhaps the best answer to this question is that it is reasonable to consider multiple alternative contrast categories for self, something being done in various studies that are not available as circulatable reports.

    Perhaps the best evidence for construct validity of the self-esteem IAT measure was its use to confirm balance-theoretical predictions of the role of self-esteem and self-concept in implicit social cognition, by Greenwald et al. (2002).

    Bosson, J. K., Swann, W. B. Jr., & Pennebaker, J. W. (2000)    . Stalking the perfect measure of implicit self-esteem: The blind men and the elephant revisited? Journal of Personality & Social Psychology. 79, 631-643.

    Karpinski, A. (2004). Measuring self-esteem using the Implicit Association Test: The role of the other. Personality and Social Psychology Bulletin, 30, 22-34.

    Pinter, B., & Greenwald, A. G. (2005)    . Clarifying the role of the "other" category in the self-esteem IAT. Experimental Psychology, 52, 74-79.

    Greenwald, A. G., Banaji, M. R., Rudman, L. A., Farnham, S. D., Nosek, B. A., & Mellott, D. S. (2002). A unified theory of implicit attitudes, stereotypes, self-esteem, and self-concept. Psychological Review, 109, 3-25.

  •   CULTURE VS. PERSON    Back to Top
      The most interesting question raised about the IAT's construct validity is the suggestion that the IAT measures associations that reside in the culture, rather than in the person. This alternative view has appeared, in different forms, in Karpinski and Hilton (2001), Olson and Fazio (2004), and Arkes and Tetlock (2004). The Banaji, Nosek, and Greenwald (2004) article and     the two articles by Nosek and Hansen (2008; in press) respond to these. Perhaps the most compelling type of evidence that the IAT measures something in the person (even if that is derived from something in the culture) has been the numerous successful uses of the IAT to measure individual differences. The large collection of predictive validity studies involving the IAT, reviewed by Greenwald et al. (cited on this page) gives an overview of evidence indicating sensitivity to individual differences, something also supported by the Greenwald et al. (2002) and Hofmann et al. (2005) articles cited elsewhere on this page.

    Karpinski, A., & Hilton, J. L. (2001). Attitudes and the implicit association test. Journal of Personality and Social Psychology, 81, 774-788.

    Olson, M. A., & Fazio, R. H. (2004). Reducing the Influence of Extrapersonal Associations on the Implicit Association Test: Personalizing the IAT. Journal of Personality & Social Psychology. 86, 653-667.

    Nosek, B. A., & Hansen, J. J. (2008). The associations in our heads belong to us: Searching for attitudes and knowledge in implicit evaluation. Cognition and Emotion, 22, 553-594.

    Nosek, B. A., & Hansen, J. J. (in press). Personalizing the Implicit Association Test increases explicit evaluation of the target concepts. European Journal of Psychological Assessment.

    Arkes, H. R., & Tetlock, P. E. (2004). Attributions of Implicit Prejudice, or "Would Jesse Jackson 'Fail' the Implicit Association Test?" Psychological Inquiry. 15, 257-278. [entire exchange]

    Banaji, M. R., Nosek, B. A., & Greenwald, A. G. (2004)    . No place for nostalgia in science: A response to Arkes and Tetlock. Psychological Inquiry, 15, 279-289. [entire exchange]

    Arkes, H. R., & Tetlock, P. E. (2004). The Implicit Prejudice Exchange: Islands of Consensus in a Sea of Controversy: Response. Psychological Inquiry, 15, 311-321. [entire exchange]
 Internal Validity    Back to Top
  (Do IAT procedures generate effects that are due to variables other than the category variations represented in the selections of stimuli used for IAT measures?)

  Most all of the following internal validity topics were first investigated in response to interpretive questions raised repeatedly by both psychologists and laypersons who experienced the IAT on the web in group demonstrations. Several of these questions were never raised seriously in print because they had been resolved (in articles cited in this section) very early in the course of research into the IAT's properties.
  •   FAMILIARITY    Back to Top
      Comments expressed as questions in pre-1998 conference and colloquium presentations, and by many laypersons who completed an IAT on the web, often oberved that, because familiar stimuli might be better liked, stimulus item familiarity was a possible source of artifact in IAT attitude measures. Several studies examined this question. Their findings established that familiarity was not a substantial source of artifact, so long as the stimuli used to represent a category in the IAT were not entirely unfamiliar (such as nonsense strings).

    Brendl, Markman, and Messner (2001) is an example of a study that used totally unfamiliar stimuli, intending these to represent pseudo-categories that should lack associations with other categories. Greenwald and Nosek (2001) concluded that such uses of totally unfamiliar stimuli would yield problematic IAT findings and should be avoided. The best explanation for the problematic nature of this procedure may be that the IAT has been shown to operate at the level of the categories that contain the stimuli used in the task. When the task stimuli fall into no existing category, the IAT appears not to work as desired. The situation might be similar if items chosen to represent a category in an IAT were poor exemplars that were difficult to classify into the intended category. This circumstance should be avoided, as it is known to produce slow responding and psychometrically inferior IAT measures.

    Rudman, L. A., Greenwald, A. G., Mellott, D. S., & Schwartz, J. L. K. (1999). Measuring the automatic components of prejudice: Flexibility and generality of the Implicit Association Test. Social Cognition, 17, 437-465.

    Dasgupta, N., McGhee, D. E., Greenwald, A. G., & Banaji, M. R. (2000). Automatic preference for White Americans: Eliminating the familiaritexplanation. Journal of Experimental Social Psychology, 36, 316-328.

    Ottaway, S. A., Hayden, D. C., & Oakes, M. A. (2001). Implicit attitudes and racism: Effects of word familiarity and frequency on the implicit association test. Social Cognition, 19, 97-144.

    Brendl, C. M., Markman, A. B., & Messner, C. (2001). How do indirect measures of evaluation work? Evaluating the inference of prejudice in the Implicit Association Test. Journal of Personality and Social Psychology, 81, 760-773.

    Dasgupta, N., Greenwald, A. G., & Banaji, M. R. (2003). The first ontological challenge to the IAT: Attitude or mere familiarity? Psychological Inquiry, 14, 238-243.

    Greenwald, A. G., & Nosek, B. A. (2001). Health of the Implicit Association Test at age 3. Zeitschrift für Experimentelle Psychologie, 48, 85-93.

  •   ORDER OF COMBINED TASKS    Back to Top
      The first IAT publication (Greenwald et al., 1998) observed that, other things being equal, strengths of associations used in the first of the IAT's two combined tasks appeared to be stronger than those used in the second. This finding was subsequently observed in numerous other studies that used a procedure of counterbalancing order of presentation of the IAT's two combined tasks.     Nosek, Greenwald, & Banaji (2005) showed the possibility of reducing this order effect by increasing the amount of practice on the 5th of the 7 blocks of the standard IAT procedure.

    Greenwald, A. G., McGhee, D. E., & Schwartz, J. L. K. (1998). Measuring individual differences in implicit cognition: The Implicit Association Test. Journal of Personality and Social Psychology, 74, 1464-1480.

    Nosek, B. A., Greenwald, A. G., & Banaji, M. R. (2005)    . Understanding and using the Implicit Association Test: II. Method variables and construct validity. Personality and Social Psychology Bulletin, 31, 166-180.
  •   PREVIOUS EXPERIENCE WITH THE IAT (order of measures)    Back to Top
      Greenwald, Nosek, & Banaji (2003) reported that IAT effects were slightly reduced in magnitude for subjects who had prior experience taking the IAT. No interpretation of this effect has yet been offered.

    Greenwald, A. G, Nosek, B. A., & Banaji, M. R. (2003). Understanding and using the Implicit Association Test: I. An improved scoring algorithm. Journal of Personality and Social Psychology, 85, 197-216.

  •   INTERTRIAL INTERVAL DURATION    Back to Top
      Greenwald et al. (1998, Experiment 1) reported that the Interval between successive trials of the IAT, in the range from 100 ms to 700 ms, had virtually no effect on magnitude of IAT effects.

    Greenwald, A. G., McGhee, D. E., & Schwartz, J. L. K. (1998). Measuring individual differences in implicit cognition: The Implicit Association Test. Journal of Personality and Social Psychology, 74, 1464-1480.

  •   ASSIGNMENT OF CATEGORIES TO RIGHT OR LEFT KEY    Back to Top
      Some have suspected that a perhaps-natural association of right side with positive valence would make attitudinal IATs vary as a function of whether the pleasant category was assigned to the left or the right side. Greenwald et al. (1998) reported no effect of assigning pleasant and unpleasant categories to the right or left side. Numerous subsequent studies have similarly counterbalanced side to which categories are assigned, with no reports that this produced variations in results.

    Greenwald, A. G., McGhee, D. E., & Schwartz, J. L. K. (1998). Measuring individual differences in implicit cognition: The Implicit Association Test. Journal of Personality and Social Psychology, 74, 1464-1480.

  •   HANDEDNESS    Back to Top
      Greenwald & Nosek (2001, p. 87) reported no effects on IAT scores of subject self-reported handedness in a large web-based data collection.

    Greenwald, A. G., & Nosek, B. A. (2001)    . Health of the Implicit Association Test at age 3. Zeitschrift für Experimentelle Psychologie, 48, 85-93.

  •   SYSTEMATIC METHOD VARIANCE AND TASK-SWITCHING    Back to Top
      This category includes the role of task-switching; cognitive fluency; and age on IAT measures. As an overview: It appears that cognitive fluency and age are both associated with general slowing of responding, which tends to increase latency difference effects (and therefore IAT measures). Effects of cognitive fluency and age on IAT measures are substantially moderated (even if not eliminated) by using the IAT's improved scoring algorithm for the IAT, which reduces the correlation of IAT effects with average latency of responding to a small value     (Greenwald, Nosek, & Banaji, JPSP, 2003). In regard to task switching: It appears that the association strengths measured by the IAT reduce task-switching costs when associated categories share a response. Consequently, task switching costs may be integral to what is measured by the IAT, rather than being an artifactual contributor to IAT effects.

    McFarland, S. G., & Crouch, Z. (2002). A cognitive skill confound on the Implicit Association Test. Social Cognition, 20, 483-510.

    Mierke, J., & Klauer, K. C. (2001). Implicit association measurement with the IAT: Evidence for effects of executive control porcesses. Zeitschrift fuer Experimentelle Psychologie. 48, 107-122.

    Mierke, J. & Klauer, K. C. (2003). Method-specific variance in the Implicit Association Test. Journal of Personality and Social Psychology, 85, 1180-1192.

    Klauer, K. C., & Mierke, J. (2005). Task-set inertia, attitude accessibility, and compatibility-order effects: New evidence for a task-set switching account of the IAT effect. Personality and Social Psychology Bulletin, in press.

    Back, M. D., Schmukle, S. C., & Egloff, B. (2005). Measuring task-switching ability in the Implicit Association Test. Experimental Psychology, 52, 167-179.

    Greenwald, A. G, Nosek, B. A., & Banaji, M. R. (2003). Understanding and using the Implicit Association Test: I. An improved scoring algorithm. Journal of Personality and Social Psychology, 85, 197-216.

    Cai, H., Sriram, N., Greenwald, A. G., & McFarland, S. G. (2004)    . The Implicit Association Test's D measure can minimize a cognitive skill confound: Comment on McFarland and Crouch (2002). Social Cognition, 22, 673-684.

  •   FAKEABILITY    Back to Top
      Several studies have examined the effects of asking subjects to fake their IAT performances. Findings reveal that it is difficult to fake IAT performances, although modest effects of faking have been observed in some of these studies. Clearly, the best strategy for faking the IAT (although few research subjects discover it on their own) is to respond slowly in whichever of the two combined tasks is easier. Research that is currently underway (nothing yet published) is seeking to identify statistical patterns that will indicate that a respondent is attempting to fake an IAT result.

    Banse, R., Seise, J., & Zerbes, N. (2001). Implicit attitudes toward homosexuality: Reliability, validity, and controllability of the IAT. Zeitschrift fur Experimentelle Psychologie, 48, 145-160. (no PDF available)

    Kim, D. Y. (2003). Voluntary controllability of the Implicit Association Test (IAT). Social Psychology Quarterly, 66, 83-96.

    Steffens, M. C. (2004). Is the Implicit Association Test Immune to Faking? Experimental Psychology, 51, 2004, 165-179.

    Egloff, B., & Schmukle, S. C. (2002). Predictive validity of an Implicit Association Test for measuring anxiety. Journal of Personality and Social Psychology, 83, 1441-1455.

    Fiedler, K., & Bluemke, M. (in press). Faking the IAT: Aided and Unaided Response Control on the Implicit Association Tests. Basic and Applied Social Psychology.

  •   STIMULUS ITEM (EXEMPLAR) SELECTION    Back to Top
      Results obtained by Steffens and Plewe (2001), Govan and Williams (2004), Mitchell, Nosek, & Banaji (2003) and Bluemke & Friese (2005) establish that it is possible to influence IAT measures by selecting stimuli in a fashion that confounds a category distinction of interest with some other category distinction. For example, Mitchell et al. showed that implicit race attitudes were influenced when category exemplars were selected to represent either (a) disliked Black and liked White or, alternately, liked Black and disliked White. Govan and Williams showed a similar result and also showed that category exemplars of flowers and insects could be varied by identifying liked insects (e.g., butterfly, firefly, grasshopper) and disliked flowers (some of which were not exactly flowers - e.g., poison ivy, nettles, weed). In a gender attitude IAT, Steffens and Plewe represented valence concepts with adjectives that were confounded with gender (e.g., beautiful, aesthetic, tasteful [positive] vs. martial, corrupt, violent [negative] or s    uccessful, logical, independent [positive] vs. prim, bitchy, and hysterical [negative]). These findings make clear that it is wise to avoid selecting stimuli so as to permit the subject to the IAT's category distinctions in more than one way. At the same time, De Houwer (2001) clearly showed that the IAT effectively tolerated a good deal of variance in an irrelevant attribute (valence) that was allowed to vary when subjects were asked to classify names as British (e.g., Princess Diana, the Queen Mother, and two mass murderers) and foreign (e.g., Einstein, Ghandi, Hitler, and Saddam Hussein).

    Steffens, M. C., & Plewe, I. (2001). Items' cross-category associations as a confounding factor in the Implicit Association Test. Zeitschrift für Experimentelle Psychologie, 48, 123-134.

    De Houwer, J., (2001). A structural and process analysis of the Implicit Association Test. Journal of Experimental Social Psychology, 37, 443-451.f

    Mitchell, J. P., Nosek, B. A., & Banaji, M. R. (2003). Contextual variations in implicit evaluation. Journal of Experimental Psychology: General, 132, 455-469.

    Govan, C. L., & Williams, K. D. (2004). Reversing or eliminating IAT effects by changing the affective valence of the stimulus items. Journal of Experimental Social Psychology, 40, 357-365.

    Bluemke, M., & Friese, M. (in press). Do features of stimuli influence IAT effects? Journal of Experimental Social Psychology

  Statistical Conclusion Validity    Back to Top
(Are the numbers of the IAT measures being interpreted appropriately?)

  Questions about interpretation of numerical values of IAT measures have been raised in two articles (both by Blanton & Jaccard, in press). The Greenwald et al. (in press) responses to these articles explain how IAT measures, although less than perfect, are being used in reasonable ways. The issues in the first pair of articles have to do mainly with whether the zero point and units of IAT measures are meaningful. The second set of articles also has to do with the meaningfulness of the IAT measure's zero point, with the added consideration of the relevance of this to testing multiplicative relationships among triads of association-strength measures as theorized by Greenwald et al. (2002). These issues have little relation to construct validity and internal validity questions.