Correcting Smith et al.’s (2018) Criticisms of All Rorschach Studies in Mihura, Meyer, Dumitrascu, and Bombel’s (2013) Meta-Analyses

Abstract

Abstract. Smith et al. (2018) describe their article as “an evaluation as to the extent that individual studies have conformed to [ Exner’s (1995a) ] proposed methodological criteria” (Abstract). However, the authors did not conduct analyses to compare research before and after Exner (1995a) in order to assess its impact nor were the set of criteria they used Exner’s. Instead, they critiqued the individual studies in Mihura and colleagues’ (2013) meta-analyses, declaring all methodologically unsound (including Exner’s). They conjectured that Mihura et al. omitted studies with less “methodological bias” that would have provided more support for Rorschach validity. I explain why most of the criteria they use to criticize the studies’ methodology are not sound. But to directly test their hypotheses, I requested their ratings of study methodology. Findings from studies they rated as having more methodological “issues” (e.g., not reporting IQ or Lambda range) or as being “application studies” – which they said should be excluded – were not less supportive of Rorschach validity as they assumed would be the case. The small effect size associations ( r < |.10|) were also in the opposite direction of which Smith et al. argued to be true, indicating that the criteria by which they evaluated other researchers’ studies were not sound. Our findings do indicate that researchers are responding to the one criterion that is clearly stated in Exner (1995a) , which is Weiner’s (1991) recommendation to report interrater reliability; before 1991, 12% of studies reported interrater reliability, which afterward jumped to 78.4%. Other claims in the article by Smith et al. are also addressed.