A prominent vein of comparative cognition research asks which cognitive abilities may be ascribed to different species. Here, we argue that the current structure of comparative cognition makes it near impossible to evaluate the accuracy of many of the claims produced by the field’s empirical research. We base this argument on six observations: i) the field is biased towards confirming more exceptional abilities in animals, ii) a likely high rate of false positive discovery, iii) the persistence of the bias towards more exceptional abilities, even in the presence of strong methodological criticism, iv) the absence of a formal method to assess evidence of absence of a cognitive ability, v) the ambiguity in definitions used to make claims, and vi) the small size of the research field. We then highlight how the widespread methodological and conceptual criticism in the comparative cognition literature does not facilitate scientific progress, rather it can impede it by creating an illusion of scientific rigour. Finally, we call for a reform of our research field that draws on recent developments in related fields and the Open Science movement.