Author:
Robson Samuel G.,Searston Rachel A.,Thompson Matthew B.,Tangen Jason M.
Abstract
AbstractDecisions in forensic science are often binary. A firearms expert must decide whether a bullet was fired from a particular gun or not. A face comparison expert must decide whether a photograph matches a suspect or not. A fingerprint examiner must decide whether a crime scene fingerprint belongs to a suspect or not. Researchers who study these decisions have therefore quantified expert performance using measurement models derived largely from signal detection theory. Here we demonstrate that the design and measurement choices researchers make can have a dramatic effect on the conclusions drawn about the performance of forensic examiners. We introduce several performance models – proportion correct, diagnosticity ratio, and parametric and non-parametric signal detection measures – and apply them to forensic decisions. We use data from expert and novice fingerprint comparison decisions along with a resampling method to demonstrate how experimental results can change as a function of the task, case materials, and measurement model chosen. We also graphically show how response bias, prevalence, inconclusive responses, floor and ceiling effects, case sampling, and number of trials might affect one’s interpretation of expert performance in forensics. Finally, we discuss several considerations for experimental and diagnostic accuracy studies: (1) include an equal number of same-source and different-source trials; (2) record inconclusive responses separately from forced choices; (3) include a control comparison group; (4) counterbalance or randomly sample trials for each participant; and (5) present as many trials to participants as is practical.
Funder
University of New South Wales
Publisher
Springer Science and Business Media LLC
Reference80 articles.
1. Albright, T. D. (2022). How to make better forensic decisions. Proceedings of the National Academy of Sciences, 119(38), e2206567119. https://doi.org/10.1073/pnas.2206567119
2. Albright, T. D. (2021). The US Department of Justice stumbles on visual perception. Proceedings of the National Academy of Sciences, 118(24), e2102702118. https://doi.org/10.1073/pnas.2102702118
3. American Association for the Advancement of Science. (2017). Forensic science assessments: A quality and gap analysis–Latent fingerprint examination. Washington, DC. https://www.aaas.org/resources/forensic-science-assessments-quality-and-gap-analysis
4. Arkes, H. R., Koehler, J. J. (2022). Inconclusives and error rates in forensic science: a signal detection theory approach, Law, Probability and Risk, 20(3), 153–168. https://doi.org/10.1093/lpr/mgac005
5. Biedermann, A., & Kotsoglou, K. N. (2021). Forensic science and the principle of excluded middle: “Inconclusive” decisions and the structure of error rate studies. Forensic Science International: Synergy, 3, 100147. https://doi.org/10.1016/j.fsisyn.2021.100147