Reproducible variability: assessing investigator discordance across 9 research teams attempting to reproduce the same observational study

Author:

Ostropolets Anna1ORCID,Albogami Yasser2,Conover Mitchell3,Banda Juan M4,Baumgartner William A5,Blacketer Clair3ORCID,Desai Priyamvada6,DuVall Scott L78,Fortin Stephen3,Gilbert James P3,Golozar Asieh9,Ide Joshua10,Kanter Andrew S1,Kern David M3,Kim Chungsoo11ORCID,Lai Lana Y H12,Li Chenyu13ORCID,Liu Feifan14,Lynch Kristine E78,Minty Evan15,Neves Maria Inês16,Ng Ding Quan17,Obene Tontel18,Pera Victor19,Pratt Nicole20,Rao Gowtham3,Rappoport Nadav21ORCID,Reinecke Ines22,Saroufim Paola23,Shoaibi Azza3,Simon Katherine24,Suchard Marc A2526,Swerdel Joel N3ORCID,Voss Erica A3,Weaver James3,Zhang Linying1ORCID,Hripcsak George127,Ryan Patrick B13

Affiliation:

1. Department of Biomedical Informatics, Columbia University Irving Medical Center , New York, New York, USA

2. Department of Clinical Pharmacy, College of Pharmacy, King Saud University , Riyadh, Saudi Arabia

3. Observational Health Data Analytics, Janssen Research & Development , Titusville, New Jersey, USA

4. Department of Computer Science, Georgia State University , Atlanta, Georgia, USA

5. Division of General Internal Medicine, Department of Medicine, University of Colorado Anschutz Medical Campus , Aurora, Colorado, USA

6. Research IT, Technology and Digital Solutions, Stanford Medicine , Stanford, California, USA

7. VA Salt Lake City Health Care System , Salt Lake City, Utah, USA

8. Department of Internal Medicine, University of Utah School of Medicine , Salt Lake City, Utah, USA

9. Odysseus Data Services , New York, New York, USA

10. Johnson & Johnson , Titusville, New Jersey, USA

11. Department of Biomedical Sciences, Ajou University Graduate School of Medicine , Suwon, South Korea

12. Department of Informatics, Imaging & Data Sciences, University of Manchester , Manchester, UK

13. Department of Biomedical Informatics, University of Pittsburgh , Pittsburgh, Pennsylvania, USA

14. Department of Population and Quantitative Health Sciences, University of Massachusetts Chan Medical School , Worcester, Massachusetts, USA

15. O’Brien Institute for Public Health, Faculty of Medicine, University of Calgary , Calgary, Canada

16. Real World Solutions, IQVIA , Durham, North Carolina, USA

17. Department of Pharmaceutical Sciences, School of Pharmacy & Pharmaceutical Sciences, University of California , Irvine, California, USA

18. Mississippi Urban Research Center, Jackson State University , Jackson, Mississippi, USA

19. Department of Medical Informatics, Erasmus University Medical Center , Rotterdam, The Netherlands

20. Quality Use of Medicines and Pharmacy Research Centre, University of South Australia , Adelaide, Australia

21. Department of Software and Information Systems Engineering, Ben-Gurion University of the Negev , Israel

22. Institute for Medical Informatics and Biometry, Carl Gustav Carus Faculty of Medicine, Technische Universität Dresden , Dresden, Germany

23. Cleveland Institute for Computational Biology, Case Western Reserve University , Cleveland, Ohio, USA

24. VA Tennessee Valley Health Care System, Vanderbilt University Medical Center , Nashville, Tennessee, USA

25. Department of Biostatistics, University of California , Los Angeles, California, USA

26. Department of Human Genetics, University of California , Los Angeles, California, USA

27. Medical Informatics Services, New York-Presbyterian Hospital , New York, New York, USA

Abstract

Abstract Objective Observational studies can impact patient care but must be robust and reproducible. Nonreproducibility is primarily caused by unclear reporting of design choices and analytic procedures. This study aimed to: (1) assess how the study logic described in an observational study could be interpreted by independent researchers and (2) quantify the impact of interpretations’ variability on patient characteristics. Materials and Methods Nine teams of highly qualified researchers reproduced a cohort from a study by Albogami et al. The teams were provided the clinical codes and access to the tools to create cohort definitions such that the only variable part was their logic choices. We executed teams’ cohort definitions against the database and compared the number of subjects, patient overlap, and patient characteristics. Results On average, the teams’ interpretations fully aligned with the master implementation in 4 out of 10 inclusion criteria with at least 4 deviations per team. Cohorts’ size varied from one-third of the master cohort size to 10 times the cohort size (2159–63 619 subjects compared to 6196 subjects). Median agreement was 9.4% (interquartile range 15.3–16.2%). The teams’ cohorts significantly differed from the master implementation by at least 2 baseline characteristics, and most of the teams differed by at least 5. Conclusions Independent research teams attempting to reproduce the study based on its free-text description alone produce different implementations that vary in the population size and composition. Sharing analytical code supported by a common data model and open-source tools allows reproducing a study unambiguously thereby preserving initial design choices.

Funder

Alnylam Pharmaceuticals, Inc

AstraZeneca

Publisher

Oxford University Press (OUP)

Subject

Health Informatics

Cited by 2 articles. 订阅此论文施引文献 订阅此论文施引文献,注册后可以免费订阅5篇论文的施引文献,订阅后可以查看论文全部施引文献

同舟云学术

1.学者识别学者识别

2.学术分析学术分析

3.人才评估人才评估

"同舟云学术"是以全球学者为主线,采集、加工和组织学术论文而形成的新型学术文献查询和分析系统,可以对全球学者进行文献检索和人才价值评估。用户可以通过关注某些学科领域的顶尖人物而持续追踪该领域的学科进展和研究前沿。经过近期的数据扩容,当前同舟云学术共收录了国内外主流学术期刊6万余种,收集的期刊论文及会议论文总量共计约1.5亿篇,并以每天添加12000余篇中外论文的速度递增。我们也可以为用户提供个性化、定制化的学者数据。欢迎来电咨询!咨询电话:010-8811{复制后删除}0370

www.globalauthorid.com

TOP

Copyright © 2019-2024 北京同舟云网络信息技术有限公司
京公网安备11010802033243号  京ICP备18003416号-3