Multidimensional Signals and Analytic Flexibility: Estimating Degrees of Freedom in Human-Speech Analyses-Reference-Cited by-同舟云学术

Multidimensional Signals and Analytic Flexibility: Estimating Degrees of Freedom in Human-Speech Analyses

Published:2023-07 Issue:3 Volume:6 Page:
ISSN:2515-2459
Container-title:Advances in Methods and Practices in Psychological Science
language:en
Short-container-title:Advances in Methods and Practices in Psychological Science

Author:

Coretta Stefano¹^ORCID,Casillas Joseph V.²,Roessig Simon³,Franke Michael⁴,Ahn Byron⁵,Al-Hoorie Ali H.⁶^ORCID,Al-Tamimi Jalal⁷,Alotaibi Najd E.⁸^ORCID,AlShakhori Mohammed K.⁹,Altmiller Ruth M.¹⁰,Arantes Pablo¹¹,Athanasopoulou Angeliki¹²,Baese-Berk Melissa M.¹³,Bailey George¹⁴^ORCID,Sangma Cheman Baira A¹²,Beier Eleonora J.¹⁵,Benavides Gabriela M.⁹,Benker Nicole¹⁶,BensonMeyer Emelia P.¹⁷,Benway Nina R.¹⁸^ORCID,Berry Grant M.¹⁹,Bing Liwen²⁰,Bjorndahl Christina²¹,Bolyanatz Mariška²²,Braver Aaron²³^ORCID,Brown Violet A.¹⁰,Brown Alicia M.²⁴,Brugos Alejna²⁵,Buchanan Erin M.²⁶^ORCID,Butlin Tanna¹²,Buxó-Lugo Andrés²⁷^ORCID,Caillol Coline²⁸,Cangemi Francesco²⁹^ORCID,Carignan Christopher³⁰,Carraturo Sita¹⁰,Caudrelier Tiphaine³¹,Chodroff Eleanor¹⁴,Cohn Michelle³²,Cronenberg Johanna¹⁶,Crouzet Olivier³³,Dagar Erica L.³⁴,Dawson Charlotte³⁵,Diantoro Carissa A.¹³^ORCID,Dokovova Marie³⁶,Drake Shiloh¹³^ORCID,Du Fengting³⁷,Dubuis Margaux³⁸,Duême Florent³¹,Durward Matthew³⁹,Egurtzegi Ander⁴⁰^ORCID,Elsherif Mahmoud M.⁴¹^ORCID,Esser Janina⁴²,Ferragne Emmanuel²⁸,Ferreira Fernanda¹⁵,Fink Lauren K.⁴³,Finley Sara⁴⁴^ORCID,Foster Kurtis¹³,Foulkes Paul¹⁴,Franzke Rosa¹⁶,Frazer-McKee Gabriel⁴⁵,Fromont Robert⁴⁶^ORCID,García Christina⁴⁷,Geller Jason⁴⁸,Grasso Camille L.⁴⁹,Greca Pia¹⁶^ORCID,Grice Martine²⁹,Grose-Hodge Magdalena S.²⁰,Gully Amelia J.¹⁴^ORCID,Halfacre Caitlin³⁷,Hauser Ivy³⁴,Hay Jen⁴⁶,Haywood Robert⁵⁰,Hellmuth Sam¹⁴,Hilger Allison I.⁵¹,Holliday Nicole⁵²,Hoogland Damar⁸,Huang Yaqian⁵³,Hughes Vincent¹⁴,Icardo Isasa Ane⁵⁴,Ilchovska Zlatomira G.⁵⁵,Jeon Hae-Sung⁵⁶,Jones Jacq³⁹,Junges Mágat N.⁵⁷,Kaefer Stephanie³⁹,Kaland Constantijn⁵⁸,Kelley Matthew C.⁵⁹^ORCID,Kelly Niamh E.³⁷,Kettig Thomas¹⁴,Khattab Ghada⁸^ORCID,Koolen Ruud⁶⁰,Krahmer Emiel⁶⁰,Krajewska Dorota⁶¹,Krug Andreas⁸,Kumar Abhilasha A.⁶²,Lander Anna⁶³,Lentz Tomas O.⁶⁰^ORCID,Li Wanyin⁵⁵,Li Yanyu⁸,Lialiou Maria⁶⁴^ORCID,Lima Ronaldo M.⁶⁵,Lo Justin J. H.³⁰^ORCID,Lopez Otero Julio Cesar⁶⁶^ORCID,Mackay Bradley⁶⁷,MacLeod Bethany⁶⁸,Mallard Mel¹⁰,McConnellogue Carol-Ann Mary⁶⁹,Moroz George⁶³,Murali Mridhula⁷⁰^ORCID,Nalborczyk Ladislas⁴⁹^ORCID,Nenadić Filip⁷¹,Nieder Jessica⁷²,Nikolić Dušan¹²,Nogueira Francisco G. S.⁷³,Offerman Heather M.⁷⁴^ORCID,Passoni Elisa⁷⁵,Pélissier Maud²⁸^ORCID,Perry Scott J.⁷⁶^ORCID,Pfiffner Alexandra M.⁷⁷,Proctor Michael⁷⁸^ORCID,Rhodes Ryan⁷⁹,Rodríguez Nicole²,Roepke Elizabeth⁸⁰,Röer Jan P.⁸¹,Sbacco Lucia⁸,Scarborough Rebecca⁸²,Schaeffler Felix⁸³,Schleef Erik⁶⁷^ORCID,Schmitz Dominic⁸⁴,Shiryaev Alexander⁶³,Sóskuthy Márton⁸⁵,Spaniol Malin⁸⁶,Stanley Joseph A.⁸⁷^ORCID,Strickler Alyssa⁸²,Tavano Alessandro⁸⁸,Tomaschek Fabian⁸⁹,Tucker Benjamin V.⁷⁶^ORCID,Turnbull Rory³⁷,Ugwuanyi Kingsley O.⁹⁰^ORCID,Urrestarazu-Porta Iñigo⁴⁰^ORCID,van de Vijver Ruben⁷²,Van Engen Kristin J.¹⁰,van Miltenburg Emiel⁶⁰^ORCID,Wang Bruce Xiao⁹¹,Warner Natasha⁹,Wehrle Simon²⁹^ORCID,Westerbeek Hans⁹²,Wiener Seth⁹³,Winters Stephen¹²,Wong Sidney G.-J.⁹⁴,Wood Anna¹³^ORCID,Wottawa Jane⁹⁵,Xu Chenzi⁹⁶^ORCID,Zárate-Sández Germán⁹⁷,Zellou Georgia³²,Zhang Cong⁸^ORCID,Zhu Jian⁹⁸,Roettger Timo B.⁹⁹^ORCID

Affiliation:

1. Department of Linguistics and English Language, University of Edinburgh, Edinburgh, United Kingdom

2. Department of Spanish and Portuguese, Rutgers University, New Brunswick, New Jersey

3. Department of Linguistics, Cornell University, Ithaca, New York

4. Department of General and Computational Linguistics, University of Tübingen, Tübingen, Germany

5. Program in Linguistics, Princeton University, Princeton, New Jersey

6. Jubail English Language and Preparatory Year Institute, Royal Commission for Jubail and Yanbu, Jubail, Saudi Arabia

7. Université Paris Cité, Laboratoire de Linguistique Formelle, CNRS, Paris, France

8. School of Education, Communication and Language Sciences, Newcastle University, Newcastle upon Tyne, United Kingdom

9. Department of Linguistics, University of Arizona, Tucson, Arizona

10. Psychological and Brain Sciences, Washington University in St. Louis, St. Louis, Missouri

11. Departamento de Letras, Universidade Federal de São Carlos, São Carlos, Brazil

12. School of Languages, Linguistics, Literatures and Cultures, University of Calgary, Calgary, Alberta, Canada

13. Department of Linguistics, University of Oregon, Eugene, Oregon

14. Department of Language and Linguistic Science, University of York, York, United Kingdom

15. Department of Psychology, University of California, Davis, Davis, California

16. Institute of Phonetics and Speech Processing, University of Munich, Munich, Germany

17. Department of Linguistics, University of Pennsylvania, Philadelphia, Pennsylvania

18. Department of Communication Sciences and Disorders, Syracuse University, Syracuse, New York

19. Department of Spanish, Villanova University, Villanova, Pennsylvania

20. Department of English Language and Linguistics, University of Birmingham, Birmingham, United Kingdom

21. Department of Philosophy, Carnegie Mellon University, Pittsburgh, Pennsylvania

22. Department of Spanish and French Studies, Occidental College, Los Angeles, California

23. Department of English, Texas Tech University, Lubbock, Texas

24. Department of Spanish and Portuguese, University of Arizona, Tucson, Arizona

25. Division of Mathematics and Computer Science, Simmons University, Boston, Massachusetts

26. Analytics, Harrisburg University of Science and Technology, Harrisburg, Pennsylvania

27. Department of Psychology, University at Buffalo, Buffalo, New York

28. CLILLAC-ARP, Université Paris Cité, Paris, France

29. IfL-Phonetik, University of Cologne, Cologne, Germany

30. Department of Speech, Hearing and Phonetic Sciences, University College London, London, United Kingdom

31. Basque Center on Cognition Brain and Language, Donostia, Spain

32. Department of Linguistics, University of California, Davis, Davis, California

33. Laboratoire de Linguistique de Nantes (UMR6310), Nantes Université, CNRS, Nantes, France

34. Department of Linguistics and TESOL, University of Texas at Arlington, Arlington, Texas

35. School of Psychology, Newcastle University, Newcastle upon Tyne, United Kingdom

36. School of Psychological Sciences and Health, University of Strathclyde, Glasgow, United Kingdom

37. School of English Literature, Language and Linguistics, Newcastle University, Newcastle upon Tyne, United Kingdom

38. Department of Comparative Language Science, University of Zurich, Zurich, Swizerland

39. Department of Linguistics, University of Canterbury, Christchurch, New Zealand

40. IKER (UMR 5478), CNRS, Bayonne, France

41. Department of Neuroscience, Psychology and Behaviour, University of Leicester, Leicester, United Kingdom

42. Statistics Group, Association for Diversity in Linguistics, Cologne, Germany

43. Department of Music, Max Planck Institute for Empirical Aesthetics, Frankfurt, Germany

44. Department of Psychology, Pacific Lutheran University, Parkland, Washington

45. Department of Languages, Linguistics, and Translation, Université Laval, Quebec City, Quebec, Canada

46. New Zealand Institute of Language, Brain and Behaviour, University of Canterbury, Christchurch, New Zealand

47. Department of Languages, Literatures, and Cultures, Saint Louis University, St. Louis, Missouri

48. Department of Psychology, Princeton University, Princeton, New Jersey

49. LPC, Aix-Marseille Université, CNRS, France

50. Ao Tawhiti Unlimited Discovery, Christchurch, New Zealand

51. Department of Speech, Language, and Hearing Sciences, University of Colorado Boulder, Boulder, Colorado

52. Department of Linguistics and Cognitive Science, Pomona College, Claremont, California

53. University of California, Los Angeles, Los Angeles, California

54. Department of Modern and Classical Languages and Literatures, California State University, Northridge, California

55. School of Psychology, University of Birmingham, Birmingham, United Kingdom

56. School of Humanities, Language and Global Studies, University of Central Lancashire, Preston, United Kingdom

57. Programa de Pós-Graduação em Letras, Federal University of Rio Grande do Sul, Porto Alegre, Brazil

58. Institute of Linguistics, University of Cologne, Cologne, Germany

59. Department of Linguistics, University of Washington, Seattle, Washington

60. Department of Communication and Cognition, Tilburg University, Tilburg, the Netherlands

61. Department of Linguistics and Basque Studies, University of the Basque Country UPV/EHU, Vitoria-Gasteiz, Spain

62. Department of Psychology, Bowdoin College, Brunswick, Maine

63. Linguistic Convergence Laboratory, HSE University, Moscow, Russia

64. Institute of German Language I Linguistics, University of Cologne, Cologne, Germany

65. Department of English Language Studies, Federal University of Ceará, Fortaleza, Brazil

66. Department of Hispanic Studies, University of Houston, Houston, Texas

67. Department of English and American Studies, University of Salzburg, Salzburg, Austria

68. School of Linguistics and Language Studies, Carleton University, Ottawa, Ontario, Canada

69. Population Health Sciences Institute, Faculty of Medical Sciences, Newcastle University, Newcastle upon Tyne, United Kingdom

70. Speech and Language Therapy, University of Strathclyde, Glasgow, United Kingdom

71. Department of Psychology, Faculty of Media and Communications, Singidunum University, Belgrade, Serbia

72. Institute of Linguistics, Heinrich Heine University Düsseldorf, Düsseldorf, Germany

73. Graduate Program in Linguistics, Federal University of Ceará, Fortaleza, Brazil

74. Department of World Languages, Literatures and Cultures, University of Arkansas, Fayetteville, Arkansas

75. Deparment of Linguistics, Queen Mary University of London, London, United Kingdom

76. Department of Linguistics, University of Alberta, Alberta, Canada

77. Department of Linguistics, University of California, Berkeley, Berkeley, California

78. Department of Linguistics, Macquarie University, Sydney, Australia

79. Center for Cognitive Science, Rutgers University, Piscataway, New Jersey

80. Department of Speech, Language, and Hearing Sciences, Saint Louis University, St. Louis, Missouri

81. Department of Psychology, Witten/Herdecke University, Witten, Germany

82. Department of Linguistics, University of Colorado Boulder, Boulder, Colorado

83. Clinical Audiology, Speech and Language Research Centre, Queen Margaret University Edinburgh, Edinburgh, United Kingdom

84. Department of English and American Studies, Heinrich Heine University Düsseldorf, Düsseldorf, Germany

85. Department of Linguistics, University of British Columbia, Vancouver, British Columbia, Canada

86. Department of Psychiatry and Psychotherapy, University Hospital Cologne, Cologne, Germany

87. Department of Linguistics, Brigham Young University, Provo, Utah

88. Department of Neuroscience, Max Planck Institute for Empirical Aesthetics, Frankfurt, Germany

89. Department of General Linguistics, University of Tübingen, Tübingen, Germany

90. Department of English and Literary Studies, University of Nigeria, Nsukka, Nigeria

91. Chinese and Bilingual Studies, Hong Kong Polytechnic University, Hong Kong SAR, China

92. Department of Languages, Literature and Communication, Utrecht University, Utrecht, the Netherlands

93. Department of Modern Languages, Carnegie Mellon University, Pittsburgh, Pennsylvania

94. Geospatial Research Institute, University of Canterbury, Christchurch, New Zealand

95. Département de Lettres Modernes, LIUM, LST, Le Mans Université, Le Mans, France

96. Faculty of Linguistics, Philology and Phonetics, University of Oxford, Oxford, United Kingdom

97. Department of Spanish, Western Michigan University, Kalamazoo, Michigan

98. School of Information, University of Michigan, Ann Arbor, Michigan

99. Department of Linguistics and Scandinavian Studies, University of Oslo, Oslo, Norway

Abstract

Recent empirical studies have highlighted the large degree of analytic flexibility in data analysis that can lead to substantially different conclusions based on the same data set. Thus, researchers have expressed their concerns that these researcher degrees of freedom might facilitate bias and can lead to claims that do not stand the test of time. Even greater flexibility is to be expected in fields in which the primary data lend themselves to a variety of possible operationalizations. The multidimensional, temporally extended nature of speech constitutes an ideal testing ground for assessing the variability in analytic approaches, which derives not only from aspects of statistical modeling but also from decisions regarding the quantification of the measured behavior. In this study, we gave the same speech-production data set to 46 teams of researchers and asked them to answer the same research question, resulting in substantial variability in reported effect sizes and their interpretation. Using Bayesian meta-analytic tools, we further found little to no evidence that the observed variability can be explained by analysts’ prior beliefs, expertise, or the perceived quality of their analyses. In light of this idiosyncratic variability, we recommend that researchers more transparently share details of their analysis, strengthen the link between theoretical construct and quantitative system, and calibrate their (un)certainty in their conclusions.

Publisher

SAGE Publications

Subject

General Psychology

Link

http://journals.sagepub.com/doi/pdf/10.1177/25152459231162567

Reference106 articles.

1. Consensus-based guidance for conducting and reporting multi-analyst studies

2. Overspecification in written instruction

3. Has the Credibility of the Social Sciences Been Credibly Destroyed? Reanalyzing the “Many Analysts, One Data Set” Project

4. Baselga A., Orme D., Villeger S., De Bortoli J., Leprieur F., Logez M. (2020). betapart: Partitioning beta diversity into turnover and nestedness components (R package version 1.5.2). https://CRAN.R-project.org/package=betapart

5. Time to get personal? The impact of researchers choices on the selection of treatment targets using the experience sampling methodology