The questionnaire method in systems research: an overview of sample sizes, response rates and statistical approaches utilized in studies

Abstract

Purpose The purpose of this study is to identify typical sample sizes and response rates in questionnaire research studies within the discipline of information systems, as well as the top statistical analyses utilized for questionnaire data in these studies. Design/methodology/approach A total of 842 articles published between the years of 2000 and 2019 were identified that met the criteria of using a questionnaire as the research method. These articles were analyzed based on the sample size, response rate (if applicable) and statistical analysis methods used. Findings The typical questionnaire study received between 136 (first quartile) and 374 (third quartile) respondents, with a median number of 217. Typical response rate ranged between 16.5% and 50.0%, with a median of 27.8%. it was found that articles published in journals included in the Social Science Citation Index had significantly larger numbers of respondents than those not included in the index, though no difference was found for response rate. Studies that utilized more advanced statistical methods (regression analysis, structural equation modeling) were found to have significantly larger sample sizes than those that utilized only descriptive statistics or t-tests. Structural equation modeling, including the partial least squares approach, was used in the largest number of studies. Originality/value This study is the first to broadly examine the typical sample size, response rates and methods of statistical analysis used in information systems questionnaire studies. The findings of this study may be useful for systems researchers in developing appropriate procedures for questionnaire-based research.