How Accurate are GPT-3’s Hypotheses About Social Science Phenomena?

Abstract

Abstract We test whether GPT-3 can accurately predict simple study outcomes in the social sciences. Ground truth outcomes were obtained by surveying 600 adult US citizens about their political attitudes. GPT-3 was prompted to predict the direction of the empirical inter-attitude correlations. Machine-generated hypotheses were accurate in 78% (zero-shot), 94% (five-shot and chained prompting), and 97% (extensive finetuning) of cases. Positive and negative correlations were balanced in the ground truth data. These results encourage the development of hypothesis engines for more challenging contexts. Moreover, they highlight the importance of addressing the numerous ethical and philosophical challenges that arise with hypothesis automation. While future hypothesis engines could potentially compete with human researchers in terms of empirical accuracy, they have inherent drawbacks that preclude full automations for the foreseeable future.