Benford’s Law applies to word frequency rank in English, German, French, Spanish, and Italian

Abstract

Benford’s Law states that, in many real-world data sets, the frequency of numbers’ first digits is predicted by the formula log(1 + (1/d)). Numbers beginning with a 1 occur roughly 30% of the time, and are six times more common than numbers beginning with a 9. We show that Benford’s Law applies to the the frequency rank of words in English, German, French, Spanish, and Italian. We calculated the frequency rank of words in the Google Ngram Viewer corpora. Then, using the first significant digit of the frequency rank, we found the FSD distribution adhered to the expected Benford’s Law distribution. Over a series of additional corpora from sources ranging from news to books to social media and across the languages studied, we consistently found adherence to Benford’s Law. Furthermore, at the user-level on social media, we found Benford’s Law holds for the vast majority of users’ collected posts and significant deviations from Benford’s Law tends to be a mark of spam bots.