Abstract
Big data require new techniques to handle the information they come with. Here we consider four datasets (email communication, Twitter posts, Wikipedia articles and Gutenberg books) and propose a novel statistical framework to predict global statistics from random samples. More precisely, we infer the number of senders, hashtags and words of the whole dataset and how their abundances (i.e. the popularity of a hashtag) change through scales from a small sample of sent emails per sender, posts per hashtag and word occurrences. Our approach is grounded on statistical ecology as we map inference of human activities into the unseen species problem in biodiversity. Our findings may have applications to resource management in emails, collective attention monitoring in Twitter and language learning process in word databases.
Funder
Progetto Dottorati - Fondazione Cassa di Risparmio di Padova e Rovigo
neXt grant
STARS grant 2019 from University of Padova
University of Padova through “Excellence Project 2018” of the Cariparo foundation
H2020 European Research Council
Italian Ministry of Education, University and Research (MIUR), “Dipartimenti di Eccellenza”
Publisher
Public Library of Science (PLoS)