Text mining of veterinary forums for epidemiological surveillance supplementation

Abstract

AbstractWeb scraping and text mining are popular computer science methods deployed by public health researchers to augment traditional epidemiological surveillance. However, within veterinary disease surveillance, such techniques are still in the early stages of development and have not yet been fully utilised. This study presents an exploration into the utility of incorporating internet-based data to better understand smallholder farming communities within the UK, by using online text extraction and the subsequent mining of this data. Web scraping of the livestock fora was conducted, with text mining and topic modelling of data in search of common themes, words, and topics found within the text, in addition to temporal analysis through anomaly detection. Results revealed that some of the key areas in pig forum discussions included identification, age management, containment, and breeding and weaning practices. In discussions about poultry farming, a preference for free-range practices was expressed, along with a focus on feeding practices and addressing red mite infestations. Temporal topic modelling revealed an increase in conversations around pig containment and care, as well as poultry equipment maintenance. Moreover, anomaly detection was discovered to be particularly effective for tracking unusual spikes in forum activity, which may suggest new concerns or trends. Internet data can be a very effective tool in aiding traditional veterinary surveillance methods, but the requirement for human validation of said data is crucial. This opens avenues of research via the incorporation of other dynamic social media data, namely Twitter, in addition to location analysis to highlight spatial patterns.