Fine-tuning strategies for classifying community-engaged research studies using transformer-based models: Classification System and Improvement Study (Preprint)

Abstract

Community-Engaged Research (CEnR) is where institutions of higher education collaborate with organizations in their communities to exchange resources and knowledge that enhance its wellbeing. Community engagement is an integral part of a university's mission; however, there are unique challenges with reporting CEnR metrics. These challenges often restrict external relations within communities as well as federally funded research programs. Capturing studies where communities are "highly engaged" allows institutions to be more informed about the prevalence of CEnR.

We propose an updated approach done on our own previous experiments to classify Community-Engaged Research, capturing distinct levels of involvement a community partner has in the overall direction of a research study.

This paper describes the use of fine-tuning methods such as discriminative learning rates and freezing layers across three transformer-based models (BERT, Bio+ClinicalBERT, and XLM-RoBERTa), adding to the empirical evidence that the utilization of fine-tuning strategies significantly improves transformer-based models. Using deep learning to classify hand labeled CEnR studies to aid the tracking of these studies to our knowledge has not been done before.

Bio+ClinicalBERT appears to be superior, achieving a 73.08% accuracy and 62.94% F1 score on the hold out set. All the models trained in these experiments outperformed our previous ones by 10-23% in F1 score and accuracy.

Transfer learning is a viable method for tracking these studies, and we were able to provide evidence that the utilization of fine-tuning strategies significantly improves transformer-based models, as well as a tool for categorizing the type and volume of engagement.