Digital library evaluation measures in academic settings: Perspectives from scholars and practitioners

Abstract

Evaluation criteria and appropriate measures are critical to the success of digital library evaluation. The key problem is a lack of specific measures, especially from a heuristic perspective, corresponding to diverse dimensions and criteria. This study explored a variety of measures for digital library evaluation focusing on their appropriateness. It also compared the similarities and differences in perceptions of the appropriateness of digital library evaluation measures of two groups. Sixty-one participants were recruited representing scholars and academic digital librarians. The participants were instructed to fill in an in-depth survey consisting of 174 measures associated with 10 evaluation dimensions and 82 criteria. The findings of this study highlight the most appropriate measures in each dimension and show significant agreement in identifying appropriate measures by digital library scholars and librarians. Differences in the rating of the measures were found between the two groups in the dimensions of “interface design,” “system and technology,” “effects on users,” “administration,” “user engagement,” and “context.” The significance of this study lies not only in the integration of the perspectives on measures from both researchers and practitioners, but also in providing an inclusive list of measures to guide practitioners to effectively evaluate digital libraries in academic settings.