Assessing Apps for Health Care Workers Using the ISYScore-Pro Scale: Development and Validation Study

Abstract

Background The presence of mobile phone and smart devices has allowed for the use of mobile apps to support patient care. However, there is a paucity in our knowledge regarding recommendations for mobile apps specific to health care professionals. Objective The aim of this study is to establish a validated instrument to assess mobile apps for health care providers and health systems. Our objective is to create and validate a tool that evaluates mobile health apps aimed at health care professionals based on a trust, utility, and interest scale. Methods A five-step methodology framework guided our approach. The first step consisted of building a scale to evaluate apps for health care professionals based on a literature review. This was followed with expert panel validation through a Delphi method of (rated) web-based questionnaires to empirically evaluate the inclusion and weight of the indicators identified through the literature review. Repeated iterations were followed until a consensus greater than 75% was reached. The scale was then tested using a pilot to assess reliability. Interrater agreement of the pilot was measured using a weighted Cohen kappa. Results Using a literature review, a first draft of the scale was developed. This was followed with two Delphi rounds between the local research group and an external panel of experts. After consensus was reached, the resulting ISYScore-Pro 17-item scale was tested. A total of 280 apps were originally identified for potential testing (140 iOS apps and 140 Android apps). These were categorized using International Statistical Classification of Diseases, Tenth Revision. Once duplicates were removed and they were downloaded to confirm their specificity to the target audience (ie, health care professionals), 66 remained. Of these, only 18 met the final criteria for inclusion in validating the ISYScore-Pro scale (interrator reliabilty 92.2%; kappa 0.840, 95% CI 0.834-0.847; P<.001). Conclusions We have developed a reproducible methodology to objectively evaluate mobile health apps targeted to health care professionals and providers, the ISYScore-Pro scale. Future research will be needed to adapt the scale to other languages and across other domains (eg, legal compliance or security).