A computational validation for the Health Concept Maturity Levels Questionnaire

Abstract

Abstract Background The healthcare market is undergoing rapid transformation, requiring the integration of user needs from the earliest stages of product and service design. Living Labs are emerging as a model for the co-creation and evaluation of user-centered innovations. In this work, we developed a health CML grid and questionnaire to assess the maturity of health concepts. Methods The research process included multiple stages, starting with the creation of the Association Innov'Autonomie - Health Concept Maturity Levels Questionnaire − 178-items (AIA/CMLHQ178), designed to evaluate health concept maturity levels. Speech acts from Health CML expert interventions were then annotated and used as data for our machine learning and deep learning models. We used the CatBoost algorithm in the first experiment to discern individual Health CML factors from speech acts to generate factor probabilities used to feed a neural network trained to take the final decision, to evaluate whether the network could accurately identify the membership factors of Health CML criteria when presented with items from the AIA/CMLHQ178 questionnaire. Results The results of the study indicate that only the models trained with the true factors are able to correctly identify the corresponding factor in the sequentially encoded texts, with the exception of the need domains whose artificial performance was measured according to sensitivity. The general performance of the different CatBoost algorithms used to predict one factor versus the other two showed similar performance. For the questionnaire, the models trained with the real factors also showed better performance in identifying the matching factors compared to the random factors. A marginal difference was observed between the "Need" and "Technology" factors. Conclusion This study provides initial evidence of content validity for the AIA/CMLHQ178, introducing a novel approach to validate psychometric instruments using machine learning and deep learning techniques. However, overlaps between "Programmatic" and "Need" factors indicate a need for improvement in the CML Health model. Future work will focus on enhancing these models and investigating their potential application in other psychometric tools.