Abstract
Background
Accelerated digitalization in the health sector requires the development of appropriate evaluation methods to ensure that digital health technologies (DHTs) are safe and effective. Software as a medical device (SaMD) is a commonly used DHT by clinicians to provide care to patients. Traditional research methods for evaluating health care products, such as randomized clinical trials, may not be suitable for DHTs, such as SaMD. However, evidence to show their safety and efficacy is needed by regulators before they can be used in practice. Clinical simulation can be used by researchers to test SaMD in an agile and low-cost way; yet, there is limited research on criteria to assess the robustness of simulations and, subsequently, their relevance for a regulatory decision.
Objective
The objective of this study was to gain consensus on the criteria that should be used to assess clinical simulation from a regulatory perspective when it is used to generate evidence for SaMD.
Methods
An eDelphi study approach was chosen to develop a set of criteria to assess clinical simulation when used to evaluate SaMD. Participants were recruited through purposive and snowball sampling based on their experience and knowledge in relevant sectors. They were guided through an initial scoping questionnaire with key themes identified from the literature to obtain a comprehensive list of criteria. Participants voted upon these criteria in 2 Delphi rounds, with criteria being excluded if consensus was not met. Participants were invited to add qualitative comments during rounds and qualitative analysis was performed on the comments gathered during the first round. Consensus was predefined by 2 criteria: if <10% of the panelists deemed the criteria as “not important” or “not important at all” and >60% “important” or “very important.”
Results
In total, 33 international experts in the digital health field, including academics, regulators, policy makers, and industry representatives, completed both Delphi rounds, and 43 criteria gained consensus from the participants. The research team grouped these criteria into 7 domains—background and context, overall study design, study population, delivery of the simulation, fidelity, software and artificial intelligence, and study analysis. These 7 domains were formulated into the simulation for regulation of SaMD framework. There were key areas of concern identified by participants regarding the framework criteria, such as the importance of how simulation fidelity is achieved and reported and the avoidance of bias throughout all stages.
Conclusions
This study proposes the simulation for regulation of SaMD framework, developed through an eDelphi consensus process, to evaluate clinical simulation when used to assess SaMD. Future research should prioritize the development of safe and effective SaMD, while implementing and refining the framework criteria to adapt to new challenges.
Reference34 articles.
1. International Medical Device Regulators ForumSoftware as a medical device (SaMD): key definitions20132024-06-04International Medical Device Regulators Forumhttps://www.imdrf.org/sites/default/files/docs/imdrf/final/technical/imdrf-tech-131209-samd-key-definitions-140901.pdf
2. Medicines and Healthcare products Regulatory AgencyConsultation on the Future Regulation of Medical Devices in the United Kingdom20212024-06-05London, UKMedicines and Healthcare products Regulatory Agencyhttps://www.gov.uk/government/consultations/consultation-on-the-future-regulation-of-medical-devices-in-the-united-kingdom
3. International Medical Device Regulators ForumSoftware as a medical device: possible framework for risk categorization and corresponding considerations20142024-06-05International Medical Device Regulators Forumhttps://www.imdrf.org/sites/default/files/docs/imdrf/final/technical/imdrf-tech-140918-samd-framework-risk-categorization-141013.pdf
4. International Medical Device Regulators ForumSoftware as a medical device (SaMD): application of quality management system20152024-06-04International Medical Device Regulators Forumhttps://www.imdrf.org/sites/default/files/2021-09/imdrf-cons-samd-aqms-150326.pdf
5. Food and Drug AdministrationSoftware as a Medical Device (SAMD): Clinical Evaluation. Guidance for Industry and Food and Drug Administration Staff20172024-06-05Silver Spring, MDFood and Drug Administrationhttps://www.fda.gov/media/100714/download?attachment