Evaluation of a Training Program to Improve Organizational Capacity for Health Systems Analytics

Abstract

Objective The Leadership in Analytics and Data Science (LEADS) course was evaluated for effectiveness. LEADS was a 6-month program for working biomedical and health informatics (BMHI) professionals designed to improve analytics skills, knowledge of enterprise applications, data stewardship, and to foster an analytics community of practice through lectures, hands-on skill building workshops, networking events, and small group projects. Methods The effectiveness of the LEADS course was evaluated using the Kirkpatrick Model by assessing pre- and postcourse knowledge, analytics capabilities, goals, practice, class lecture reaction, and change in the size of participant professional networks. Differences in pre- and postcourse responses were analyzed with a Wilcoxon signed rank test to determine significance, and effect sizes were computed using a z-statistic. Results Twenty-nine students completed the course with 96% of respondents reporting that they were “very” or “extremely” likely to recommend the course. Participants reported improvement in several analytics capabilities including Epic data warehousing (p = 0.017), institutional review board policy (p = 0.005), and data stewardship (p = 0.007). Changes in practice patterns mirrored those in self-reported capability. On average, the participant professional network doubled. Conclusion LEADS was the first course targeted to working BMHI professional at a large academic medical center to have a formal effectiveness evaluation be published in the literature. The course achieved the goals of expansion of BMHI knowledge, skills, and professional networks. The LEADS course provides a template for continuing education of working BMHI professionals.