Implementation and evaluation of portfolio assessment in the “Introduction to Clinical Medicine” course at a Korean medical school: analysis of the content validity index and inter-rater reliability

Abstract

Abstract Background Portfolio assessment is a comprehensive educational method that is crucial for fostering professionalism, knowledge, and skills. This study addresses the need for reliable evaluation methods in medical education, specifically focusing on the “Introduction to Clinical Medicine” (ICM) course and emphasizing ways of overcoming challenges related to portfolio assessment, such as ensuring evaluators' responsibilities and the reliability of the results.Methods The study included 124 portfolios from the ICM course at E University Medical School. Seven experienced evaluators received training to promote consistent scoring. The evaluation process consisted of three stages: analytical scoring, one-on-one feedback, and comprehensive evaluation. Inter-rater reliability was examined using correlation coefficients and kappa statistics. Content validity was evaluated using the content validity index (CVI).Results The inter-rater reliability analysis demonstrated a strong positive correlation between evaluators, along with high kappa coefficients, indicating a substantial level of agreement. The content validity of the assessment was shown by CVI scores of 0.90 or higher for all items, confirming the validity of the established portfolio evaluation criteria.Conclusions The study validated the reliability and content validity of portfolio assessment in the ICM course, demonstrating its consistency with educational objectives. The results indicate that portfolio assessment could be an effective and credible evaluation method in medical education. To broaden its applicability, recommendations are made to reduce evaluator subjectivity and improve the quality of the overall evaluation process.