A Consensus-Building Approach to Improve Inter-Rater Reliability in OSCE Evaluations by Nutrition Teachers

Authors

Li Chun Hsieh
Chia Yi Lin
Hsing Chun Lin
Chao Yen Huang

Abstract

Background: Inter-rater inconsistencies in OSCE scoring among trained nutrition teachers may compromise the reliability of assessments and the quality of teaching. Objectives: This study aimed to develop a simulation-based consensus training approach as a faculty development strategy to enhance inter-rater reliability among clinical nutrition teachers. Methods: Twelve teachers independently evaluated an OSCE video before and after a structured training process using high-fidelity simulation videos embedded with deliberate errors. Raters engaged in guided consensus-building discussions to align scoring criteria. Interrater agreement was measured using Kendall's coefficient of concordance (W). Results: The inter-rater agreement improved from moderate (W = 0.461) to substantial (W = 0.733, p < 0.05), indicating enhanced scoring consistency following the training intervention. Conclusions: Simulation-based rater training combined with structured consensus-building is efficacious in improving OSCE scoring reliability among clinical teachers. This approach supports consistent assessment practices and may be applied in broader health professions education settings.

First Page

102

Last Page

107

DOI

10.6145/jme.202506_29(2).0003

Recommended Citation

Hsieh, Li Chun; Lin, Chia Yi; Lin, Hsing Chun; and Huang, Chao Yen (2025) "A Consensus-Building Approach to Improve Inter-Rater Reliability in OSCE Evaluations by Nutrition Teachers," Journal of Medical Education: Vol. 29: Iss. 2, Article 4.
DOI: [https://doi.org/]10.6145/jme.202506_29(2).0003
Available at: https://jme.researchcommons.org/journal/vol29/iss2/4