Applicability and fairness of the oral examination in undergraduate psychiatry training in South Africa

Abstract

Objective: There are several methods of evaluating medical students’ performance, such as written examination, oral examination and objective structured clinical examination (OSCE). Many studies have focused on the reliability and validity of these methods but few studies have explored comparison between these methods. Psychiatry is the only subject at the University of Stellenbosch where the final assessment consists of solely an oral component. The aim of the study was to compare students’ final overall and discipline specific examination marks (i.e. in the other subjects) with the examination marks in psychiatry, and to determine if content or structure (e,g. oral, written or OSCE format) of examination impacts more on the student performance in the examination. Method: 343 final year medical students were included. All undertook their psychiatry rotation at the University of Stellenbosch, South Africa during 2008 and 2009. Data of marks obtained in all the disciplines during 2008 and 2009 were collected and class marks were compared with their final examination marks across all disciplines. Bland-Altman plots were used to assess the level of agreement between the class and examination marks. Cases below the lower threshold were compared to all other cases across all disciplines. The odds ratio for group status was calculated for gender distribution of examiners. Results: The psychiatry class mark and final oral examination mark provided similar measures within a width of 31.5. Cases below the threshold had poorer performance in two other disciplines. The gender distribution of the examiners (female-female) significantly increased the odds ratio for poorer performance in the oral examination.Conclusion: The results suggest that a group of students underperform in their final examination independent of method of evaluation and that the gender of examiners impacts on examination marks. Therefore future research should focus on identifying and modifying factors (including choice of examiner combinations) that contribute to the poor performance of medical students in their final examination, in order to help students perform better. Gender distribution of examiners should also be considered when examinations are structured and designed.

Keywords: Oral; Medical; Student; Gender; Examiner; Performance; OSCE