Applicability of two standard setting methods for enhancing the reporting of assessment results within the South African education context

Abstract

The study reported on here contributes to the growing body of knowledge on the use of standard setting methods for improving the reporting and utility value of assessment results in South Africa as well as for addressing the conceptual shortcomings of the Curriculum and Assessment Policy Statement (CAPS) reporting framework. Using data from the “verification” version of the Annual National Assessments (ANAs), we explored relevant technical and conceptual factors to consider for the application of standard setting methods. Two sets of panellists were trained to generate cut scores for Grade 6 mathematics and English First Additional Language (FAL), one using the Angoff method and the other the Objective Standard Setting (OSS) method. The findings indicate that the 2 methods generated different sets of cut scores across the performance levels for both subjects. While these cut scores had significant implications for the percentage of learners classified at each performance level, they were consistent with findings from other studies. We also identified 4 key factors to address when undertaking standard setting exercises: engagement with test content, resource requirements, requisite expertise and software, and collective accountability. We conclude that standard setting approaches should be the preferred option to the CAPS reporting framework when reporting assessment results in South Africa. More importantly, the decision on the most appropriate method for the South African context depends largely on the extent to which the 4 key factors identified can be addressed.