Using subjective judgment to determine the validity of a tutorial performance evaluation instrument

Abstract

Evaluating students’ learning performance is dependent on assessment criteria from which valid inferences can be made about student learning. An existing 36-item instrument used to evaluate baccalaureate nursing students’ performance in problem-based learning tutorials was presented to experts in nursing for their subjective judgement of item validity. Quantitative analysis of data sets from experts’ judgements was used to construct a valid measurement scale for evaluating students’ tutorial performance. The objectives of the study were to determine the content validity of items in a tutorial performance-evaluation (TPE) instrument and to determine the construct validity of items through paired comparison of main and sub-items in the instrument. Academic experts (n = 8) from two South African universities were selected by means of purposive, maximum variation sampling. Data were collected in three rounds of the Delphi technique, which incorporated the Subjective Judgement Model for paired comparison of instrument items. Experts’ ratings were captured on a visual analogue scale for each item. Relative item weights were determined using
paired comparisons. Statistical analysis resulted in ratio scale data, each item being assigned a ratio relative to its weight. It was concluded that quantitative analysis of subjective judgements is useful to determine the construct validity of items through paired comparison of items in a TPE instrument. This article presents the methodological perspectives of subjective judgement to establish instrument validity.

Opsomming:Die evaluering van studente se leervermoë is afhanklik van die waardebepalingskriteria waarvan geldige afleidings betreffende die student se leerervaring gemaak kan word. ’n Bestaande instrument
met 36 items waarmee baccalaureus-verpleegkundestudente se prestasie in die probleemgebaseerde leertutoriale geëvalueer is, is aan kundiges in verpleegkunde gegee vir subjektiewe beoordeling van die geldigheid van die items. ’n Geldige meetinstrument vir die evaluering van studente
se tutoriale prestasie is ontwerp deur van die kwantitatiewe ontleding van die datastelle op grond van die kundiges se oordeel gebruik te maak. Die doelwitte van die studie was om die inhoudsgeldigheid van items in ’n evalueringsinstrument van tutoriale prestasie te bepaal en om die
konstrukgeldigheid van items te bepaal deur die gepaarde vergelyking van hoof- en sub-items in die instrument. Akademiese kundiges (n = 8) van twee Suid-Afrikaanse universiteite is deur middel van doelgerigte, maksimale variasie-steekproeftrekking geselekteer. Data is deur middel van drie rondtes van die Delphi-tegniek ingesamel, wat die subjektiewe oordeelmodel vir gepaarde vergelyking van die instrumentitems ingesluit het. Die kundiges se beoordeling is op ’n visueleanaloë-skaal
vir elke item weergegee. Relatiewe itemgewigte is deur middel van gepaarde vergelyking bepaal. Statistiese ontleding het verhoudingskaaldata tot gevolg gehad, en elke item is van ’n verhouding
relatief tot die gewig voorsien. Daar is bevind dat kwantitatiewe ontleding van subjektiewe beoordeling bruikbaar is om die geldigheid van ’n konstruk deur gepaarde vergelyking van items in ’n evalueringsinstrument van tutoriale prestasie te bepaal. Hierdie artikel bied die metodologiese
perspektiewe van die subjektiewe beoordeling aan om die geldigheid van die instrument te bepaal.