Impact measurement: quantitatively determining the improvement in students’ academic literacy levels at a South African university

Abstract

Academic literacy interventions are becoming increasingly important in a country where the secondary education system no longer  adequately prepares students for the literacy demands of higher education. This article investigates whether there was an improvement
in students’ academic literacy levels between the onset and completion of an academic literacy module at a South African university. This is done by using a combination of instruments selected from a proposed evaluation design for academic literacy interventions, suggested by Fouché, Van Dyk and Butler (2016). A pre-test / post-test design is used, where, firstly, students’ results in a validated and reliable  generic academic literacy test are considered. Secondly, students’ writing abilities are assessed by means of two instruments: a rubric and quantitative measures. Finally, students’ academic literacy marks are correlated to other variables, and interpreted within the context of the study, to give additional insight into the impact of the academic literacy course. Findings indicate that students showed an  improvement across a wide array of academic literacy abilities, in particular their ability to use source material in their writing assignments, and their usage of a wider range of academic vocabulary. However, there were also areas where students did not display any
improvement. Based on the experience of implementing various evaluation instruments, several recommendations are made on how future researchers could avoid pitfalls that were encountered in this study.