An analysis of validity of a rubric in an across the curriculum programme of English as a foreign language

Abstract

This investigation reports on a study of the validation of a rubric for oral exams in a context of standardised evaluation at university level in Chile regarding three aspects of validity: construct validity, face validity and reliability. The study examined the rubric applied in an across the curriculum English programme at USACH. The data was collected from three different formation programmes: Ingeniería en informática, Pedagogía en Lenguaje, Enfermería. The focus of the investigation relies on understanding the elements that contribute to the three aspects of the rubric’s validity. This is a mixed-method study that collected self-reports by relevant stakeholders (students and raters) by applying a survey focused on their perception of the three aspects of the rubric’s validity. Additionally, the actual rubric and associated documents (course programmes, student scores, etc.) were also analysed by applying a matrix of aspects of the three aspects of validity. The self-reports and the rubric were analysed in search for evidence that the rubric reflected adequately the correct functioning of the theoretical constructs of the instrument (construct validity, face validity and reliability). By making preliminary observations, it was expected that perceptions of stakeholders regarding the aspects of validity of the common rubric would reveal low levels of construct validity and reliability, and consequently, of face validity. The results showed that the rubric, indeed, presented problems regarding construct validity and reliability, but surprisingly, not regarding face validity, exposing the positive appreciation of the rubric of both stakeholders. Further research suggests for this issue to be addressed explicitly by considering the observation of variables that could explain this positive evaluation.