The Effect of Rubric on Rater's Severity and Bias in TVET Laboratory Practice Assessment: Analysis Using Many-Facet Rasch Measurement

Abstract

Performance assessments such as laboratory practice in Technical and Vocational Education and Training (TVET) are difficult to measure because they are subjective in their characteristics, which can cause bias. The use of a rubric scoring scale is suitable for measuring students' practical competency as it could translate the qualitative criteria into quantitative forms on a grading scale. The purpose of this study is to analyse the effect of rater severity for the performance of component installation of an electronic circuit project using Many-Facet Rasch Measurement (MFRM). Nine raters have examined the competency performance of 68 students using a four-point rating scale rubric consisting of 16 items. This study was conducted at three vocational colleges. The results showed that all raters measured with different severities but very high consistency. The effect of rater severity did not exist at the group level but exhibited at the individual level. It can also be examined that all raters were biased on the item and also towards students, either by being severe or lenient. Thus, the rubric cannot eliminate the effect of bias during the assessment. However, it can help raters to be consistent so that they have the same understanding when translating the rubric. These findings have implications for the psychometric quality of the electronic circuit project performance assessment.