rater-reliability
Filtering by topic rater-reliability(1)Clear all filters
- PaperDOAJ — Language assessment1 May 2026
Comparative judgment and writing quality: reliability and validity across expert, peer, crowdsourced, and LLM-generated judgments
Alaa Alzarahni, Hassan Alshumrani, Faten Alarjani, Adel Alfaifi
The study examined the reliability and validity of comparative judgment (CJ) for assessing Arabic essays across expert, peer, crowdsourced, and GPT-4o raters. Results showed acceptable reliability and validity for experts, crowdsourced workers, and GPT-4o, but only limited reliability for peers. The findings support the use of CJ with diverse rater types and extend its application to new assessment contexts.
Original abstract
Abstract Prior research has demonstrated the strong reliability and concurrent validity of the comparative judgement (CJ) method for assessing writing quality. However, the factors considered when making these judgements are less investigated, as well as their potential variation across raters with different backgrounds. In Study 1, we asked four participant groups to rate the writing quality of 90 Arabic essays using the CJ method. Regression and correlational analysis showed that expert, crowdsource, and GPT-4o’s CJ ratings exhibited acceptable levels of reliability and concurrent validity, while peers demonstrated only some reliability. In Study 2, we analyzed decision comments left by three groups (experts, crowdsourced workers, peers) after making their judgments. We found evidence for construct validity among experts and crowdsourced workers. These results provide evidence supporting the reliability and validity of the CJ method across three distinct rater groups (experts, crowdsource workers, GPT-4o) and extend its application to a novel assessment context.