llm-scoring
Filtering by topic llm-scoring(1)Clear all filters
- PaperJournal of Learning Analytics18 May 2026
Learning-Aware Reliability Estimation for Tutor Skill Assessment Using Large Language Models
Conrad Borchers, Danielle R. Thomas, Jionghao Lin, Kenneth R. Koedinger
A novel Rasch-based split-half method adjusts reliability estimates for learning gains in pre-post assessments scored by large language models. Applied to GPT-4 scoring of 985 tutors' open-ended responses, the method achieved satisfactory reliability (0.733) with as few as 14 items, and identified three skill subdimensions: socio-emotional, cognitive, and fairness-related tutoring skills.
Original abstract
Assessment is foundational to learning analytics, especially in evaluating instructional interventions and guiding improvement in online learning environments. With the growing use of large language models (LLMs) to score open-ended responses, questions arise about the reliability of these model-generated scores, particularly in short pre-post formats where learners are expected to improve. This study introduces a novel method for estimating test reliability that adjusts for learning gains using a Rasch-based split-half approach. We validated this approach through simulation under realistic conditions of missing data and score change, showing tangible improvements in reliability estimation compared to baseline methods. Applying this method to a dataset of 985 tutors completing 12 online lessons, we find that GPT-4-based scoring achieves satisfactory reliability, with open-ended responses (0.733) outperforming multiple-choice items (0.652). Both item types jointly yielded the highest reliability (0.774). Hence, as few as 14 open-ended items (across an average of 3-4 completed lessons) were sufficient to surpass common reliability thresholds of 0.7 or higher. Principal component analysis revealed a skill structure with a strong primary dimension shared across almost all lessons and interpretable subdimensions—socio-emotional, cognitive, and fairness-related tutoring skills—supporting a bifactor-like model. These findings demonstrate that GPT-4 and similar LLMs can be effectively used for formative assessment of complex instructional skills in online and personalized learning contexts, provided their reliability is empirically verified. This study contributes an open-source, learning-aware framework for scalable and reliable AI-supported assessment in learning analytics contexts.