automated-essay-scoring
Filtering by topic automated-essay-scoring(5)Clear all filters
- PaperarXiv — AI in Education (cs.CY)15 Jul 2026
When Rubrics Change: Cross-Rubric Generalization for Critical Thinking Essay Scoring
Nischal Ashok Kumar, Payu Wittawatolarn, Sana Kang, Marisa C. Peczuh et al.
The paper investigates cross-rubric generalization in automated essay scoring, where models trained on essays scored under one rubric must perform on new rubrics targeting different aspects. Using a trait-based intermediate representation and target-essay supervision, the approach improves macro F1 by 5% in the hardest setting. Their best open-source Llama-based model outperforms GPT-5-mini prompting by 2.1% and trails GPT-5 by 1.9%.
Original abstract
Automated essay scoring (AES) research has largely focused on cross-prompt generalization, where essays from unseen prompts are scored while the scoring criteria are typically held constant. In practice, however, educators may revise or even introduce new rubrics in their scoring task, to evaluate different aspects of essays. We study cross-rubric generalization: training on essays labeled under one set of rubrics and evaluating on previously unseen rubrics, which target different aspects of the essay. We use a Large Language Model (LLM) fine-tuning framework with two components: rubric-agnostic intermediate representations, called traits, and target-essay supervision under seen rubrics during training. On an AES dataset augmented with multiple rubric-defined labels of student critical thinking skills, we find that traits improve macro F1 by 5.0% over a baseline without traits in the hardest setting, where both target rubrics and target essays are unseen during training. We further find that increasing target-essay supervision improves performance, with our best fine-tuned open-source Llama-based model outperforming GPT-5-mini prompting by 2.1% macro F1 and trailing GPT-5 by 1.9%. These results show that trait-based intermediate structure and controlled supervision improve generalization to unseen rubrics.
- PaperAssessing Writing26 Jun 2026
Assessing the reliability and validity of large language models in automatic essay scoring
SCOTT CROSSLEY, Langdon Holmes, Wesley Morris
Large language model (LLM) based automated essay scoring (AES) systems were developed and tested for reliability, agreement with human raters, and convergent validity using linguistic components. Both representation and generative LLM AES systems showed strong reliability and agreement, but the representation system exhibited differential correlations with human scores regarding text length and type-token ratio, while the generative system showed no differences. The findings support the use of LLM-based AES in standardized writing assessments for secondary students, though further research is needed.
Original abstract
With the advent of artificial intelligence, large language model (LLM) based Automated Essay Scoring (AES) systems have been developed that can consistently make human-like decisions that do not depend fully on surface level linguistic features. However, research into the use of LLM-based AES systems is limited and little is known about the reliability, agreement, or validity of the systems. The goal of this study was to provide evidence for the reliability, agreement, and validity of LLM-based AES systems in a standardized writing assessment used for secondary school students. Both representation and generative LLM-based AES systems were developed to score persuasive essays and assessed for reliability. Then the agreement of the developed AES systems with human raters was assessed through correlational analyses. We used extrinsic convergent validation approaches to examine if the human and LLM scores correlated with linguistic components. Results indicate strong reliability and agreement for the LLM scores. In terms of convergent validity, initial correlational analyses indicated that the representation LLM AES system showed differential correlations with the human scores in terms of a text length and type-token ratio component. This result contrasts with the correlational results from the generative LLM AES model, which indicated no differences in associations between the model and human scores with regards to the linguistic components.
- PaperAssessing Writing2 Jun 2026
Reassessing automated essay scoring with large language models: Evidence from API and GUI interfaces
Jieun Kim, Daniel Holden
This study reassesses automated essay scoring using large language models, comparing evidence from API and GUI interfaces to evaluate their effectiveness.
- PaperarXiv — Language & NLP (cs.CL)25 May 2026
Does Continued Pretraining on a Learner Corpus Improve Automated Essay Scoring on English Proficiency Tests? Evidence from EFCAMDAT
Duy Anh Nguyen
A study investigates whether continued pretraining on a learner corpus (EFCAMDAT) improves automated essay scoring (AES) for English proficiency tests. Full-corpus pretraining yielded mixed results across models and datasets, but targeting pretraining to proficiency-aligned subsets (e.g., B1-B2 data for FCE) improved in-domain scoring more reliably. However, these gains did not consistently transfer to other test datasets.
Original abstract
Recent automated essay scoring (AES) studies increasingly use pretrained transformer models, but these models are usually pretrained on general-domain English and may under-represent second-language learner writing. This study investigates whether domain-adaptive continued pretraining (DAPT) on the EFCAMDAT learner corpus improves transformer-based AES for English proficiency tests. We apply DAPT to three transformer encoders and evaluate them on FCE and IELTS in both in-domain scoring and few-shot cross-dataset transfer. Full-corpus DAPT produces mixed results across models, datasets, and metrics. Further analyses suggest that these mixed effects are partly explained by mismatches in proficiency, genre, and communicative purpose between EFCAMDAT and the downstream datasets. A proficiency-based ablation shows that targeted DAPT using CEFR-aligned subsets improves downstream scoring more reliably than full-corpus DAPT, especially for FCE with B1--B2 data. However, these gains do not consistently improve cross-dataset transfer. Overall, the findings suggest that continued pretraining on a learner-writing corpus can benefit in-domain AES for English assessment when the pretraining data is sufficiently aligned with the downstream assessment settings. However, it does not automatically improve transferability across different English proficiency test datasets.
- PaperETS Research Report Series20 Apr 2026
On the Representation of Racial and Ethnic Subgroups in AI-generated Texts: A Case Study in Automated Essay Scoring
Akshay Badola, Mo Zhang, Chen Li
Using GPT-4 and GPT-4o to generate essays for specific racial/ethnic subgroups from example essays, this study finds that the generated racial distribution does not match the real distribution. Augmenting automated essay scoring training data with these LLM-generated essays, even when race is mispredicted, reduces bias without harming performance.
Original abstract
In this study, we assess the capability of LLMs in generating essays of a specific race/ethnicity after being given example essays and rubric, and investigate the efficacy of data augmented in this manner for Automated Essay Scoring with respect to model performance and bias. In a series of experiments, we use models GPT-4 and GPT-4o, and ask them to generate essays from a given subgroup after inferring the race/ethnicity of the writer. We find that while LLMs can be directed to generate essays for specific demographic groups, the inferred racial and ethnic distribution in the generated data does not closely mirror the actual distribution observed in the source dataset. We augment existing data for underrepresented subgroups with LLM generated data separated into two groups with correct LLM race prediction and with incorrect race prediction and assess the improvement in agreement with human scores with quadratic weighted Kappa and bias mitigation as change in standardized mean difference. Our analysis shows that while LLMs struggle to predict the race accurately from given samples, augmentation with such data can be helpful to mitigate bias regardless. Suggested citation: Badola, A., Zhang, M., & Li, Chen. (in press). On the representation of racial and ethnic subgroups in AI-generated texts: A case study in automated essay scoring. ETS Research Report Series. https://doi.org/10.64634/ac01td58