learner-corpus
Filtering by topic learner-corpus(3)Clear all filters
- PaperComputers & Education14 Jul 2026
LLM-derived metrics in second language writing assessment: an explainable AI approach
Jingying Hu, Yan Cong
This study derived surprisal, perplexity, and embedding-based similarity from large language models to quantify linguistic predictability and semantic coherence in second language writing. These metrics, computed on Chinese learner essays, decreased with proficiency for Traditional Chinese-focused models, while embedding similarity increased. Combining LLM-derived metrics with classical linguistic features improved proficiency classification, supporting transparent and scalable AI assessment for underrepresented languages.
Original abstract
Large language models (LLMs) are increasingly used in educational technology for automated writing assessment, yet most applications rely on prompt-based scoring and feedback generation, which often lack transparency, reproducibility, and interpretability. This study investigates whether model-internal LLM representations can provide interpretable and reproducible metrics for second language (L2) writing assessment. We derived surprisal and perplexity from next-token prediction to quantify linguistic predictability and embedding-based similarity to measure semantic coherence across sentences. These metrics were computed at the token, sentence, and discourse levels using three pretrained Chinese language models and evaluated on 1,196 essays written by Chinese L2 learners across four proficiency levels. Their relationships with 11 established linguistic measures of fluency, lexical sophistication, phraseological complexity, and syntactic complexity were also examined. Results showed that surprisal and perplexity generally decreased with proficiency for the two Traditional Chinese-focused models, indicating greater linguistic predictability in more proficient writing, whereas the multilingual model showed weaker sensitivity. Embedding-based similarity increased with proficiency, reflecting stronger semantic coherence. Combining LLM-derived metrics with classical linguistic features improved proficiency classification and prediction beyond either feature set alone. Correlation and qualitative analyses further demonstrated that the proposed metrics capture complementary aspects of writing while revealing conditions under which their interpretations become less reliable. These findings demonstrate the value of interpretable, model-derived metrics for transparent, reproducible, and scalable AI-supported L2 writing assessment, particularly for underrepresented learner populations and lower-resource languages.
- PaperarXiv — Language & NLP (cs.CL)25 May 2026
Does Continued Pretraining on a Learner Corpus Improve Automated Essay Scoring on English Proficiency Tests? Evidence from EFCAMDAT
Duy Anh Nguyen
A study investigates whether continued pretraining on a learner corpus (EFCAMDAT) improves automated essay scoring (AES) for English proficiency tests. Full-corpus pretraining yielded mixed results across models and datasets, but targeting pretraining to proficiency-aligned subsets (e.g., B1-B2 data for FCE) improved in-domain scoring more reliably. However, these gains did not consistently transfer to other test datasets.
Original abstract
Recent automated essay scoring (AES) studies increasingly use pretrained transformer models, but these models are usually pretrained on general-domain English and may under-represent second-language learner writing. This study investigates whether domain-adaptive continued pretraining (DAPT) on the EFCAMDAT learner corpus improves transformer-based AES for English proficiency tests. We apply DAPT to three transformer encoders and evaluate them on FCE and IELTS in both in-domain scoring and few-shot cross-dataset transfer. Full-corpus DAPT produces mixed results across models, datasets, and metrics. Further analyses suggest that these mixed effects are partly explained by mismatches in proficiency, genre, and communicative purpose between EFCAMDAT and the downstream datasets. A proficiency-based ablation shows that targeted DAPT using CEFR-aligned subsets improves downstream scoring more reliably than full-corpus DAPT, especially for FCE with B1--B2 data. However, these gains do not consistently improve cross-dataset transfer. Overall, the findings suggest that continued pretraining on a learner-writing corpus can benefit in-domain AES for English assessment when the pretraining data is sufficiently aligned with the downstream assessment settings. However, it does not automatically improve transferability across different English proficiency test datasets.
- PaperDOAJ — Language assessment1 Mar 2026
Multi-tier linguistic feature engineering for CEFR classification: a comprehensive analysis of deterministic and machine learning-based features
Tendai M. Chikake, Elena M. Bazanova, Anna V. Gorizontova
Analyzed 133 linguistic features for automated CEFR classification using a two-tier architecture combining deterministic and machine-learning features. The best model achieved 66.72% exact accuracy and 94.53% adjacent accuracy on 3,198 learner texts, with strongest signals from morphological complexity and lexical sophistication.
Original abstract
We analyzed 133 linguistic features for automated proficiency classification under the Common European Framework of Reference (CEFR) in a two-tier architecture: deterministic Tier 1 (lexical, morphological, and syntactic measures) and machine-learning-based Tier 2 (semantic coherence, topic structure, cohesion, and error-analysis signals). Experiments were conducted on a corpus of 3,205 learner texts from mixed sources, with triangulated validation against expert-verified Cambridge examination subsets. The materials were collected in 2022--2025 and included a substantial institutional corpus of over 3,000 essays and other writing texts produced by students of Moscow Institute of Physics and Technology (MIPT) studying English as a foreign language and regularly assessed by our AI-powered testing system ISTOK (Intelligent System for Testing General Language Competencies). Feature matrices were standardized after missing-value handling (fold-local median imputation for cross-validation and zero-fill for held-out reporting). In supervised evaluation, the best Tier 1+2 model reaches 66.72% exact accuracy (macro F1 = 0.69) and 94.53% adjacent accuracy (within one CEFR level) on a 3,198-sample CEFR-labeled benchmark split; an extended comparison including preliminary Tier 3 features achieves 67.50%. Unsupervised analyses show strong structure for extreme levels (A1 99.5% purity; C2 82.4% purity) and moderate alignment with professional Cambridge labels (Adjusted Rand Index = 0.303). We report block ablations and compact subset searches, with strongest signals from morphological complexity and lexical sophistication, and consistent incremental gains from error-based features. The results provide a validated, interpretable feature inventory and practical guidance for feature selection in automated language assessment systems.