learning-analytics
Filtering by topic learning-analytics(3)Clear all filters
- PaperComputers and Education: Artificial Intelligence17 Jul 2026
Comparing human and LLM ordered coding of qualitative data: How coding differences cascade through temporal analysis
Kamila Misiejuk, Sonsoles López-Pernas, Eduardo A. Oliveira, Brendan Eagan et al.
This study compares human and large language model (LLM) ordered coding of qualitative learner data, revealing systematic differences that cascade through temporal analysis. Two evaluation approaches assess coding quality and demonstrate a context-aware prompting method. Results show significant disparities in structural, transitional, and code-level metrics, cautioning against relying on LLM outputs for automated feedback in learning analytics.
Original abstract
Automating the process of qualitatively coding text data from learners has been a long-standing ambition of learning analytics researchers since it represents an essential step toward delivering timely and scalable feedback. Automating this process is especially challenging in the case of ordered coding schemes —necessary for temporal analytical methods— where one text utterance can be assigned more than one qualitative code and the assignment order matters. This problem goes beyond multi-class and multi-label classification and, therefore, cannot be easily tackled using classic language models such as BERT. Recent advances in generative artificial intelligence, especially with the advent of large language models, have —allegedly— created a substantial step forward in making the goal of automatically coding complex temporal data attainable. However, little is yet known about how to implement this process in a way that most closely resembles human coding, i.e., taking into account the context in which the textual data appears for accurate interpretation. Moreover, due to the complexity of the data and its shape, the accuracy of the results cannot be computed using classic accuracy metrics. This study makes two main contributions: first, it presents two evaluation approaches for assessing the quality of ordered data coding and the usability of LLM in automatically coding ordered processes; and second, it demonstrates a method of LLM prompting that leverages a consistent context window. Our results reveal systematic and statistically significant differences between LLM and human coding across structural, transitional, and code-level metrics for binary and ordered tasks. As classification errors can propagate through automated feedback systems, relying on LLM outputs risks amplifying inaccuracies and producing misleading interpretations of learning processes.
- PaperJournal of Learning Analytics8 Jul 2026
Toward Reliable Estimation of Algorithmic Bias for Minority Groups
Jaeyoon Choi, Shamya Karumbaiah, Jeffrey Matayoshi
Predictive models in learning analytics often show performance disparities across demographic groups, but reliable estimation of group bias is hampered by small group sizes and sampling error, especially for marginalized students. Using simulations and real-world data, the study recommends bootstrapping for confidence intervals, using multiple metrics, and moving beyond p-values to improve bias estimation.
Original abstract
While predictive models are widely used in learning analytics, several studies have shown that the performance of these models can vary significantly across different demographic groups of students. The first step to audit for and mitigate these group biases is to accurately estimate them. However, the current practices for identifying and measuring group bias often suffer from reliability issues. In this paper, we use simulations and real-world data analysis to explore statistical factors that impact the reliable estimate of group bias and suggest approaches to improve their statistical robustness. Our analysis revealed that small group sizes lead to high variability in group bias estimation due to sampling error -- an issue that is more likely to impact students from historically marginalized communities. We then suggest statistical approaches, such as bootstrapping, to construct confidence intervals for a more reliable estimation of group bias. Based on our findings, we encourage future learning analytics researchers to ensure sufficiently large group sizes, construct confidence intervals rather than relying on p-values, use at least two metrics, and move beyond the dichotomy of the presence or absence of bias for a more comprehensive evaluation of group bias.
- PaperBritish Journal of Educational Technology3 Jul 2026
A balance between stability and flexibility: Adaptive patterns of self‐regulated learning processes shape game‐based learning
Elizabeth B. Cloude, Stefan E. Huber, Jingwei Wei, Bianca Esmanhoto et al.
This study analyzes self-regulated learning (SRL) as a complex system in game-based learning (GBL), using multimodal data to examine patterns and interactions among cognitive, affective, metacognitive, and motivational processes. Results show a quadratic relationship between the regularity of cognitive, affective, and metacognitive state transitions and learning outcomes, indicating an optimal balance between stability and flexibility. Physiological measures of motivation did not predict learning performance, suggesting motivation's role may be mediated through cognitive and metacognitive engagement.
Original abstract
To ensure learning efficiency in game‐based learning (GBL), learners must regulate cognitive, affective, metacognitive and motivational (CAMM) processes, collectively known as self‐regulated learning (SRL). SRL is dynamic and non‐linear, characterized by regulatory patterns and CAMM interactions that lead to macro‐level SRL behaviours. In this paper, we explore SRL as a complex system by analysing patterns and interactions among CAMM processes during GBL and examine their relation to learning outcomes. Thirty‐seven ( n = 37) healthy adults used Antidote COVID‐19, a GBL environment designed to increase emotional engagement and biology knowledge. To define momentary CAMM processes, subjective measures of cognitive, affective and metacognitive states via think‐ and emote‐alouds were synchronized with two physiological correlates of motivation: intensity of facial expressions (arousal) and skin conductance response. Sequence analysis and recurrence quantification defined the regularity of patterns within CAMM components, and transfer entropy estimated concurrent interactions between CAMM components. Our results indicated a quadratic relationship between the turbulence (a measure of regularity) of cognitive, affective and metacognitive (CAM) state transitions (via think‐ and emote‐aloud methods) provided the best fit, explaining 31% of variability in learning. This highlights an optimal balance between stability and flexibility in CAM state transitions that were beneficial for learning. However, physiological correlates of motivation were not predictive regarding learning performance. This lack of predictive capability of the considered measures may reflect their limitations in capturing nuanced motivation dynamics relevant to learning or suggest that motivation's role in learning is predominantly mediated through cognitive and metacognitive engagement rather than directly influencing outcomes. Practitioner notes What is already known about this topic Game‐based learning (GBL) is a promising intervention for improving science knowledge, but it places high self‐regulatory demands on learners. Learners must manage cognitive, affective, motivational and metacognitive (CAMM) processes to ensure GBL efficiency; otherwise known as self‐regulated learning (SRL). Prior research has shown that CAMM processes are each important for learning in games, but they are often studied in isolation rather than as interacting processes over time. What this paper adds This study conceptualizes SRL in GBL as a complex system utilizing mixed multimodal data, emphasizing patterns and interactions among CAMM processes rather than static averages. Results show that learning is maximized when learners exhibit an optimal balance between stability and flexibility in their cognitive, affective and metacognitive state transitions, neither overly rigid nor overly random regulation patterns. Cognitive, affective and metacognitive processes (assessed via think–/emote‐alouds) were strongly related to learning gain, whereas physiological indicators of motivation alone were not predictive. Implications for practice and/or policy Designers of GBL environments should support adaptive regulation, encouraging learners to flexibly shift SRL strategies and emotions while maintaining coherence in their learning process. Educators and researchers should be cautious about relying solely on physiological measures (eg, arousal) for assessing learning effectiveness, as these may not capture meaningful regulatory dynamics. Policies and evaluation frameworks for educational games should prioritize tools and analytics that capture process‐level SRL patterns over time, rather than focusing exclusively on outcomes or isolated behavioural indicators.