educational-equity
Filtering by topic educational-equity(1)Clear all filters
- PaperJournal of Learning Analytics8 Jul 2026
Toward Reliable Estimation of Algorithmic Bias for Minority Groups
Jaeyoon Choi, Shamya Karumbaiah, Jeffrey Matayoshi
Predictive models in learning analytics often show performance disparities across demographic groups. This study identifies that small group sizes, common among marginalized students, cause unreliable bias estimates due to sampling error. The authors recommend using confidence intervals, multiple metrics, and larger samples to improve bias assessment.
Original abstract
While predictive models are widely used in learning analytics, several studies have shown that the performance of these models can vary significantly across different demographic groups of students. The first step to audit for and mitigate these group biases is to accurately estimate them. However, the current practices for identifying and measuring group bias often suffer from reliability issues. In this paper, we use simulations and real-world data analysis to explore statistical factors that impact the reliable estimate of group bias and suggest approaches to improve their statistical robustness. Our analysis revealed that small group sizes lead to high variability in group bias estimation due to sampling error -- an issue that is more likely to impact students from historically marginalized communities. We then suggest statistical approaches, such as bootstrapping, to construct confidence intervals for a more reliable estimation of group bias. Based on our findings, we encourage future learning analytics researchers to ensure sufficiently large group sizes, construct confidence intervals rather than relying on p-values, use at least two metrics, and move beyond the dichotomy of the presence or absence of bias for a more comprehensive evaluation of group bias.