group-bias
Filtering by topic group-bias(1)Clear all filters
- PaperJournal of Learning Analytics8 Jul 2026
Toward Reliable Estimation of Algorithmic Bias for Minority Groups
Jaeyoon Choi, Shamya Karumbaiah, Jeffrey Matayoshi
The paper investigates statistical factors affecting reliable estimation of algorithmic bias across student demographic groups, finding that small group sizes lead to high variability. It recommends using bootstrapping to construct confidence intervals, ensuring sufficiently large groups, using multiple metrics, and moving beyond binary assessments of bias.
Original abstract
While predictive models are widely used in learning analytics, several studies have shown that the performance of these models can vary significantly across different demographic groups of students. The first step to audit for and mitigate these group biases is to accurately estimate them. However, the current practices for identifying and measuring group bias often suffer from reliability issues. In this paper, we use simulations and real-world data analysis to explore statistical factors that impact the reliable estimate of group bias and suggest approaches to improve their statistical robustness. Our analysis revealed that small group sizes lead to high variability in group bias estimation due to sampling error -- an issue that is more likely to impact students from historically marginalized communities. We then suggest statistical approaches, such as bootstrapping, to construct confidence intervals for a more reliable estimation of group bias. Based on our findings, we encourage future learning analytics researchers to ensure sufficiently large group sizes, construct confidence intervals rather than relying on p-values, use at least two metrics, and move beyond the dichotomy of the presence or absence of bias for a more comprehensive evaluation of group bias.