cognitive-taxonomy
Filtering by topic cognitive-taxonomy(2)Clear all filters
- PaperarXiv — AI in Education (cs.CY)2 Jul 2026
Automated grading of Linux/bash examinations using large language models: a four-level cognitive taxonomy approach
Manuel Alonso-Carracedo, Ruben Fernandez-Boullon, Pedro Celard, Francisco J. Rodriguez-Martinez et al.
Four frontier large language models (GPT, Claude Opus, Gemini, GLM) were evaluated for grading Linux/bash command responses using a four-level cognitive taxonomy. Gemini 3.0 Pro with rubric-guided prompting achieved the highest agreement with expert instructors (ICC=0.888). The results show that question complexity predicts grading difficulty, and rubric quality affects agreement more than model choice.
Original abstract
Scalable and reliable grading of command-line examinations remains a challenge in computing education, where rising enrolments make manual marking difficult and rule-based autograders cannot handle partial credit, equivalent solutions, or syntactic variation. This paper evaluates whether four frontier Large Language Models (GPT, Claude Opus, Gemini, and GLM) can approximate expert judgment when grading short Linux/bash command responses. The study adopts a four-level cognitive taxonomy that combines cognitive complexity and operational impact, ranging from information retrieval (L1) and basic file manipulation (L2) to structural operations (L3) and advanced system management (L4). The models were tested with two prompt variants, a minimal baseline and a rubric-enhanced version, on 1200 real responses from second-year Computer Engineering students independently graded by three expert instructors. Gemini~3.0 Pro with rubric-guided prompting achieved the highest human-AI agreement (ICC(3,1) = 0.888, MAE = 0.10, Bland-Altman bias = -0.014). Agreement declined consistently as taxonomy level increased, with the largest discrepancies at higher levels. Across all models, rubric quality had a larger effect than provider choice, with structured prompts consistently improving agreement. These results show that question complexity is a reliable predictor of the difficulty LLMs face in grading accurately, and they establish a principled, taxonomy-based framework for determining which questions are suitable for AI-assisted grading and which require human review, while also providing a transferable evaluation protocol and prompt templates.
- PaperarXiv — AI in Education (cs.CY)30 Jun 2026
CogTax: A Four-Level Cognitive Taxonomy for Command-Line Computing Education
Manuel Alonso-Carracedo, Ruben Fernandez-Boullon, Pedro Celard, Francisco J. Rodriguez-Martinez et al.
The paper introduces CogTax, a four-level cognitive taxonomy for command-line computing education that integrates cognitive complexity with operational impact to address the real-world consequences of learner actions. The taxonomy combines Bloom's Revised Taxonomy with four levels of system impact (observational, reversible, structural, administrative) and demonstrates automatic classification of commands with 89% accuracy using a hybrid syntactic-semantic classifier.
Original abstract
As computing education expands beyond traditional programming into operational domains such as systems administration and command-line environments, existing pedagogical frameworks struggle to capture a dimension that is critical in these contexts: the real-world consequences of learner actions. Existing cognitive taxonomies classify learning objectives by mental operations but do not account for system impact, leaving a critical gap in command-line education where conceptually simple commands can have severe consequences. This work presents CogTax, a four-level cognitive taxonomy that integrates two dimensions: cognitive complexity, derived from Bloom's Revised Taxonomy, and operational impact, which distinguishes observational, reversible, structural, and administrative operations. The four progressive levels range from safe read-only inspection to advanced system management requiring integration of multiple abstract models. Then, the taxonomy level is defined as the maximum of these dimensions, ensuring that both conceptual understanding and operational awareness are addressed. CogTax gives instructors a principled framework for sequencing course material and calibrating assessment difficulty, and gives students an explicit reference for self-assessment and gap identification. To demonstrate that taxonomy levels are automatically assignable, making the framework scalable without manual expert annotation, a classifier that combines syntactic representations derived from abstract syntax trees with semantic embeddings is trained. Evaluated on 585 expert-annotated Linux/bash commands, this combined approach achieves 89% accuracy, outperforming either representation alone, and demonstrates cross-language extensibility through structural equivalences across command languages.