design-problems
Filtering by topic design-problems(1)Clear all filters
- PaperarXiv — AI in Education (cs.CY)13 Jul 2026
LLM-Generated Design Problems for Assessing Higher-Order Thinking in Project-Based Learning
Ahmad D. Suleiman, Daqing Hou, Maliha Noushin Raida
Design problems (DPs)—scenario-based prompts requiring application of project concepts in new contexts—offer a way to assess higher-order thinking in project-based learning. Surveys of 31 instructors and evaluation of 80 LLM-generated DPs showed that instructors value DPs but find creation time-consuming, while LLMs can generate high-quality prompts with strong expert agreement. Student performance on DPs correlated weakly with traditional grades, indicating DPs capture distinct aspects of higher-order thinking and can complement conventional assessments.
Original abstract
Project-based learning (PjBL) is common in computing education, but traditional assessments of PjBL often fail to capture higher-order thinking (HOT), especially in transfer contexts. This study introduces "design problems" (DPs): concise, scenario-based prompts that require applying project concepts in new situations, to address this gap. We examined instructor perceptions, the ability of large language models (LLMs) to generate DPs, and student experiences. Surveys of 31 instructors, evaluation of 80 LLM-generated DPs, and student performance data showed that while instructors value DPs, creation effort is a barrier. LLMs helped by producing high-quality prompts with strong expert agreement. Students rated DPs from different LLMs similarly, and their performance on DP tasks showed negligible correlation with traditional project grades, suggesting DPs may capture distinct aspects of HOT. Keystroke data also suggested deeper cognitive engagement of students through planning and revision behaviors. Overall, DPs appear to be a useful complement to traditional assessments, especially in situations where AI use or collaboration may undermine individual learning.