llm-in-education
Filtering by topic llm-in-education(3)Clear all filters
- PaperLanguage Testing6 Jul 2026
Human or Machine? Evaluating Second Language Speaking Performance in Paired Discussions with a Large Language Model-Driven Spoken Dialogue System vs. a Human Interlocutor
Shangchao Min, Zhuohan Hou, Yanxin Wang
A study compared second language speaking performance in paired discussions with an LLM-driven spoken dialogue system versus a human interlocutor. No significant overall differences were found, but the LLM condition showed slightly lower pronunciation and language use scores, slower speech rate, reduced lexical diversity yet higher syntactic complexity, and more initiative in discussion management. The findings suggest LLM-driven SDSs can serve as usable interlocutors but require refinement to better capture interactive listening.
Original abstract
The use of large language models (LLMs) in language assessment, particularly in spoken dialogue systems (SDSs) for assessing speaking, remains at an early stage. This study explored the use of an LLM-driven SDS to assess second language speaking ability. Using a within-participant design, we compared the paired discussion performance of 30 participants interacting with a self-built LLM-driven SDS (E-Talk) versus a human interlocutor, focusing on the interlocutor effect on test scores and fine-grained linguistic and interactional features. Results did not yield substantive differences in oral performance across the two conditions, although the interactions with the LLM-driven SDS displayed slightly lower scores for pronunciation and language use, slower speech rate, and reduced lexical diversity alongside increased syntactic complexity, and greater initiative in introducing new ideas, prompting responses, and guiding discussions toward negotiation, coupled with weaker interactive listening. These findings suggest that LLM-driven SDSs can serve as a potential, usable interlocutor in dialogic speaking assessments, eliciting key aspects of speaking ability. That said, further refinement is needed to better capture interactive listening and collaborative meaning construction, highlighting the importance of interpreting SDS-based performance relative to its specific interactional affordances.
- PaperarXiv — AI in Education (cs.CY)29 Jun 2026
Less Deliberate in Teams: Student LLM Use Across Individual and Collaborative Work
Sehrish Basir Nizamani, Zannah Ziew, Saad Nizamani, Khyati Goyal
A semester-long study of 96 undergraduate computing students found that large language model (LLM) usage dropped significantly when students moved from individual work to team projects. Students wrote fewer and simpler prompts, used fewer intentional prompting strategies, and were less careful in verifying AI-generated output. These results suggest that collaborative contexts reduce deliberate LLM engagement beyond what task type alone explains.
Original abstract
As large language models (LLMs) become common in computing courses, we need to understand how the social setting shapes how students use them. This paper reports findings from a semester-long study of 96 undergraduate students who completed six assignments, alternating between individual homework and team project milestones. We tracked LLM usage, prompting habits, and how students verified AI-generated output across all six assignments. LLM usage dropped by 42.7 percentage points when students moved from individual work to their first team milestone, then partly recovered in later team tasks. Students also wrote fewer and simpler prompts, used fewer intentional prompting strategies, and checked LLM output less carefully. The share of students who ran tests on AI-generated code fell by 19.4 percentage points during team assignments and never fully rebounded. A within-student analysis found that 18.9% of students who consistently used LLMs on their own stopped using them entirely in teams, while only 3.2% went the other direction. These results suggest that collaborative context is associated with reduced deliberate LLM engagement beyond what task type alone can explain. The moment students form teams appears to be a critical and currently unsupported turning point in computing course design.
- PaperarXiv — AI in Education (cs.CY)29 Jun 2026
Teaching Prompt-Based Programming with LLMs: A 45-Minute Lesson with Guided Practice for End-User Programmers
Keith Tran, Samiha Marwan, Thomas Price
A 45-minute prompt-based programming intervention with guided practice was evaluated against a control activity. Engineering students in the experimental group showed greater gains in prompting self-efficacy and slight improvement in ability to specify computational goals to LLMs. The results suggest that even a brief intervention can modestly improve prompting skills, but more time may be needed for significant gains.
Original abstract
Prompt-based programming, a new modality enabled by large language models (LLMs), allows users to express computational goals through natural language rather than traditional code. While this approach lowers barriers to entry, especially for non-CS learners, it does not eliminate the need for foundational CS skills. Learners often struggle to communicate their intent clearly to LLMs, resulting in vague or underspecified prompts. Prior work has documented the need for explicit prompting for both CS and non-CS learners. However, it remains less clear how such instruction can fit into busy classrooms or how much time is needed to produce meaningful gains. In this paper, we evaluated a 45-minute prompt-based programming intervention, consisting of a lesson with guided practice, against a business-as-usual CS lab activity (code tracing) of equal length, representing a class without prompt-focused instruction. We conducted a randomized controlled study with 55 engineering students. We found that students in the experimental condition improved more on average (though not significantly more) from pre- to post-test than the control group (+10.8 vs +1.1 percentage points) and showed significantly greater average gains in prompting self-efficacy (+35.4 vs +21.9 percentage points). Our results suggest it is likely that a brief intervention can improve learners' ability to specify computational goals to LLMs. However, the effect was modest, suggesting that prompting skills may require more time and practice to develop. We provide a lightweight lesson that requires no prior CS background and can be readily dropped into existing courses.