ai-assessment
Filtering by topic ai-assessment(2)Clear all filters
- PaperEdArXiv (OSF Preprints)20 Jul 2026
The Verification Paradox
Greg O'Keefe
The paper identifies three verification problems students face when using generative AI: correctness, depth, and calibration, with calibration being most consequential as it is a failure to recognize that anything requires checking. It argues that AI-sourced claims lack an accountable party, making this structural and permanent, and that oral defenses cannot fix the gap because ownership and validity are independent variables.
Original abstract
When a student uses generative AI to produce academic work, the capacity to verify that work fails for two parties, for two reasons, at two moments. The teacher, examining the finished artifact, cannot determine who performed the thinking. The student, situated inside the process of production, cannot determine whether the material the system has supplied is correct, deep, or safe to build upon. This paper concerns the second failure. It argues that a student operating outside their own domain of competence faces three distinct verification problems — correctness, depth, and calibration — and that the third is the most consequential, because it is not a failure to check but a failure to recognize that anything requires checking. The paper then argues that this predicament is not reducible to the familiar condition of novice ignorance. AI-sourced claims differ structurally from textbook-sourced claims in a way that is independent of quality: no accountable party stands behind any particular sentence, which removes the external correction mechanism that has historically made novice learning survivable. Following recent work on the testimony gap, the paper shows this difference to be structural rather than incidental, and therefore permanent: it does not resolve as the underlying systems improve. Finally, the paper argues that the most natural institutional response — verifying student understanding through oral defense — cannot close the gap, because ownership and validity are independent variables. A student can genuinely own a false claim. Any instrument that certifies ownership while remaining blind to validity does not merely fail to detect the problem; it launders it, converting an unverified claim into credentialed knowledge.
- PaperEdArXiv (OSF Preprints)20 Jul 2026
The Forcing Function: Ownership, Provenance, and Oral Defense in AI-Assisted Student Work
Greg O'Keefe
This paper argues that detection-based responses to generative AI are structurally flawed because finished writing lacks reliable process evidence. It proposes an assessment redesign centered on oral defense as a forcing function that requires students to demonstrate ownership through verified provenance and independent synthesis, tailored for secondary classrooms with small class sizes.
Original abstract
Institutional responses to generative AI have largely relied on detection. Detection has failed, and Papers One and Two of this series argue that the failure is structural rather than technical: a finished piece of writing does not carry reliable evidence of the process that produced it, and no improvement in detection tools alters this. This paper proposes a different response. Rather than attempting to establish after the fact whether a student used AI, it redesigns the assessment so that success depends on a capacity AI cannot supply on the student's behalf. The instrument is oral defense, though not as a means of detecting deception. Oral defense is analyzed here as a forcing function: it alters what a student must do in advance in order to succeed, and therefore shapes behaviour before the assessment rather than adjudicating it afterward. The paper develops three connected components. The first is a research protocol that assigns AI a narrow role, the location of material, and reserves for the student the two operations that constitute ownership: verifying what each source argues, and constructing a position from the verified set. The second is a rule of exclusion: a claim whose primary source the student cannot obtain and read may not function as a premise in the argument, even in hedged form. The third is a two-probe defense that tests these two operations separately, incorporating a live friction point that cannot be anticipated and scripted. Ownership, on this account, is not authorship of the prose. It is verified provenance together with demonstrated independent synthesis. A student who used AI extensively and satisfies both conditions passes; a student who used no AI and satisfies neither fails. This is not a loophole in the framework but its central commitment. The paper is addressed to the secondary classroom rather than the university lecture hall, and the scope is integral to the argument rather than incidental. The mechanism developed here depends on an instructor who knows their students across a term, on class sections of roughly twenty to thirty, and on a developmental stage at which the knowledge being assessed is still forming rather than already established and merely being applied. Existing AI-resilient assessment frameworks, examined in Section 5, are designed for university cohorts in the hundreds, where that knowledge is assumed largely in place. The two scopes are not competing solutions to a single problem; they are solutions to different problems, and this paper claims only its own.