simulation-based-assessment
Filtering by topic simulation-based-assessment(1)Clear all filters
- PaperAssessment & Evaluation in Higher Education4 Jul 2026
Observer-based assessment in authentic management simulations: multi-source validity evidence and construct boundaries
Joerg Hruby
This study validates observer-based live assessments in authentic business simulations using Messick's unified validity framework. Multi-source evidence from 42 students in international strategy battles showed strong internal consistency and convergence with peer ratings, indicating that observer ratings effectively capture socially visible strategic performance rather than latent cognition.
Original abstract
Business schools increasingly use experiential simulations to develop applied strategic thinking, intercultural communication, and managerial decision-making. However, the validity of observer-based live assessments remains under-theorised, as existing research has focused mainly on written assignments and presentations. Following Messick’s unified validity framework, this study conceptualised observer ratings as score interpretations that require empirical justification from multiple sources of evidence to be valid. A multi-source, mixed-methods design integrated observer rubric evaluations, peer assessments, pre- and post-program surveys, and reflective reports from multicultural student teams participating in international business strategy battles involving headquarters–subsidiary negotiations (treatment: N = 42; control: N = 31; peer ratings: N = 412; reflection reports: N = 80). The results indicated strong internal consistency, meaningful convergence between observer and peer ratings, and substantial developmental gains across the program’s duration. Convergence patterns suggest that observer and peer systems capture the socially visible dimensions of enacted strategic performance more effectively than latent cognitive processes. The findings position observer-based assessments as a structured validity argument for externally enacted strategic performance in authentic higher-education simulations.