domain-experts
Filtering by topic domain-experts(1)Clear all filters
- PaperAssessing Writing2 Jun 2026
Examining the relevance of three TOEFL Essentials writing tasks to the accounting profession: The role of domain experts
Ute Knoch, Jason Fan, Michael Davey, Sally O’Hagan et al.
The study investigates how domain experts (accountants) judge the relevance of three TOEFL Essentials writing tasks to the accounting profession. Findings show the 'Write an Email' task is considered most relevant and 'Build a Sentence' least, but experts focused on a narrow set of task features and sometimes made comparisons that did not align between test and workplace demands. The results highlight methodological considerations for involving domain experts in test validation.
Original abstract
Large-scale English language proficiency tests are increasingly used to make decisions about professional registration despite not originally being developed to make predictions about language use in the workplace. Domain experts can play a valuable role as informants in establishing the relevance of test tasks to a specific TLU domain. However, this practice has seldom been critically examined. In particular, no studies to date have examined the nature of the interview questions used when engaging with domain experts in this type of research. The current study was designed with two aims: to (1) explore the relevance of three writing tasks (i.e., Build a Sentence, Write for an Academic Discussion, Write an Email) from the TOEFL Essentials test to the accounting profession and (2) evaluate the judgements of domain expert participants. Twenty accountants from non-English speaking backgrounds as well as three accounting educators were interviewed for the study, drawing on a methodology with broad, general questions. The data was analysed qualitatively to identify (a) to what extent the participants considered the three tasks relevant and (b) what task features they attended to when commenting on the relevance of the tasks. The findings showed that the participants generally found the Build a Sentence the least relevant of the three writing tasks, and the Write an Email task the most relevant. When reviewing the task features the participants judged as relevant to writing demands in their workplace, it was shown they focused on a small/narrow range of features. They also engaged in ‘misinterpretations’, comparing aspects of test and workplace tasks that did not align (e.g., comparing a writing task to events in a spoken meeting). The findings are discussed in terms of domain expert involvement in validation research.