domain-adaptation
Filtering by topic domain-adaptation(1)Clear all filters
- PaperComputers and Education: Artificial Intelligence30 Jun 2026
Enhancing domain adaptation of LLM via model composition in solving medical exam questions
Yicong Liang, Di Zou, Haoran Xie, Fu Lee Wang
The study explores using model composition (CALM) to adapt large language models for medical exam question answering without full fine-tuning. MedCALM, which connects a pretrained LLM with a medical-specialized augmenting model via multi-head cross-attention, outperforms baselines on medical multiple-choice questions. The results highlight that cross-attention between the last few layers of the two models yields optimal performance.
Original abstract
The advent of LLMs has sparked interest among educators across various fields, particularly in medical education. However, the current pretrained large language models (LLMs) can not be directly applied to answer medical exam questions, as they require adaptation to specific domains via further fine-tuning. Fine-tuning a pretrained large language model requires intensive computing resources, making it impractical for injecting new capabilities into the LLM. In this work, we investigate the framework of applying the approach Composition to Augment Language Models (CALM) for medical domain adaptation in answering real-world exam questions. In particular, we augment a pre-trained large language model by further fine-tuning it on an open-ended question-answering dataset related to medical exams to inject medical domain knowledge. We develop the model MedCALM to enhance medical question-answering performance by incorporating another individual LLM with a medical-specialized augmenting model. The experiment is conducted on a dataset related to real-world medical exam questions, and the experimental results demonstrate that MedCALM outperforms the compared baselines. The results show that utilizing the multi-head cross-attention module to connect two large language models improves performance in answering multiple-choice questions. In addition, selecting the position of layers for connection has a significant impact on the effectiveness when composing the anchor model and the augmenting model. Our experiment suggests that cross-attention between the last few layers of the two models achieves promising performance.