vision-language-models
Filtering by topic vision-language-models(2)Clear all filters
- PaperarXiv — Language & NLP (cs.CL)18 May 2026
EgoBabyVLM: Benchmarking Cross-Modal Learning from Naturalistic Egocentric Video Data
Dongyan Lin, Phillip Rust, Angel Villar Corrales, Alvin W. M. Tan et al.
Current vision-language models fail to learn from naturalistic egocentric video data like human infants do, as they rely on tightly aligned curated datasets. The authors introduce EgoBabyVLM and Machine-DevBench to benchmark cross-modal learning, showing that VLMs cannot exploit weakly-aligned signals that dominate real-world egocentric input.
Original abstract
Children acquire language grounding with remarkable robustness from limited visuo-linguistic input in ways that surpass today's best large multimodal models. Recent research suggests current vision-language models (VLMs) trained on curated web data fail to generalize to the sparse, weakly-aligned egocentric streams produced by wearable devices, embodied agents, and infant head-cams -- and no fixed evaluation pipeline exists for measuring progress on this regime. We train VLMs on datasets with varying degrees of semantic alignment between visual and linguistic inputs, including naturalistic infant and adult egocentric videos, and evaluate them with a comprehensive suite spanning multimodal language grounding and unimodal vision and language tasks. At the core of this suite is Machine-DevBench, a corpus-grounded benchmark of lexical and grammatical competence, automatically generated from the model's training vocabulary across logarithmic frequency bins to eliminate the train/eval mismatch and low statistical power of prior developmental benchmarks. Our results show that current VLM paradigms hinge on the tight semantic alignment of curated data and fail to exploit the weakly-aligned signal that dominates naturalistic egocentric input -- the very regime in which humans thrive. To motivate progress, we introduce the EgoBabyVLM Challenge to drive the development of models capable of grounded language learning from the kind of naturalistic data that human infants experience.
- PaperarXiv — Language & NLP (cs.CL)12 May 2026
Allegory of the Cave: Measurement-Grounded Vision-Language Learning
Kepeng Xu, Li Xu, Gang He, Wenxin Yu
This paper proposes measurement-grounded vision-language learning, using RAW camera data instead of standard RGB images to improve model grounding. The method, PRISM-VL, achieves higher BLEU, ROUGE-L, and LLM-Judge accuracy by preserving sensor evidence lost during RGB rendering. While not directly about education, the approach could inform multimodal learning tools for language teaching.
Original abstract
Vision-language models typically reason over post-ISP RGB images, although RGB rendering can clip, suppress, or quantize sensor evidence before inference. We study whether grounding improves when the visual interface is moved closer to the underlying camera measurement. We formulate measurement-grounded vision-language learning and instantiate it as PRISM-VL, which combines RAW-derived Meas.-XYZ inputs, camera-conditioned grounding, and Exposure-Bracketed Supervision Aggregation for transferring supervision from RGB proxies to measurement-domain observations. Using a quality-controlled 150K instruction-tuning set and a held-out benchmark targeting low-light, HDR, visibility-sensitive, and hallucination-sensitive cases, PRISM-VL-8B reaches 0.6120 BLEU, 0.4571 ROUGE-L, and 82.66\% LLM-Judge accuracy, improving over the RGB Qwen3-VL-8B baseline by +0.1074 BLEU, +0.1071 ROUGE-L, and +4.46 percentage points. These results suggest that part of VLM grounding error arises from information lost during RGB rendering, and that preserving measurement-domain evidence can improve multimodal reasoning.