MLD at ACL 2026

Researchers from the Multimodal Language Department (MLD) participated in the 64th Annual Meeting of the Association for Computational Linguistics (ACL 2026), held in San Diego, USA, from July 2–7, 2026. At one of the world's leading conferences in natural language processing and computational linguistics, MLD researchers presented their latest work on multimodal communication, sign languages, human cognition, and artificial intelligence.

Perspectival words in humans and multimodal language models

Dota Tianai Dong gave an oral presentation at ACL 2026 on the paper "Using Perspectival Words Is Harder Than Vocabulary Words for Humans—and Even More So for Multimodal Language Models," co-authored with Yifan Luo, Po-Ya Angela Wang, Aslı Özyürek, and Paula Rubio-Fernandez.

Multimodal language models (MLMs) are becoming increasingly capable of human-like communication, yet their use of everyday perspectival words remains poorly understood. 

This study investigates how humans and multimodal language models use different types of words: vocabulary (e.g., ’boat’ or ’cup’), possessives (e.g., ’mine’ vs. ’yours’), and demonstratives (e.g., ’this one’ vs. ’that one’). 

The results show that perspectival words such as “mine,” “yours,” “this one,” and “that one” are more challenging than basic vocabulary words for both humans and models. However, the difficulty is much greater for multimodal language models. While models perform relatively well on vocabulary, they show clear limitations with possessives and even stronger difficulties with demonstratives. 

Evaluating vision-language models on sign language

Onur Keleş presented a poster at ACL 2026 on the paper "The Visual Iconicity Challenge: Evaluating Vision-Language Models on Sign Language Form-Meaning Mapping."

Iconicity, the resemblance between linguistic form and meaning, is pervasive in sign languages, offering a natural testbed for visual grounding in vision–language models (VLMs). The study introduces the Visual Iconicity Challenge, a video-based benchmark that adapts psycholinguistic measures to evaluate VLMs on three tasks: (i) phonological sign-form prediction, (ii) transparency (inferring meaning from visual form), and (iii) graded iconicity ratings.

The team assessed 17 state-of-the-art VLMs in zero- and few-shot settings on Sign Language of the Netherlands (NGT) and compared them to human baselines. VLMs mirror human phonological difficulty patterns (e.g., handshape harder than location) and achieve moderate to strong alignment with human iconicity ratings. However, they still fail to infer lexical meaning from visual form alone and show a systematic object-based bias that inverts the human preference for action-based signs. Crucially, models with stronger phonological form prediction correlate better with human iconicity judgments, indicating shared sensitivity to visually grounded structure. 

The findings validate these diagnostic tasks, show that explicit reasoning narrows the open-to-closed-model calibration gap, and motivate human-centric signals for modelling iconicity in multimodal models.

Advancing multimodal communication and AI

These two papers contribute to ongoing discussions at the intersection of multimodal communication, sign language research, pragmatics, and artificial intelligence. They show how insights from human language and cognition can help evaluate and improve computational models.

For more information about ACL 2026, including the full conference programme, see the ACL26 Digital Conference Handbook.pdf - Google Drive