Publications

Abstract

Dynamic time warping (DTW) is a well-known algorithm used to assess the similarity between signals of varying lengths. Initially developed for automatic speech recognition, DTW has found applications in psycholinguistics, particularly in analyzing gesture form similarity. An open question in this domain is how effectively DTW captures gesture form similarity. Here, we validate DTW against human annotations of gesture form similarity across two multimodal interaction corpora and explore its utility as an automatic, continuous measure of gesture form similarity. Our findings reveal weak to moderate correlations between DTW distance and the number of similar gesture features – such as handshape, movement, orientation, and position – suggesting that DTW serves as a useful proxy for gesture form similarity. Additionally, we highlight the importance of qualitative analysis of raw data and DTW predictions in enhancing DTW’s predictive accuracy. Our study offers a rigorous validation of DTW as a measure of gesture form similarity and presents a detailed framework for preprocessing motion tracking data and calculating DTW distance. While none of the methods is perfect, the combination of automatic and manual measures provides a comprehensive approach to understanding and measuring gesture form similarity.

Abstract

Background
Impaired plasticity has been implicated in major depressive disorder (MDD), which is reflected in impaired automatic sensory detection as measured using mismatch negativity (MMN). Intermittent theta-burst stimulation (iTBS) is used to treat MDD and its mechanism of action has been suggested to be plasticity dependent. Thus, we aimed at evaluating MMN as a MDD biomarker and predictor of antidepressant effect of iTBS.

Methods
This study comprised 46 patients with MDD and 64 healthy controls. The participants completed an auditory duration deviant MMN paradigm. In patients, MMN was recorded before and after a two-week intervention of twice-daily stimulation with a prolonged iTBS protocol, or sham, over the dorsomedial prefrontal cortex (DMPFC). Depressive symptoms were assessed with the affective subscale of the Brief Psychiatric Rating Scale (BPRS-affective) and through the self-report version of the Montgomery-Åsberg Depression Rating Scale (MADRS-S).

Results
There were no differences in MMN amplitude at baseline between patients and controls, nor any correlations between depressive symptoms and MMN amplitude. MMN amplitude did not change after iTBS. However, in the active iTBS group a larger baseline MMN amplitude correlated significantly with greater reductions of depressive symptoms following iTBS, Δ-MADRS-S (r = 0.48, p = 0.021) and Δ-BPRS-affective (r = 0.46, p = 0.026). No correlation between baseline MMN amplitude and change of depressive symptoms was observed in the sham group.

Conclusion
Larger MMN amplitudes at baseline were associated with greater improvement in depressive symptoms only after active iTBS over the DMPFC. Indicating potential predictive properties of MMN in DMPFC iTBS in MDD, warranting replication in larger samples.

Abstract

Rapid Invisible Frequency Tagging (RIFT) is a recent advance in frequency tagging that exploits novel, high-frequency displays to modulate luminance at imperceptibly high frequencies. RIFT goes beyond low-frequency tagging by allowing researchers to track neural responses to rhythmic stimulation while avoiding perceptual confounds. RIFT is thus a promising method to address central questions in rhythmic cognition, including attention, multimodal integration, and the neural mechanisms underlying oscillatory coordination in perception. However, setting up a RIFT study involves several technical and conceptual considerations. In an effort to make RIFT more accessible, we provide a comprehensive guide for implementing RIFT in cognitive neuroscience. On the basis of the joint experiences and data-driven insights of multiple laboratories, we provide practical recommendations derived from empirical datasets to improve reproducibility, covering hardware requirements, stimulus design, analysis approaches, and interpretation of results. We hope that this guide helps readers to both identify the conceptual areas where RIFT offers promising insights and navigate the technical caveats that come with the approach.

Abstract

Human communication often involves spontaneous teaching, a capacity central to cultural evolution, in which representational gestures (e.g. showing how to break an egg) play a crucial role in conveying semantic information. While representational gestures have been studied towards young children, less is known about how they differ for older children versus adults and their cross-cultural variation. We asked 16 Italian and 16 Dutch adults, likely to differ in their gesture use, to demonstrate two novel logic puzzles to 9–10-year-old children and to other adults. Results show that even though Italian participants produced more representational gestures overall than Dutch participants, neither group increased them for children compared to adults. However, across both cultures, two-handed representational gestures were more frequent in child-directed demonstrations. Within representational gestures, Italians used fewer bracketed gestures (gestures in which one hand remains on hold while the other moves) than Dutch participants for adults, possibly to support grounding of reference on previous information. However, both groups used bracketed gestures for children at similar rates. Overall, the findings reveal that even though cultures differ in their gesture use, similar strategies enhancing iconicity seem to be employed in demonstrations to children.

Abstract

The emergence of human language is a complex process, and to investigate the role of iconicity in this, researchers have combined insights from computational models with empirical observations from laboratory experiments. This chapter provides an overview of the most important insights on the interaction between iconicity and other linguistic properties such as combinatoriality and systematicity. In the experimental and computational work reported, it is shown how iconicity can affect the way in which emerging languages are learned and used. The chapter also discusses how computational methods can help to better understand the gradient and subjective nature of iconicity.

Abstract

Multimodal language models (MLMs) increasingly demonstrate human-like communication, yet their use of everyday perspectival words remains poorly understood. To address this gap, we compare humans and MLMs in their use of three word types, which we predict impose increasing cognitive demands: vocabulary (e.g., ’boat’ or ’cup’), possessives (e.g., ’mine’ vs. ’yours’), and demonstratives (e.g., ’this one’ vs. ’that one’). Testing seven MLMs against human participants, we find that perspectival words are harder than vocabulary words for both groups. The gap is even larger for MLMs: while models approach human-level performance on using vocabulary, they exhibit clear deficits with possessives and even greater difficulties with demonstratives. Ablation analyses point to limitations in perspective-taking and spatial reasoning as key sources of these gaps in MLMs. Instruction-based prompting helps close the gap for possessives but still leaves demonstratives far below human performance. These results show that, unlike vocabulary, perspectival words pose a greater challenge in human communication—and this difficulty is further amplified in MLMs, revealing a crucial shortfall in their pragmatic and social-cognitive abilities.

Abstract

Disfluencies in speech frequently occur before the production of longer and more complex speech content. Listeners are thought to use the distribution of disfluencies in the comprehension of speech to inform their predictions. Here, we investigated whether the presence of disfluencies in speech affects word processing also in naturalistic listening conditions. Participants (n = 36) listened to the spoken recall of the events of a television series while undergoing fMRI. We modelled word processing effort using parametric modulations for word length, frequency, entropy, as well as surprisal and presence/absence of a disfluency. To investigate the effects of disfluencies on word processing, we tested the interaction between disfluency and frequency, and disfluency and surprisal. Words preceded by a disfluency were associated with increased activity in the left and right STG. Lower word frequency was associated with an increase in activity in the left mid STG. Increased word surprisal elicited a similar distribution of activity, with bilateral superior temporal activation. The effect of surprisal was reduced after a disfluency in a cluster in the left posterior temporal lobe, while the effect of frequency increased following disfluencies in the left superior temporal gyrus and the left inferior frontal cortex. Therefore, the presence of a disfluency affects the response to upcoming input, suggesting that it prepares the listener for higher complexity in the upcoming speech, by potentially allocating increased attention resources that facilitate integration in context.

Abstract

When speakers refer to objects in the world, they frequently overinform. Contrary to classical theories in linguistics, we hypothesise that overinformativeness is an efficient means of facilitating listener comprehension: speakers use redundancy to provide their listeners with search-efficient perceptual information. In Experiment One (N = 72), we operationalise search efficiency as the ease or difficulty of perceptual discrimination. We borrow methods from psychophysics to manipulate discriminability across attributes (material, colour) and across sensory modalities (audition, vision). We show that across both attributes and modalities, speakers overinform more often when the redundant information helps the listener perceptually discriminate the referent, thus aiding listener-search. Contrary to our expectations, we also find that speakers disproportionately overinform with colour information (relative to material). In Experiment Two (N = 97), we investigate the disproportionate use of colour directly, addressing explanations that appeal to colour’s perceptual salience by 1) dampening colour’s perceptual distinctiveness, and 2) controlling linguistic complexity both in terms of language production and retrieval effort. Contrary to widespread explanations, these factors cannot explain the disproportionate use of colour: colour is privileged in reference.

Abstract

Traditional psycholinguistic approaches to language have examined production and comprehension in isolation. However, these processes are tightly intertwined and embedded in social interactions. In this Review, we summarize empirical work that highlights the behavioural and cognitive complexities of communicating meaning in face-to-face conversation and that should be captured by psycholinguistic accounts and paradigms. To begin, we consider the implications of conceptualizing language as a situated joint action. Then, we summarize work on three defining features of conversation. First, visual bodily signals play an integral role in composing and comprehending meaning and achieving mutual understanding. Second, addressee feedback signals understanding or difficulty understanding, and the monitoring of interlocutors for such signals adds demands on cognitive resources. Third, multi-party interactions require participants to keep track of and adapt to multiple people’s understanding, signals and shared knowledge. In closing, we point to issues that require further research and the development of experimental paradigms that can capture defining features of face-to-face conversation while maintaining experimental control.

Abstract

Language processing is facilitated by the prediction of upcoming linguistic content. Natural speech is full of disfluencies that have potential predictive power for the listener. The extent to which different pragmatic cues, such as disfluencies, are integrated with semantic context to inform predictions is, however, currently unknown. Here we combine electroencephalography (EEG), eye tracking and virtual reality to investigate to what extent predictive processing during naturalistic language comprehension is influenced by hesitation disfluencies in speech. Participants (n= 64) listened to sentences spoken by a virtual agent in various virtual scenes (e.g., office, street) while their eye-movements and EEG were recorded. Spoken sentences were predictable or unpredictable based on the verb constraints, and were either fluent or disfluent with a hesitation disfluency (“uhh”) preceding the noun phrase. Referents were visible or absent in the scene to be congruent or incongruent with listeners’ predictions, respectively. In predictable but not unpredictable sentences, fixations towards the referent increased before it was mentioned, which confirmed that participants predicted the referent. After hesitation onset, fixations towards the referent no longer increased but shifted towards the virtual speaker. EEG data at noun onset (N400 amplitudes and alpha and theta frequency power) provided no evidence that listeners discard their prediction after hearing a hesitation disfluency. Rather, the disfluency further facilitates predictable noun processing, possibly through mechanisms of increased attention and processing time.

Abstract

Children often use gestures to express concepts before expressing them in speech, particularly in domains rich in visual-spatial information. For example, spatial relations such as left-right are cognitively and linguistically challenging for children. Consequently, 8-year-olds struggle to convey these complex relations verbally, but they frequently rely on gestures to describe these spatial concepts informatively. This study builds on this prior work on the descriptions of left-right relations to investigate further (1) the differing functions of gestures (complementary or redundant) in relation to speech in both children and adults; (2) the change in semantic information conveyed in both of these types of gestures in childhood to adulthood, aiming to shed light on the interaction between spatial language and spatial cognitive development. Eight-year-old and adult monolingual Turkish speakers described pictures of objects in left-right spatial relations shown among a quartet of other types of spatial relations between these objects. Results demonstrated that when describing left-right relations between objects, children, compared to adults, provided more under-informative descriptions (i.e., using “side” instead of “left-right”) in speech but used complementary gestures to convey missing information multimodally. Adults already used informative spatial terms in speech and used gestures mostly redundantly. Moreover, children showed a preference for iconic gestures depicting the relative locations of objects over directional pointing gestures indicating single locations, especially when gestures complemented speech. In contrast, adults showed no reliable preference for either gesture type. These results indicate the significance of gestures as mechanisms of change, alongside speech, in spatial language and cognition, particularly in the context of describing cognitively complex left-right spatial relations between objects.

Abstract

Iconicity, the resemblance between linguistic form and meaning, is pervasive in sign languages, offering a natural testbed for visual grounding in vision–language models (VLMs). We introduce the Visual Iconicity Challenge, a video-based benchmark that adapts psycholinguistic measures to evaluate VLMs on three tasks: (i) phonological sign-form prediction, (ii) transparency (inferring meaning from visual form), and (iii) graded iconicity ratings. We assess 17 state-of-the-art VLMs in zero- and few-shot settings on Sign Language of the Netherlands and compare them to human baselines. VLMs mirror human phonological difficulty patterns (e.g., handshape harder than location) and achieve moderate to strong alignment with human iconicity ratings. However, they still fail to infer lexical meaning from visual form alone and show a systematic object-based bias that inverts the human preference for action-based signs. Crucially, models with stronger phonological form prediction correlate better with human iconicity judgments, indicating shared sensitivity to visually grounded structure. Our findings validate these diagnostic tasks, show that explicit reasoning narrows the open-to-closed-model calibration gap, and motivate human-centric signals for modelling iconicity in multimodal models.

Abstract

A hallmark of human cognition is the ability to segment streams of dynamic stimuli into discrete event units. Currently, most of the evidence about event segmentation comes from English speakers. Could cross-linguistic differences in how events are encoded affect how speakers of different languages form cognitive event units? Here, we address this question by comparing how English and Turkish speakers segment and represent motion events. We find robust differences in the way speakers of these two languages form linguistic event units when describing these events (Experiment 1). Nevertheless, these differences do not affect non-linguistic event unit formation as measured by a classic event segmentation task (Experiment 2) and an event enumeration task (Experiment 3). We conclude that conceptual and linguistic event units are related but distinct levels of event structure. These results have implications for theories of event cognition, as well as the broad relation between language and thought.

Abstract

Co-speech gestures emerge from the interaction between visuospatial experience and speech formulation, but what happens when visual experience is absent from birth? Blind individuals gesture in ways consistent with their native language, albeit they gesture less frequently. However, nothing is known about whether their gesture form differs from that of sighted individuals. Using computer vision technologies, this study examines whether gesture kinematics are shaped by visuospatial experience. In an auditory motion-description task, we analyzed spontaneous path gestures—depicting motion trajectories—produced by 20 congenitally blind, 21 blindfolded, and 21 sighted Turkish speakers. We assessed the gestures’ alignment with actual motion paths, along with duration, size, and speed. Blind participants produced gestures that were more precisely aligned with the motion trajectories, as well as larger and longer. These findings show that altered spatial cognition in blindness enhances gesture precision. The study underscores the role of sensorimotor experience in shaping gesture form.

Abstract

Vision-Language models (VLMs) can now produce fluent, socially appropriate dialogue, leading to interest in whether they have Theory of Mind (ToM). While recent work suggests that VLMs still lack coherent mental-state reasoning, this work has focused on classical propositional belief representations. Here we test a complementary, communication-relevant form of ToM: attention-based social micro-processes that support referential communication in the here and now (i.e., selecting efficient descriptions to identify a particular object for someone else). In face-to-face communication, people strategically add redundant color adjectives to facilitate the listener's visual search, and omit them when they provide no benefit. We evaluate whether VLMs show the same strategy in a referential communication paradigm where the usefulness of redundant color words varies based on the set size and color distribution of objects. We find that VLMs can produce successful referential expressions but lack the attention-guiding strategies that make human communication so efficient. This suggests that VLMs lack the more implicit representations of attention that people use in everyday communication.

Abstract

* = Joint first authorship I risultati evidenziano l’utilità e l’adattabilità dei materiali nella pratica clinica logopedica (con alcune differenze in funzione del setting di impiego e della popolazione coinvolta) e incoraggia la prosecuzione del progetto e l’implementazione dei materiali, con l’obiettivo di favorire la partecipazione sociale delle PcA.

Abstract

Common ground is the information that the participants in a conversation treat as background information for the purposes of their interaction. We review two traditions of research on common ground. The formal tradition, consisting mainly of theoretical linguists and philosophers of language, has developed increasingly sophisticated formal models of common ground to generate predictions about an expanding range of empirical phenomena. Meanwhile, the psycholinguistic tradition has focused on a narrower range of phenomena while developing more realistic theories of the psychological mechanisms that allow us to select and represent common ground. After summarizing these two traditions, we consider several reasons why they should be reintegrated, and we argue that the best way to bring them back together would be to adopt a cognitive-pluralist approach, whereby language users have access to a variety of mechanisms for managing background information, which are more or less available and efficient depending on the communicative situation and the kind of information mentally represented as well as the cognitive demands of each mechanism.

Abstract

All languages in the world have demonstrative terms such as ‘this’ and ‘that’ in English, which have traditionally been treated as spatial words. Here we aim to provide experimental evidence that demonstrative choice is jointly determined by spatial considerations (e.g., whether the referent is near or far) and socio-cognitive factors (e.g.,the listener’s attention focus). We also test whether demonstrative choice varies depending on the speaker’s use of pointing, to provide evidence for a multimodal account of demonstrative systems. We focus on the Turkish system and compare it with the Spanish one to better understand the cross-linguistic variability of 3-term demonstrative systems. Corpus studies have suggested that the Turkish proximal ‘bu’ and distal ‘o’ mark a spatial contrast between near and far space, whereas the medial ‘s¸u’ is used to direct the listener’s attention to a new referent. Supporting this analysis, an online experiment using a picture-based demonstrative-choice task revealed that the medial form ‘s¸u’ was preferred when the listener was looking at the wrong object. The results of a second experiment using video stimuli further showed that the medial ‘s¸u’ was preferred when the speaker pointed to the referent to direct the listener’s attention, whereas the proximal demonstrative was used in near space and the distal in far space, mostly in joint attention and without pointing. The results of a third experiment in Spanish showed radically different patterns of demonstrative-pointing use. The medial ‘ese’ was preferred in joint attention, whereas the proximal ‘este’ and distal ‘aquel’ were selected to direct the listener’s attention towards the intended referent but without an effect of pointing. Our results confirm that demonstrative choice within a given system is determined by both spatial and socio-cognitive factors, interacting with pointing patterns and varying across languages. Leveraging recent experimental work in several languages, we interpret these findings as further evidence for the weighted parameters framework (e.g., referent position and listener attention), which explains demonstrative choice beyond previous categorical analyses.

Abstract

Autistic individuals often show differential sensory perception, including hypo- or hypersensitivities to sound. Previous research also suggests that autistic individuals often have difficulty processing intentional and affective cues in speech acoustics. However, general speech processing difficulties remain underexplored. We investigated self-reported auditory perception using the Speech, Spatial, and Qualities of Hearing Questionnaire among autistic (self-identifying (n = 18) and clinically diagnosed (n = 45)) and non-autistic adults (N = 66). The study was conducted in the Netherlands, but the questionnaire and call for participation were in English and open to anyone regardless of country of residence. Both clinically diagnosed and self-identifying individuals with autism reported significantly lower scores on the Speech, Spatial, and Qualities of Hearing Questionnaire score and on the Speech subscale compared with non-autistic individuals, indicating challenges in overall quality of auditory perception, speech comprehension. Clinically diagnosed individuals also showed lower scores on the quality and spatial subscales compared with non-autistic individuals. Post hoc analysis further suggested that speech hearing is particularly challenging for many autistic individuals. In addition, our finding that self-identifying and clinically diagnosed autistic individuals show similar patterns of hearing difficulties emphasizes the need for more inclusive research practices that collect the experiences of all the individuals in the autistic community in the study of sensory perception in autism.

Abstract

Simultaneous constructions present a unique linguistic property in sign languages where multiple body articulators (hands, torso, head, facial expression, and eye gaze) are used to structure linguistic information not only linearly, as is done in speech, but also simultaneously. Simultaneous constructions allow a more direct encoding of scenes and events by using various iconic strategies to depict multiple event elements (e.g. referents and their actions), their spatial relations and temporal overlap, mirroring how they are perceived in the world; that is, simultaneously. The chapter provides an overview of how iconicity is operationalized in simultaneous constructions and the methods used to study their function, and discusses the history and future directions of the study of iconicity as a structuring principle in sign languages.

Abstract

According to relevance theory, communication relies on speakers’ ability to signal relevant information, which addressees use to infer meaning efficiently. Most research within the relevance theoretic framework has examined how relevance is marked in speech, treating it as the primary channel for signaling informative content. Yet, little is known about whether, how, and in what contexts speakers highlight information in co-speech gestures. Here, we investigate how speakers use ostensive cues—specifically, visual and verbal deixis (e.g. gaze toward a gesture, demonstratives like “this,” “that,” or “here”)—to highlight the communicative relevance of iconic gestures in child- versus adult-directed communication and whether addressees show sensitivity to these cues. Sixteen Italian adults explained the rules of two logic puzzles to a child and another adult. Results show that speakers highlight more gestures for children than for adults, primarily by using visual deixis. Furthermore, addressees of both age groups, adults and children, were more likely to shift their gaze to highlighted than to non-highlighted gestures. These findings demonstrate that speakers dynamically adapt multimodal cues to highlight iconic gestures for addressees, providing empirical support for extending relevance theory to a multimodal view of language.

Abstract

Gaze is a key non-verbal cue in face-to-face interaction, yet we know relatively little about how people visually explore a robot’s face during conversation. In human-human interactions (HHI), gaze allocation is shaped by conversational role and task demands: speakers typically avert their gaze from their partner’s face more than listeners do, and listeners often shift gaze from the eyes to the mouth to support speech understanding. In human-robot interactions (HRI), it is often implicitly assumed that gaze to humanoid robots follows similar patterns, but this has rarely been tested quantitatively at the level of specific facial regions. In this late-breaking report, we report a secondary analysis of an existing HRI dataset with usable eye-tracking data from 31 participants who took part in semi-structured interviews with a social robot (Furhat). Using MediaPipe Face Mesh on participant’s egocentric video from eye tracking glasses, we segmented the robot’s face into eye, mouth, and full-face regions of interest (ROI), and quantified how participants distributed their gaze at each ROI over the entire interaction, and separately for speaking and listening. Participants spent most of the interaction looking at the robot’s face; within the face, the eyes and mouth were the main targets, and gaze to these regions increased during listening, especially for the mouth. This pattern aligns with the central findings from HHI and offers empirical evidence for assumed similarities in gaze allocation between HHI and HRI. In an exploratory analysis, we additionally examined how the robot’s own gaze behaviour, with or without human-like gaze aversions, shaped gaze to the eyes and mouth. We discuss how these findings inform the interpretation of gaze as an implicit engagement cue in HRI. Finally, we provide baseline references and show how ROI-based analyses can enrich future gaze studies in HRI.

Abstract

Recent cognitive neuroscience research has uncovered some similarities and some differences between general semantic memory (GS; e.g., knowledge about family in general) and personal semantic memory (PS; e.g., knowledge about my family in particular). To better understand the representational content and cognitive processes of PS and their relation to general semantics, we adapted a staple of General semantic memory research, the Property Generation task. In a first study, we randomly assigned 240 adult participants to a traditional General semantics perspective (e.g., listing the properties of families and bedrooms in general) or to a PS perspective (e.g., my family, my bedroom) in a between-subjects design. In a second, replication study, 124 participants completed the task in a within-subject design, taking each perspective for different concepts. Relative to the General semantics condition, the PS condition was associated with more features; these were more semantically distant from each other, and included adjectives more frequently and nouns less frequently. However, the two conditions had substantial (~46%) overlap in the frequency of their features. The findings contribute to the identification of the cognitive differences (e.g., semantic richness) and similarities (e.g., correspondence in semantic features) between personal and general semantic memory.

Abstract

To navigate in and communicate about the continuous world we experience, our minds segment this experience into discrete event units. Yet, languages differ in how they package core aspects of events into linguistic units. Here, we ask how event units in language and cognition relate to each other, and how this relation might change during language acquisition. To do so, we focus on motion events and compare child and adult speakers of Turkish—a verb-framed language encoding motion events in multiple linguistic units with distinct units for each path segment. In a linguistic task, there were systematic differences in the number of linguistic units used for expressing motion paths when describing events with versus without direction changes in adults and to a lesser extent in 5-year-olds but not in 4-year-olds. In a non-linguistic eye-tracked dwell-time task, both children and adults had similar visual attention profiles for events with and without direction changes. These findings indicate that although linguistic event units become increasingly language-specific with age, cognitive event units remain stable and independent of linguistic encoding. These findings show that people flexibly shift between different levels of granularity when segmenting events in language and cognition. Further, this flexibility seems to emerge in children as young as 4 to 5 years old.

Abstract

How do children allocate visual attention to scenes as they prepare to describe them multimodally in speech and co-speech gesture? In an eye-tracking study, Turkish-speaking 8-year-old children viewed four-picture displays depicting the same two objects in different spatial relations as they prepared to describe target pictures depicting left-right relations. Children's visual attention was sustained on the target picture when they were planning descriptions that expressed the spatial relation multimodally in speech and gesture, but not unimodally in speech only. This pattern persisted regardless of the semantic relation between speech and gesture (i.e., for both complementary gestures that disambiguated speech and redundant gestures that supplemented already unambiguous speech). Importantly, visual attention patterns did not differ across description types while children were previewing the displays before message preparation. These results indicate that multimodal message preparation might place different demands on visual attention than unimodal message preparation, possibly due to the affordances of gestures for expressing spatial relations.

Abstract

While events can be identified based on both perceptual and higher-level information, the contribution of language-related factors remains unclear. Here, we examine whether recent linguistic experience modulates event role identification scenes by asking whether verbs with different argument and thematic structures cue abstract event representations that guide subsequent event cognition. Participants briefly viewed agent–patient events (for 300ms) preceded by a transitive verb, an intransitive verb, or a nonsense word control, and then completed a probe-based role identification task. Transitive verb primes facilitated recognition of event roles, yielding faster responses in role-matching trials relative to intransitive and control conditions. These findings show that linguistic cues influence event role identification, especially when their argument and thematic structure aligns with event structures.

Abstract

The philosopher G.E. Moore first observed that making a statement and then denying that one knows or believes that statement is unacceptable. For example, "It is raining, but I don’t think that" is defective. Across six experiments (n = 600), this study investigates the nature and extent of this unacceptability as a way to adjudicate between alternative theoretical explanations of this defectiveness. Results confirm that Moorean conjunctions are judged more acceptable than semantic contradictions (e.g., "It is raining, but it isn’t") but less so than felicitous conjunctions (e.g., "It is raining, but it’s okay"), and this is so regardless of whether Moorean conjunctions are produced by a person or by an artificial agent such as ChatGPT. However, if the first conjunct is anchored on a perspective other than the speaker’s, Moorean conjunctions increase in acceptability (e.g., "The train arrives at noon, but I think it won’t"). Several features can drive this perspective-shift, yet the acceptability of Moorean conjunctions did not match that of felicitous sentences in our sample. Our findings support the view that the unacceptability is a byproduct of mindreading, a key part of the social cognition that is crucial to using and understanding language.

Abstract

This workshop proposes looking at errors in humans and robots. The workshop will focus along three axes: Human, Robot and Interaction induced errors. Errors in Human-Robot Interaction (HRI) pose a crucial challenge for the deployment of robots. Errors can affect the safety and trust of users, therefore it is important for empowering society that these robots can handle errors robustly. The workshop intends to foster interdisciplinary discussion among researchers in robotics, HCI, cognitive science, and social sciences, to encourage the development of robust, user-centered, and socially responsible HRI systems. The invited speakers, paper presentations and discussions will focus around topics of detecting errors, handling errors, adapting to errors, restoring trust, using intentional errors, communication strategies, issues of expectation as well as look at design, interdisciplinary and ethical approaches to research. In this way we will help inform research into errors, and develop robotic systems capable of robust interaction and collaboration.

Abstract

The present intervention study examines how parents' use of gestures changes the way they provide math talk to their children, and whether increased use of parental gestures promotes children's math skills. In this parent-administered book reading intervention study, three- to four-year old Turkish-speaking children (N = 56, Mage = 49.8 months, SD = 3.6) and their parents were randomly assigned to one of the following conditions: parents asked to use gestures while reading a numerical language book (NL + G, n = 19), parents asked not to use gestures while reading a numerical language book (NL-G, n = 18), no specific instruction on gesture use to parents while reading a book without numerical language (L, n = 19). Children were assessed on four math tasks (verbal counting, cardinality, nonverbal, and verbal arithmetic) before and after the intervention phase. Parents also read their assigned book to children in both the pre- and post-intervention sessions. Results showed that parents provided more math talk when they were assigned to NL + G and NL-G compared to L in the post-intervention reading session. Moreover, parents in the NL + G group provided more math talk than those in the NL-G group. Children assigned to the NL + G group showed better improvement in their verbal arithmetic skills than those in the NL-G group. No other significant improvements in child performance were found. These results suggest that parental gesture use was associated with higher levels of math input and children's math skills exclusively in the context of the verbal arithmetic task. The possible mechanisms and contributions of providing math talk to children through different modalities are discussed.

Abstract

While bilingual studies have extensively explored crosslinguistic transfer across various domains, prosody remains relatively underexplored. This study addresses the gap by contributing experimental data on prosodic transfer at lexical and pragmatic levels in heritage speakers of Turkish in Germany. Given typological differences between Turkish and German prosody, this population is ideal for examining whether the heritage language (Turkish) is influenced by the majority language (German) in processing prosodic information across linguistic levels. In a recent study, Zora et al. examined how majority language speakers of Turkish interpret prosodic cues, specifically lexical stress and prosodic focus, when assessing sentence acceptability. Forty participants rated sentences containing prosodic violations, revealing that lexical stress violations elicited lower acceptability than prosodic focus violations. This suggests that lexical-level prosody, rather than pragmatic-level prosody, plays a more central role in spoken comprehension for majority language speakers of Turkish. Building on these findings, this study applies the same paradigm to 40 heritage Turkish speakers in Germany. Results show the reverse pattern, such that prosodic focus violations were rated less acceptable than lexical stress violations. Typologically, this aligns with Germanic languages, where prosodic focus holds greater weight in interpretation under the same experimental paradigm. The results further revealed that proficiency modulated sensitivity to prosodic information, such that high-proficiency speakers of Turkish judged both stress and focus violations as incorrect, whereas low-proficiency speakers judged only focus violations as incorrect. These findings highlight the dynamic nature of prosodic transfer, potentially from German to Turkish, and the significant role of language dominance and proficiency in shaping bilingual prosodic processing.

Abstract

Children's emotional development is linked to language development for typically developing children and deaf children with native sign language exposure. However, approximately 90% of deaf children are born to hearing parents who are not familiar with sign language. These deaf children begin learning a sign language when they attend a school for the deaf. Late sign language exposure has negative consequences on several aspects of language development. We investigate whether acquiring sign language late affects children's emotion recall and channel of emotion expression. After watching a silent video depicting emotions, late- and native-signing deaf children retold the story in Turkish Sign Language. Results showed that late signers recalled fewer emotions and used fewer signs and facial expressions compared to native signers. Manual gestures (non-sign hand movements), head and body movements did not differ across groups. The findings suggest that late sign language acquisition negatively impacts deaf children's ability to recall and express emotions, highlighting the importance of early language exposure for the development of emotion recall.

Abstract

Thematic roles in language (Agents, Patients) are considered to be hierarchically organized in terms of their salience, and this hierarchy is rooted in their counterparts as event participants in cognition. Here, we examine the relative salience of Agents over Patients in two-participant causative events in Turkish-speaking 3- to 5-year-old children. We also test if this asymmetry is modulated by the animacy of the Patient (human vs. inanimate object) and specific to the presence of a coherent event. In an eye-tracked change detection task, changes to Agents were detected more accurately (and after fewer fixations) than changes to inanimate Patients when there was a coherent event. This asymmetry disappeared when the Patient was animate (for accuracy) and when event coherence was disrupted (for both accuracy and fixations). These findings suggest an interplay of event roles and animacy in Agent preference.

Abstract

There is an increasing trend towards evaluating NLP models with LLMs instead of human judgments, raising questions about the validity of these evaluations, as well as their reproducibility in the case of proprietary models. We provide JUDGE-BENCH, an extensible collection of 20 NLP datasets with human annotations covering a broad range of evaluated properties and types of data, and comprehensively evaluate 11 current LLMs, covering both open-weight and proprietary models, for their ability to replicate the annotations. Our evaluations show substantial variance across models and datasets. Models are reliable evaluators on some tasks, but overall display substantial variability depending on the property being evaluated, the expertise level of the human judges, and whether the language is human or model-generated. We conclude that LLMs should be carefully validated against human judgments before being used as evaluators.

Abstract

Speech is inherently variable, requiring listeners to apply adaptation mechanisms to deal with the variability. A proposed perceptual adaptation mechanism is recalibration, whereby listeners learn to adjust cognitive representations of speech sounds based on disambiguating contextual information. Most studies on the role of recalibration in speech perception have focused on variability in particular speech segments (e.g., consonants/vowels), and speech has mostly been studied with a focus on talking heads. However, speech is often accompanied by visual bodily signals like hand gestures, and is thus multimodal. Moreover, variability in speech extends beyond segmental aspects alone and also affects prosodic aspects, like lexical stress. We currently do not understand well how listeners adjust their representations of lexical stress patterns to different speakers. In four experiments, we investigated recalibration of lexical stress perception, driven by lexico-orthographical information (Experiment 1) and by manual beat gestures (Experiments 2–4). Across experiments, we observed that these two types of disambiguating information (presented in an audiovisual exposure phase) led listeners to adjust their representations of lexical stress, with lasting consequences for subsequent spoken word recognition (in an audio-only test phase). However, evidence for generalization of this recalibration to new words was only found in the third experiment, suggesting that generalization may be limited. These results highlight that recalibration is a plausible mechanism for suprasegmental speech adaption in everyday communication and show that even the timing of simple hand gestures can have a lasting effect on auditory speech perception.

Abstract

This paper presents a model specification for group comparisons regarding a functional trend over time within a trial and learning across a series of trials in intensive binary longitudinal eye-tracking data. The functional trend and learning effects are modeled using by-variable smooth functions. This model specification is formulated as a generalized additive mixed model, which allowed for the use of the freely available mgcv package (Wood in Package ‘mgcv.’ https://cran.r-project.org/web/packages/mgcv/mgcv.pdf, 2023) in R. The model specification was applied to intensive binary longitudinal eye-tracking data, where the questions of interest concern differences between individuals with and without brain injury in their real-time language comprehension and how this affects their learning over time. The results of the simulation study show that the model parameters are recovered well and the by-variable smooth functions are adequately predicted in the same condition as those found in the application.

Abstract

Narrative discourse impairments are well documented in individuals with moderate-severe traumatic brain injury (TBI). Studies of narrative discourse (i.e., story generation, story retelling) in this population have frequently focused on impairment of semantic relations across utterances and the larger discourse context (e.g., cohesion, coherence, story grammar). Less attention has been given to the temporal organization of narrative retelling in TBI. We applied temporal contiguity analyses, a technique traditionally used to characterize temporal organization of free recall of wordlists, to quantify the temporal organization of participants' story retellings with respect to the order in which the narrator originally presented the story details. We also conducted a parallel analysis of temporal contiguity of wordlist recall using data from the Rey Auditory Verbal Learning test. Participants with moderate-severe TBI and non-injured peers demonstrated above chance temporal organization and a tendency to make short transitions in the forward direction when recalling items in both the narrative recall and wordlist recall task. However, these effects were significantly reduced in the TBI group. Overall, their free recall performance was less temporally clustered, and they were more likely to make larger jumps between story details (or words in the wordlist recall task) than their non-injured peers when recalling stories. Examining free recall at multiple timepoints revealed that while repetition (i.e., multiple presentations of the wordlist) increased temporal organization of recall, long delays (i.e., one week) decreased temporal organization for both the TBI and non-injured groups. We propose that reduced temporal organization of narrative recall in individuals with moderate-severe TBI is linked to impairments in the declarative relational memory system. In line with retrieved-context models of free recall, memory disruption not only impacts the total number of story details recalled, but also the ability to use temporal context to encode and retrieve items in a sequentially organized way.

Abstract

Increasing evidence suggests that interlocutors use visual communicative signals to form predictions about unfolding utterances, but there is little data on the predictive potential of facial signals in conversation. In an online experiment with virtual agents, we examine whether facial signals produced by an addressee may allow speakers to anticipate the response to a question before it is given. Participants (n = 80) viewed videos of short conversation fragments between two virtual humans. Each fragment ended with the Questioner asking a question, followed by a pause during which the Responder looked either straight at the Questioner (baseline), or averted their gaze, or accompanied the straight gaze with one of the following facial signals: brow raise, brow frown, nose wrinkle, smile, squint, mouth corner pulled back (dimpler). Participants then indicated on a 6-point scale whether they expected a “yes” or “no” response. Analyses revealed that all signals received different ratings relative to the baseline: brow raises, dimplers, and smiles were associated with more positive responses, gaze aversions, brow frowns, nose wrinkles, and squints with more negative responses. Qur findings show that interlocutors may form strong associations between facial signals and upcoming responses to questions, highlighting their predictive potential in face-to-face conversation.

Abstract

Previous research has suggested the importance of relational language and working memory in children’s relational reasoning. The tendency to use language (e.g., using more relational than object-focused language, prioritizing focal objects over background in linguistic descriptions) could reflect children’s biases toward the relational versus object-based solutions in a relational match-to-sample (RMTS) task. In the lack of any apparent object match as a foil option, object-focused children might rely on other cognitive mechanisms (i.e., working memory) to choose a relational match in the RMTS task. The current study examined the interactive roles of language- and working memory-related sources of variation in Turkish-learning preschoolers’ relational reasoning. We collected data from 4- and 5-year-olds (N = 41) via Zoom in the RMTS task, a scene description task, and a backward word span task. Generalized binomial mixed effects models revealed that children who used more relational language and background-focused scene descriptions performed worse in the relational reasoning task. Furthermore, children with less frequent relational language use and focal object descriptions of the scenes benefited more from working memory to succeed in the relational reasoning task. These results suggest additional working memory demands for object-focused children to choose relational matches in the RMTS task, highlighting the importance of examining the interactive effects of different cognitive mechanisms on relational reasoning.

Abstract

In face-to-face interaction, we use multiple modalities, including speech and gestures, to communicate information and resolve references to objects. However, how representational co-speech gestures refer to objects remains understudied from a computational perspective. In this work, we address this gap by introducing a multimodal reference resolution task centred on representational gestures, while simultaneously tackling the challenge of learning robust gesture embeddings. We propose a self-supervised pre-training approach to gesture representation learning that grounds body movements in spoken language. Our experiments show that the learned embeddings align with expert annotations and have significant predictive power. Moreover, reference resolution accuracy further improves when (1) using multimodal gesture representations, even when speech is unavailable at inference time, and (2) leveraging dialogue history. Overall, our findings highlight the complementary roles of gesture and speech in reference resolution, offering a step towards more naturalistic models of human-machine interaction.

Abstract

When speakers refer to objects in the world, they often overinform: provide their listener with redundant adjectival information. Contrary to classical theories in linguistics, recent theories have framed overinformativeness as an efficient means of grounding reference in perceptual information of high discriminability to facilitate listener comprehension. However, the generalisability of such theories is constrained by the methodological challenge associated with reliably manipulating the perceptual discriminability of naturalistic stimuli. Here, we overcome these methodological challenges, using methods from psychophysics to manipulate the perceptual discriminability of colour and material attributes in a reference-production experiment. We provide a robust validation of the view that overinformative reference is driven by speakers grounding expressions in attributes of high discriminability. However, we also find that colour information is privileged above and beyond such factors of discriminability.

Abstract

Children need to learn the demands of their native language in the early vocabulary development phase. In this dynamic process, parental multimodal input may shape neurodevelopmental trajectories while also being tailored by child-related factors. Moving beyond typically characterized group profiles, in this article, we synthesize growing evidence on the effects of parental multimodal input (amount, quality, or absence), domain-specific input (space and math), and language-specific input (causal verbs and sound symbols) on preterm, full-term, and deaf children's early vocabulary development, focusing primarily on research with children learning Turkish and Turkish Sign Language. We advocate for a theoretical perspective, integrating neonatal characteristics and parental input, and acknowledging the unique constraints of languages.

Abstract

Language is inherently multimodal. In spoken languages, combined spoken and visual signals (e.g., co-speech gestures) are an integral part of linguistic structure and language representation. This requires an extension of the parallel architecture, which needs to include the visual signals concomitant to speech. We present the evidence for the multimodality of language. In addition, we propose that distributional semantics might provide a format for integrating speech and co-speech gestures in a common semantic representation.

Abstract

It has long been known that we use words to perform speech acts foundational to everyday conversation, such as requesting, informing, proposing, or complaining. However, the natural environment of human language is face-to-face interaction where we use words and an abundance of visual signals to communicate. The multimodal nature of human language is increasingly recognised in the language and cognitive sciences. In line with this turn of the tide, findings demonstrate that facial signals significantly contribute to communicating intentions and that they may facilitate pragmatically appropriate responding in the fast-paced environment of conversation. In light of this, the notion of speech acts no longer seems appropriate, highlighting the need for a modality-neutral conception, such as social action.

Abstract

Interlocutors often use the semantics of comprehended speech to inform the semantics of planned speech. Do representations of the comprehension and planning stimuli interact? In this EEG study, we used rapid invisible frequency tagging (RIFT) to better understand the attentional distribution to representations of comprehension and speech planning stimuli, and how they interact in the neural signal. To do this, we leveraged the picture-word interference (PWI) paradigm with delayed naming, where participants simultaneously comprehend auditory distractors (auditory [f1]; tagged at 54 Hz) while preparing to name related or unrelated target pictures (visual [f2]; tagged at 68 Hz). RIFT elicits steady-state evoked potentials, which reflect allocation of attention to the tagged stimuli. When representations of the tagged stimuli interact, increased power has been observed at the intermodulation frequency resulting from an interaction of the base frequencies (f2 ± f1; Drijvers et al., 2021). Our results showed clear power increases at 54 Hz and 68 Hz during the tagging window, but no power difference between the related and unrelated condition. Interestingly, we observed a larger power difference in the intermodulation frequency (compared to baseline) in the unrelated compared to the related condition (68 Hz − 54 Hz: 14 Hz), indicating stronger interaction between unrelated auditory and visual representations. Our results go beyond standard PWI results by showing that participants’ difficulties in the related condition do not arise from allocating attention to the pictures or distractors. Instead, processing difficulties arise during interaction of the concepts or lemmas invoked by the two stimuli, thus, we conclude, that interaction might be downregulated in the related condition.

Abstract

Psycholinguistic investigations provide invaluable empirical utility in theorizing and typologizing phonological phenomena. Instrumental approaches to the sound structure of Turkish have proven to be no exception here, contributing independent and multi-faceted evidence towards theory building and testing. Two areas of Turkish phonology in relation to suprasegmental structure and prominence patterns, namely word-level prosody (Section 2) and prominence and rhythmic phenomena at the level of the sentence and beyond (Section 3) have particularly fueled psycholinguistically motivated empirical studies. This chapter will approach representational and processing-related issues in each of these and provide a review of pertinent perception and production studies, touching upon phonetic and developmental investigations insofar as they have implications for mental representations or processing.

Abstract

*Joint first authorship.
Language acquisition unfolds within inherently multimodal contexts, where communication is expressed and perceived through diverse channels embedded in social interactions. For hearing children, this involves integrating speech with gesture; for deaf children, language develops through fully visual modalities. Such observations necessitate a paradigm shift from speech-centric models to a holistic framework that equally values all modalities, whether in spoken or signed languages. This framework must account not only for the multimodal scaffolding of input and interaction but also for individual and contextual diversity, including the cultural and cognitive variabilities children bring to language learning contexts. Responding to commentaries on our target article, this paper refines and expands the multimodal language framework, emphasizing its capacity to integrate the interactive richness of input and the heterogeneous contexts and individual variations shaping language acquisition.

Abstract

*=shared first authorship
Children across the world acquire their first language(s) naturally, regardless of typology or modality (e.g. sign or spoken). Various attempts have been made to explain the puzzle of language acquisition using several approaches, trying to understand to what extent it can be explained by what children bring to language-learning situations as well as what they learn from the input and the interactive context. However, most of these approaches consider only speech development, thus ignoring the inherently multimodal nature of human language. As a multimodal view of language is becoming more widely adopted for the study of adult language, a multimodal approach to language acquisition is inevitable. Not only do children have the capacity to learn spoken and sign language equally easily, but spoken language acquisition consists of learning to coordinate linguistic expressions in both modalities, that is, in both speech and gesture. To provide a step forward in this direction, this article aims to synthesize findings from research studies that take a multimodal perspective on language acquisition in different sign and spoken languages, including the development of speech and accompanying gestures. Our review shows that while some aspects of language acquisition seem to be modality-independent, others might differ according to the affordances of each modality when used separately as well as together (either in sign, speech, and/or gesture). We argue that these findings need to be integrated into our understanding of language acquisition. We also identify which areas need future research for both spoken and sign language acquisition, taking into account not only multimodal but also cross-linguistic variation.

Abstract

One of the earliest sources of mathematical input arises in dyadic parent–child interactions. However, the emphasis has been on parental input only in speech and how input varies across different environmental and child-specific factors remains largely unexplored. Here, we investigated the relationship among parental math input modality and type, children’s gestational status (being preterm vs. full-term born), and vocabulary development. Using book-reading as a medium for parental math input in dyadic interaction, we coded specific math input elicited by Turkish-speaking parents and their 26-month-old children (N = 58, 24 preterms) for speech-only and multimodal (speech and gestures combined) input. Results showed that multimodal math input, as opposed to speech-only math input, was uniquely associated with gestational status, expressive vocabulary, and the interaction between the two. Full-term children with lower expressive vocabulary scores received more multimodal input compared to their preterm peers. However, there was no association between expressive vocabulary and multimodal math input for preterm children. Moreover, cardinality was the most frequent type for both speech-only and multimodal input. These findings suggest that the specific type of multimodal math input can be produced as a function of children’s gestational status and vocabulary development.

Abstract

Purpose: The purpose of this viewpoint is to advocate for increased study of 3654244brcommon ground and audience design processes in multiparty communication in traumatic brain injury (TBI).
Method: Building on discussions at the 2024 International Cognitive-Communication Disorders Conference, we review common ground and audience design processes in dyadic and multiparty communication. We discuss how the diffuse profiles of neural and cognitive deficits place individuals with TBI at increased risk for keeping track of who knows what in group settings and using that knowledge to flexibly adapt their communication behaviors.
Results: We routinely engage in social communication in groups of three or more people at work, school, and social functions. While academic, vocational, and interpersonal domains are all areas where individuals with TBI are at risk for negative outcomes, we know very little about the impact of TBI on group, or multiparty, communication.
Conclusions: The empirical study of common ground and audience design in multiparty communication in TBI presents a promising new direction in characterizing the impact of TBI on social communication, uncovering the underlying mechanisms of cognitive-communication disorders, and may lead to new inter-ventions aimed at improving success in navigating group communication at work and school, and in interpersonal relationships.

Abstract

Creating a virtual avatar with semantically coherent gestures that are aligned with speech is a challenging task. Existing gesture generation research mainly focused on generating rhythmic beat gestures, neglecting the semantic context of the gestures. In this paper, we propose a novel approach for semantic grounding in co-speech gesture generation that integrates semantic information at both fine-grained and global levels. Our approach starts with learning the motion prior through a vector-quantized variational autoencoder. Built on this model, a second-stage module is applied to automatically generate gestures from speech, text-based semantics and speaker identity that ensures consistency between the semantic relevance of generated gestures and co-occurring speech semantics through semantic coherence and relevance modules. Experimental results demonstrate that our approach enhances the realism and coherence of semantic gestures. Extensive experiments and user studies show that our method outperforms state-of-the-art approaches across two benchmarks in co-speech gesture generation in both objective and subjective metrics.

Abstract

Libraries are very often considered the hallway to developing knowledge. However, the lack of adequate staff within Indian libraries makes catering to the visitors’ needs difficult. Previous systems that have sought to address libraries’ needs through automation have mostly been limited to storage and fetching aspects while lacking in their interaction aspect. We propose to address this issue by incorporating social robots within Indian libraries that can communicate and address the visitors’ queries in a multi-modal fashion attempting to make the experience more natural and appealing while helping reduce the burden on the librarians. In this paper, we propose and deploy a Furhat robot as a robot librarian by programming it on certain core librarian functionalities. We evaluate our system with a physical robot librarian (N = 26). The results show that the robot librarian was found to be very informative and overall left with a positive impression and preference.

Abstract

This preregistered study examined whether visual experience influences conceptual representations by examining both gestural expression and feature listing. Gestures—mostly driven by analog mappings of visuospatial and motoric experiences onto the body—offer a unique window into conceptual representations and provide complementary information not offered by language-based features, which have been the focus of previous work. Thirty congenitally or early blind and 30 sighted Turkish speakers produced silent gestures and features for concepts from semantic categories that differentially rely on experience in visual (non-manipulable objects and animals) and motor (manipulable objects) information. Blind individuals were less likely than sighted individuals to produce gestures for non-manipulable objects and animals, but not for manipulable objects. Overall, the tendency to use a particular gesture strategy for specific semantic categories was similar across groups. However, blind participants relied less on drawing and personification strategies depicting visuospatial aspects of concepts than sighted participants. Feature-listing revealed that blind participants share considerable conceptual knowledge with sighted participants, but their understanding differs in fine-grained details, particularly for animals. Thus, while concepts appear broadly similar in blind and sighted individuals, this study reveals nuanced differences, too, highlighting the intricate role of visual experience in conceptual representations.

Abstract

The increased interest in developing next-gen social robots has raised questions about the factors affecting the perception of robot emotions. This study investigates the impact of robot appearances (human-like, mechanical) and face regions (full-face, eye-region) on human perception of robot emotions. A between-subjects user study (N = 305) was conducted where participants were asked to identify the emotions being displayed in videos of robot faces, as well as a human baseline. Our findings reveal three important insights for effective social robot face design in Human-Robot Interaction (HRI): Firstly, robots equipped with a back-projected, fully animated face – regardless of whether they are more human-like or more mechanical-looking – demonstrate a capacity for emotional expression comparable to that of humans. Secondly, the recognition accuracy of emotional expressions in both humans and robots declines when only the eye region is visible. Lastly, within the constraint of only the eye region being visible, robots with more human-like features significantly enhance emotion recognition.

Abstract

This paper explores image modeling from the frequency space and introduces DCTdiff, an end-to-end diffusion generative paradigm that efficiently models images in the discrete cosine transform (DCT) space. We investigate the design space of DCTdiff and reveal the key design factors. Experiments on different frameworks (UViT, DiT), generation tasks, and various diffusion samplers demonstrate that DCTdiff outperforms pixel-based diffusion models regarding generative quality and training efficiency. Remarkably, DCTdiff can seamlessly scale up to 512×512 resolution without using the latent diffusion paradigm and beats latent diffusion (using SD-VAE) with only 1/4 training cost. Finally, we illustrate several intriguing properties of DCT image modeling. For example, we provide a theoretical proof of why ‘image diffusion can be seen as spectral autoregression’, bridging the gap between diffusion and autoregressive models. The effectiveness of DCTdiff and the introduced properties suggest a promising direction for image modeling in the frequency space. The code is at https://github.com/forever208/DCTdiff.

Abstract

The study investigated whether the perspective of multimodal input in visuospatial maps predicts children’s spatial performance, particularly verbal recall and direction-following behavior. 5-year-old monolingual Turkish children were engaged in the Directions Task, which included visuospatial maps and videos of a speaker describing routes on maps in three conditions: Speech-Gesture combination with a front-facing view, Speech-Gesture combination with an upper back angle, and Speech-only condition with a front-facing view for control. Children were asked to verbally recall and draw the route described in the videos. They also engaged in perspective-taking, mental rotation, and relational reasoning tasks. Results showed that children’s verbal recall, but not necessarily behavioral recall, was enhanced by receiving multimodal directions. Moreover, children’s relational reasoning and perspective-taking abilities modulate their verbal recall performances. The results of this study underline the importance of multimodal input and presentation perspective in enhancing children’s spatial performance.

Abstract

Gestures express redundant or complementary information to speech they accompany by depicting visual and spatial features of referents. In doing so, they recruit both spatial and verbal cognitive resources that underpin the processing of visual semantic information and its integration with speech. The relation between spatial and verbal skills and gesture comprehension, where gestures may serve different roles in relation to speech is yet to be explored. This study examined the role of spatial and verbal skills in processing gestures that expressed redundant or complementary information to speech during the comprehension of spatial relations between objects. Turkish-speaking adults (N=74) watched videos describing the spatial location of objects that involved perspective-taking (left-right) or not (on-under) with speech and gesture. Gestures either conveyed redundant information to speech (e.g., saying and gesturing “left”) or complemented the accompanying demonstrative in speech (e.g., saying “here,” gesturing “left”). We also measured participants’ spatial (the Corsi block span and the mental rotation tasks) and verbal skills (the digit span task). Our results revealed nuanced interactions between these skills and spatial language comprehension, depending on the modality in which the information was expressed. One insight emerged prominently. Spatial skills, particularly spatial working memory capacity, were related to enhanced comprehension of visual semantic information conveyed through gestures especially when this information was not present in the accompanying speech. This study highlights the critical role of spatial working memory in gesture processing and underscores the importance of examining the interplay among cognitive and contextual factors to understand the complex dynamics of multimodal language.

Abstract

Emotional facial expressions often take place during communicative face-to-face interactions. Yet little is known as to whether natural spoken processing can be modulated by emotional expressions during online processing. Furthermore, the functional independence of syntactic processing from other cognitive and affective processes remains a long-standing debate in the literature. To address these issues, this study investigated the influence of masked emotional facial expressions on syntactic speech processing. Participants listened to sentences that could contain morphosyntactic anomalies while a masked emotional expression was presented for 16 ms (i.e., subliminally) just preceding the critical word. A larger Left Anterior Negativity (LAN) amplitude was observed for both emotional faces (i.e., happy and angry) compared to neutral ones. Moreover, a larger LAN amplitude was found for angry faces than for happy faces. Finally, a reduced P600 amplitude was observed only for angry faces when compared to neutral faces. Collectively, the results presented here indicate that first-pass syntactic parsing is influenced by emotional visual stimuli even under masked conditions and that this effect extends also to later linguistic processes. These findings constitute evidence in favor of an interactive view of language processing as integrated within a complex and integrated system for human communication.

Abstract

Prior research shows that self-referential information (e.g., seeing one's own face) is prioritized in human cognition. However, the brain signal variability underlying self-processing remains scarcely treated in the literature. Additionally, less is known about whether the processing of self-referential visual content can be modulated by facial expressions of emotion, as these resemble more natural situations than neutral expressions. This study therefore investigated the brain signal variability underlying self-referential visual processing and its possible interaction with emotional facial expressions, as indexed by multiscale entropy analysis (MSE). This metric captures the temporal complexity or variability contained in neural patterns at varying timescales. Thirty-two participants were presented with distinctive facial identities (self, friend, and unknown) displaying different facial expressions (happy, neutral, and angry) and performed an identity recognition task. Our results showed that brain signal variability decreases in response to self-faces compared to other identities. Similarly, brain signal variability also decreases for friend faces relative to unknown faces. This reduction in complexity could be indicative of greater efficiency during the preferential processing of personally relevant stimuli. Furthermore, the data observed here show that self-processing is unaffected by facial expressions of emotion, suggesting an independent processing of identity from more dynamic facial information, particularly when the task demands are focused on identity recognition. These results provide novel evidence of the moment-to-moment brain signal variability involved in the identity of the self and others. The evidence presented here adds to a growing literature highlighting the relevance of neural variability for understanding brain-behavior relationships.

Abstract

Pragmatic phenomena are characterized by extreme variability, which makes it difficult to draw sound generalizations about the role of social cognition in pragmatic language by and large. I introduce cultural evolutionary pragmatics as a new framework for the study of the interdependence between language and social cognition, and point at the study of common-ground management across languages and ages as a way to test the reliance of pragmatic language on social cognition. I illustrate this new research line with three experiments on article use by second language speakers, whose mother tongue lacks articles. These L2 speakers are known to find article use challenging and it is often argued that their difficulties stem from articles being pragmatically redundant. Contrary to this view, the results of this exploratory study support the view that proficient article use requires automatizing basic socio-cognitive processes, offering a window into the interdependence between language and social cognition.

Abstract

How might social cognition help us communicate through language? At what levels does this interaction occur? In classical views, social cognition is independent of language, and integrating the two can be slow, effortful, and error-prone. But new research into word level processes reveals that communication
is brimming with social micro-processes that happen in real time, guiding even the simplest choices like how we use adjectives, articles, and demonstratives. We interpret these findings in the context of advances in theoretical models of social cognition and propose a Communicative Mind-Tracking
framework, where social micro-processes aren’t a secondary process in how we use language—they are fundamental to how communication works.

Abstract

This chapter introduces cultural evolutionary pragmatics as an empirical research programme that investigates the interdependence of language and social cognition as a product of cultural evolution. According to the positive feedback loop hypothesis, these two distinctively human capacities are connected in a positive reinforcement cycle, whereby the development of one skill boosts the development of the other. It is hypothesized that language and social cognition co-develop in ontogeny and co-evolve in diachrony through the acquisition, mature use, and cultural evolution of reference systems (including, e.g., demonstratives, articles, pronouns, and proper names). The interdependence of reference systems and social cognition must therefore be investigated across three parallel timescales—language acquisition, language use, and language change. The potential of this research programme is illustrated with a cross-linguistic study of demonstrative choice and its relation to monitoring the listener’s attention in referential communication.

Abstract

Sign languages—the conventional languages of deaf communities—have been considered to provide a window into answering some questions regarding language emergence and evolution. In particular, iconicity, defined as the ‘existence of a structure-preserving mapping between mental models of linguistic form and meaning’, is generally regarded as a precursor to the arbitrary and segmental categorical structures found in spoken languages. However, iconic structures are omnipresent in sign languages at all levels of linguistic organization. Thus, there is a necessity for a more nuanced understanding of iconicity and its trajectory in language evolution. In this chapter, we outline different quantitative and qualitative methods to study iconicity and how one can operationalize them at lexical and discourse levels to investigate the role of iconicity in the evolution of sign languages.

Abstract

Purpose:
This study aimed to identify how a set of predefined factors modulates phoneme articulation time within a speaker.
Method:
We used a custom in-lab system that records lip muscle activity through electromyography signals, aligned with the produced speech, to measure phoneme articulation time. Twenty Spanish-speaking participants (12 females) were evaluated while producing sequences of a consonant–vowel syllable, with each sequence consisting of repeated articulations of either /pa/ or /pu/. Before starting the sequences, participants underwent a priming step with either a fast or slow speech rate. Additionally, the general attentional state level was assessed at the beginning, middle, and end of the protocol. To analyze the variability in the duration of /p/ and vowel articulation, we fitted individual linear mixed-models considering three factors: general attentional state level, priming rate, and coarticulation effects (for /p/, i.e., followed by /a/ or /u/) or phoneme identity (for vowels, i.e., being /a/ or /u/).
Results:
We found that the level of general attentional state positively correlated with production time for both the consonant /p/ and the vowels. Additionally, /p/ production was influenced by the nature of the following vowel (i.e., coarticulation effects), while vowel production time was affected by the primed speech rate.
Conclusions:
Phoneme duration appears to be influenced by both stable, speaker-specific characteristics (idiosyncratic traits) and internal, state-dependent factors related to the speaker's condition at the time of speech production. While some factors affect both consonants and vowels, others specifically modify only one of these types.

Abstract

This study investigates the role of action bias in the acquisition of classifier constructions by deaf children acquiring Turkish Sign Language (TİD). While classifier handshapes are morphologically complex and iconic, deaf children (aged 7–9) were found to prefer handling classifiers (reflecting the actions performed by agents) more than signing adults, even in contexts requiring entity classifiers (reflecting the visual properties of their referents). The findings reveal that children’s frequent use of action-based lexical signs for nouns influenced their classifier preferences, suggesting a cognitive bias toward motoric representations. Furthermore, our results suggest the use of handling classifiers in intransitive contexts — even by adult signers — thus indicating a new type of variability in classifier use, which has not been reported for other sign languages before. These results provide new insights into how iconicity and lexical context shape the developmental trajectory of classifier constructions in sign language acquisition.

Abstract

Purpose:
Typical late-onset Alzheimer's disease (AD) compromises episodic memory, the ability to encode new events and recollect past events. Much of the research on episodic memory in AD has relied on lab-based memory tasks (e.g., word lists, short stories). It is unclear how well these tasks characterize the impact of episodic memory impairments across different domains of everyday life, including conversational memory. The goal of the review was to establish what is known about conversational memory in AD, that is, memory for the content of conversations one overhears or in which one participates, such as utterances said and corresponding referents (e.g., “I remember discussing medical decisions with my children and I said […] and they responded […]”).
Method:
In this scoping review, we followed the JBI Manual for Evidence Synthesis and PRISMA Extension for Scoping Reviews guidelines to conduct the scoping review. We retained 121 reports on conversation and three reports on conversational memory out of the 8,351 unique records found on PubMed, CINAHL, PsycINFO, Web of Science, Cochrane, and Embase and in relevant reviews. Included reports had to involve conversation in any format and on any topic, include an AD population, be peer-reviewed, be in English, and be published between 1990 and 2022.
Results:
None of the studies investigated memory for spontaneous conversation. Although most studies on conversation did report on key characteristics of the interactional context (e.g., level of structure, number and category of conversational partners), studies also left several important details unspecified, such as hearing/vision (omitted in 67% studies) and diagnostic process (omitted in 33% studies). Studies described a broad range of behaviors during conversation, with most concerning verbal behaviors (e.g., repetitions, disfluency, ambiguity) and only 29% nonverbal behaviors (e.g., facial expression, head and hand gestures, eye gaze). Given the rarity of studies on conversational memory, we primarily summarized existing knowledge about conversation and methodological considerations to inspire hypotheses for future research on conversational memory in AD and to illuminate decisions regarding study design.
Conclusions:
This review revealed a wide gap in knowledge on conversational memory in AD and offers a path to accelerating research on the topic. Conversational memory may be an important factor in promoting independence, participation in health care, and social well-being.

Abstract

In face-to-face conversation, people use speech and gesture to convey meaning. Seeing gestures alongside speech facilitates comprehenders’ language processing, but crucially, the mechanisms underlying this facilitation remain unclear. We investigated whether comprehenders use the semantic information in gestures, typically preceding related speech, to predict upcoming meaning. Dutch adults listened to questions asked by a virtual avatar. Questions were accompanied by an iconic gesture (e.g., typing) or meaningless control movement (e.g., arm scratch) followed by a short pause and target word (e.g., “type”). A Cloze experiment showed that gestures improved explicit predictions of upcoming target words. Moreover, an EEG experiment showed that gestures reduced alpha and beta power during the pause, indicating anticipation, and reduced N400 amplitudes, demonstrating facilitated semantic processing. Thus, comprehenders use iconic gestures to predict upcoming meaning. Theories of linguistic prediction should incorporate communicative bodily signals as predictive cues to capture how language is processed in face-to-face interaction.

Abstract

Despite the importance of greetings for opening social interactions, their multimodal coordination processes remain poorly understood. We used a naturalistic, lab-based setup where pairs of unacquainted participants approached and greeted each other while unaware their greeting behavior was studied. We measured the prevalence and time course of multimodal behaviors potentially culminating in a handshake, including motor behaviors (e.g., walking, standing up, hand movements like raise, grasp, and retraction), gaze patterns (using eye tracking glasses), and speech (close and distant verbal salutations). We further manipulated the visibility of partners’ eyes to test its effect on gaze. Our findings reveal that gaze to a partner's face increases over the course of a greeting, but is partly averted during approach and is influenced by the visibility of partners’ eyes. Gaze helps coordinate handshakes, by signaling intent and guiding the grasp. The timing of adjacency pairs in verbal salutations is comparable to the precision of floor transitions in the main body of conversations, and varies according to greeting phase, with distant salutation pair parts featuring more gaps and close salutation pair parts featuring more overlap. Gender composition and a range of multimodal behaviors affect whether pairs chose to shake hands or not. These findings fill several gaps in our understanding of greetings and provide avenues for future research, including advancements in social robotics and human−robot interaction.

Abstract

When engaged in spoken conversation, speakers convey meaning using both speech and visual signals, such as facial expressions and manual gestures. An important question is how information is distributed in utterances during face-to-face interaction when information from visual signals is also present. In a corpus of casual Dutch face-to-face conversations, we focus on spoken questions in particular because they occur frequently, thus constituting core building blocks of conversation. We quantified information density (i.e. lexical entropy and surprisal) and the number and relative duration of facial and manual signals. We tested whether lexical information density or the number of visual signals differed between the first and last halves of questions, as well as whether the number of visual signals occurring in the less-predictable portion of a question was associated with the lexical information density of the same portion of the question in a systematic manner. We found that information density, as well as number of visual signals, were higher in the first half of questions, and specifically lexical entropy was associated with fewer, but longer visual signals. The multimodal front-loading of questions and the complementary distribution of visual signals and high entropy words in Dutch casual face-to-face conversations may have implications for the parallel processes of utterance comprehension and response planning during turn-taking.

Abstract

During conversation, speakers coordinate and synergize their behaviors at multiple levels, and in different ways. The extent to which individuals converge or diverge in their behaviors during interaction may relate to interpersonal differences relevant to social interaction, such as empathy as measured by the empathy quotient (EQ). An association between interpersonal difference in empathy and interpersonal entrainment could help to throw light on how interlocutor characteristics influence interpersonal entrainment. We investigated this possibility in a corpus of unconstrained conversation between dyads. We used dynamic time warping to quantify entrainment between interlocutors of head motion, hand motion, and maximum speech f0 during question–response sequences. We additionally calculated interlocutor differences in EQ scores. We found that, for both head and hand motion, greater difference in EQ was associated with higher entrainment. Thus, we consider that people who are dissimilar in EQ may need to “ground” their interaction with low-level movement entrainment. There was no significant relationship between f0 entrainment and EQ score differences.

Abstract

In spoken languages, children acquire locative terms in a cross-linguistically stable order. Terms similar in meaning to in and on emerge earlier than those similar to front and behind, followed by left and right. This order has been attributed to the complexity of the relations expressed by different locative terms. An additional possibility is that children may be delayed in expressing certain spatial meanings partly due to difficulties in discovering the mappings between locative terms in speech and spatial relation they express. We investigate cognitive and mapping difficulties in the domain of spatial language by comparing how children map spatial meanings onto speech versus visually motivated forms in co-speech gesture across different spatial relations. Twenty-four 8-year-old and 23 adult native Turkish-speakers described four-picture displays where the target picture depicted in-on, front-behind, or left-right relations between objects. As the complexity of spatial relations increased, children were more likely to rely on gestures as opposed to speech to informatively express the spatial relation. Adults overwhelmingly relied on speech to informatively express the spatial relation, and this did not change across the complexity of spatial relations. Nevertheless, even when spatial expressions in both speech and co-speech gesture were considered, children lagged behind adults when expressing the most complex left-right relations. These findings suggest that cognitive development and mapping difficulties introduced by the modality of expressions interact in shaping the development of spatial language.

Abstract

Children vary in mathematical skills even before formal schooling. The current study investigated how parental math beliefs, parents’ math anxiety, and children's spontaneous gestures contribute to preschool-aged children’s math performance. Sixty-three Turkish-reared children (33 girls, Mage = 49.9 months, SD = 3.68) were assessed on verbal counting, cardinality, and arithmetic tasks (nonverbal and verbal). Results showed that parental math beliefs were related to children’s verbal counting, cardinality and arithmetic scores. Children whose parents have higher math beliefs along with low math anxiety scored highest in the cardinality task. Children’s gesture use was also related to lower cardinality performance and the relation between parental math beliefs and children’s performance became stronger when child gestures were absent. These findings highlight the importance of parent and child-related contributors in explaining the variability in preschool-aged children’s math skills.

Abstract

Prosody underpins various linguistic domains ranging from semantics and syntax to discourse. For instance, prosodic information in the form of lexical stress modifies meanings and, as such, syntactic contexts of words as in Turkish kaz-má "pickaxe" (noun) versus káz-ma "do not dig" (imperative). Likewise, prosody indicates the focused constituent of an utterance as the noun phrase filling the wh-spot in a dialogue like What did you eat? I ate----. In the present study, we investigated the relevance of such prosodic variations for discourse comprehension in Turkish. We aimed at answering how lexical stress and prosodic focus mismatches on critical noun phrases-resulting in grammatical anomalies involving both semantics and syntax and discourse-level anomalies, respectively-affect the perceived correctness of an answer to a question in a given context. To that end, 80 native speakers of Turkish, 40 participating in a psychometric experiment and 40 participating in an EEG experiment, were asked to judge the acceptability of prosodic mismatches that occur either separately or concurrently. Psychometric results indicated that lexical stress mismatch led to a lower correctness score than prosodic focus mismatch, and combined mismatch received the lowest score. Consistent with the psychometric data, EEG results revealed an N400 effect to combined mismatch, and this effect was followed by a P600 response to lexical stress mismatch. Conjointly, these results suggest that every source of prosodic information is immediately available and codetermines the interpretation of an utterance; however, semantically and syntactically relevant lexical stress information is assigned more significance by the language comprehension system compared with prosodic focus information.

Abstract

Prosody not only distinguishes “lexical” meaning but also plays a key role in information packaging by highlighting the most relevant constituent of the discourse, namely, “focus” information. The present study investigated the role of lexical and focus functions of prosody in the coherent interpretation of linguistic input. To this end, we manipulated the correctness of prosodic markers in the context and scrutinized how listeners evaluate these violations—whether they result in lexical or focus anomalies—using psychometric and EEG measures. Psychometric data from 40 participants indicated that prosodic violations were judged as incorrect by the listeners both at the lexical and focus levels, with focus level violations leading to lower correctness scores than lexical level violations, and combined violations receiving the lowest scores. EEG data from 20 participants documented a strong N400 effect (350–550 msec) in response to combined violations, and a late posterior negativity (600–900 msec) present only for combined violations and focus-level violations. Consistent with the psychometric data, the EEG data suggest that prosodic violations at the focus level result in higher costs for comprehension than prosodic violations at the lexical level, whereas combined prosodic violations most significantly disrupt the interpretation. Taken together, these findings suggest that the language comprehension system is sensitive to accurate representations of both lexical and information structure prosody, and benefits from the interaction between them; however, they are weighted differently based on their relevance for a functioning spoken communication.

Abstract

In Swedish, words are associated with either of two pitch contours labelled as Accent 1 and Accent 2. At least one of them is taken to be phonologically and cognitively marked. Besides encoding lexical tonal distinctions, these accents reflect intonational prominence. Drawing on data from psychometric and electroencephalographic (EEG) measures, we scrutinized the functional load of the accents for the processing of linguistic input, and explored any potential processing differences between Accent 1 and Accent 2. Experimental stimuli consisted of one hundred sets of auditory dialogues, where test words were accented either appropriately or inappropriately within their respective contexts. Native speakers of Central Swedish were tasked with judging the correctness of sentences containing the test words, actively in the psychometric paradigm and passively in the EEG paradigm. Psychometric data from forty participants revealed that accent violations exerted a statistically significant negative impact on correctness judgements. Both Accent 1 and Accent 2 violations were deemed as incorrect by the listeners, indicating that listeners use both of them to arrive at the correct interpretation of the linguistic input. Moreover, there was a statistically significant difference in the perceived correctness of violations depending on the accent pattern. Accent 2 violations received a lower rating for correctness in comparison to Accent 1 violations, suggesting that listeners show more sensitivity to accent violations in Accent 2 words than in Accent 1 words. EEG data from twenty participants were in accordance with the psychometric data, and documented larger negative ERP responses, observed at both early and later latencies, to Accent 2 violations compared to Accent 1 violations, reflecting neurocognitive difficulty associated with the processing of linguistic input. Put differently, the application of wrong accent pattern for Accent 2 words resulted in higher costs for spoken communication than Accent 1 words, which is in line with the notion that Accent 2 is marked both phonologically and cognitively in Central Swedish. This pattern of results provides evidence that the brain not only extracts and utilizes pitch accents for a coherent interpretation of the linguistic input but also treats them differently depending on their phonological and cognitive markedness.

Abstract

When we communicate with others, we often repeat aspects of each other's communicative behavior such as sentence structures and words. Such behavioral alignment has been mostly studied for speech or text. Yet, language use is mostly multimodal, flexibly using speech and gestures to convey messages. Here, we explore the use of alignment in speech (words) and co-speech gestures (iconic gestures) in a referential communication task aimed at finding labels for novel objects in interaction. In particular, we investigate how people flexibly use lexical and gestural alignment to create shared labels for novel objects and whether alignment in speech and gesture are related over time. The present study shows that interlocutors establish shared labels multimodally, and alignment in words and iconic gestures are used throughout the interaction. We also show that the amount of lexical alignment positively associates with the amount of gestural alignment over time, suggesting a close relationship between alignment in the vocal and manual modalities.

Abstract

Recognizing the limitations of standard vision assessments in capturing the real-world capabilities of individuals with low vision, we investigated the potential of the Seguin Form Board Test (SFBT), a widely-used intelligence assessment employing a visuo-haptic shape-fitting task, as an estimator of vision's practical utility. We present findings from 23 children from India, who underwent treatment for congenital bilateral dense cataracts, and 21 control participants. To assess the development of functional visual ability, we conducted the SFBT and the standard measure of visual acuity, before and longitudinally after treatment. We observed a dissociation in the development of shape-fitting and visual acuity. Improvements of patients' shape-fitting preceded enhancements in their visual acuity after surgery and emerged even with acuity worse than that of control participants. Our findings highlight the importance of incorporating multi-modal and cognitive aspects into evaluations of visual proficiency in low-vision conditions, to better reflect vision's impact on daily activities.

Abstract

Background
Language is multimodal and situated in rich visual contexts. Language is also incremental, unfolding moment-to-moment in real time, yet few studies have examined how spoken language interacts with gesture and visual context during multimodal language processing. Gesture is a rich communication cue that is integrally related to speech and often depicts concrete referents from the visual world. Using eye-tracking in an adapted visual world paradigm, we examined how participants with and without moderate-severe traumatic brain injury (TBI) use gesture to resolve temporary referential ambiguity.

Methods
Participants viewed a screen with four objects and one video. The speaker in the video produced sentences (e.g., “The girl will eat the very good sandwich”), paired with either a meaningful gesture (e.g., sandwich-holding gesture) or a meaningless grooming movement (e.g., arm scratch) at the verb “will eat.” We measured participants’ gaze to the target object (e.g., sandwich), a semantic competitor (e.g., apple), and two unrelated distractors (e.g., piano, guitar) during the critical window between movement onset in the gesture modality and onset of the spoken referent in speech.

Results
Both participants with and without TBI were more likely to fixate the target when the speaker produced a gesture compared to a grooming movement; however, relative to non-injured participants, the effect was significantly attenuated in the TBI group.

Discussion
We demonstrated evidence of reduced speech-gesture integration in participants with TBI relative to non-injured peers. This study advances our understanding of the communicative abilities of adults with TBI and could lead to a more mechanistic account of the communication difficulties adults with TBI experience in rich communication contexts that require the processing and integration of multiple co-occurring cues. This work has the potential to increase the ecological validity of language assessment and provide insights into the cognitive and neural mechanisms that support multimodal language processing.

Abstract

Developing efficient electromagnetic interference shielding materials has become significantly important in present times. This paper reports a series of (Ba1-xCax)(Ti0.9Sn0.1)O3 (BCTS) ((x =0, 0.01, 0.05, & 0.1)ceramics synthesized by conventional method which were studied for electromagnetic interference shielding (EMI) applications in X-band (8-12.4 GHz). EMI shielding properties and all S parameters (S11 & S12) of BCTS ceramic pellets were measured in the frequency range (8-12.4 GHz) using a Vector Network Analyser (VNA). The BCTS ceramic pellets for x = 0.05 showed maximum total effective shielding of 46 dB indicating good shielding behaviour for high-frequency applications. However, the development of lead-free ceramics with different concentrations usually requires iterative experiments resulting in, longer development cycles and higher costs. To address this, we used a machine learning (ML) strategy to predict the EMI shielding for different concentrations and experimentally verify the concentration predicted to give the best EMI shielding. The ML model predicted BCTS ceramics with concentration (x = 0.06, 0.07, 0.08, and 0.09) to have higher shielding values. On experimental verification, a shielding value of 58 dB was obtained for x = 0.08, which was significantly higher than what was obtained experimentally before applying the ML approach. Our results show the potential of using ML in accelerating the process of optimal material development, reducing the need for repeated experimental measures significantly.

Abstract

Studies of the impact of brain injury on memory processes often focus on the quantity and episodic richness of those recollections. Here, we argue that the organization of one's recollections offers critical insights into the impact of brain injury on functional memory. It is well-established in studies of word list memory that free recall of unrelated words exhibits a clear temporal organization. This temporal contiguity effect refers to the fact that the order in which word lists are recalled reflects the original presentation order. Little is known, however, about the organization of recall for semantically rich materials, nor how recall organization is impacted by hippocampal damage and memory impairment. The present research is the first study, to our knowledge, of temporal organization in semantically rich narratives in three groups: (1) Adults with bilateral hippocampal damage and severe declarative memory impairment, (2) adults with bilateral ventromedial prefrontal cortex (vmPFC) damage and no memory impairment, and (3) demographically matched non-brain-injured comparison participants. We find that although the narrative recall of adults with bilateral hippocampal damage reflected the temporal order in which those narratives were experienced above chance levels, their temporal contiguity effect was significantly attenuated relative to comparison groups. In contrast, individuals with vmPFC damage did not differ from non-brain-injured comparison participants in temporal contiguity. This pattern of group differences yields insights into the cognitive and neural systems that support the use of temporal organization in recall. These data provide evidence that the retrieval of temporal context in narrative recall is hippocampal-dependent, whereas damage to the vmPFC does not impair the temporal organization of narrative recall. This evidence of limited but demonstrable organization of memory in participants with hippocampal damage and amnesia speaks to the power of narrative structures in supporting meaningfully organized recall despite memory impairment.

Abstract

Purpose:
Traumatic brain injury (TBI) is a leading cause of death and disability among adults in the United States. There is evidence to suggest the peripheral vestibular system is vulnerable to damage in individuals with TBI. However, there are limited prospective studies that describe the type and frequency of vestibular impairment in individuals with chronic moderate–severe TBI (> 6 months postinjury).

Method:
Cervical and ocular vestibular evoked myogenic potentials (VEMPs) and video head impulse test (vHIT) were used to assess the function of otolith organ and horizontal semicircular canal (hSCC) pathways in adults with chronic moderate–severe TBI and in noninjured comparison (NC) participants. Self-report questionnaires were administered to participants with TBI to determine prevalence of vestibular symptoms and quality of life associated with those symptoms.

Results:
Chronic moderate–severe TBI was associated with a greater degree of impairment in otolith organ, rather than hSCC, pathways. About 63% of participants with TBI had abnormal VEMP responses, compared to only ~10% with abnormal vHIT responses. The NC group had significantly less abnormal VEMP responses (~7%), while none of the NC participants had abnormal vHIT responses. As many as 80% of participants with TBI reported vestibular symptoms, and up to 36% reported that these symptoms negatively affected their quality of life.

Conclusions:
Adults with TBI reported vestibular symptoms and decreased quality of life related to those symptoms and had objective evidence of peripheral vestibular impairment. Vestibular testing for adults with chronic TBI who report persistent dizziness and imbalance may serve as a guide for treatment and rehabilitation in these individuals.

Abstract

Conversation requires a substantial amount of coordination between dialogue participants, from managing turn taking to negotiating mutual understanding. Part of this coordination effort surfaces as the reuse of linguistic behaviour across speakers, a process often referred to as alignment. While the presence of linguistic alignment is well documented in the literature, several questions remain open, including the extent to which patterns of reuse across speakers have an impact on the emergence of labelling conventions for novel referents. In this study, we put forward a methodology for automatically detecting shared lemmatised constructions---expressions with a common lexical core used by both speakers within a dialogue---and apply it to a referential communication corpus where participants aim to identify novel objects for which no established labels exist. Our analyses uncover the usage patterns of shared constructions in interaction and reveal that features such as their frequency and the amount of different constructions used for a referent are associated with the degree of object labelling convergence the participants exhibit after social interaction. More generally, the present study shows that automatically detected shared constructions offer a useful level of analysis to investigate the dynamics of reference negotiation in dialogue.

Abstract

In face-to-face dialogues, the form-meaning relationship of co-speech gestures varies depending on contextual factors such as what the gestures refer to and the individual characteristics of speakers. These factors make co-speech gesture representation learning challenging. How can we learn meaningful gestures representations considering gestures’ variability and relationship with speech? This paper tackles this challenge by employing self-supervised contrastive learning techniques to learn gesture representations from skeletal and speech information. We propose an approach that includes both unimodal and multimodal pre-training to ground gesture representations in co-occurring speech. For training, we utilize a face-to-face dialogue dataset rich with representational iconic gestures. We conduct thorough intrinsic evaluations of the learned representations through comparison with human-annotated pairwise gesture similarity. Moreover, we perform a diagnostic probing analysis to assess the possibility of recovering interpretable gesture features from the learned representations. Our results show a significant positive correlation with human-annotated gesture similarity and reveal that the similarity between the learned representations is consistent with well-motivated patterns related to the dynamics of dialogue interaction. Moreover, our findings demonstrate that several features concerning the form of gestures can be recovered from the latent representations. Overall, this study shows that multimodal contrastive learning is a promising approach for learning gesture representations, which opens the door to using such representations in larger-scale gesture analysis studies.

Abstract

Gestures are integral components of face-to-face communication. They unfold over time, often following predictable movement phases of preparation, stroke, and re-
traction. Yet, the prevalent approach to automatic gesture detection treats the problem as binary classification, classifying a segment as either containing a gesture or not, thus failing to capture its inherently sequential and contextual nature. To address this, we introduce a novel framework that reframes the task as a multi-phase sequence labeling problem rather than binary classification. Our model processes sequences of skeletal movements over time windows, uses Transformer encoders to learn contextual embeddings, and leverages Conditional Random Fields to perform sequence labeling. We evaluate our proposal on a large dataset of diverse co-speech gestures in task-oriented face-to-face dialogues. The results consistently demonstrate that our method significantly outperforms strong baseline models in detecting gesture strokes. Furthermore, applying Transformer encoders to learn contextual embeddings from movement sequences substantially improves gesture unit detection. These results highlight our framework’s capacity to capture the fine-grained dynamics of co-speech gesture phases, paving the way for more nuanced and accurate gesture detection and analysis.

Abstract

Purpose:
Assessing fluency in aphasia is diagnostically important for determining aphasia type and severity and therapeutically important for determining appropriate treatment targets. However, wide variability in the measures and criteria used to assess fluency, as revealed by a recent survey of clinicians (Gordon & Clough, 2022), results in poor reliability. Furthermore, poor specificity in many fluency measures makes it difficult to identify the underlying impairments. Here, we introduce the Flu-ID Aphasia, an evidence-based tool that provides a more informative method of assessing fluency by capturing the range of behaviors that can affect the flow of speech in aphasia.

Method:
The development of the Flu-ID was based on prior evidence about factors underlying fluency (Clough & Gordon, 2020; Gordon & Clough, 2020) and clinical perceptions about the measurement of fluency (Gordon & Clough, 2022). Clinical utility is maximized by automated counting of fluency behaviors in an Excel template. Reliability is maximized by outlining thorough guidelines for transcription and coding. Eighteen narrative samples representing a range of fluency were coded independently by the authors to examine the Flu-ID's utility, reliability, and validity.

Results:
Overall reliability was very good, with point-to-point agreement of 86% between coders. Ten of the 12 dimensions showed good to excellent reliability. Validity analyses indicated that Flu-ID scores were similar to clinician ratings on some dimensions, but differed on others. Possible reasons and implications of the discrepancies are discussed, along with opportunities for improvement.

Conclusions:
The Flu-ID assesses fluency in aphasia using a consistent and comprehensive set of measures and semi-automated procedures to generate individual fluency profiles. The profiles generated in the current study illustrate how similar ratings of fluency can arise from different underlying impairments. Supplemental materials include an analysis template, extensive guidelines for transcription and coding, a completed sample, and a quick reference guide.

Abstract

Linguistic communication is an intrinsically social activity that enables us to share thoughts across minds. Many complex social uses of language can be captured by domain-general representations of other minds (i.e., mentalistic representations) that externally modulate linguistic meaning through Gricean reasoning. However, here we show that representations of others’ attention are embedded within language itself. Across ten languages, we show that demonstratives—basic grammatical words (e.g.,“this”/“that”) which are evolutionarily ancient, learned early in life, and documented in all known languages—are intrinsic attention tools. Beyond their spatial meanings, demonstratives encode both joint attention and the direction in which the listenermmust turn to establish it. Crucially, the frequency of the spatial and attentional uses of demonstratives varies across languages, suggesting that both spatial and mentalistic representations are part of their conventional meaning. Using computational modeling, we show that mentalistic representations of others’ attention are internally encoded in demonstratives, with their effect further boosted by Gricean reasoning. Yet, speakers are largely unaware of this, incorrectly reporting that they primarily capture spatial representations. Our findings show that representations of other people’s cognitive states (namely, their attention) are embedded in language and suggest that the most basic building blocks of the linguistic system crucially rely on social cognition.

Abstract

Indian Sign Language has limited resources for developing machine learning and data-driven approaches for automated language processing. Though text/audio-based language processing techniques have shown colossal research interest and tremendous improvements in the last few years, Sign Languages still need to catch up due to the need for more resources. To bridge this gap, in this work, we propose iSign: a benchmark for Indian Sign Language (ISL) Processing. We make three primary contributions to this work. First, we release one of the largest ISL-English datasets with more than video-sentence/phrase pairs. To the best of our knowledge, it is the largest sign language dataset available for ISL. Second, we propose multiple NLP-specific tasks (including SignVideo2Text, SignPose2Text, Text2Pose, Word Prediction, and Sign Semantics) and benchmark them with the baseline models for easier access to the research community. Third, we provide detailed insights into the proposed benchmarks with a few linguistic insights into the working of ISL. We streamline the evaluation of Sign Language processing, addressing the gaps in the NLP research community for Sign Languages. We release the dataset, tasks and models via the following website: https://exploration-lab.github.io/iSign/

Abstract

When learning a spoken second language (L2), words overlapping in form and meaning with one’s native language (L1) help break into the new language. When non-signing speakers learn a sign language as L2, such forms are absent because of the modality differences (L1:speech, L2:sign). In such cases, non-signing speakers might use iconic form-meaning mappings in signs or their own gestural experience as gateways into the to-be-acquired sign language. Here, we investigated how both these factors may contribute jointly to the acquisition of sign language vocabulary by hearing non-signers. Participants were presented with three types of sign in NGT (Sign Language of the Netherlands): arbitrary signs, iconic signs with high or low gesture overlap. Signs that were both iconic and highly overlapping with gestures boosted learning most at first exposure, and this effect remained the day after. Findings highlight the influence of modality-specific factors supporting the acquisition of a signed lexicon.

Abstract

Expressing Left-Right relations is challenging for speaking-children. Yet, this challenge was absent for signing-children, possibly due to iconicity in the visual-spatial modality of expression. We investigate whether there is also a modality advantage when speaking-children’s co-speech gestures are considered. Eight-year-old child and adult hearing monolingual Turkish speakers and deaf signers of Turkish-Sign-Language described pictures of objects in various spatial relations. Descriptions were coded for informativeness in speech, sign, and speech-gesture combinations for encoding Left-Right relations. The use of co-speech gestures increased the informativeness of speakers’ spatial expressions compared to speech-only. This pattern was more prominent for children than adults. However, signing-adults and children were more informative than child and adult speakers even when co-speech gestures were considered. Thus, both speaking- and signing-children benefit from iconic expressions in visual modality. Finally, in each modality, children were less informative than adults, pointing to the challenge of this spatial domain in development.

Abstract

Speakers tend to engage in adaptive behavior, known as entrainment, when they reuse their partner's linguistic representations, including lexical, acoustic prosodic, semantic, or syntactic structures during a conversation. Studies have explored the relationship between entrainment and social factors such as likeability, task success, and rapport. Still, limited research has investigated the relationship between entrainment and gaze. To address this gap, we conducted a within-subjects user study (N = 33) to test if gaze behavior of a robotic head affects entrainment of subjects toward the robot on four linguistic dimensions: lexical, syntactic, semantic, and acoustic-prosodic. Our results show that participants entrain more on lexical and acoustic-prosodic features when the robot exhibits well-timed gaze aversions similar to the ones observed in human gaze behavior, as compared to when the robot keeps staring at participants constantly. Our results support the predictions of the computers as social actors (CASA) model and suggest that implementing well-timed gaze aversion behavior in a robot can lead to speech entrainment in human-robot interactions.

Abstract

Somatic practices frequently use imagery, typically via verbal instructions, to scaffold sensorimotor organization and experience, a phenomenon we term “introjection”. We argue that introjection is an imagery practice in which sensorimotor and conceptual aspects are co-orchestrated, suggesting the necessity of crosstalk between somatics, phenomenology, psychology, embodied-enactive cognition, and linguistic research on embodied simulation. We presently focus on the scarcely addressed details of the process necessary to enact instructions of a literal or metaphoric nature through the body. Based on vignettes from dance, Feldenkrais, and Taichi practice, we describe introjection as a complex form of processual sense-making, in which context-interpretive, mental, attentional and physical sub-processes recursively braid. Our analysis focuses on how mental and body-related processes progressively align, inform and augment each other. This dialectic requires emphasis on the active body, which implies that uni-directional models (concept ⇒ body) are inadequate and should be replaced by interactionist alternatives (concept ⇔ body). Furthermore, we emphasize that both the source image itself and the body are specifically conceptualized for the context through constructive operations, and both evolve through their interplay. At this level introjection employs representational operations that are embedded in enactive dynamics of a fully situated person.