Publications

Abstract

Co-speech gestures emerge from the interaction between visuospatial experience and speech formulation, but what happens when visual experience is absent from birth? Blind individuals gesture in ways consistent with their native language, albeit they gesture less frequently. However, nothing is known about whether their gesture form differs from that of sighted individuals. Using computer vision technologies, this study examines whether gesture kinematics are shaped by visuospatial experience. In an auditory motion-description task, we analyzed spontaneous path gestures—depicting motion trajectories—produced by 20 congenitally blind, 21 blindfolded, and 21 sighted Turkish speakers. We assessed the gestures’ alignment with actual motion paths, along with duration, size, and speed. Blind participants produced gestures that were more precisely aligned with the motion trajectories, as well as larger and longer. These findings show that altered spatial cognition in blindness enhances gesture precision. The study underscores the role of sensorimotor experience in shaping gesture form.

Abstract

Dynamic time warping (DTW) is a well-known algorithm used to assess the similarity between signals of varying lengths. Initially developed for automatic speech recognition, DTW has found applications in psycholinguistics, particularly in analyzing gesture form similarity. An open question in this domain is how effectively DTW captures gesture form similarity. Here, we validate DTW against human annotations of gesture form similarity across two multimodal interaction corpora and explore its utility as an automatic, continuous measure of gesture form similarity. Our findings reveal weak to moderate correlations between DTW distance and the number of similar gesture features – such as handshape, movement, orientation, and position – suggesting that DTW serves as a useful proxy for gesture form similarity. Additionally, we highlight the importance of qualitative analysis of raw data and DTW predictions in enhancing DTW’s predictive accuracy. Our study offers a rigorous validation of DTW as a measure of gesture form similarity and presents a detailed framework for preprocessing motion tracking data and calculating DTW distance. While none of the methods is perfect, the combination of automatic and manual measures provides a comprehensive approach to understanding and measuring gesture form similarity.

Abstract

Rapid Invisible Frequency Tagging (RIFT) is a recent advance in frequency tagging that exploits novel, high-frequency displays to modulate luminance at imperceptibly high frequencies. RIFT goes beyond low-frequency tagging by allowing researchers to track neural responses to rhythmic stimulation while avoiding perceptual confounds. RIFT is thus a promising method to address central questions in rhythmic cognition, including attention, multimodal integration, and the neural mechanisms underlying oscillatory coordination in perception. However, setting up a RIFT study involves several technical and conceptual considerations. In an effort to make RIFT more accessible, we provide a comprehensive guide for implementing RIFT in cognitive neuroscience. On the basis of the joint experiences and data-driven insights of multiple laboratories, we provide practical recommendations derived from empirical datasets to improve reproducibility, covering hardware requirements, stimulus design, analysis approaches, and interpretation of results. We hope that this guide helps readers to both identify the conceptual areas where RIFT offers promising insights and navigate the technical caveats that come with the approach.

Abstract

Art can evoke strong emotional responses in humans. Here, we examine genetic contributions to chills, a marker of such responses. We gather self-reports from a genotyped sample of thousands of partly related individuals from the Netherlands (n = 15,606). Using genomic relationships based on common single-nucleotide polymorphism (SNP) data, we find that up to 29% of the variation in proneness to aesthetic (visual art and poetry) and music chills can be explained by familial relatedness effects, one-fourth of which is attributed to SNP variation. Furthermore, we reveal a moderate genetic correlation of .58 between aesthetic and music chills, pointing to shared genetic variation affecting susceptibility to strong emotional responses across different art forms. Finally, we find that a polygenic index (PGI) for openness to experience (n = 220,015) is associated with susceptibilities to both aesthetic and music chills. Our results show that additive genetic variation, but also familial relatedness beyond shared common SNPs, contributes to proneness to chills from artistic, poetic, and musical expressions. These results open up a promising path towards studying the human attitude towards art, via both state-of-the-art genomics and intergenerational models of transmission.

Abstract

Word frequency (WF) and name agreement (NA) affect a word’s accessibility during speech production. Speakers are faster to name pictures with high-frequency (e.g. dog) compared to low-frequency names (e.g. rhinoceros) and those that a group of speakers tend to agree on the name of (high NA; e.g. arm) than those that they do not (low NA; e.g. sofa, couch). Recent accounts of lexical access suggest that the structure of the mental lexicon is flexible and changes with exposure. Consistent with this view, repetition priming studies have shown that low-frequency and low NA items benefit from repetition more than high-frequency and high NA items. But there is little evidence that repetition has long-term effects on WF and NA. We tested this issue in a two-session (online) picture naming study. In Session 1, participants named pictures varying in WF and NA three times each, and so we could test the short-term effects of repetition on WF and NA. We tested long-term effects of repetition by having participants name the same old items 1 week later in Session 2, together with new items that they had not named previously. In Session 1 the WF effect was eliminated by repetition, while the NA effect was reduced but still present. Thus, previous naming affected both the WF and NA effects. However, both effects reappeared in Session 2. These findings suggest that previous naming can reduce the WF and NA effect, thus affecting how easy it is to produce a word, but these effects are relatively short-lived.

Abstract

The emergence of human language is a complex process, and to investigate the role of iconicity in this, researchers have combined insights from computational models with empirical observations from laboratory experiments. This chapter provides an overview of the most important insights on the interaction between iconicity and other linguistic properties such as combinatoriality and systematicity. In the experimental and computational work reported, it is shown how iconicity can affect the way in which emerging languages are learned and used. The chapter also discusses how computational methods can help to better understand the gradient and subjective nature of iconicity.

Abstract

Multimodal language models (MLMs) increasingly demonstrate human-like communication, yet their use of everyday perspectival words remains poorly understood. To address this gap, we compare humans and MLMs in their use of three word types, which we predict impose increasing cognitive demands: vocabulary (e.g., ’boat’ or ’cup’), possessives (e.g., ’mine’ vs. ’yours’), and demonstratives (e.g., ’this one’ vs. ’that one’). Testing seven MLMs against human participants, we find that perspectival words are harder than vocabulary words for both groups. The gap is even larger for MLMs: while models approach human-level performance on using vocabulary, they exhibit clear deficits with possessives and even greater difficulties with demonstratives. Ablation analyses point to limitations in perspective-taking and spatial reasoning as key sources of these gaps in MLMs. Instruction-based prompting helps close the gap for possessives but still leaves demonstratives far below human performance. These results show that, unlike vocabulary, perspectival words pose a greater challenge in human communication—and this difficulty is further amplified in MLMs, revealing a crucial shortfall in their pragmatic and social-cognitive abilities.

Abstract

We examined the morphosyntactic prediction ability of child heritage speakers and the role of reading skills and language experience in predictive processing. Using visual world eye-tracking, we focused on predictive use of case-marking cues in Turkish with monolingual (N=49, Mage=83 months) and heritage children, who were early bilinguals of Turkish and Dutch (N=30, Mage=90 months). We found quantitative differences in magnitude of the prediction ability of monolingual and heritage children; however, their overall prediction ability was on par. The heritage speakers’ prediction ability was facilitated by their reading skills in Dutch, but not in Turkish as well as by their heritage language exposure, but not by engagement in literacy activities. These findings emphasize the facilitatory role of reading skills and spoken language experience in predictive processing. This study is the first to show that in a developing bilingual mind, effects of reading-on-prediction can take place across modalities and across languages.

Abstract

Background

The factors that contribute to language alternation by bilingual speakers with aphasia have been debated. Some studies suggest that atypical language mixing results from impairments in language control, while others posit that mixing is a way to enhance communicative effectiveness. To address this question, most prior research examined the appropriateness of language mixing in connected speech tasks.
Aims

The goal of this study was to provide new insight into the question whether language mixing in aphasia reflects a strategy to enhance verbal effectiveness or involuntary behaviour resulting from impaired language control.
Methods & procedures

Semi-structured web-based interviews with bilingual speakers with aphasia (N = 19) with varying language backgrounds were conducted. The interviews were transcribed and coded for: (1) Self-reports regarding language control and compensation, (2) instances of language mixing, and (3) in two cases, instances of repair initiation.
Outcomes & results

The results showed that several participants reported language control difficulties but that the knowledge of additional languages could also be recruited to compensate for lexical retrieval problems. Most participants showed no or very few instances of mixing and the observed mixes appeared to adhere to the pragmatic context and known functions of switching. Three participants exhibited more marked switching behaviour and reported corresponding difficulties with language control. Instances of atypical mixing did not coincide with clear problems initiating conversational repair.
Conclusions

Our study highlights the variability in language mixing patterns of bilingual speakers with aphasia. Furthermore, most of the individuals in the study appeared to be able to effectively control their languages, and to alternate between their languages for compensatory purposes. Control deficits resulting in atypical language mixing were observed in a small number of participants.

Abstract

Speakers sometimes produce lexical errors, such as saying “salt” instead of “pepper.” This study aimed to better understand the origin of lexical errors by assessing whether they arise from a hasty selection and premature decision to speak (premature selection hypothesis) or from momentary attentional disengagement from the task (attentional lapse hypothesis). We analyzed data from a speeded picture naming task (Lampe et al., 2023) and investigated whether lexical errors are produced as fast as target (i.e., correct) responses, thus arising from premature selection, or whether they are produced more slowly than target responses, thus arising from lapses of attention. Using ex-Gaussian analyses, we found that lexical errors were slower than targets in the tail, but not in the normal part of the response time distribution, with the tail effect primarily resulting from errors that were not coordinates, that is, members of the target’s semantic category. Moreover, we compared the coordinate errors and target responses in terms of their word-intrinsic properties and found that they were overall more frequent, shorter, and acquired earlier than targets. Given the present findings, we conclude that coordinate errors occur due to a premature selection but in the context of intact attentional control, following the same lexical constraints as targets, while other errors, given the variability in their nature, may vary in their origin, with one potential source being lapses of attention.

Abstract

Vocal individuality has important biological functions in mammals: at crucial stages of development, it ensures feeding and is a prerequisite for auditory-based mother–pup recognition. Is vocal individuality only shaped by maturation or also by the degree of conspecific acoustic input? Here, we test how the neonatal auditory environment shapes development and individualization of calls in harbour seal pups (Phoca vitulina). To simulate low- versus high-conspecific acoustic density, 18 pups heard playbacks of calls from either 2 or 30 conspecifics. We recorded calls before and after playback exposure and extracted 12 acoustic parameters. Supervised machine learning and discriminant function analyses showed greater individual distinctiveness in both groups after playback, indicating a developmental trend toward individualization. Notably, pups exposed to less-variable input showed higher individuality. Euclidean distances on call parameters showed that both groups diverged from the playback signals. Distances on within-pup and between-pups housed together revealed opposite trajectories in the two groups: after the exposure, less-variable auditory input determined steadier individual calls, while more-variable auditory input scattered calls across the acoustic space. Altogether, our findings indicate that auditory input modulates vocal development in pups, making harbour seals a promising model for unravelling how neonatal environment affects vocal plasticity in a non-human mammal.

Abstract

Previous studies have evaluated the role of the microRNA-137 (miRNA137) regulatory pathway in schizophrenia by using in silico or in vitro predicted target genes. These approaches do not capture the dynamic spatiotemporal nature of the miRNA137 regulatory pathway or tend to overestimate direct miRNA binding sites. To provide a more accurate representation of the miRNA137 pathway during human brain development, we evaluated the biological functionality of direct ex vivo miRNA137 targets previously documented in the early prenatal and adult brain. We studied the role of differential expression and genetic predisposing variation to schizophrenia and related disorders within these miRNA137 targets by gene set enrichment analyses (GSEA) and gene-set based polygenic score predictions in an independent schizophrenia case-control cohort. All results were compared to those from in silico or in vitro predicted targets. Only direct miRNA137 targets at prenatal and adult human brain displayed significant enrichments in synaptic and neuronal functions and captured the transcriptomic and genetic predisposing variation to schizophrenia. Furthermore, while adult target genes showed a more consistent enrichment in predisposing variation to schizophrenia and bipolar disorder in case/control polygenic models, prenatal target genes predicted negative symptomatology in schizophrenia patients. Our results suggest that using direct, temporally specific miRNA137 targets significantly improved the detection of biological mechanisms underlying its relationship with psychosis, clarified the association between schizophrenia and related conditions, and suggested association with specific symptomatology domains at different developmental stages of the disorder.

Abstract

The nature of the link between speech perception and production is extensively debated. One question in this debate concerns the possible use of shared representations. Talker representations encode information about particular talkers’ speech and are formed through perception. We present three experiments that investigate the potential role of talker representations in speech production as measured by their impact on phonetic convergence to voice fundamental frequency (F0). In Experiment 1, female native Dutch speakers (N = 32) performed a baseline reading task, followed by a synchronous speech task where they were instructed to synchronize (temporally) to a pre-recorded model talker. The model talker’s voice was pitch-shifted to have high or low F0, and half of the sample performed the task in each F0 condition. Participants’ F0 values converged toward that of the model talker. In Experiment 2, we added an exposure task before synchronous speech, during which participants (N = 32) were familiarized with the model talker at high or low F0. At test, they heard the model talker at the familiar F0. Finally, in Experiment 3, we reversed the F0 manipulation between exposure and test. Thus, participants (N = 32) were familiarized with either high or low F0, but heard the opposite F0 condition during synchronous speech. The data revealed that while congruent and conflicting talker information both diminish convergence magnitude, only conflicting talker information reduces the probability of convergence. These results indicate that talker information acquired exclusively through perception is used during production, suggesting shared representations across perception and production.

Abstract

Background

Mastering gross motor abilities in early infancy and culturally defined actions (e.g. self-care routines) in late infancy can initiate cascading developmental changes that affect language learning. Here, we adopt a genetic perspective to investigate underlying processes, implicating either shared or “gateway” mechanisms, where the latter enable children to interact with their environment.
Methods

Selecting heritable traits (h2, heritability), we studied infant gross motor (6 months) and self-care/symbolic (15 months) skills as predictors of 10 language outcomes (15–38 months) in genotyped children from the Avon Longitudinal Study of Parents and Children (N ≤ 7,017). Language measures were combined into three interrelated language factors (LF) using structural equation modeling (SEM), corresponding to largely different age windows (LF15M, LF24M, LF38M, 51.3% total explained variance). Developmental genomic and non-genomic relationships across measures were dissected with Cholesky decompositions using genetic-relationship-matrix structural equation modeling (GRM-SEM) as part of a multivariate approach.
Results

Gross motor abilities at 6 months (h2 = 0.18 (SE = .06)) and self-care/symbolic actions at 15 months (h2 = 0.18 (SE = .06)) were modestly heritable, as well as the three derived language factor scores (LFS15M-h2 = 0.12 (SE = .05), LFS24M-h2 = 0.21 (SE = .06), LFS38M-h2 = 0.17 (SE = .05)), enabling genetic analyses. Developmental genetic models (GRM-SEM) showed that gross motor abilities (6 months) share genetic influences with self-care/symbolic actions (15 months, factor loading λ; λ = 0.22 (SE = .09)), but not with language performance (p ≥ .05). In contrast, genetic influences underlying self-care/symbolic actions, independent of early gross motor skills, were related to all three language factors (LFS15M-λ = 0.26 (SE = .09), LFS24M-λ = 0.28 (SE = .10), LFS38M-λ = 0.30 (SE = .10)). Multivariate models studying individual language outcomes provided consistent results, both for genomic and non-genomic influences.
Conclusions

Genetically encoded processes linking gross motor behaviour in young infants to self-care/symbolic actions in older infants are different from those linking self-care/symbolic actions to emerging language abilities. These findings are consistent with a developmental cascade where motor control enables children to engage in novel social interactions, but children's social learning abilities foster language development.

Abstract

Rhythm and language-related traits are phenotypically correlated, but their genetic overlap is largely unknown. Here, we leveraged two large-scale genome-wide association studies performed to shed light on the shared genetics of rhythm (N=606,825) and dyslexia (N=1,138,870). Our results reveal an intricate shared genetic and neurobiological architecture, and lay groundwork for resolving longstanding debates about the potential co-evolution of human language and musical traits.

Abstract

Humans acquire word knowledge through producing and comprehending spoken and written language. Word learning continues into adulthood and knowledge accumulates across the lifespan. Therefore, receptive vocabulary size is often conceived of as a proxy for linguistic experience and plays a central role in assessing individuals’ language proficiency. There is currently no valid open access test available for assessing receptive vocabulary size in German-speaking adults. We addressed this gap and developed the German Auditory and Image Vocabulary Test (GAudI). In the GAudI, participants are presented with spoken test words and have to indicate their meanings by selecting the corresponding picture from a set of four alternatives. Here we describe the development of the test and provide evidence for its validity. Specifically, we report a study in which 168 German-speaking participants completed the GAudI and five other tests tapping into linguistic experience: one test measuring print exposure, two tests measuring productive vocabulary, one test assessing knowledge of book language grammar, and a test of receptive vocabulary that was normed in adolescents. The psychometric properties of the GAudI and its relationships to the other tests demonstrate that it is a suitable tool for measuring receptive vocabulary size. We offer an open-access digital test environment that can be used for research purposes, accessible via https://ems13.mpi.nl/bq4_customizable_de/researchers_welcome.php.

Abstract

Individuals vary substantially in their language skills. The Individual Differences in Language Skills Test Battery (IDLaS) is a tool to assess variability in (1) linguistic experience, (2) general cognitive skills implicated in language, including nonverbal processing speed, working memory, and nonverbal reasoning, and (3) linguistic processing skills, including word- and sentence-level production and comprehension. The test battery was initially developed for Dutch language users. Building on this work, we recently developed a German version (IDLaS-DE). IDLaS-DE consists of 30 behavioral tests that have been validated in a large group of German speakers, aged between 18 and 30 years. In addition, we have developed a web platform that researchers interested in assessing language and general cognitive skills can use for their research purposes. Here, we provide a guide for creating and running customized studies online via this platform. The IDLaS-DE web platform and all its services are free of charge and accessible at https://www.mpi.nl/idlas-de.

Abstract

Human communication is inherently multimodal. Auditory speech, but also visual cues can be used to understand another talker. Most studies of audiovisual speech perception have focused on the perception of speech segments (i.e., speech sounds). However, less is known about the influence of visual information on the perception of suprasegmental aspects of speech like lexical stress. In two experiments, we investigated the influence of different visual cues (e.g., facial articulatory cues and beat gestures) on the audiovisual perception of lexical stress. We presented auditory lexical stress continua of disyllabic Dutch stress pairs together with videos of a speaker producing stress on the first or second syllable (e.g., articulating VOORnaam or voorNAAM). Moreover, we combined and fully crossed the face of the speaker producing lexical stress on either syllable with a gesturing body producing a beat gesture on either the first or second syllable. Results showed that people successfully used visual articulatory cues to stress in muted videos. However, in audiovisual conditions, we were not able to find an effect of visual articulatory cues. In contrast, we found that the temporal alignment of beat gestures with speech robustly influenced participants' perception of lexical stress. These results highlight the importance of considering suprasegmental aspects of language in multimodal contexts.

Abstract

Speech is inherently variable, requiring listeners to apply adaptation mechanisms to deal with the variability. A proposed perceptual adaptation mechanism is recalibration, whereby listeners learn to adjust cognitive representations of speech sounds based on disambiguating contextual information. Most studies on the role of recalibration in speech perception have focused on variability in particular speech segments (e.g., consonants/vowels), and speech has mostly been studied with a focus on talking heads. However, speech is often accompanied by visual bodily signals like hand gestures, and is thus multimodal. Moreover, variability in speech extends beyond segmental aspects alone and also affects prosodic aspects, like lexical stress. We currently do not understand well how listeners adjust their representations of lexical stress patterns to different speakers. In four experiments, we investigated recalibration of lexical stress perception, driven by lexico-orthographical information (Experiment 1) and by manual beat gestures (Experiments 2–4). Across experiments, we observed that these two types of disambiguating information (presented in an audiovisual exposure phase) led listeners to adjust their representations of lexical stress, with lasting consequences for subsequent spoken word recognition (in an audio-only test phase). However, evidence for generalization of this recalibration to new words was only found in the third experiment, suggesting that generalization may be limited. These results highlight that recalibration is a plausible mechanism for suprasegmental speech adaption in everyday communication and show that even the timing of simple hand gestures can have a lasting effect on auditory speech perception.

Abstract

Speech comprehension involves more than just identifying speech sounds. It also requires the use of prosodic cues, which can be conveyed auditorily (e.g., intonation), but also visually (e.g., prominence-lending beat gestures). Prior studies on unimodal speech emphasize the critical role of prosody in comprehension, in higher level processing (e.g., pragmatics), and even in lexical access. For instance, people are faster at identifying a word when it is prosodically accented than when it is unaccented. This study tested whether beat gestures, serving as visual prominence cues, can similarly aid lexical access even in situations where other cues are already highly supportive of word recognition (e.g., semantically constraining sentences). Moreover, we investigated if this facilitation effect would be modulated by the (mis)alignment of the beat gesture with the word-internal prominence (i.e., stressed syllables). To answer this question, we presented participants with videos of a talker producing semantically constraining sentences containing a critical disyllabic sentence-final target in a lexical decision task. The target was either produced without a gesture or accompanied by a beat gesture aligned to the stressed or unstressed syllable. Response times showed that participants were generally faster when the target was presented together with a beat gesture, regardless of its within-word alignment. Moreover, we found that this facilitatory effect was larger for words than pseudowords. These results provide evidence that beat gestures—even when they are not essential for successful speech comprehension—affect lexical access in highly constraining contexts.

Abstract

This study evaluates a recent account of islands and parasitic gaps, which proposes that island violations are unacceptable in part because they contain a referring argument in the predicate that contributes discourse-processing complexity. In two acceptability judgment experiments in Dutch, participants rated three types of filler-gap constructions that were preceded by a context manipulating the discourse accessibility of a referring argument in the target construction. The constructions differed in the location of the gap, which was realized as the complement of the matrix verb (regular filler-gap dependencies), as the complement of the verb in an adjunct clause (adjunct island violations), or as both (parasitic gap constructions). Adjunct clauses were untensed in Experiment 1 and tensed in Experiment 2. In both experiments, island violations were rated as unacceptable, regardless of whether the referring argument was discourse-accessible or discourse-novel. Parasitic gap constructions, which do not contain a referring argument in the predicate, were rated as acceptable, but only when the parasitic gap was located in an untensed clause. Reviewing these results from syntactic and discourse-processing perspectives, we conclude that the difference between islands and parasitic gap constructions is not a matter of discourse-processing complexity. The data instead support a primarily syntactic account of parasitic gaps.

Abstract

Studies of perception have long shown that the brain adds information to its sensory analysis of the physical environment. A touchstone example for humans is language use: to comprehend a physical signal like speech, the brain must add linguistic knowledge, including syntax. Yet, syntactic rules and representations are widely assumed to be atemporal (i.e., abstract and not bound by time), so they must be translated into time-varying signals for speech comprehension and production. Here, we test 3 different models of the temporal spell-out of syntactic structure against brain activity of people listening to Dutch stories: an integratory bottom-up parser, a predictive top-down parser, and a mildly predictive left-corner parser. These models build exactly the same structure but differ in when syntactic information is added by the brain—this difference is captured in the (temporal distribution of the) complexity metric “incremental node count.” Using temporal response function models with both acoustic and information-theoretic control predictors, node counts were regressed against source-reconstructed delta-band activity acquired with magnetoencephalography. Neural dynamics in left frontal and temporal regions most strongly reflect node counts derived by the top-down method, which postulates syntax early in time, suggesting that predictive structure building is an important component of Dutch sentence comprehension. The absence of strong effects of the left-corner model further suggests that its mildly predictive strategy does not represent Dutch language comprehension well, in contrast to what has been found for English. Understanding when the brain projects its knowledge of syntax onto speech, and whether this is done in language-specific ways, will inform and constrain the development of mechanistic models of syntactic structure building in the brain.

Abstract

Speech error monitoring recruits the medial frontal cortex (MFC) region in the human brain. Error monitoring-related activity in the MFC has been interpreted both in terms of conflict monitoring and feedback-driven control, but as similar regions of the MFC are implicated in various levels of behavioral control ranging from basic motor movement control to high-level cognitive control functions, a more comprehensive account is needed. Moreover, as speech errors and other actions that involve varying control demands engage a widespread yet partially overlapping set of regions of the MFC, such an account should ideally explain the anatomical distribution of error-related functional activations within the MFC. Here we wanted to assess the hypothesis that the MFC has a similar role in the evaluation of action outcomes for motor and mental actions, operating along a rostral-caudal gradient of higher-lower degree of cognitive control demands involving prediction errors from both sensory and epistemic sources. To this end, we conducted an individual-specific annotation of task-fMRI BOLD activation peaks related to overt speech error monitoring (i.e. that involve the largest degree of cognitive control demands, Study I and II), tongue movement monitoring (i.e. that involve an intermediate degree of cognitive control demands) and tongue movement (i.e. that involve the lowest degree of cognitive control demands, Study II) in the MFC region. Results revealed overlapping clusters across the three contrasts across the MFC, but importantly both the number of peaks and their relative position along the rostral caudal axis were consistent with a hierarchical rostral caudal processing gradient in the MFC. While tongue movement showed more caudal activation in the MFC, overt speech error monitoring showed more rostral activation, and tongue movement monitoring patterned in between. Furthermore, the combined results of both studies suggested that activation peaks were located more dorsally for participants that had a paracingulate gyrus, replicating a previously documented effect for movement and further supporting a common functional role of the MFC across very distinct actions.

Abstract

The duration of animal vocalizations varies between and within species. Which mammals can learn to control this duration? Such respiratory production learning is a scarcely studied subcomponent of vocal learning. Here, we test the hypothesis that harbor seals (Phoca vitulina) are capable of respiratory production learning by testing whether a harbor seal can be trained to i) actively control its vocalization’s duration in two directions (short and long), and ii) exceed the pre-experimental vocalization’s duration (min = 0.202 s, max = 2.621 s). The seal learned to produce uninterrupted vocalizations spanning more than two orders of magnitude in duration, from 79 ms to 9.23 s. Our findings demonstrate a remarkable level of respiratory control in a harbor seal: this respiratory production learning encompasses an extensive range of sound durations and arises at a young age. Producing durations that span such a magnitude is hardly reported in the non-human animal literature; this capacity may be orthogonal to other vocal learning modules and should be tested in more species, both vocal production learners and non-learners.

Abstract

This chapter introduces the origins and development of our current anatomical terminology. It scrutinizes the historical evolution and etymological significance of the over 1900 official anatomical terms in the current nomenclature, underscoring their impact on the contemporary comprehension of cognitive neuroanatomy. The chapter traces unification efforts from the Basel Nomina Anatomica in 1895 to the 1998 Terminologia Anatomica, noting challenges arising from outdated terminology in light of recent anatomical advancements.

Highlighting the influence of terminologies on interpretations of brain anatomy, the chapter explores several anatomical mapping methods such as surface, sectional, connectional, and functional anatomy. It illuminates discrepancies and controversies, exemplified by divergent interpretations of the number of brain lobes and the definitions of 'Broca' and 'Wernicke' areas.

The chapter explores anatomical terms' historical and cultural underpinnings, encompassing mythonyms, eponyms, and cultural influences on nomenclature. It critically examines the implications of these terminologies on contemporary research and shows that Large Language Models mirror these discrepancies. It underscores the need for more inclusive and culturally sensitive approaches in anatomical education.

Lastly, we advocate for updating anatomical nomenclature, suggesting that a deeper understanding of these terminologies could provide insights and aid in resolving ongoing debates in the field. This examination sheds light on historical knowledge and emphasizes the dynamic interplay between language, culture, and anatomy in shaping our comprehension of the neurobiology of the brain and how we navigate neuroanatomy in the 21st century.

Abstract

Infant-Directed Speech (IDS) and Adult-Directed Speech (ADS) are two registers that can differ across multiple linguistic domains and social contexts. In languages where the phonemes’ acoustic clarity is modified, there is a typical assumption in the literature that these phonemes are either hyper- or hypo-articulated. These modifications have received considerable attention in language research, serving as the basis of proposals concerning the potential didactic and communicative (e.g., affective) functions of IDS. The current study adds to this literature by examining how vowels and consonants are modified in IDS in a previously unstudied African language with an unusual phoneme inventory: Tashlhiyt Berber. Seven caregivers were recorded interacting with: (a) infants between the ages of 0;6 and 1;5, and (b) an adult experimenter who is a native speaker of Tashlhiyt. In IDS, we found longer vowel duration for content words and exaggerated pitch levels across the board, successfully replicating cross-linguistic patterns. Counter to the hyper-articulation hypothesis, there was more vowel overlap in IDS, while no differences were observed between IDS and ADS for overall vowel space size. We additionally found that speakers tend to completely neutralize certain consonant contrasts in IDS, rendering them indistinguishable to the infant. Our results are most consistent with the proposal that caregivers use IDS to guide and maintain their infant’s attention and to convey positive affect.

Abstract

When language abilities in aphasia are assessed in clinical and research settings, the standard practice is to examine each language of a multilingual person separately. But many multilingual individuals, with and without aphasia, mix their languages regularly when they communicate with other speakers who share their languages. We applied a novel approach to scoring language production of a multilingual person with aphasia. Our aim was to discover whether the assessment outcome would differ meaningfully when we count accurate responses in only the target language of the assessment session versus when we apply a translanguaging framework, that is, count all accurate responses, regardless of the language in which they were produced. The participant is a Farsi-German-English speaking woman with chronic moderate aphasia. We examined the participant’s performance on two picture-naming tasks, an answering wh-question task, and an elicited narrative task. The results demonstrated that scores in English, the participant’s third-learned and least-impaired language did not differ between the two scoring methods. Performance in German, the participant’s moderately impaired second language benefited from translanguaging-based scoring across the board. In Farsi, her weakest language post-CVA, the participant’s scores were higher under the translanguaging-based scoring approach in some but not all of the tasks. Our findings suggest that whether a translanguaging-based scoring makes a difference in the results obtained depends on relative language abilities and on pragmatic constraints, with additional influence of the linguistic distances between the languages in question.

Abstract

Early-life abilities involved in perceiving, producing and engaging with music (musicality) may shape later (social) communication and language abilities. Here, we investigate phenotypic and genetic relationships linking musicality and communication abilities by studying information from preschool and school-aged children of the Avon Longitudinal Study of Parents and Children (N = 4169–6737 per measure, age 0.5–17 years). Using structural models, we identified relationships between latent musicality and speech- and cognition-related variables (r > 0.30). Consistently, polygenic scores for rhythmicity in adulthood (PGSrhythmicity) showed associations with preschool and school-age musicality (incremental-Nagelkerke-R2 = 0.006-0.011, p < 0.0025), as well as school-age communication and cognition-related measures (incremental-R2 = 0.04–1%, p < 0.0025). Studying the directionality of genetic effects using a mediation framework, we found evidence supporting a developmental pathway linking preschool musicality to school-age speech-/syntax-related abilities, as captured by PGSrhythmicity (shared effect: β = 0.0051(SE = 0.0021), p = 0.015). Associations were found conditional on general cognition and genetically unrelated to educational attainment, suggesting robust developmental links between early musicality and later speech-related communication performance.

Abstract

In conversation, future speakers often plan speech simultaneously with comprehension, which means that they must divide attentional resources between these processes. In this EEG study, we used responses to linguistic attention probes (i.e., syllable “BA” presented during spoken sentences) to track temporal variations in attention to comprehension. Participants were asked to listen to prerecorded sentences with expected or unexpected sentence-final words. Each sentence was presented twice, once with and once without the attention probe starting 100 msec after the target word onset. Participants saw a picture 50 msec before the target word. Depending on the test block (picture naming or button press), participants either named the picture or pressed the space bar, both after an 850-msec delay. The probes elicited a negative potential approximately 100 msec after probe onset (i.e., an attention probe effect) in all probe conditions. Unexpectedly, neither word expectancy nor speech planning influenced the timing or strength of the attention probe effect. This indicates that expectancy of words in Dutch does not affect the allocation of attention toward these words 100 msec after their onset (i.e., the time of the probe presentation). Interestingly, engaging in speech planning does not seem to divert attentional resources away from comprehension at the moment of probe presentation. These findings imply that listeners are able to effectively distribute their attentional resources between comprehension and speech planning and carry out these processes at the same time. Considering these unexpected findings, using attention probes might not be the best approach to capture variations in temporal attention in dual-task paradigms.

Abstract

Experimental research has yielded many important psycholinguistic findings in bi-/multilingualism. However, cognitive computational models of sentence processing were limited to the single-language case until recently. In this methodological review, we discuss cognitive modelling work that uses the Bilingual Dual-path model to simulate experimental research on bilingual sentence production, processing, and development. As a detailed example of such work, we then report on new simulations conducted with the model. Finally, we suggest directions for future cognitive modelling research using this model.

Abstract

Sentence production is a stage-like process of mapping a conceptual representation to the linear speech signal via grammatical rules. While the typological diversity of languages is vast and thus must necessarily influence sentence production, psycholinguistic studies of diverse languages are comparatively rare. Here, we present data from a sentence planning and production study in Pitjantjatjara, an Australian Indigenous language that has highly flexible word order. Forty-nine (N = 49) native speakers described pictures of two-participant scenes while their eye-movements were recorded. Participants produced all possible orders of agent, patient, and verb. There was a general preference to produce agent-initial orders, but word order was influenced by the semantic properties of agent and patient referents (± human). Analyses of participants’ eye-movements revealed early relational encoding of the entire event, whereby speakers distributed their attention between agent and patient referents in a manner that is different than typically observed in languages that have more restricted word order options. Relational encoding was influenced by the word order that participants eventually produced. The results provide evidence to suggest that sentence planning in Pitjantjatjara is a hierarchical process, in which early relational encoding creates a wholistic conceptualization of an event, possibly driven by pressure to decide upon one of many possible word orders.

Abstract

Prediction is argued to be a key feature of human cognition, including in syntactic processing. Prediction error has been linked to dynamic changes in syntactic representations in theoretical models of language processing. This mechanism is termed error-based learning. Evidence from syntactic priming research supports error-based learning accounts; however, measuring prediction error itself has not been a research focus. Here we present a study exploring the use of pupillometry as a measure of prediction error during syntactic priming. We found a larger pupil response to the more complex and less expected passive structure. In addition, the pupil response predicted priming while being weakly dependent on changes in expectations over the experiment. We conclude that the pupil response is not only sensitive to syntactic complexity in comprehension, but there is some evidence that its magnitude is related to the adjustment of dynamic mental representations for syntax that lead to syntactic priming.

Abstract

Creating a virtual avatar with semantically coherent gestures that are aligned with speech is a challenging task. Existing gesture generation research mainly focused on generating rhythmic beat gestures, neglecting the semantic context of the gestures. In this paper, we propose a novel approach for semantic grounding in co-speech gesture generation that integrates semantic information at both fine-grained and global levels. Our approach starts with learning the motion prior through a vector-quantized variational autoencoder. Built on this model, a second-stage module is applied to automatically generate gestures from speech, text-based semantics and speaker identity that ensures consistency between the semantic relevance of generated gestures and co-occurring speech semantics through semantic coherence and relevance modules. Experimental results demonstrate that our approach enhances the realism and coherence of semantic gestures. Extensive experiments and user studies show that our method outperforms state-of-the-art approaches across two benchmarks in co-speech gesture generation in both objective and subjective metrics.

Abstract

Acknowledging that the parent–child attachment is a dyadic relationship, we investigated differences between pairs of parents and preschool children based on gender configurations in the association between attachment and problem behavior. We looked at mother–daughter, mother–son, father–daughter, and father–son dyads, but also compared mothers and fathers, daughters and sons, and same versus different gender pairs. We employed multigroup structural equation modeling to explore moderation effects of gender in a sample of 446 independent pairs of parents and preschool children (2–5 years old) from the Netherlands. A stronger association between both secure and avoidant attachment and internalizing problems was found for father–son dyads compared to father–daughter dyads. A stronger association between both secure and avoidant attachment and externalizing problems was found for mother–son dyads compared to mother–daughter and father–daughter dyads. Sons showed a stronger negative association between secure attachment and externalizing problems, a stronger positive association between avoidant attachment and externalizing problems, and a stronger negative association between secure attachment and internalizing problems compared to daughters. These results provide evidence for gender moderation and demonstrate that a dyadic approach can reveal patterns of associations that would not be recognized if parent and child gender effects were assessed separately.

Abstract

*All authors contributed equally to this work
Meaningful gestures enhance degraded speech comprehension in neurotypical adults, but it is unknown whether this is the case for neurodivergent populations, such as autistic individuals. Previous research demonstrated atypical multisensory and speech-gesture integration in autistic individuals, suggesting that integrating speech and gestures may be more challenging and less beneficial for speech comprehension in adverse listening conditions in comparison to neurotypicals. Conversely, autistic individuals could also benefit from additional cues to comprehend speech in noise, as they encounter difficulties in filtering relevant information from noise. We here investigated whether gestural enhancement of degraded speech comprehension differs for neurotypical (n = 40, mean age = 24.1) compared to autistic (n = 40, mean age = 26.8) adults. Participants watched videos of an actress uttering a Dutch action verb in clear or degraded speech accompanied with or without a gesture, and completed a free-recall task. Gestural enhancement was observed for both autistic and neurotypical individuals, and did not differ between groups. In contrast to previous literature, our results demonstrate that autistic individuals do benefit from gestures during degraded speech comprehension, similar to neurotypicals. These findings provide relevant insights to improve communication practices with autistic individuals and to develop new interventions for speech comprehension.

Abstract

Speech is often accompanied by gestures. Since beat gestures—simple nonreferential up-and-down hand movements—frequently co-occur with prosodic prominence, they can indicate stress in a word and hence influence spoken-word recognition. However, little is known about the reverse influence of auditory speech on visual perception. The current study investigated whether lexical stress has an effect on the perceived timing of hand beats. We used videos in which a disyllabic word, embedded in a carrier sentence (Experiment 1) or in isolation (Experiment 2), was coupled with an up-and-down hand beat, while varying their degrees of asynchrony. Results from Experiment 1, a novel beat timing estimation task, revealed that gestures were estimated to occur closer in time to the pitch peak in a stressed syllable than their actual timing, hence reducing the perceived temporal distance between gestures and stress by around 60%. Using a forced-choice task, Experiment 2 further demonstrated that listeners tended to perceive a gesture, falling midway between two syllables, on the syllable receiving stronger cues to stress than the other, and this auditory effect was greater when gestural timing was most ambiguous. Our findings suggest that f0 and intensity are the driving force behind the temporal attraction effect of stress on perceived gestural timing. This study provides new evidence for auditory influences on visual perception, supporting bidirectionality in audiovisual interaction between speech-related signals that occur in everyday face-to-face communication.

Abstract

This paper explores image modeling from the frequency space and introduces DCTdiff, an end-to-end diffusion generative paradigm that efficiently models images in the discrete cosine transform (DCT) space. We investigate the design space of DCTdiff and reveal the key design factors. Experiments on different frameworks (UViT, DiT), generation tasks, and various diffusion samplers demonstrate that DCTdiff outperforms pixel-based diffusion models regarding generative quality and training efficiency. Remarkably, DCTdiff can seamlessly scale up to 512×512 resolution without using the latent diffusion paradigm and beats latent diffusion (using SD-VAE) with only 1/4 training cost. Finally, we illustrate several intriguing properties of DCT image modeling. For example, we provide a theoretical proof of why ‘image diffusion can be seen as spectral autoregression’, bridging the gap between diffusion and autoregressive models. The effectiveness of DCTdiff and the introduced properties suggest a promising direction for image modeling in the frequency space. The code is at https://github.com/forever208/DCTdiff.

Abstract

Changes in the activity profile of cortical neurons are due to effects at the scale of local and long-range networks. Accordingly, abrupt transitions in the state of cortical neurons—a phenomenon known as Up-Down states—have been attributed to variation in the activity of afferent neurons. However, cellular physiology and morphology may also play a role in causing Up-Down states. This study examines the impact of dendritic nonlinearities, particularly those mediated by voltage-dependent NMDA receptors, on the response of cortical neurons to balanced excitatory/inhibitory synaptic inputs. Using a neuron model with two segregated dendritic compartments, we compared cells with and without dendritic nonlinearities. NMDA receptors boosted somatic firing in the balanced condition and increased the correlation between membrane potentials across the compartments of the neuron model. Dendritic nonlinearities elicited strong bimodality in the distribution of the somatic potential when the cell was driven with cortical-like input. Moreover, dendritic nonlinearities could detect small input fluctuations and lead to Up-Down states whose statistics and dynamics closely resemble electrophysiological data. Up-Down states also occurred in recurrent networks with oscillatory firing activity, as in anaesthetized animal models, when dendritic NMDA receptors were partially disabled. These findings suggest that there is a dissociation between cellular and network-level features that could both contribute to the emergence of Up-Down states. Our study highlights the complex interplay between dendritic integration and activity-driven dynamics in the origin of cortical bistability.

Abstract

When talking, speakers naturally produce hand movements (co-speech gestures) that contribute to communication. Evidence in Dutch suggests that the timing of simple up-and-down, non-referential “beat” gestures influences spoken word recognition: the same auditory stimulus was perceived as CONtent (noun, capitalized letters indicate stressed syllables) when a beat gesture occurred on the first syllable, but as conTENT (adjective) when the gesture occurred on the second syllable. However, these findings were based on a small number of minimal pairs in Dutch, limiting the generalizability of the findings. We therefore tested this effect in Spanish, where lexical stress is highly relevant in the verb conjugation system, distinguishing bailo, “I dance” with word-initial stress from bailó, “she danced” with word-final stress. Testing a larger sample (N = 100), we also assessed whether individual differences in working memory capacity modulated how much individuals relied on the gestures in spoken word recognition. The results showed that, similar to Dutch, Spanish participants were biased to perceive lexical stress on the syllable that visually co-occurred with a beat gesture, with the effect being strongest when the acoustic stress cues were most ambiguous. No evidence was found for by-participant effect sizes to be influenced by individual differences in phonological or visuospatial working memory. These findings reveal gestural-speech coordination impacts lexical stress perception in a language where listeners are regularly confronted with such lexical stress contrasts, highlighting the impact of gestures’ timing on prominence perception and spoken word recognition.

Abstract

With the increasing popularity of social media networks as platforms for news distribution, there is a growing concern that news is becoming more subjective due to adaptation to the shareability and engagement-driven logic of social media. The present study investigated whether this is the case for British news media by comparing the subjectivity of Facebook news posts versus news articles on the websites of four major British news outlets. Subjectivity was estimated on a sentence level using a machine learning-based model trained on native English speakers’ judgements. The results revealed that there was no across-the-board tendency to increased subjectivity and that different types of news sources (popular vs. quality) use different strategies in adapting to social media. Facebook posts from quality news outlets showed no significant difference in subjectivity compared to the articles on their websites. However, for popular news sources, Facebook posts were found to be noticeably more subjective than the articles, suggesting that a common strategy when posting on Facebook is a subjective meta-commentary on the news content, potentially used as clickbait. Thus, subjectification of British news on Facebook seems to mainly be happening in popular news sources.

Abstract

In face-to-face conversation, people use speech and gesture to convey meaning. Seeing gestures alongside speech facilitates comprehenders’ language processing, but crucially, the mechanisms underlying this facilitation remain unclear. We investigated whether comprehenders use the semantic information in gestures, typically preceding related speech, to predict upcoming meaning. Dutch adults listened to questions asked by a virtual avatar. Questions were accompanied by an iconic gesture (e.g., typing) or meaningless control movement (e.g., arm scratch) followed by a short pause and target word (e.g., “type”). A Cloze experiment showed that gestures improved explicit predictions of upcoming target words. Moreover, an EEG experiment showed that gestures reduced alpha and beta power during the pause, indicating anticipation, and reduced N400 amplitudes, demonstrating facilitated semantic processing. Thus, comprehenders use iconic gestures to predict upcoming meaning. Theories of linguistic prediction should incorporate communicative bodily signals as predictive cues to capture how language is processed in face-to-face interaction.

Abstract

Do aesthetic preferences for images of neatly organized compositions (e.g., images collected on blogs like Things Organized Neatly©) generalize across cultures? In an earlier study, focusing on stimulus and personal properties related to order and complexity, Western participants indicated their preference for one of two simultaneously presented images (100 pairs). In the current study, we compared the data of the native Dutch-speaking participants from this earlier sample (N = 356) to newly collected data from a native Chinese-speaking sample (N = 220). Overall, aesthetic preferences were quite similar across cultures. When relating preferences for each sample to ratings of order, complexity, soothingness, and fascination collected from a Western, mainly Dutch-speaking sample, the results hint at a cross-culturally consistent preference for images that Western participants rate as more ordered, but a cross-culturally diverse relation between preferences and complexity.

Abstract

When we communicate with others, we often repeat aspects of each other's communicative behavior such as sentence structures and words. Such behavioral alignment has been mostly studied for speech or text. Yet, language use is mostly multimodal, flexibly using speech and gestures to convey messages. Here, we explore the use of alignment in speech (words) and co-speech gestures (iconic gestures) in a referential communication task aimed at finding labels for novel objects in interaction. In particular, we investigate how people flexibly use lexical and gestural alignment to create shared labels for novel objects and whether alignment in speech and gesture are related over time. The present study shows that interlocutors establish shared labels multimodally, and alignment in words and iconic gestures are used throughout the interaction. We also show that the amount of lexical alignment positively associates with the amount of gestural alignment over time, suggesting a close relationship between alignment in the vocal and manual modalities.

Abstract

Comparative neurobiology allows us to investigate relationships between phylogeny and the brain and understand the evolution of traits. Bats constitute an attractive group of mammalian species for comparative studies, given their large diversity in behavioural phenotypes, brain morphology, and array of specialised traits. Currently, the order Chiroptera contains over 1,450 species within 21 families and spans ca. 65 million years of evolution. To date, 194 Neotropical bat species (ca. 13% of the total number of species around the world) have been recorded in Central America. This study includes qualitative and quantitative macromorphological descriptions of the brains of 12 species from six families of Neotropical bats. These analyses, which include histological neuronal staining of two species from different families (Phyllostomus hastatus and Saccopteryx bilineata), show substantial diversity in brain macromorphology including brain shape and size, exposure of mesencephalic regions, and cortical and cerebellar fissure depth. Brain macromorphology can in part be explained by phylogeny as species within the same family are more similar to each other. However, macromorphology cannot be explained by evolutionary time alone as brain differences between some phyllostomid bats are larger than between species from the family Emballonuridae despite being of comparable diverging distances in the phylogenetic tree. This suggests that faster evolutionary changes in brain morphology occurred in phyllostomids — although a larger number of species needs to be studied to confirm this. Our results show the rich diversity in brain morphology that bats provide for comparative and evolutionary studies.

Abstract

We present a reference genome assembly from an individual male Rhynchonycteris naso (Chordata; Mammalia; Chiroptera; Emballonuridae). The genome sequence is 2.46 Gb in span. The majority of the assembly is scaffolded into 22 chromosomal pseudomolecules, with the Y sex chromosome assembled.

Abstract

Language is supported by a distributed network of brain regions with a particular contribution from the left hemisphere. A multi-level understanding of this network requires studying the genetic architecture of its functional connectivity and hemispheric asymmetry. We used resting state functional imaging data from 29,681 participants from the UK Biobank to measure functional connectivity between 18 left-hemisphere regions implicated in multimodal sentence-level processing, as well as their homotopic regions in the right-hemisphere, and interhemispheric connections. Multivariate genome-wide association analysis of this total network, based on common genetic variants (with population frequencies above 1%), identified 14 loci associated with network functional connectivity. Three of these loci were also associated with hemispheric differences of intrahemispheric connectivity. Polygenic dispositions to lower language-related abilities, dyslexia and left-handedness were associated with generally reduced leftward asymmetry of functional connectivity, but with some trait- and connection-specific exceptions. Exome-wide association analysis based on rare, protein-altering variants (frequencies < 1%) suggested 7 additional genes. These findings shed new light on the genetic contributions to language network connectivity and its asymmetry based on both common and rare genetic variants, and reveal genetic links to language-related traits and hemispheric dominance for hand preference.

Abstract

The observation that the neural correlates of reading are left-lateralized is ubiquitous in the cognitive neuroscience and neuropsychological literature. Still, reading is served by a constellation of neural units, and the extent to which these units are consistently left-lateralized is unclear. In this regard, the functional lateralization of the fusiform gyrus is of particular interest, by virtue of its hypothesized role as a “visual word form area”. A quantitative Activation Likelihood Estimation meta-analysis was conducted on activation foci from 35 experiments investigating silent reading, and both a whole-brain and a bayesian ROI-based approach were used to assess the lateralization of the data submitted to meta-analysis. Perirolandic areas showed the highest level of left-lateralization, the fusiform cortex and the parietal cortex exhibited only a moderate pattern of left-lateralization, while in the occipital, insular cortices and in the cerebellum the lateralization turned out to be the lowest observed. The relatively limited functional lateralization of the fusiform gyrus was further explored in a regression analysis on the lateralization profile of each study. The functional lateralization of the fusiform gyrus during reading was positively associated with the lateralization of the precentral and inferior occipital gyri and negatively associated with the lateralization of the triangular portion of the inferior frontal gyrus and of the temporal pole. Overall, the present data highlight how lateralization patterns differ within the reading network. Furthermore, the present data highlight how the functional lateralization of the fusiform gyrus during reading is related to the degree of functional lateralization of other language brain areas.

Abstract

While infants' sensitivity to visual speech cues and the benefit of these cues have been well-established by behavioural studies, there is little evidence on the effect of visual speech cues on infants' neural processing of continuous auditory speech. In this study, we investigated whether visual speech cues, such as the movements of the lips, jaw, and larynx, facilitate infants' neural speech tracking. Ten-month-old Dutch-learning infants watched videos of a speaker reciting passages in infant-directed speech while electroencephalography (EEG) was recorded. In the videos, either the full face of the speaker was displayed or the speaker's mouth and jaw were masked with a block, obstructing the visual speech cues. To assess neural tracking, speech-brain coherence (SBC) was calculated, focusing particularly on the stress and syllabic rates (1–1.75 and 2.5–3.5 Hz respectively in our stimuli). First, overall, SBC was compared to surrogate data, and then, differences in SBC in the two conditions were tested at the frequencies of interest. Our results indicated that infants show significant tracking at both stress and syllabic rates. However, no differences were identified between the two conditions, meaning that infants' neural tracking was not modulated further by the presence of visual speech cues. Furthermore, we demonstrated that infants' neural tracking of low-frequency information is related to their subsequent vocabulary development at 18 months. Overall, this study provides evidence that infants' neural tracking of speech is not necessarily impaired when visual speech cues are not fully visible and that neural tracking may be a potential mechanism in successful language acquisition.

Abstract

The environment in which infants learn language is multimodal and rich with social cues. Yet, the effects of such cues, such as eye contact, on early speech perception have not been closely examined. This study assessed the role of ostensive speech, signalled through the speaker's eye gaze direction, on infants’ word segmentation abilities. A familiarisation-then-test paradigm was used while electroencephalography (EEG) was recorded. Ten-month-old Dutch-learning infants were familiarised with audio-visual stories in which a speaker recited four sentences with one repeated target word. The speaker addressed them either with direct or with averted gaze while speaking. In the test phase following each story, infants heard familiar and novel words presented via audio-only. Infants’ familiarity with the words was assessed using event-related potentials (ERPs). As predicted, infants showed a negative-going ERP familiarity effect to the isolated familiarised words relative to the novel words over the left-frontal region of interest during the test phase. While the word familiarity effect did not differ as a function of the speaker's gaze over the left-frontal region of interest, there was also a (not predicted) positive-going early ERP familiarity effect over right fronto-central and central electrodes in the direct gaze condition only. This study provides electrophysiological evidence that infants can segment words from audio-visual speech, regardless of the ostensiveness of the speaker's communication. However, the speaker's gaze direction seems to influence the processing of familiar words.

Abstract

The neural mechanisms underpinning the dynamic switching of a listener’s attention between speakers are not well understood. Here we addressed this issue in a natural conversation involving 21 triadic adult groups. Results showed that when the listener’s attention dynamically switched between speakers, neural synchronization with the to-be-attended speaker was significantly enhanced, whereas that with the to-be-ignored speaker was significantly suppressed. Along with attention switching, semantic distances between sentences significantly increased in the to-be-ignored speech. Moreover, neural synchronization negatively correlated with the increase in semantic distance but not with acoustic change of the to-be-ignored speech. However, no difference in neural synchronization was found between the listener and the two speakers during the phase of sustained attention. These findings support the attenuation model of attention, indicating that both speech signals are processed beyond the basic physical level. Additionally, shifting attention imposes a cognitive burden, as demonstrated by the opposite fluctuations of interpersonal neural synchronization.

Abstract

Human language offers a variety of ways to create meaning, one of which is referring to entities, objects, or events in the world. One such meaning maker is understanding to whom or to what a pronoun in a discourse refers to. To understand a pronoun, the brain must access matching entities or concepts that have been encoded in memory from previous linguistic context. Models of language processing propose that internally stored linguistic concepts, accessed via exogenous cues such as phonological input of a word, are represented as (a)synchronous activities across a population of neurons active at specific frequency bands. Converging evidence suggests that delta band activity (1–3 Hz) is involved in temporal and representational integration during sentence processing. Moreover, recent advances in the neurobiology of memory suggest that recollection engages neural dynamics similar to those which occurred during memory encoding. Integrating from these two research lines, we here tested the hypothesis that neural dynamic patterns, especially in delta frequency range, underlying referential meaning representation, would be reinstated during pronoun resolution. By leveraging neural decoding techniques (i.e., representational similarity analysis) on a magnetoencephalogram data set acquired during a naturalistic story-listening task, we provide evidence that delta-band activity underlies referential meaning representation. Our findings suggest that, during spoken language comprehension, endogenous linguistic representations such as referential concepts may be proactively retrieved and represented via activation of their underlying dynamic neural patterns.

Abstract

Harbor seals (Phoca vitulina) are more solitary than many other pinnipeds. Yet, they are capable of vocal learning, a form of social learning. Most extant literature examines social animals when investigating social learning, despite sociality not being a prerequisite. Here, we report two formerly silent harbor seals who initiated vocalizations, after having repeatedly observed a conspecific receiving food rewards for vocalizing. Our observations suggest both social and vocal learning in a group of captive harbor seals, a species that lives semi-solitarily in the wild. We propose that, in this case, social learning acted as a shortcut to acquiring food rewards compared to the comparatively costly asocial learning.

Abstract

Background

Which mammals show vocal learning abilities, e.g., can learn new sounds, or learn to use sounds in new contexts? Vocal usage and comprehension learning are submodules of vocal learning. Specifically, vocal usage learning is the ability to learn to use a vocalization in a new context; vocal comprehension learning is the ability to comprehend a vocalization in a new context. Among mammals, harbor seals (Phoca vitulina) are good candidates to investigate vocal learning. Here, we test whether harbor seals are capable of vocal usage and comprehension learning.

Results

We trained two harbor seals to (i) switch contexts from a visual to an auditory cue. In particular, the seals first produced two vocalization types in response to two hand signs; they then transitioned to producing these two vocalization types upon the presentation of two distinct sets of playbacks of their own vocalizations. We then (ii) exposed the seals to a combination of trained and novel vocalization stimuli. In a final experiment, (iii) we broadcasted only novel vocalizations of the two vocalization types to test whether seals could generalize from the trained set of stimuli to only novel items of a given vocal category. Both seals learned all tasks and took ≤ 16 sessions to succeed across all experiments. In particular, the seals showed contextual learning through switching the context from former visual to novel auditory cues, vocal matching and generalization. Finally, by responding to the played-back vocalizations with distinct vocalizations, the animals showed vocal comprehension learning.

Conclusions

It has been suggested that harbor seals are vocal learners; however, to date, these observations had not been confirmed in controlled experiments. Here, through three experiments, we could show that harbor seals are capable of both vocal usage and comprehension learning.

Abstract

Although there is a large body of research assessing whether exposure to narratives boosts social cognition immediately afterward, not much research has investigated the underlying mechanism of this putative effect. This experiment investigates the possibility that reading a narrative increases social curiosity directly afterward, which might explain the short-term boosts in social cognition reported by some others. We developed a novel measure of state social curiosity and collected data from participants (N = 222) who were randomly assigned to read an excerpt of narrative fiction or expository nonfiction. Contrary to our expectations, we found that those who read a narrative exhibited less social curiosity afterward than those who read an expository text. This result was not moderated by trait social curiosity. An exploratory analysis uncovered that the degree to which texts present readers with social targets predicted less social curiosity. Our experiment demonstrates that reading narratives, or possibly texts with social content in general, may engage and fatigue social-cognitive abilities, causing a temporary decrease in social curiosity. Such texts might also temporarily satisfy the need for social connection, temporarily reducing social curiosity. Both accounts are in line with theories describing how narratives result in better social cognition over the long term.

Abstract

Earlier work has explored spoken word production during irrelevant background speech such as intelligible and unintelligible word lists. The present study compared how different types of irrelevant background speech (word lists vs. sentences) influenced spoken word production relative to a quiet control condition, and whether the influence depended on the intelligibility of the background speech. Experiment 1 presented native Dutch speakers with Chinese word lists and sentences. Experiment 2 presented a similar group with Dutch word lists and sentences. In both experiments, the lexical selection demands in speech production were manipulated by varying name agreement (high vs. low) of the to-be-named pictures. Results showed that background speech, regardless of its intelligibility, disrupted spoken word production relative to a quiet condition, but no effects of word lists versus sentences in either language were found. Moreover, the disruption by intelligible background speech compared with the quiet condition was eliminated when planning low name agreement pictures. These findings suggest that any speech, even unintelligible speech, interferes with production, which implies that the disruption of spoken word production is mainly phonological in nature. The disruption by intelligible background speech can be reduced or eliminated via top–down attentional engagement.

Abstract

* Ole Goltermann and Gökberk Alagöz contributed equally.
Primate brain evolution has involved prominent expansions of the cerebral cortex, with largest effects observed in the human lineage. Such expansions were accompanied by fine-grained anatomical alterations, including increased cortical folding. However, the molecular bases of evolutionary alterations in human sulcal organization are not yet well understood. Here, we integrated data from recently completed large-scale neuroimaging genetic analyses with annotations of the human genome relevant to various periods and events in our evolutionary history. These analyses identified single-nucleotide polymorphism (SNP) heritability enrichments in fetal brain human-gained enhancer (HGE) elements for a number of sulcal structures, including the central sulcus, which is implicated in human hand dexterity. We zeroed in on a genomic region that harbors DNA variants associated with left central sulcus shape, an HGE element, and genetic loci involved in neurogenesis including ZIC4, to illustrate the value of this approach for probing the complex factors contributing to human sulcal evolution.

Abstract

Formants, or resonance frequencies of the upper vocal tract, are an essential part of acoustic communication. Articulatory gestures—such as jaw, tongue, lip, and soft palate movements—shape formant structure in human vocalizations, but little is known about how nonhuman mammals use those gestures to modify formant frequencies. Here, we report a case study with an adult male harbor seal trained to produce an arbitrary vocalization composed of multiple repetitions of the sound wa. We analyzed jaw movements frame-by-frame and matched them to the tracked formant modulation in the corresponding vocalizations. We found that the jaw opening angle was strongly correlated with the first (F1) and, to a lesser degree, with the second formant (F2). F2 variation was better explained by the jaw angle opening when the seal was lying on his back rather than on the belly, which might derive from soft tissue displacement due to gravity. These results show that harbor seals share some common articulatory traits with humans, where the F1 depends more on the jaw position than F2. We propose further in vivo investigations of seals to further test the role of the tongue on formant modulation in mammalian sound production.

Abstract

Common genetic variation has been associated with multiple symptoms in Autism Spectrum Disorder (ASD). However, our knowledge of shared genetic factor structures contributing to this highly heterogeneous neurodevelopmental condition is limited. Here, we developed a structural equation modelling framework to directly model genome-wide covariance across core and non-core ASD phenotypes, studying autistic individuals of European descent using a case-only design. We identified three independent genetic factors most strongly linked to language/cognition, behaviour and motor development, respectively, when studying a population-representative sample (N=5,331). These analyses revealed novel associations. For example, developmental delay in acquiring personal-social skills was inversely related to language, while developmental motor delay was linked to self-injurious behaviour. We largely confirmed the three-factorial structure in independent ASD-simplex families (N=1,946), but uncovered simplex-specific genetic overlap between behaviour and language phenotypes. Thus, the common genetic architecture in ASD is multi-dimensional and contributes, in combination with ascertainment-specific patterns, to phenotypic heterogeneity.

Abstract

To understand the neurocognitive mechanisms that underlie heterogeneity in cognitive ageing, recent scientific efforts have led to a growing public availability of imaging cohort data. The Advanced BRain Imaging on ageing and Memory (ABRIM) project aims to add to these existing datasets by taking an adult lifespan approach to provide a cross-sectional, normative database with a particular focus on connectivity, myelinization and iron content of the brain in concurrence with cognitive functioning, mechanisms of reserve, and sleep-wake rhythms. ABRIM freely shares MRI and behavioural data from 295 participants between 18–80 years, stratified by age decade and sex (median age 52, IQR 36–66, 53.20% females). The ABRIM MRI collection consists of both the raw and pre-processed structural and functional MRI data to facilitate data usage among both expert and non-expert users. The ABRIM behavioural collection includes measures of cognitive functioning (i.e., global cognition, processing speed, executive functions, and memory), proxy measures of cognitive reserve (e.g., educational attainment, verbal intelligence, and occupational complexity), and various self-reported questionnaires (e.g., on depressive symptoms, pain, and the use of memory strategies in daily life and during a memory task). In a sub-sample (n = 120), we recorded sleep-wake rhythms using an actigraphy device (Actiwatch 2, Philips Respironics) for a period of 7 consecutive days. Here, we provide an in-depth description of our study protocol, pre-processing pipelines, and data availability. ABRIM provides a cross-sectional database on healthy participants throughout the adult lifespan, including numerous parameters relevant to improve our understanding of cognitive ageing. Therefore, ABRIM enables researchers to model the advanced imaging parameters and cognitive topologies as a function of age, identify the normal range of values of such parameters, and to further investigate the diverse mechanisms of reserve and resilience.

Abstract

We investigated prediction skills of adult heritage speakers and the role of written and spoken language experience on predictive processing. Using visual world eye-tracking, we focused on predictive use of case-marking cues in verb-medial and verb-final sentences in Turkish with adult Turkish heritage speakers (N = 25) and Turkish monolingual speakers (N = 24). Heritage speakers predicted in verb-medial sentences (when verb-semantic and case-marking cues were available), but not in verb-final sentences (when only case-marking cues were available) while monolinguals predicted in both. Prediction skills of heritage speakers were modulated by their spoken language experience in Turkish and written language experience in both languages. Overall, these results strongly suggest that verb-semantic information is needed to scaffold the use of morphosyntactic cues for prediction in heritage speakers. The findings also support the notion that both spoken and written language experience play an important role in predictive spoken language processing.

Abstract

Expressing Left-Right relations is challenging for speaking-children. Yet, this challenge was absent for signing-children, possibly due to iconicity in the visual-spatial modality of expression. We investigate whether there is also a modality advantage when speaking-children’s co-speech gestures are considered. Eight-year-old child and adult hearing monolingual Turkish speakers and deaf signers of Turkish-Sign-Language described pictures of objects in various spatial relations. Descriptions were coded for informativeness in speech, sign, and speech-gesture combinations for encoding Left-Right relations. The use of co-speech gestures increased the informativeness of speakers’ spatial expressions compared to speech-only. This pattern was more prominent for children than adults. However, signing-adults and children were more informative than child and adult speakers even when co-speech gestures were considered. Thus, both speaking- and signing-children benefit from iconic expressions in visual modality. Finally, in each modality, children were less informative than adults, pointing to the challenge of this spatial domain in development.

Abstract

Multimodal rhythmic signalling abounds across animal taxa. Studying its mechanisms and functions can highlight adaptive components in highly complex rhythmic behaviours, like dance and music. Pinnipeds, such as the harbour seal, Phoca vitulina, are excellent comparative models to assess rhythmic capacities. Harbour seals engage in rhythmic percussive behaviours which, until now, have not been described in detail. In our study, eight zoo-housed harbour seals (two pups, two juveniles and four adults) were passively monitored by audio and video during their pupping/breeding season. All juvenile and adult animals performed percussive signalling with their fore flippers in agonistic conditions, both on land and in water. Flipper slap sequences produced on the ground or on the seals' bodies were often highly regular in their interval duration, that is, were quasi-isochronous, at a 200–600 beats/min pace. Three animals also showed significant lateralization in slapping. In contrast to slapping on land, display slapping in water, performed only by adult males, showed slower tempo by one order of magnitude, and a rather motivic temporal structure. Our work highlights that percussive communication is a significant part of harbour seals' behavioural repertoire. We hypothesize that its forms of rhythm production may reflect adaptive functions such as regulating internal states and advertising individual traits.

Abstract

We examined the extent to which cognate facilitation effects occurred in simultaneous bilingual children’s production and comprehension and how these were modulated by language dominance and language context. Bilingual Dutch-German children, ranging from Dutch-dominant to German-dominant, performed picture naming and auditory lexical decision tasks in single-language and dual-language contexts. Language context was manipulated with respect to the language of communication (with the experimenter and in instructional videos) and by means of proficiency tasks. Cognate facilitation effects emerged in both production and comprehension and interacted with both dominance and context. In a single-language context, stronger cognate facilitation effects were found for picture naming in children’s less dominant language, in line with previous studies on individual differences in lexical activation. In the dual-language context, this pattern was reversed, suggesting inhibition of the dominant language at the decision level. Similar effects were observed in lexical decision. These findings provide evidence for an integrated bilingual lexicon in simultaneous bilingual children and shed more light on the complex interplay between lexicon-internal and lexicon-external factors modulating the extent of lexical cross-linguistic influence more generally.

Abstract

In a between-language lexical priming study, we examined to what extent the two languages in a simultaneous bilingual child's lexicon interact, while taking individual differences in language exposure into account. Primary-school-aged Dutch–Greek bilinguals performed a primed picture selection task combined with eye-tracking. They matched pictures to auditorily presented Dutch target words preceded by Greek prime words. Their reaction times and eye movements were recorded. We tested for effects of between-language phonological priming, translation priming, and phonological priming through translation. Priming effects emerged in reaction times and eye movements in all three conditions, at different stages of processing, and unaffected by language exposure. These results extend previous findings for bilingual toddlers and bilingual adults. Processing similarities between these populations indicate that, across different stages of development, bilinguals have an integrated lexicon that is accessed in a language-nonselective way and is susceptible to interactions within and between different types of lexical representation.

Abstract

Classical galactosaemia (CG) is a hereditary disease in galactose metabolism that despite dietary treatment is characterized by a wide range of cognitive deficits, among which is language production. CG brain functioning has been studied with several neuroimaging techniques, which revealed both structural and functional atypicalities. In the present study, for the first time, we compared the oscillatory dynamics, especially the power spectrum and time–frequency representations (TFR), in the electroencephalography (EEG) of CG patients and healthy controls while they were performing a language production task. Twenty-one CG patients and 19 healthy controls described animated scenes, either in full sentences or in words, indicating two levels of complexity in syntactic planning. Based on previous work on the P300 event related potential (ERP) and its relation with theta frequency, we hypothesized that the oscillatory activity of patients and controls would differ in theta power and TFR. With regard to behavior, reaction times showed that patients are slower, reflecting the language deficit. In the power spectrum, we observed significant higher power in patients in delta (1–3 Hz), theta (4–7 Hz), beta (15–30 Hz) and gamma (30–70 Hz) frequencies, but not in alpha (8–12 Hz), suggesting an atypical oscillatory profile. The time-frequency analysis revealed significantly weaker event-related theta synchronization (ERS) and alpha desynchronization (ERD) in patients in the sentence condition. The data support the hypothesis that CG language difficulties relate to theta–alpha brain oscillations.

Abstract

Anecdotal evidence suggests that learning a new foreign language (FL) makes you forget previously learned FLs. To seek empirical evidence for this claim, we tested whether learning words in a previously unknown L3 hampers subsequent retrieval of their L2 translation equivalents. In two experiments, Dutch native speakers with knowledge of English (L2), but not Spanish (L3), first completed an English vocabulary test, based on which 46 participant-specific, known English words were chosen. Half of those were then learned in Spanish. Finally, participants’ memory for all 46 English words was probed again in a picture naming task. In Experiment 1, all tests took place within one session. In Experiment 2, we separated the English pre-test from Spanish learning by a day and manipulated the timing of the English post-test (immediately after learning vs. 1 day later). By separating the post-test from Spanish learning, we asked whether consolidation of the new Spanish words would increase their interference strength. We found significant main effects of interference in naming latencies and accuracy: Participants speeded up less and were less accurate to recall words in English for which they had learned Spanish translations, compared with words for which they had not. Consolidation time did not significantly affect these interference effects. Thus, learning a new language indeed comes at the cost of subsequent retrieval ability in other FLs. Such interference effects set in immediately after learning and do not need time to emerge, even when the other FL has been known for a long time.

Abstract

Background

Forced switching between languages poses demands on control abilities, which may be difficult to meet for bilinguals with aphasia. Freely choosing languages has been shown to increase naming efficiency in healthy bilinguals, and lexical accessibility was found to be a predictor for language choice. The overlap between bilingual language switching and other types of switching is yet unclear.

Aims

This study aimed to examine the benefits of free language choice for bilinguals with aphasia and to investigate the overlap of between- and within-language switching abilities.

Methods & Procedures

Seventeen bilinguals with aphasia completed a questionnaire and four web-based picture naming tasks: single-language naming in the first and second language separately; voluntary switching between languages; cued and predictable switching between languages; cued and predictable switching between phrase types in the first language. Accuracy and naming latencies were analysed using (generalised) linear mixed-effects models.

Outcomes & Results

The results showed higher accuracy and faster naming for the voluntary switching condition compared to single-language naming and cued switching. Both voluntary and cued language switching yielded switch costs, and voluntary switch costs were larger. Ease of lexical access was a reliable predictor for voluntary language choice. We obtained no statistical evidence for differences or associations between switch costs in between- and within-language switching.

Conclusions

Several results point to benefits of voluntary language switching for bilinguals with aphasia. Freely mixing languages improved naming accuracy and speed, and ease of lexical access affected language choice. There was no statistical evidence for overlap of between- and within-language switching abilities. This study highlights the benefits of free language choice for bilinguals with aphasia.