Publications

Abstract

Natural human interaction involves the fast-paced exchange of speaker turns. Crucially, if a next speaker waited with planning their turn until the current speaker was finished, language production models would predict much longer turn transition times than what we observe. Next speakers must therefore prepare their turn in parallel to listening. Visual signals likely play a role in this process, for example by helping the next speaker to process the ongoing utterance and thus prepare an appropriately-timed response.

To understand how visual signals contribute to the timing of turn-taking, and to move beyond the mostly qualitative studies of gesture in conversation, we examined unconstrained, computer-mediated conversations between 20 pairs of participants while systematically manipulating speaker visibility. Using motion tracking and manual gesture annotation, we assessed 1) how visibility affected the timing of turn transitions, and 2) whether use of co-speech gestures and 3) the communicative kinematic features of these gestures were associated with changes in turn transition timing.

We found that 1) decreased visibility was associated with less tightly timed turn transitions, and 2) the presence of gestures was associated with more tightly timed turn transitions across visibility conditions. Finally, 3) structural and salient kinematics contributed to gesture’s facilitatory effect on turn transition times.

Our findings suggest that speaker visibility--and especially the presence and kinematic form of gestures--during conversation contributes to the temporal coordination of conversational turns in computer-mediated settings. Furthermore, our study demonstrates that it is possible to use naturalistic conversation and still obtain controlled results.

Abstract

In this thesis I approach language as a neurobiological system. I defend a sequence processing perspective on language and on the function of Broca's region in the left inferior frontal gyrus (LIFG). This perspective provides a way to express common structural aspects of language, music and action, which all engage the LIFG. It also facilitates the comparison of human language and structured sequence processing in animals. Research on infants, song-birds and non-human primates suggests an interesting role for non-adjacent dependencies in language acquisition and the evolution of language. In a series of experimental studies using a sequence processing paradigm called artificial grammar learning (AGL), we have investigated sequences with adjacent and non-adjacent dependencies. Our behavioral and transcranial magnetic stimulation (TMS) studies show that healthy subjects successfully discriminate between grammatical and non-grammatical sequences after having acquired aspects of a grammar with nested or crossed non-adjacent dependencies implicitly. There were no indications of separate acquisition/processing mechanisms for sequence processing of adjacent and non-adjacent dependencies, although acquisition of non-adjacent dependencies takes more time. In addition, we studied the causal role of Broca‟s region in processing artificial syntax. Although syntactic processing has already been robustly correlated with activity in Broca's region, the causal role of Broca's region in syntactic processing, in particular syntactic comprehension has been unclear. Previous lesion studies have shown that a lesion in Broca's region is neither a necessary nor sufficient condition to induce e.g. syntactic deficits. Subsequent to transcranial magnetic stimulation of Broca‟s region, discrimination of grammatical sequences with non-adjacent dependencies from non-grammatical sequences was impaired, compared to when a language irrelevant control region (vertex) was stimulated. Two additional experiments show perturbation of discrimination performance for grammars with adjacent dependencies after stimulation of Broca's region. Together, these results support the view that Broca‟s region plays a causal role in implicit structured sequence processing.

Abstract

In this dissertation the perceptual relevance of prevoicing in Dutch was investigated. Prevoicing is the presence of vocal fold vibration during the closure of initial voiced plosives (negative voice onset time). The presence or absence of prevoicing is generally used to describe the difference between voiced and voiceless Dutch plosives. The first experiment described in this dissertation showed that prevoicing is frequently absent in Dutch and that several factors affect the production of prevoicing. A detailed acoustic analysis of the voicing distinction identified several acoustic correlates of voicing. Prevoicing appeared to be by far the best predictor. Perceptual classification data revealed that prevoicing was indeed the strongest cue that listeners use when classifying plosives as voiced or voiceless. In the cases where prevoicing was absent, other acoustic cues influenced classification, such that some of these tokens were still perceived as being voiced. In the second part of this dissertation the influence of prevoicing variation on spoken-word recognition was examined. In several cross-modal priming experiments two types of prevoicing variation were contrasted: a difference between the presence and absence of prevoicing (6 versus 0 periods of prevoicing) and a difference in the amount of prevoicing (12 versus 6 periods). All these experiments indicated that primes with 12 and 6 periods of prevoicing had the same effect on lexical decisions to the visual targets. The primes without prevoicing had a different effect, but only when their voiceless counterparts were real words. Phonetic detail appears to influence lexical access only when it is useful: In Dutch, the presence versus absence of prevoicing is informative, while the amount of prevoicing is not.

Abstract

Translated and expanded version of 'Privileged syntactic arguments, pivots and controllers

Abstract

The aim of this thesis was to gain more insight into spoken-word comprehension and the influence of sentence-contextual information on these processes using ERPs. By manipulating critical words in semantically constraining sententes, in semantic or syntactic sense, and examining the consequences in the electrophysiological signal (e.g., elicitation of ERP components such as the N400, N200, LAN, and P600), three questions were tackled: I At which moment is context information used in the spoken-word recognition process? II What is the temporal relationship between lexical selection and integration of the meaning of a spoken word into a higher-order level representeation of the preceding sentence? III What is the time course of the processing of different sources of linguistic information obtained from the context, such as phonological, semantic and syntactic information, during spoken-word comprehension? From the results of this thesis it can be concluded that sentential context already exerts an influence on spoken-word processing at approximately 200 ms after word onset. In addition, semantic integration is attempted before a spoken word can be selected on the basis of the acoustic signal, i.e. before lexical selection is completed. Finally, knowledge of the syntactic category of a word is not needed before semantic integration can take place. These findings, therefore, were interpreted as providing evidence for an account of cascaded spoken-word processing that proclaims an optimal use of contextual information during spoken-word identification. Optimal use is accomplished by allowing for semantic and syntactic processing to take place in parallel after bottom-up activation of a set of candidates, and lexical integration to proceed with a limited number of candidates that still match the acoustic input

Abstract

Introduction: What’s happening in the brains of two people having a conversation? One reasonable guess is that in the fMRI scanner we’d see most of their brains light up. Another is that their EEG will be a total mess, reflecting dozens of interacting neuronal systems. Conversation recruits all of the basic language systems reviewed in this book. It also heavily taxes cognitive systems more likely to be found in handbooks of memory, attention and control, or social cognition (Brownell & Friedman, 2001). With most conversations going beyond the single utterance, for instance, they place a heavy load on episodic memory, as well as on the systems that allow us to reallocate cognitive resources to meet the demands of a dynamically changing situation. Furthermore, conversation is a deeply social and collaborative enterprise (Clark, 1996; this volume), in which interlocutors have to keep track of each others state of mind and coordinate on such things as taking turns, establishing common ground, and the goals of the conversation.

Abstract

Relative clauses present an interesting challenge for theories of the syntaxsemantics interface, because one element functions simultaneously in the matrix and relative clauses. The exact nature of the challenge depends on whether the relative clause is externally-headed or internallyheaded. Standard analyses of relative clauses are grounded in the analysis of Englishtype externally-headed constructions involving a relative pronoun, e.g. The horse which the man bought was a good horse, despite its typological rarity, and such accounts typically involve movement rules, both overt and covert, and phonologically null elements. The analysis of internally-headed relative clauses often involves the positing of an abstract structure including a null external head, with covert movement of the internal head to that position. The purpose of this paper is to show that the essential features of both types of relative clause can be captured in a syntactic theory that eschews movement rules and phonologically null elements, Role and Reference Grammar. It will be argued that a single set of linking principles can handle the syntax-to-semantics linking for both types. Keywords: Externally-headed relative clauses; internally-headed relative clauses; Role and Reference Grammar; linking syntax and semantics

Abstract

In a small village in the north of Bali called Bengkala, relatively many people inherit deafness. The Balinese therefore refer to this village as Desa Kolok, which means 'deaf village'. Connie de Vos studied Kata Kolok, the sign language of this village, and the ways in which the language recruits space to talk about both spatial and non-spatial matters. he small village community Bengkala in the north of Bali has almost 3,000 inhabitants. Of all the inhabitants, 57% use sign language, with varying degrees of fluency. But of this signing community (between 1,200 and 1,800 signers, depending on your definition of 'signer'), only 4% are deaf. So, not only do the deaf people of Bengkala use the sign language Kata Kolok, but also the majority of the hearing population.
"I've worked with deaf people from all over Asia, Europe, and also some signers in America," says Connie de Vos of MPI's Language and Cognition Department, and Centre for Language Studies (RU). "What sets apart this particular deaf village is that deaf individuals are highly integrated within the village clans. There is really a huge proportion of hearing signers." The sign language currently functions in all major aspects of village life and has been acquired from birth by multiple generations of deaf, native signers. According to De Vos, Kata Kolok is a fully-fledged sign language in every sense of the word. As a collaborative project, she has initiated inclusive deaf education within the village and now Kata Kolok is used as the primary language of instruction. De Vos' primary finding is that Kata Kolok discourse uses a different system of referring to space than other sign languages. Spatial relations are represented by a so-called "absolute frame of reference", based on geographic locations and wind directions. "All sign languages, as we know, use relative constructions for spatial relations. They use signs comparable to words like 'left' and 'right' instead of 'east' and 'west'. Kata Kolok does the latter. Kata Kolok signers appear to have an internal compass to continually register their position in space."De Vos is the first sign linguist who has documented Kata Kolok extensively. She spent more than a year in the village and collected over a hundred hours of video material of spontaneous conversations. "One of the things I've noticed is that language doesn't really emerge out of nothing," she says. "Signers adopt a local gesture system and transform it into a new and much more systematic sign language. A lot of the signs refer to concepts they're familiar with. That's why hearing signers have no difficulties in picking up Kata Kolok. Kata Kolok unites the hearing and the deaf.

Abstract

This dissertation focuses on syntactic comprehension problems in patients with Broca's aphasia from an electrophysiological perspective. The central objective of this dissertation was to further explore what syntax-related event-related brain potential (ERP) effects can reveal about the nature of the deficit that underlies syntactic comprehension problems in patients with Broca's aphasia. Chapter two to four describe experiments in which event-related brain potentials were recorded while subjects (Broca patients, non-aphasic patients with a right-hemisphere lesion, and healthy elderly controls) were presented with sentences that contained either violations of syntactic constraints or were syntactically correct. Chapter two investigates ERP effects of subject-verb agreement violations in the different subject groups, and seeks to answer the following questions: Do agrammatic comprehenders show sensitivity to subject-verb agreement violations as indicated by a syntax-related ERP effect? In addition, does the severity of the syntactic comprehension impairment in the Broca patients affect the ERP responses? Chapter three describes an investigation of whether Broca patients show sensitivity to violations of word order as indicated by a syntax-related ERP effect, and whether the ERP responses in the Broca patients are affected by the severity of their syntactic comprehension impairment. Chapter four reports on ERP effects of violations of word-category. In addition, also a semantic violation condition was added to track possible dissociations in the sensitivity to semantic and syntactic information in the Broca patients. Chapter five describes the development of a paradigm in which the electrophysiological approach and the classical sentence-picture matching approach are combined. In this chapter, the ERP method is applied to study on-line thematic role assignment in Broca patients during sentence-picture matching. Also the relation between ERP effects and behavioral responses is pursued. Finally, Chapter 6 provides a summary of the main findings of the experiments and a general discussion.

Abstract

The primary goal of psycholinguistic research is to understand the architectures and mechanisms that underlie human language comprehension and production. This entails an understanding of how linguistic knowledge is represented and organized in the brain and a theory of how that knowledge is accessed when we use language. Research has traditionally emphasized purely linguistic aspects of on-line comprehension, such as the influence of lexical, syntactic, semantic and discourse constraints, and their tim -course. It has become increasingly clear, however, that nonlinguistic information, such as the visual environment, are also actively exploited by situated language comprehenders.

Abstract

In order to decode the message of a speaker, listeners have to recognize individual words in the speaker's utterance.

Abstract

Many people speak a second language next to their mother tongue. How do they learn this language and how does the brain process it compared to the native language? A second language can be learned without explicit instruction. Our brains automatically pick up grammatical structures, such as word order, when these structures are repeated frequently during learning. The learning takes place within hours or days and the same brain areas, such as frontal and temporal brain regions, that process our native language are very quickly activated. When people master a second language very well, even the same neuronal populations in these language brain areas are involved. This is especially the case when the grammatical structures are similar. In conclusion, it appears that a second language builds on the existing cognitive and neural mechanisms of the native language as much as possible.

Abstract

Here we review evidence from cognitive neuroscience for a tight relation between language and action in the brain. We focus on two types of relation between language and action. First, we investigate whether the perception of speech and speech sounds leads to activation of parts of the cortical motor system also involved in speech production. Second, we evaluate whether understanding action-related language involves the activation of parts of the motor system. We conclude that whereas there is considerable evidence that understanding language can involve parts of our motor cortex, this relation is best thought of as inherently flexible. As we explain, the exact nature of the input as well as the intention with which language is perceived influences whether and how motor cortex plays a role in language processing.

Abstract

ISO Technical Committee 37, Terminology and other language and content resources, established an ISO 12620:2009 based Data Category Registry (DCR), called ISOcat (see http://www.isocat.org), to foster semantic interoperability of linguistic resources. However, this goal can only be met if the data categories are reused by a wide variety of linguistic resource types. A resource indicates its usage of data categories by linking to them. The small DC Reference XML vocabulary is used to embed links to data categories in XML documents. The link is established by an URI, which servers as the Persistent IDentifier (PID) of a data category. This paper discusses the efforts to mimic the same approach for RDF-based resources. It also introduces the RDF quad store based Relation Registry RELcat, which enables ontological relationships between data categories not supported by ISOcat and thus adds an extra level of linguistic knowledge.

Abstract

5.1 Einführung in den Forschungsbereich Die Psycholinguistik ist der Bereich der Linguistik, der sich mit dem Zusammenhang zwischen menschlicher Sprache und dem Denken und anderen mentalen Prozessen beschäftigt, d.h. sie stellt sich einer Reihe von essentiellen Fragen wie etwa (1) Wie schafft es unser Gehirn, im Wesentlichen akustische und visuelle kommunikative Informationen zu verstehen und in mentale Repräsentationen umzusetzen? (2) Wie kann unser Gehirn einen komplexen Sachverhalt, den wir anderen übermitteln wollen, in eine von anderen verarbeitbare Sequenz von verbalen und nonverbalen Aktionen umsetzen? (3) Wie gelingt es uns, in den verschiedenen Phasen des Lebens Sprachen zu erlernen? (4) Sind die kognitiven Prozesse der Sprachverarbeitung universell, obwohl die Sprachsysteme derart unterschiedlich sind, dass sich in den Strukturen kaum Universalien finden lassen?

Abstract

In recent decades, neuroimaging studies on the neural infrastructure of language are usually (or mostly) conducted with certain on-line language processing tasks. These functional neuroimaging studies helped to localize the language areas in the brain and to investigate the brain activity during explicit language processing. However, little is known about what is going on with the language areas when the brain is ‘at rest’, i.e., when there is no explicit language processing running. Taking advantage of the fcMRI and DTI techniques, this thesis is able to investigate the language function ‘off-line’ at the neuronal network level and the connectivity among language areas in the brain. Based on patient studies, the traditional, classical model on the perisylvian language network specifies a “Broca’ area – Arcuate Fasciculus – Werinicke’s area” loop (Ojemann 1991). With the help of modern neuroimaging techniques, researchers have been able to track language pathways that involve more brain structures than are in the classical model, and relate them to certain language functions. In such a background, a large part of this thesis made a contribution to the study of the topology of the language networks. It revealed that the language networks form a topographical functional connectivity pattern in the left hemisphere for the right-handers. This thesis also revealed the importance of structural hubs, such as Broca’s and Wernicke’s areas, which have more connectivity to other brain areas and play a central role in the language networks. Furthermore, this thesis revealed both functionally and structurally lateralized language networks in the brain. The consistency between what is found in this thesis and what has been known from previous functional studies seems to suggest, that the human brain is optimized and ‘ready’ for the language function even when there is currently no explicit language-processing running.

Abstract

Classifiers (currently also called 'depicting handshapes'), are observed in almost all signed languages studied to date and form a well-researched topic in sign language linguistics. Yet, these elements are still subject to much debate with respect to a variety of matters. Several different categories of classifiers have been posited on the basis of their semantics and the linguistic context in which they occur. The function(s) of classifiers are not fully clear yet. Similarly, there are differing opinions regarding their structure and the structure of the signs in which they appear. Partly as a result of comparison to classifiers in spoken languages, the term 'classifier' itself is under debate. In contrast to these disagreements, most studies on the acquisition of classifier constructions seem to consent that these are difficult to master for Deaf children. This article presents and discusses all these issues from the viewpoint that classifiers are linguistic elements.