Publications

Abstract

This chapter highlights that most language behavior consists of speaking and listening. The chapter also reveals differences and similarities between speaking and listening. The laboratory study of word production raises formidable problems; ensuring that a particular word is produced may subvert the spontaneous production process. Word production is investigated via slips and tip-of-the-tongue (TOT), primarily via instances of processing failure and via the technique of via the picture-naming task. The methodology of word production is explained in the chapter. The chapter also explains the phenomenon of interaction between various stages of word production and the process of speech recognition. In this context, it explores the difference between sound and meaning and examines whether or not the comparisons are appropriate between the processes of recognition and production of spoken words. It also describes the similarities and differences in the structure of the recognition and production systems. Finally, the chapter highlights the common issues in recognition and production research, which include the nuances of frequency of occurrence, morphological structure, and phonological structure.

Abstract

Dutch listeners were slower to make judgements about the semantic relatedness between a spoken target word (e.g. atLEET, 'athlete') and a previously presented visual prime word (e.g. SPORT 'sport') when the spoken word was mis-stressed. The adverse effect of mis-stressing confirms the role of stress information in lexical recognition in Dutch. However, although the erroneous stress pattern was always initially compatible with a competing word (e.g. ATlas, 'atlas'), mis-stressed words did not produced high false alarm rates in unrelated pairs (e.g. SPORT - atLAS). This suggests that stress information did not completely rule out segmentally matching but suprasegmentally mismatching words, a finding consistent with spoken-word recognition models involving multiple activation and inter-word competition.

Abstract

Two lexical decision studies examined the effects of single-phoneme mismatches on lexical activation in spoken-word recognition. One study was carried out in English, and involved spoken primes and visually presented lexical decision targets. The other study was carried out in Dutch, and primes and targets were both presented auditorily. Facilitation was found only for spoken targets preceded immediately by spoken primes; no facilitation occurred when targets were presented visually, or when intervening input occurred between prime and target. The effects of vowel mismatches and consonant mismatches were equivalent.

Abstract

Simulations explored the inability of the TRACE model of spoken-word recognition to model the effects on human listening of acoustic-phonetic mismatches in word forms. The source of TRACE's failure lay not in its interactive connectivity, not in the presence of interword competition, and not in the use of phonemic representations, but in the need for continuously optimised interpretation of the input. When an analogue of TRACE was allowed to cycle to asymptote on every slice of input, an acceptable simulation of the subcategorical mismatch data was achieved. Even then, however, the simulation was not as close as that produced by the Merge model.

Abstract

Different linguistic structures allow us to highlight distinct aspects of a situation. The aim of this survey is to investigate similarities and differences in the expression of situations or events as “stative” (maintaining a state), “inchoative” (adopting a state) and “agentive” (causing something to be in a state). The questionnaire focuses on the encoding of stative, inchoative and agentive possibilities for the translation equivalents of a set of English verbs.

Abstract

Expressions of human activities and states are a rich area for cross-linguistic comparison. Some languages of the world treat human posture verbs (e.g., sit, lie, kneel) as a special class of predicates, with distinct formal properties. This survey examines lexical, semantic and grammatical patterns for posture verbs, with special reference to contrasts between “stative” (maintaining a posture), “inchoative” (adopting a posture), and “agentive” (causing something to adopt a posture) constructions. The enquiry is thematically linked to the more general questionnaire 'Intransitive Predicate Form Class Survey'.

Abstract

The causal correlations between human genetic variants and linguistic (typological) features could represent the mechanism required for gradual, accretionary models of language evolution. The causal link is mediated by the process of cultural transmission of language across generations in a population of genetically biased individuals. The particular case of Tone, ASPM and Microcephalin is discussed as an illustration. It is proposed that this type of genetically-influenced linguistic bias, coupled with a fundamental role for genetic and linguistic diversities, provides a better explanation for the evolution of language and linguistic universals.

Abstract

This article surveys what is currently known about the complex interplay between genetics and the language sciences. It focuses not only on the genetic architecture of language and speech, but also on their interactions on the cultural and evolutionary timescales. Given the complexity of these issues and their current state of flux and high dynamism, this article surveys the main findings and topics of interest while also briefly introducing the main relevant methods, thus allowing the interested reader to fully appreciate and understand them in their proper context. Of course, not all the relevant publications and resources are mentioned, but this article aims to select the most relevant, promising, or accessible for nonspecialists.

Abstract

Retrieval of autobiographical memories is facilitated by activation of perceptuo-motor aspects of the experience, for example a congruent body position at the time of the experiencing and the time of retelling (Dijkstra, Kaschak, & Zwaan, 2007). The present study examined whether similar retrieval facilitation occurs when the direction of motor action is congruent with the valence of emotional memories. Consistent with evidence that people mentally represent emotions spatially (Casasanto, in press), participants moved marbles between vertically stacked boxes at a higher rate when the direction of movement was congruent with the valence of the memory they retrieved (e.g., upward for positive memories, downward for negative memories) than when direction and valence were incongruent (t(22)=4.24, p<.001). In addition, valence-congruent movements facilitated access to these memories, resulting in shorter retrieval times (t(22)=2.43, p<.05). Results demonstrate bidirectional influences between the emotional content of autobiographical memories and irrelevant motor actions.

Abstract

In this paper we discuss the influence of semantically unexpected information on the prosodic realization of contrast.
For this purpose, we examine the interplay between unexpectedness and various discourse factors that have been claimed to enhance the accentuation of contrastive
information: contrast direction, syntactic status, and discourse distance. We conducted a production experiment in Dutch in which speakers described scenes consisting of moving fruits with unnatural colors. We found that a general cognitive factor such as the unexpectedness of a property has a strong impact on the intonational marking of contrast, over and above the influence of the immediate discourse context.

Abstract

In this paper, we discuss the interplay of factors that influence the intonational marking of contrast in Dutch. In particular, we examine how prominence is expressed at the prosodic level when semantically abnormal information conflicts with contrastive information. For this purpose, we conducted a production experiment in Dutch in which speakers described scenes containing fruits with unnatural colors. We found that semantically abnormal information invokes cognitive prominence which corresponds to intonational prominence. Moreover, the results show that abnormality may overrule the accentual marking of information structural categories such as contrastive focus. If semantically abnormal information becomes integrated into the larger discourse context, its prosodic prominence decreases in favor of the signaling of information structural categories such as contrastive focus.

Abstract

Empirical studies addressing the age factor in second language acquisition have mainly been concerned with a comparison of end state data (from learners before and after the closure of a putative Critical Period for language acquisition) to the native speaker norm. Based on longitudinal corpus data, this paper investigates the affect of age on end state, rate and the process of acquisition and addresses the question of whether different grammatical domains are equally affected. To this end, the paper presents summarized findings from the acquisition of word order and inflectional morphology in L2 German by Russian learners of different ages and discusses theoretical implications that can be drawn from this evidence.

Abstract

This entry provides some orientation and task suggestions on how to explore the perceptual world of your field site and the interaction between the cultural world and the sensory lexicon in your community. The material consists of procedural texts; soundscapes; other documentary and observational tasks. The goal of this task is to explore the perceptual world of your field site and the interaction between the cultural world and the sensory lexicon in your community.

Abstract

In conversation, people regularly deal with problems of speaking, hearing, and understanding. We report on a cross-linguistic investigation of the conversational structure of other-initiated repair (also known as collaborative repair, feedback, requests for clarification, or grounding sequences). We take stock of formats for initiating repair across languages (comparable to English huh?, who?, y’mean X?, etc.) and find that different languages make available a wide but remarkably similar range of linguistic resources for this function. We exploit the patterned variation as evidence for several underlying concerns addressed by repair initiation: characterising trouble, managing responsibility, and handling knowledge. The concerns do not always point in the same direction and thus provide participants in interaction with alternative principles for selecting one format over possible others. By comparing conversational structures across languages, this paper contributes to pragmatic typology: the typology of systems of language use and the principles that shape them.

Abstract

Numerous experiments show that space and musical pitch are
closely linked in people's minds. However, the exact nature of
space-pitch associations and their neuronal underpinnings are
not well understood. In an fMRI experiment we investigated
different types of spatial representations that may underlie
musical pitch. Participants judged stimuli that varied in
spatial height in both the visual and tactile modalities, as well
as auditory stimuli that varied in pitch height. In order to
distinguish between unimodal and multimodal spatial bases of
musical pitch, we examined whether pitch activations were
present in modality-specific (visual or tactile) versus
multimodal (visual and tactile) regions active during spatial
height processing. Judgments of musical pitch were found to
activate unimodal visual areas, suggesting that space-pitch
associations may involve modality-specific spatial
representations, supporting a key assumption of embodied
theories of metaphorical mental representation.

Abstract

Previous studies have demonstrated that native listeners
modify their interpretation of a speech sound when a talker
produces an ambiguous sound in order to quickly tune into a
speaker, but there is hardly any evidence that non-native
listeners employ a similar mechanism when encountering
ambiguous pronunciations. So far, one study demonstrated
this lexically-guided perceptual learning effect for nonnatives,
using phoneme categories similar in the native
language of the listeners and the non-native language of the
stimulus materials. The present study investigates the question
whether phoneme category retuning is possible in a nonnative
language for a contrast, /l/-/r/, which is phonetically
differently embedded in the native (Dutch) and nonnative
(English) languages involved. Listening experiments indeed
showed a lexically-guided perceptual learning effect.
Assuming that Dutch listeners have different phoneme
categories for the native Dutch and non-native English /r/, as
marked differences between the languages exist for /r/, these
results, for the first time, seem to suggest that listeners are not
only able to retune their native phoneme categories but also
their non-native phoneme categories to include ambiguous
pronunciations.

Abstract

Der vorliegende Beitrag zum Kieler Symposium1 stellt die Resultate eines Teilbereichs meiner Arbeit zum Guarani vor, nämlich einen Vorschlag zur Analyse der Personenpräfixe dieser Sprache und der mit ihnen verbundenen grammatischen Kategorien. Die im Titel angedeutete lexikographische Fragestellung bedarf einer näheren Erläuterung, die ich im Zusammenhang mit einer kurzen Darstellung der Motivation für meine Untersuchungen geben will

Abstract

With the present essay I pursue a threefold aim as will be explained in the following paragraphs. Since I cannot expect my readers to be familiar with the language studied, Guaran´ı, more information about this language will be given in the next subsection.

Abstract

This paper describes the verbal Tense-Aspect-Mood system of Awetí (Tupian, Central Brazil) in a Word-and-Paradigm approach. One classification of Awetí verb forms contains clear aspect categories. A second set of independent classifications renders at least four moods and contains a third major TAM classification, factuality, that has one mainly temporal category Future, while others are partially or wholly modal. Structural categories reflect the formal composition of the forms. Some forms are synthetic, ‘marked’ only by means of affixes, but many are analytic, containing auxiliary particles. With selected sample forms we demonstrate in detail the interplay of structural and functional categories in Awetí verb paradigms.

Abstract

Biologically inspired spiking networks are an important tool to study the nature of computation and cognition in neural systems. In this work, we investigate the representational capacity of spiking networks engaged in an identity mapping task. We compare two schemes for encoding symbolic input, one in which input is injected as a direct current and one where input is delivered as a spatio-temporal spike pattern. We test the ability of networks to discriminate their input as a function of the number of distinct input symbols. We also compare performance using either membrane potentials or filtered spike trains as state variable. Furthermore, we investigate how the circuit behavior depends on the balance between excitation and inhibition, and the degree of synchrony and regularity in its internal dynamics. Finally, we compare different linear methods of decoding population activity onto desired target labels. Overall, our results suggest that even this simple mapping task is strongly influenced by design choices on input encoding, state-variables, circuit characteristics and decoding methods, and these factors can interact in complex ways. This work highlights the importance of constraining computational network models of behavior by available neurobiological evidence.

Abstract

How involved in an event is a person that possesses one of the event participants? Some languages can treat such “external possessors” as very closely involved, even marking them on the verb along with core roles such as subject and object. Other languages only allow possessors to be expressed as non-core participants. This task explores possibilities for the encoding of possessors and other related roles such as beneficiaries. The materials consist of a sequence of thirty drawings designed to elicit target construction types.

Abstract

This chapter reviews the computational processes that are responsible for recognizing word forms in the speech stream. We outline the different stages in a processing hierarchy from the extraction of general acoustic features, through speech‐specific prelexical processes, to the retrieval and selection of lexical representations. We argue that two recurring properties of the system as a whole are abstraction and adaptability. We also present evidence for parallel processing of information on different timescales, more specifically that segmental material in the speech stream (its consonants and vowels) is processed in parallel with suprasegmental material (the prosodic structures of spoken words). We consider evidence from both psycholinguistics and neurobiology wherever possible, and discuss how the two fields are beginning to address common computational problems. The challenge for future research in speech perception will be to build an account that links these computational problems, through functional mechanisms that address them, to neurobiological implementation.

Abstract

This field guide is for eliciting information about grammatical resources used in describing perceptual events and perception-based properties and states. A list of leading questions outlines an underlying semantic space for events/states of perception, against which language-specific constructions may be defined. It should be used as an entry point into a flexible exploration of the structures and constraints which are specific to the language you are working on. The goal is to provide a cross-linguistically comparable description of the constructions of a language used in describing perceptual events and states. The core focus is to discover any sensory asymmetries, i.e., ways in which different sensory modalities are treated differently with respect to these constructions.

Abstract

This chapter discusses some of the elements of human sociality that serve as the social and cognitive infrastructure or preconditions for the use of requests and other kinds of recruitments in interaction. The notion of an agent with goals is a canonical starting point, though importantly agency tends not to be wholly located in individuals, but rather is socially distributed. This is well illustrated in the case of requests, in which the person or group that has a certain goal is not necessarily the one who carries out the behavior towards that goal. The chapter focuses on the role of semiotic (mostly linguistic) resources in negotiating the distribution of agency with request-like actions, with examples from video-recorded interaction in Lao, a language spoken in Laos and nearby countries. The examples illustrate five hallmarks of requesting in human interaction, which show some ways in which our ‘manipulation’ of other people is quite unlike our manipulation of tools: (1) that even though B is being manipulated, B wants to help, (2) that while A is manipulating B now, A may be manipulated in return later; (3) that the goal of the behavior may be shared between A and B, (4) that B may not comply, or may comply differently than requested, due to actual or potential contingencies, and (5) that A and B are accountable to one another; reasons may be asked for, and/or given, for the request. These hallmarks of requesting are grounded in a prosocial framework of human agency.

Abstract

People of all cultures have some degree of concern with categorizing types of communicative social action. All languages have words with meanings like speak, say, talk, complain, curse, promise, accuse, nod, wink, point and chant. But the exact distinctions they make will differ in both quantity and quality. How is communicative social action categorised across languages and cultures? The goal of this task is to establish a basis for cross-linguistic comparison of native metalanguages for social action.

Abstract

This Field Manual entry has been superceded by the 2009 version: https://doi.org/10.17617/2.883564

Abstract

This article introduces a new speech corpus, the Nijmegen Corpus of Casual Czech (NCCCz), which contains more than 30 hours of high-quality recordings of casual conversations in Common Czech, among ten groups of three male and ten groups of three female friends. All speakers were native speakers of Czech, raised in Prague or in the region of Central Bohemia, and were between 19 and 26 years old. Every group of speakers consisted of one confederate, who was instructed to keep the conversations lively, and two speakers naive to the purposes of the recordings. The naive speakers were engaged in conversations for approximately 90 minutes, while the confederate joined them for approximately the last 72 minutes. The corpus was orthographically annotated by experienced transcribers and this orthographic transcription was aligned with the speech signal. In addition, the conversations were videotaped. This corpus can form the basis for all types of research on casual conversations in Czech, including phonetic research and research on how to improve automatic speech recognition. The corpus will be freely available

Abstract

With the aim to clarify the definition of humans as “linguistic animals”, in the present paper I functionally distinguish three types of language competences: i) language as a general biological tool for communication, ii) “perceptual syntax”, iii) propositional language. Following this terminological distinction, I review pivotal findings on animals' communication systems, which constitute useful evidence for the investigation of the nature of three core components of humans' faculty of language: semantics, syntax, and theory of mind. In fact, despite the capacity to process and share utterances with an open-ended structure is uniquely human, some isolated components of our linguistic competence are in common with nonhuman animals. Therefore, as I argue in the present paper, the investigation of animals' communicative competence provide crucial insights into the range of cognitive constraints underlying humans' ability of language, enabling at the same time the analysis of its phylogenetic path as well as of the selective pressures that have led to its emergence.

Abstract

The aim of this study is to investigate the word-learning phenomenon utilizing a new model that integrates three processes: a) extracting a word out of a continuous sounds sequence, b) inducing referential meanings, c) mapping a word onto its intended referent, with the possibility to extend the acquired word over a potentially infinite sets of objects of the same semantic category, and over not-previously-heard utterances. Previous work has examined the role of statistical learning and/or of prosody in each of these processes separately. In order to examine the multilayered word-learning task, we integrate these two strands of investigation into a single approach. We have conducted the study on adults and included six different experimental conditions, each including specific perceptual manipulations of the signal. In condition 1, the only cue to word-meaning mapping was the co-occurrence between words and referents (“statistical cue”). This cue was present in all the conditions. In condition 2, we added infant-directed-speech (IDS) typical pitch enhancement as a marker of the target word and of the statistical cue. In condition 3 we placed IDS typical pitch enhancement on random words of the utterances, i.e. inconsistently matching the statistical cue. In conditions 4, 5 and 6 we manipulated respectively duration, a non-prosodic acoustic cue and a visual cue as markers of the target word and of the statistical cue. Systematic comparisons between learning performance in condition 1 with the other conditions revealed that the word-learning process is facilitated only when pitch prominence consistently marks the target word and the statistical cue…

Abstract

Relative clauses are a syntactic device to create complex sentences and they make language structurally productive. Despite a considerable number of experimental studies, it is still largely unclear how children learn relative clauses and how these are processed in the language system. Researchers at the MPI for Psycholinguistics used a computational learning model to gain novel insights into these issues. The model explains the differential development of relative clauses in English as well as cross-linguistic differences

Abstract

The Cha’palaa language of Ecuador (Barbacoan) features verbal morphology for marking knowledge-based categories that, in usage, show a variant of the cross-linguistically recurrent pattern of ‘egophoric distribution': specific forms associate with speakers in contrast to others in statements and with addressees in contrast to others in questions. These are not person markers, but rather are used by speakers to portray their involvement in states of affairs as active, agentive participants (ego) versus other types of involvement (non-ego). They interact with person and argument structure, but through pragmatic ‘person sensitivities’ rather than through grammatical agreement. Not only does this pattern appear in verbal morphology, it also can be observed in alternations of predicate construction types and case alignment, helping to show how egophoric marking is a pervasive element of Cha'palaa's linguistic system. This chapter gives a first account of egophoricity in Cha’palaa, beginning with a discussion of person sensitivity, egophoric distribution, and issues of flexibility of marking with respect to degree of volition or control. It then focuses on a set of intransitive experiencer (or ‘endopathic') predicates that refer to internal states which mark egophoric values for the undergoer role, not the actor role, showing ‘quirky’ accusative marking instead of nominative case. It concludes with a summary of how egophoricity in Cha'palaa interacts with issues of argument structure in comparison to a language with person agreement, here represented by examples from Cha’palaa’s neighbor Ecuadorian Highland Quechua.

Abstract

This chapter connects the grammar of the first person collective pronoun in the Cha’palaa language of Ecuador with its use in interaction for collective reference and social category membership attribution, addressing the problem posed by the fact that non-singular pronouns do not have distributional semantics (“speakers”) but are rather associational (“speaker and relevant associates”). It advocates a cross-disciplinary approach that jointly considers elements of linguistic form, situated usages of those forms in instances of interaction, and the broader ethnographic context of those instances. Focusing on large-scale and relatively stable categories such as racial and ethnic groups, it argues that looking at how speakers categorize themselves and others in the speech situation by using pronouns provides empirical data on the status of macro-social categories for members of a society

Abstract

Structural imaging based on computerized tomography (CT) and magnetic resonance imaging (MRI) has progressively replaced traditional post‐mortem studies in the process of identifying the neuroanatomical basis of language. In the clinical setting, the information provided by structural imaging has been used to confirm the exact diagnosis and formulate an individualized treatment plan. In the research arena, neuroimaging has permitted to understand neuroanatomy at the individual and group level. The possibility to obtain quantitative measures of lesions has improved correlation analyses between severity of symptoms, lesion load, and lesion location. More recently, the development of structural imaging based on diffusion MRI has provided valid solutions to two major limitations of more conventional imaging. In stroke patients, diffusion can visualize early changes due to a stroke that are otherwise not detectable with more conventional structural imaging, with important implications for the clinical management of acute stroke patients. Beyond the sensitivity to early changes, diffusion imaging tractography presents the possibility of visualizing the trajectories of individual white matter pathways connecting distant regions. A pathway analysis based on tractography is offering a new perspective in neurolinguistics. First, it permits to formulate new anatomical models of language function in the healthy brain and allows to directly test these models in the human population without any reliance on animal models. Second, by defining the exact location of the damage to specific white matter connections we can understand the contribution of different mechanisms to the emergence of language deficits (e.g., cortical versus disconnection mechanisms). Finally, a better understanding of the anatomical variability of different language networks is helping to identify new anatomical predictors of language recovery. In this chapter we will focus on the principles of structural MRI and, in particular, diffusion imaging and tractography and present examples of how these methods have informed our understanding of variance in language performances in the healthy brain and language deficits in patient populations.

Abstract

Reading is an audiovisual process that requires the learning of systematic links between graphemes and phonemes. It is thus possible that reading impairments reflect an audiovisual processing deficit. In this study, we compared audiovisual processing in adults with developmental dyslexia and adults without reading difficulties. We focused on differences in cross-modal temporal sensitivity both for speech and for non-speech events. When compared to adults without reading difficulties, adults with developmental dyslexia presented a wider temporal window in which unsynchronized speech events were perceived as synchronized. No differences were found between groups for the non-speech events. These results suggests a deficit in dyslexia in the perception of cross-modal temporal synchrony for speech events.

Abstract

We analyze the problem of response suggestion in a closed domain along a real-world scenario of a digital library. We present a text-processing pipeline to generate question-answer pairs from chat transcripts. On this limited amount of training data, we compare retrieval-based, conditioned-generation, and dedicated representation learning approaches for response suggestion. Our results show that retrieval-based methods that strive to find similar, known contexts are preferable over parametric approaches from the conditioned-generation family, when the training data is limited. We, however, identify a specific representation learning approach that is competitive to the retrieval-based approaches despite the training data limitation.

Abstract

We present multi-modal adversarial autoencoders for recommendation and evaluate them on two different tasks: citation recommendation and subject label recommendation. We analyze the effects of adversarial regularization, sparsity, and different input modalities. By conducting 408 experiments, we show that adversarial regularization consistently improves the performance of autoencoders for recommendation. We demonstrate, however, that the two tasks differ in the semantics of item co-occurrence in the sense that item co-occurrence resembles relatedness in case of citations, yet implies diversity in case of subject labels. Our results reveal that supplying the partial item set as input is only helpful, when item co-occurrence resembles relatedness. When facing a new recommendation task it is therefore crucial to consider the semantics of item co-occurrence for the choice of an appropriate model.

Abstract

Researchers have long avoided neurophysiological experiments of overt speech production due to the suspicion that artifacts caused by muscle activity may lead to a bad signal-to-noise ratio in the measurements. However, the need to actually produce speech may influence earlier processing and qualitatively change speech production processes and what we can infer from neurophysiological measures thereof. Recently, however, overt speech has been successfully investigated using EEG, MEG, and fMRI. The aim of this Research Topic is to draw together recent research on the neurophysiological basis of language production, with the aim of developing and extending theoretical accounts of the language production process. In this Research Topic of Frontiers in Language Sciences, we invite both experimental and review papers, as well as those about the latest methods in acquisition and analysis of overt language production data. All aspects of language production are welcome: i.e., from conceptualization to articulation during native as well as multilingual language production. Focus should be placed on using the neurophysiological data to inform questions about the processing stages of language production. In addition, emphasis should be placed on the extent to which the identified components of the electrophysiological signal (e.g., ERP/ERF, neuronal oscillations, etc.), brain areas or networks are related to language comprehension and other cognitive domains. By bringing together electrophysiological and neuroimaging evidence on language production mechanisms, a more complete picture of the locus of language production processes and their temporal and neurophysiological signatures will emerge.

Abstract

This paper reports on a multilingual investigation into the effects of different masker types on native and non-native perception in a VCV consonant recognition task. Native listeners outperformed 7 other language groups, but all groups showed a similar ranking of maskers. Strong first language (L1) interference was observed, both from the sound system and from the L1 orthography. Universal acoustic-perceptual tendencies are also at work in both native and non-native sound identifications in noise. The effect of linguistic distance, however, was less clear: in large multilingual studies, listener variables may overpower other factors.

Abstract

We present a solution to the problem of online speaker/signer diarization - the task of determining "who spoke/signed when?". Our solution is based on the idea that gestural activity (hands and body movement) is highly correlated with uttering activity. This correlation is necessarily true for sign languages and mostly true for spoken languages. The novel part of our solution is the use of motion history images (MHI) as a likelihood measure for probabilistically detecting uttering activities. MHI is an efficient representation of where and how motion occurred for a fixed period of time. We conducted experiments on 4.9 hours of a publicly available dataset (the AMI meeting data) and 1.4 hours of sign language dataset (Kata Kolok data). The best performance obtained is 15.70% for sign language and 31.90% for spoken language (measurements are in DER). These results show that our solution is applicable in real-world applications like video conferences.

Abstract

We demonstrate how the problem of speaker diarization can be solved using both gesture and speaker parametric models. The novelty of our solution is that we approach the speaker diarization problem as a speaker recognition problem after learning speaker models from speech samples corresponding to gestures (the occurrence of gestures indicates the presence of speech and the location of gestures indicates the identity of the speaker). This new approach offers many advantages: comparable state-of-the-art performance, faster computation and more adaptability. In our implementation, parametric models are used to model speakers' voice and their gestures: more specifically, Gaussian mixture models are used to model the voice characteristics of each person and all persons, and gamma distributions are used to model gestural activity based on features extracted from Motion History Images. Tests on 4.24 hours of the AMI meeting data show that our solution makes DER score improvements of 19% on speech-only segments and 4% on all segments including silence (the comparison is with the AMI system).

Abstract

Prior research on language identification focused primarily on text and speech. In this paper, we focus on the visual modality and present a method for identifying sign languages solely from short video samples. The method is trained on unlabelled video data (unsupervised feature learning) and using these features, it is trained to discriminate between six sign languages (supervised learning). We ran experiments on video samples involving 30 signers (running for a total of 6 hours). Using leave-one-signer-out cross-validation, our evaluation on short video samples shows an average best accuracy of 84%. Given that sign languages are under-resourced, unsupervised feature learning techniques are the right tools and our results indicate that this is realistic for sign language identification.

Abstract

To understand the underlying principles of categorisation and classification of sensory input semantic analyses must be based on both language and culture. The senses are not only physiological phenomena, but they are also linguistic, cultural, and social. The goal of this task is to explore and describe sociocultural patterns relating language of perception, ideologies of perception, and perceptual practice in our speech communities.

Abstract

What does semantic similarity between two concepts mean? How could we measure it? The way in which semantic similarity is calculated might differ depending on the theoretical notion of semantic representation. In an eye-tracking reading experiment, we investigated whether two widely used semantic similarity measures (based on featural or distributional representations) have distinctive effects on sentence reading times. In other words, we explored whether these measures of semantic similarity differ qualitatively. In addition, we examined whether visually perceived spatial distance interacts with either or both of these measures. Our results showed that the effect of featural and distributional representations on reading times can differ both in direction and in its time course. Moreover, both featural and distributional information interacted with spatial distance, yet in different sentence regions and reading measures. We conclude that featural and distributional representations are distinct components of semantic representation.

Abstract

Recent evidence from eye tracking during reading showed that non-referential spatial distance presented in a visual context can modulate semantic interpretation of similarity relations rapidly and incrementally. In two eye-tracking reading experiments we extended these findings in two important ways; first, we examined whether other semantic domains (social relations) could also be rapidly influenced by spatial distance during sentence comprehension. Second, we aimed to further specify how abstract language is co-indexed with spatial information by varying the syntactic structure of sentences between experiments. Spatial distance rapidly modulated reading times as a function of the social relation expressed by a sentence. Moreover, our findings suggest that abstract language can be co-indexed as soon as critical information becomes available for the reader.

Abstract

This chapter explores what L2 learners' gestures reveal about L2 grammar. The focus is on learners’ difficulties with maintaining reference in discourse caused by their incomplete mastery of pronouns. The study highlights the systematic parallels between properties of L2 speech and gesture, and the parallel effects of grammatical development in both modalities. The validity of a communicative account of interlanguage grammar in this domain is tested by taking the cohesive properties of the gesture-speech ensemble into account. Specifically, I investigate whether learners use gestures to compensate for and to license over-explicit reference in speech. The results rule out a communicative account for the spoken variety of maintained reference. In contrast, cohesive gestures are found to be multi-functional. While the presence of cohesive gestures is not communicatively motivated, their spatial realisation is. It is suggested that gestures are exploited as a grammatical communication strategy to disambiguate speech wherever possible, but that they may also be doing speaker-internal work. The methodological importance of considering L2 gestures when studying grammar is also discussed.

Abstract

Gestures, the symbolic movements speakers perform while they speak, are systematically related to speech and language at multiple levels, and reflect cognitive and linguistic activities in non-trivial ways. This chapter presents an overview of what gestures can tell us about the processes of second language acquisition. It focuses on two key aspects, (a) gestures and the developing language system and (b) gestures and learning, and discusses some implications of an expanded view of language acquisition that takes gestures into account.

Abstract

Three groups of monolingual listeners, with Standard Chinese, Dutch and Hungarian as their native language, judged pairs of trisyllabic stimuli which differed only in their itch pattern. The segmental structure of the stimuli was made up by the experimenters and presented to subjects as being taken from a little-known language spoken on a South Pacific island. Pitch patterns consisted of a single rise-fall located on or near the second syllable. By and large, listeners selected the stimulus with the higher peak, the later eak, and the higher end rise as the one that signalled a question, regardless of language group. The result is argued to reflect innate, non-linguistic knowledge of the meaning of pitch variation, notably Ohala’s Frequency Code. A significant difference between groups is explained as due to the influence of the mother tongue.