Publications

Abstract

We introduce a new method for measuring prediction and language-vision interactions: tracking the trajectories of hand-reaching movements in Virtual Reality (VR) environments. Spatiotemporal trajectory tracking of hand-reaching movements in VR offers an ecologically valid yet controlled medium for conducting experiments in an environment that mirrors characteristics of real-world behaviors. Importantly, it enables tracking the continuous dynamics of processing on a single-trial level. In an exploratory experiment L2 speakers heard predictive or non-predictive sentences (e.g., “The barber cuts the hair” vs. “The coach remembers the hair”). Participants’ task was to move their hands as quickly and as accurately as possible towards the object most relevant to the sentence. We measured reaction times (RTs) and hand-reaching trajectories as indicators of predictive behavior. There was a main effect of predictability: Predictable items were touched faster than unpredictable ones. Importantly, uncertainty was captured using spatiotemporal survival analysis by prolonged fluctuations in upward and downward vertical hand movements before making a final move to target or distractor. Self-correction of prediction errors was revealed by participants switching the direction of hand-reaching movements mid-trial. We conclude that the Virtual Hand Paradigm enables measuring the onset and dynamics of predictive behavior in real time in single and averaged trial data and captures (un)certainty about target objects and the self-correction of prediction error online in ‘close to real-world’ experimental settings. The new method has great potential to provide additional insights about time-course and intermediate states of processing, provisional interpretations and partial target commitments that go beyond other state-of-the-art methods.

Abstract

We examined the morphosyntactic prediction ability of child heritage speakers and the role of reading skills and language experience in predictive processing. Using visual world eye-tracking, we focused on predictive use of case-marking cues in Turkish with monolingual (N=49, Mage=83 months) and heritage children, who were early bilinguals of Turkish and Dutch (N=30, Mage=90 months). We found quantitative differences in magnitude of the prediction ability of monolingual and heritage children; however, their overall prediction ability was on par. The heritage speakers’ prediction ability was facilitated by their reading skills in Dutch, but not in Turkish as well as by their heritage language exposure, but not by engagement in literacy activities. These findings emphasize the facilitatory role of reading skills and spoken language experience in predictive processing. This study is the first to show that in a developing bilingual mind, effects of reading-on-prediction can take place across modalities and across languages.

Abstract

Dealing with text is crucial in modern societies. However, not everyone acquires sufficient literacy skills during school education. This systematic review summarizes and synthesizes research on adults with low literacy skills (ALLS) in alphabetic writing systems, includes results from behavioral and neurobiological studies, and compares these findings with those on developmental dyslexia given that this developmental disorder is one possible explanation for low literacy skills in adulthood. Twenty-seven studies focusing on the cognitive profile of ALLS met the three predefined criteria of reading level, age, and education. Results showed that ALLS performed worse than literate adults in various tasks at skill and information processing level, and exhibited structural and functional differences at the neurobiological level. The cognitive profile of ALLS was closer to that of primary school children than of literate adults. However, relative to children, ALLS’ literacy skills relied less on phonological and more on orthographic strategies. A narrative comparison of results with meta-analyses on developmental dyslexia showed large, though not complete, overlap in the cognitive profiles. The present results helps to better understand the literacy skills and reading-related cognitive functions of ALLS and may support the development of tailored interventions directed to the specific cognitive difficulties ALLS have.

Abstract

Previous studies, using the Continuous Flash Suppression (CFS) paradigm, observed that (Western) university students are better able to detect otherwise invisible pictures of objects when they are presented with the corresponding spoken word shortly before the picture appears. Here we attempted to replicate this effect with non-Western university students in Goa (India). A second aim was to explore the performance of (non-Western) meditators practicing Sudarshan Kriya Yoga in Goa in the same task. Some previous literature suggests that meditators may excel in some tasks that tap visual attention, for example by exercising better endogenous and exogenous control of visual awareness than non-meditators. The present study replicated the finding that congruent spoken cue words lead to significantly higher detection sensitivity than incongruent cue words in non-Western university students. Our exploratory meditator group also showed this detection effect but both frequentist and Bayesian analyses suggest that the practice of meditation did not modulate it. Overall, our results provide further support for the notion that spoken words can activate low-level category-specific visual features that boost the basic capacity to detect the presence of a visual stimulus that has those features. Further research is required to conclusively test whether meditation can modulate visual detection abilities in CFS and similar tasks.

Abstract

In the present paper we describe the Enhanced Literate Mind (ELM) hypothesis. As individuals learn to read and write, they are, from then on, exposed to extensive written-language input and become literate. We propose that acquisition and proficient processing of written language (‘literacy’) leads to, both, increased language knowledge as well as enhanced language and non-language (perceptual and cognitive) skills. We also suggest that all neurotypical native language users, including illiterate, low literate, and high literate individuals, share a Basic Language Cognition (BLC) in the domain of oral informal language. Finally, we discuss the possibility that the acquisition of ELM leads to some degree of ‘knowledge parallelism’ between BLC and ELM in literate language users, which has implications for empirical research on individual and situational differences in spoken language processing.

Abstract

Literacy is in decline in many parts of the world, accompanied by drops in associated cognitive skills (including IQ) and an increasing susceptibility to fake news. It is possible that the recent explosive growth and widespread deployment of Large Language Models (LLMs) might exacerbate this trend, but there is also a chance that LLMs can help turn things around. We argue that cognitive science is ideally suited to help steer future literacy development in the right direction by challenging and informing current educational practices and policy. Cognitive scientists have the right interdisciplinary skills to study, analyze, evaluate, and change LLMs to facilitate their critical use, to encourage turn-taking that promotes rather than hinders literacy, to support literacy acquisition in diverse and equitable ways, and to scaffold potential future changes in what it means to be literate. We urge cognitive scientists to take up this mantle—the future impact of LLMs on human literacy skills is too important to be left to the large, predominately US-based tech companies.

Abstract

We investigated prediction skills of adult heritage speakers and the role of written and spoken language experience on predictive processing. Using visual world eye-tracking, we focused on predictive use of case-marking cues in verb-medial and verb-final sentences in Turkish with adult Turkish heritage speakers (N = 25) and Turkish monolingual speakers (N = 24). Heritage speakers predicted in verb-medial sentences (when verb-semantic and case-marking cues were available), but not in verb-final sentences (when only case-marking cues were available) while monolinguals predicted in both. Prediction skills of heritage speakers were modulated by their spoken language experience in Turkish and written language experience in both languages. Overall, these results strongly suggest that verb-semantic information is needed to scaffold the use of morphosyntactic cues for prediction in heritage speakers. The findings also support the notion that both spoken and written language experience play an important role in predictive spoken language processing.

Abstract

Experiments 1 and 2 examined the time-course of retrieval of phonological, visual-shape and semantic knowledge as Dutch participants listened to sentences and looked at displays of four pictures. Given a sentence with beker, `beaker', for example, the display contained phonological (a beaver, bever), shape (a bobbin, klos), and semantic (a fork, vork) competitors. When the display appeared at sentence onset, fixations to phonological competitors preceded fixations to shape and semantic competitors. When display onset was 200 ms before (e.g.) beker, fixations were directed to shape and then semantic competitors, but not phonological competitors. In Experiments 3 and 4, displays contained the printed names of the previously-pictured entities; only phonological competitors were fixated preferentially. These findings suggest that retrieval of phonological, shape and semantic knowledge in the spoken-word and picture-recognition systems is cascaded, and that visual attention shifts are co-determined by the time-course of retrieval of all three knowledge types and by the nature of the information in the visual environment.

Abstract

Visual attention can be directed immediately, as a spoken word unfolds, towards conceptually related but nonassociated objects, even if they mismatch on other dimensions that would normally determine which objects in the scene were appropriate referents for the unfolding word (Huettig & Altmann, 2005). Here we demonstrate that the mapping between language and concurrent visual objects can also be mediated by visual-shape relations. On hearing "snake", participants directed overt attention immediately, within a visual display depicting four objects, to a picture of an electric cable, although participants had viewed the visual display with four objects for approximately 5 s before hearing the target word - sufficient time to recognize the objects for what they were. The time spent fixating the cable correlated significantly with ratings of the visual similarity between snakes in general and this particular cable. Importantly, with sentences contextually biased towards the concept snake, participants looked at the snake well before the onset of "snake", but they did not look at the visually similar cable until hearing "snake". Finally, we demonstrate that such activation can, under certain circumstances (e.g., during the processing of dominant meanings of homonyms), constrain the direction of visual attention even when it is clearly contextually inappropriate. We conclude that language-mediated attention can be guided by a visual match between spoken words and visual objects, but that such a match is based on lexical input and may not be modulated by contextual appropriateness.