Publications

Abstract

We introduce a new method for measuring prediction and language-vision interactions: tracking the trajectories of hand-reaching movements in Virtual Reality (VR) environments. Spatiotemporal trajectory tracking of hand-reaching movements in VR offers an ecologically valid yet controlled medium for conducting experiments in an environment that mirrors characteristics of real-world behaviors. Importantly, it enables tracking the continuous dynamics of processing on a single-trial level. In an exploratory experiment L2 speakers heard predictive or non-predictive sentences (e.g., “The barber cuts the hair” vs. “The coach remembers the hair”). Participants’ task was to move their hands as quickly and as accurately as possible towards the object most relevant to the sentence. We measured reaction times (RTs) and hand-reaching trajectories as indicators of predictive behavior. There was a main effect of predictability: Predictable items were touched faster than unpredictable ones. Importantly, uncertainty was captured using spatiotemporal survival analysis by prolonged fluctuations in upward and downward vertical hand movements before making a final move to target or distractor. Self-correction of prediction errors was revealed by participants switching the direction of hand-reaching movements mid-trial. We conclude that the Virtual Hand Paradigm enables measuring the onset and dynamics of predictive behavior in real time in single and averaged trial data and captures (un)certainty about target objects and the self-correction of prediction error online in ‘close to real-world’ experimental settings. The new method has great potential to provide additional insights about time-course and intermediate states of processing, provisional interpretations and partial target commitments that go beyond other state-of-the-art methods.

Abstract

We examined the morphosyntactic prediction ability of child heritage speakers and the role of reading skills and language experience in predictive processing. Using visual world eye-tracking, we focused on predictive use of case-marking cues in Turkish with monolingual (N=49, Mage=83 months) and heritage children, who were early bilinguals of Turkish and Dutch (N=30, Mage=90 months). We found quantitative differences in magnitude of the prediction ability of monolingual and heritage children; however, their overall prediction ability was on par. The heritage speakers’ prediction ability was facilitated by their reading skills in Dutch, but not in Turkish as well as by their heritage language exposure, but not by engagement in literacy activities. These findings emphasize the facilitatory role of reading skills and spoken language experience in predictive processing. This study is the first to show that in a developing bilingual mind, effects of reading-on-prediction can take place across modalities and across languages.

Abstract

Dealing with text is crucial in modern societies. However, not everyone acquires sufficient literacy skills during school education. This systematic review summarizes and synthesizes research on adults with low literacy skills (ALLS) in alphabetic writing systems, includes results from behavioral and neurobiological studies, and compares these findings with those on developmental dyslexia given that this developmental disorder is one possible explanation for low literacy skills in adulthood. Twenty-seven studies focusing on the cognitive profile of ALLS met the three predefined criteria of reading level, age, and education. Results showed that ALLS performed worse than literate adults in various tasks at skill and information processing level, and exhibited structural and functional differences at the neurobiological level. The cognitive profile of ALLS was closer to that of primary school children than of literate adults. However, relative to children, ALLS’ literacy skills relied less on phonological and more on orthographic strategies. A narrative comparison of results with meta-analyses on developmental dyslexia showed large, though not complete, overlap in the cognitive profiles. The present results helps to better understand the literacy skills and reading-related cognitive functions of ALLS and may support the development of tailored interventions directed to the specific cognitive difficulties ALLS have.

Abstract

Introduction to the special issue 'The Effects of Literacy on Cognition and Brain Functioning'

Abstract

The cause of developmental dyslexia is still unknown despite decades of intense research. Many causal explanations have been proposed, based on the range of impairments displayed by affected individuals. Here we draw attention to the fact that many of these impairments are also shown by illiterate individuals who have not received any or very little reading instruction. We suggest that this fact may not be coincidental and that the performance differences of both illiterates and individuals with dyslexia compared to literate controls are, to a substantial extent, secondary consequences of either reduced or suboptimal reading experience or a combination of both. The search for the primary causes of reading impairments will make progress if the consequences of quantitative and qualitative differences in reading experience are better taken into account and not mistaken for the causes of reading disorders. We close by providing four recommendations for future research.

Abstract

Do people routinely pre-activate the meaning and even the phonological form of upcoming words? The most acclaimed evidence for phonological prediction comes from a 2005 Nature Neuroscience publication by DeLong, Urbach and Kutas, who observed a graded modulation of electrical brain potentials (N400) to nouns and preceding articles by the probability that people use a word to continue the sentence fragment (‘cloze’). In our direct replication study spanning 9 laboratories (N=334), pre-registered replication-analyses and exploratory Bayes factor analyses successfully replicated the noun-results but, crucially, not the article-results. Pre-registered single-trial analyses also yielded a statistically significant effect for the nouns but not the articles. Exploratory Bayesian single-trial analyses showed that the article-effect may be non-zero but is likely far smaller than originally reported and too small to observe without very large sample sizes. Our results do not support the view that readers routinely pre-activate the phonological form of predictable words.

Abstract

Implicit up/down words, such as bird and foot, systematically influence performance on visual tasks involving immediately following targets in compatible vs. incompatible locations. Recent studies have observed that the semantic relation between prime words and target pictures can strongly influence the size and even the direction of the effect: Semantically related targets are processed faster in congruent vs. incongruent locations (location-specific priming), whereas unrelated targets are processed slower in congruent locations. Here, we used eye-tracking to investigate the moment-to-moment processes underlying this pattern. Our reaction time results for related targets replicated the location-specific priming effect and showed a trend towards interference for unrelated targets. We then used growth curve analysis to test how up/down words and their match vs. mismatch with immediately following targets in terms of semantics and vertical location influences concurrent saccadic eye movements. There was a strong main effect of spatial association on linear growth with up words biasing changes in y-coordinates over time upwards relative to down words (and vice versa). Similar to the RT data, this effect was strongest for semantically related targets and reversed for unrelated targets. Intriguingly, all conditions showed a bias in the congruent direction in the initial stage of the saccade. Then, at around halfway into the saccade the effect kept increasing in the semantically related condition, and reversed in the unrelated condition. These results suggest that online processing of up/down words triggers direction-specific oculomotor processes that are dynamically modulated by the semantic relation between prime words and targets.