Presentations

Abstract

Theories of reading propose that the quality of word form representations affects reading comprehension. One claim is that synchronous retrieval of orthographic and phonological representations leads to better performance than asynchronous retrieval. Based on this account, one may hypothesize that synchronous rather than asynchronous presentation of orthographic and phonological forms should be beneficial when establishing the mapping between both, as it should lead to tighter couplings. We tested this hypothesis in two multi-session experiments, where participants studied isolated words of a tonal language unknown to them, Chinese. During study, written (using Pinyin transcription) and spoken word forms were presented simultaneously or in asynchronous fashion (audio-first, written-first). In both experiments, we observed an advantage for asynchronous over synchronous presentation at test, with audio-first presentation being most beneficial. These results suggest that the timing of written and spoken word forms has profound effects on the ease of learning a new tonal language.

Abstract

Prominent theories of predictive language processing assume that language production processes are used to anticipate upcoming linguistic input during comprehension (Dell & Chang, 2014; Pickering & Garrod, 2013). Here, we explore the converse case: Does a task set including production in addition to comprehension encourage prediction, compared to a task only including comprehension? To test this hypothesis, we conducted a cross-modal naming experiment (Experiment 1) including an object naming task and a self-paced reading experiment (Experiment 2) that did not include overt production. We used the same predictable (N = 40) and non-predictable (N = 40) sentences in both experiments. The sentences consisted of a fixed agent, a transitive verb and a predictable or non-predictable target word (The man drinks a beer vs. The man buys a beer). Most of the empirical work on prediction used sentences in which the target words were highly predictable (often with a mean cloze probability > .8) and thus it is little surprising that participants engaged in predictive language processing very easily. In the current sentences, the mean cloze probability in the predictable sentences was .39 (ranging from .06 to .8; zero in the non-predictable sentences). If comprehenders are more likely to engage in predictive processing when the task set involves production, we should observe more pronounced effects of prediction in Experiment 1 as compared to Experiment 2. If production does not enhance prediction, we should observe similar effects of prediction in both experiments. In Experiment 1, participants (N = 54) listened to recordings of the sentences which ended right before the spoken target word. Coinciding with the end of the playback, a picture of the target word was shown which the participants were asked to name as fast as possible. Analyses of their naming latencies revealed a statistically significant naming advantage of 106 ms on predictable over non-predictable trials. Moreover, we found that the objects’ naming advantage was predicted by the target words’ cloze probability in the sentences (r = .411, p = .016). In Experiment 2, the same sentences were used in a self-paced reading experiment. To allow for testing of potential spill-over effects, we added a neutral prepositional phrase (buys a beer from the bar keeper/drinks a beer from the shop) to each sentence. Participants (N = 54) read the sentences word-by-word, advancing by pushing the space bar. On 30% of the trials, comprehension questions were used to keep up participants' focus on comprehending the sentences. Analyses of participants’ target and post-target reading times revealed numerical advantages of 6 ms and 20 ms, respectively, in the predictable as compared to the non-predictable condition. However, in both cases, this difference was not statistically reliable (t = .757, t = 1.43) and the significant positive correlation between an item’s naming advantage and its cloze probability as seen in Experiment 1 was absent (r = .037, p = .822). Importantly, the analysis of participants' responses to the comprehension questions, showed that they understood the sentences (mean accuracy = 93%). To conclude, although both experiments used the same sentences, we observed effects of prediction only when the task included production. In Experiment 2, no evidence for anticipation was found although participants clearly understood the sentences and the method has previously been shown to be sensitive to measure prediction effects (Van Berkum et al., 2005). Our results fit with a recent study by Gollan et al. (2011) who found only a small processing advantage of predictive over non-predictive sentences in reading (using highly predictable sentences with a cloze probability > . 87) but a strong prediction effect when participants read the same sentences and carried out an additional object naming task (see also Griffin & Bock, 1998). Taken together, the studies suggest that the comprehenders' task set exerts a powerful influence on the likelihood and magnitude of predictive language processing. When the task set involves language production, as is often the case in natural conversation, comprehenders might engage in prediction to a stronger degree than in pure comprehension tasks. Being able to predict words another person is about to say might optimize the comprehension process and enable smooth turn-taking.

Abstract

A central claim of embodied theories of cognition is that sensory representations are
routinely activated and influence language processing even in the absence of relevant
sensory input (cf. Pulvermüller, 2005; Wassenburg & Zwaan, 2010). We tested the influence
of color representations during language processing in three visual world eye tracking
experiments. The method is particularly well suited to investigate this issue because the
availability of relevant visual input can be manipulated.
We made use of the phenomena that when participants hear a word that refers to a
visual object or printed word they quickly direct their eye gaze to objects or printed words
which are similar (e.g. semantically or visually) to the heard word. We used a look and listen
task which previously has been shown to be sensitive to such relationships between spoken
words and visual items. In Experiment 1, on experimental trials, participants listened to
sentences containing a critical target word associated with a prototypical color (e.g.
'...spinach...') as they inspected a visual display with four words printed in black font. One of
the four printed words was associated with the same prototypical color (e.g. green) as the
spoken target word (e.g. FROG). On experimental trials, the spoken target word did not have
a printed word counterpart (SPINACH was not present in the display). In filler trials (70% of
trials) the target was present in the display and attracted significantly more overt attention
than the unrelated distractors. In experimental trials color competitors were not looked at
more than the distractors. In Experiment 2 the printed words were replaced with line
drawings of the objects. In order to direct the attentional focus of our participants toward
color features we used a within-participants counter-balanced design and alternated color
and greyscale trials randomly throughout the experiment. Therefore, on one trial our
participants heard a word such as 'spinach' and saw a frog (colored in green) in the visual
display. On the next trial however they saw a banana (in greyscale) on hearing 'canary'
(bananas and canaries are typically yellow), etc. The presence (or absence) of color was
thus a salient property of the experiment. Participants looked more at color competitors than
unrelated distractors on hearing the target word in the color trials but not in the greyscale
trials, i.e. on hearing 'spinach' they looked at the green frog but not the greyscale frog.
Experiment 3 was identical to Experiment 2, except that the visual display was removed at
the sentence onset, after a longer preview. This experiment examined whether the continued
presence of color in the immediate visual environment was necessary for the observation of
color-mediated eye movements. Eye movements directed towards the now blank screen
were recorded as the sentence unfolded (cf. Spivey & Geng, 2001). In the filler trials,
participants looked significantly more at the locations where the targets, rather than the
distractors, had been previously presented as the target words acoustically unfolded. In the
experimental trials, the locations where the color competitors had previously been presented
did not attract increased attention (neither in color nor greyscale trials).
These data demonstrate that language-mediated eye movements are only influenced
by color relations between spoken words and visually displayed items if color is present in the immediate visual environment. We conclude that color representations are unlikely to be
routinely activated in language processing. Our findings provide strong constraints for
embodied theories of cognition which assume that sensory representations influence language processing even in the absence of relevant sensory input. These results fit best with the notion that the main role of sensory representations in language processing is a different one, namely to contextualize language in the immediate environment, connecting language to the here and now.

Abstract

The notion that predicting upcoming linguistic information in language comprehension makes use of the production system has recently received much attention (e.g., Chang et al., 2006; Dell & Chang, 2014; Federmeier, 2007; Pickering & Garrod, 2007, 2013; Van Berkum et al., 2005). So far there has been little experimental evidence for a relation between prediction and production. I will discuss the results of several recent eye-tracking experiments with toddlers (Mani & Huettig, 2012) and adults (Rommers et al. submitted, Hintz et al., in prep.) which provide some support for the view that production abilities are linked to language-mediated anticipatory eye movements. These data however also indicate that production-based prediction is situation-dependent and only one of many mechanisms supporting prediction. Taken together, these results suggest that multiple-mechanism accounts are required to provide a complete picture of anticipatory language processing.