Context-dependent mapping of linguistic and color representations challenges strong forms of embodiment

Huettig, F., & Guerra, E. (2014). Context-dependent mapping of linguistic and color representations challenges strong forms of embodiment. Talk presented at the 20th Architectures and Mechanisms for Language Processing Conference (AMLAP 2014). Edinburgh, UK. 2014-09-03 - 2014-09-06.
A central claim of embodied theories of cognition is that sensory representations are routinely activated and influence language processing even in the absence of relevant sensory input (cf. Pulvermüller, 2005; Wassenburg & Zwaan, 2010). We tested the influence of color representations during language processing in three visual world eye tracking experiments. The method is particularly well suited to investigate this issue because the availability of relevant visual input can be manipulated. We made use of the phenomena that when participants hear a word that refers to a visual object or printed word they quickly direct their eye gaze to objects or printed words which are similar (e.g. semantically or visually) to the heard word. We used a look and listen task which previously has been shown to be sensitive to such relationships between spoken words and visual items. In Experiment 1, on experimental trials, participants listened to sentences containing a critical target word associated with a prototypical color (e.g. '...spinach...') as they inspected a visual display with four words printed in black font. One of the four printed words was associated with the same prototypical color (e.g. green) as the spoken target word (e.g. FROG). On experimental trials, the spoken target word did not have a printed word counterpart (SPINACH was not present in the display). In filler trials (70% of trials) the target was present in the display and attracted significantly more overt attention than the unrelated distractors. In experimental trials color competitors were not looked at more than the distractors. In Experiment 2 the printed words were replaced with line drawings of the objects. In order to direct the attentional focus of our participants toward color features we used a within-participants counter-balanced design and alternated color and greyscale trials randomly throughout the experiment. Therefore, on one trial our participants heard a word such as 'spinach' and saw a frog (colored in green) in the visual display. On the next trial however they saw a banana (in greyscale) on hearing 'canary' (bananas and canaries are typically yellow), etc. The presence (or absence) of color was thus a salient property of the experiment. Participants looked more at color competitors than unrelated distractors on hearing the target word in the color trials but not in the greyscale trials, i.e. on hearing 'spinach' they looked at the green frog but not the greyscale frog. Experiment 3 was identical to Experiment 2, except that the visual display was removed at the sentence onset, after a longer preview. This experiment examined whether the continued presence of color in the immediate visual environment was necessary for the observation of color-mediated eye movements. Eye movements directed towards the now blank screen were recorded as the sentence unfolded (cf. Spivey & Geng, 2001). In the filler trials, participants looked significantly more at the locations where the targets, rather than the distractors, had been previously presented as the target words acoustically unfolded. In the experimental trials, the locations where the color competitors had previously been presented did not attract increased attention (neither in color nor greyscale trials). These data demonstrate that language-mediated eye movements are only influenced by color relations between spoken words and visually displayed items if color is present in the immediate visual environment. We conclude that color representations are unlikely to be routinely activated in language processing. Our findings provide strong constraints for embodied theories of cognition which assume that sensory representations influence language processing even in the absence of relevant sensory input. These results fit best with the notion that the main role of sensory representations in language processing is a different one, namely to contextualize language in the immediate environment, connecting language to the here and now.
Publication type
Publication date

Share this page