Falk Huettig

Presentations

Displaying 1 - 41 of 41
  • Favier, S., Meyer, A. S., & Huettig, F. (2018). Does literacy predict individual differences in the syntactic processing of spoken language?. Poster presented at the 1st Workshop on Cognitive Science of Culture, Lisbon, Portugal.
  • Favier, S., Meyer, A. S., & Huettig, F. (2018). Does reading ability predict individual differences in spoken language syntactic processing?. Poster presented at the International Meeting of the Psychonomics Society 2018, Amsterdam, The Netherlands.
  • Favier, S., Meyer, A. S., & Huettig, F. (2018). How does literacy influence syntactic processing in spoken language?. Talk presented at Psycholinguistics in Flanders (PiF 2018). Gent, Belgium. 2018-06-04 - 2018-06-05.
  • Garrido Rodriguez, G., Huettig, F., Norcliffe, E., Brown, P., & Levinson, S. C. (2018). Participant assignment to thematic roles in Tzeltal: Eye tracking evidence from sentence comprehension in a verb-initial language. Talk presented at Architectures and Mechanisms for Language Processing (AMLaP 2018). Berlin, Germany. 2018-09-06 - 2018-09-08.
  • Huettig, F. (2018). How learning to read changes mind and brain [keynote]. Talk presented at Architectures and Mechanisms for Language Processing-Asia (AMLaP-Asia 2018). Telangana, India. 2018-02-01 - 2018-02-03.
  • Ostarek, M., Van Paridon, J., Hagoort, P., & Huettig, F. (2018). Multi-voxel pattern analysis reveals conceptual flexibility and invariance in language. Poster presented at the 10th Annual Meeting of the Society for the Neurobiology of Language (SNL 2018), Québec City, Canada.
  • Hintz, F., Meyer, A. S., & Huettig, F. (2014). Mechanisms underlying predictive language processing. Talk presented at the 56. Tagung experimentell arbeitender Psychologen [TeaP, Conference on Experimental Psychology]. Giessen, Germany. 2014-03-31 - 2014-04-02.
  • Hintz, F., Meyer, A. S., & Huettig, F. (2014). Prediction using production or production engaging prediction?. Poster presented at the 20th Architectures and Mechanisms for Language Processing Conference (AMLAP 2014), Edinburgh (UK).

    Abstract

    Prominent theories of predictive language processing assume that language production processes are used to anticipate upcoming linguistic input during comprehension (Dell & Chang, 2014; Pickering & Garrod, 2013). Here, we explore the converse case: Does a task set including production in addition to comprehension encourage prediction, compared to a task only including comprehension? To test this hypothesis, we conducted a cross-modal naming experiment (Experiment 1) including an object naming task and a self-paced reading experiment (Experiment 2) that did not include overt production. We used the same predictable (N = 40) and non-predictable (N = 40) sentences in both experiments. The sentences consisted of a fixed agent, a transitive verb and a predictable or non-predictable target word (The man drinks a beer vs. The man buys a beer). Most of the empirical work on prediction used sentences in which the target words were highly predictable (often with a mean cloze probability > .8) and thus it is little surprising that participants engaged in predictive language processing very easily. In the current sentences, the mean cloze probability in the predictable sentences was .39 (ranging from .06 to .8; zero in the non-predictable sentences). If comprehenders are more likely to engage in predictive processing when the task set involves production, we should observe more pronounced effects of prediction in Experiment 1 as compared to Experiment 2. If production does not enhance prediction, we should observe similar effects of prediction in both experiments. In Experiment 1, participants (N = 54) listened to recordings of the sentences which ended right before the spoken target word. Coinciding with the end of the playback, a picture of the target word was shown which the participants were asked to name as fast as possible. Analyses of their naming latencies revealed a statistically significant naming advantage of 106 ms on predictable over non-predictable trials. Moreover, we found that the objects’ naming advantage was predicted by the target words’ cloze probability in the sentences (r = .411, p = .016). In Experiment 2, the same sentences were used in a self-paced reading experiment. To allow for testing of potential spill-over effects, we added a neutral prepositional phrase (buys a beer from the bar keeper/drinks a beer from the shop) to each sentence. Participants (N = 54) read the sentences word-by-word, advancing by pushing the space bar. On 30% of the trials, comprehension questions were used to keep up participants' focus on comprehending the sentences. Analyses of participants’ target and post-target reading times revealed numerical advantages of 6 ms and 20 ms, respectively, in the predictable as compared to the non-predictable condition. However, in both cases, this difference was not statistically reliable (t = .757, t = 1.43) and the significant positive correlation between an item’s naming advantage and its cloze probability as seen in Experiment 1 was absent (r = .037, p = .822). Importantly, the analysis of participants' responses to the comprehension questions, showed that they understood the sentences (mean accuracy = 93%). To conclude, although both experiments used the same sentences, we observed effects of prediction only when the task included production. In Experiment 2, no evidence for anticipation was found although participants clearly understood the sentences and the method has previously been shown to be sensitive to measure prediction effects (Van Berkum et al., 2005). Our results fit with a recent study by Gollan et al. (2011) who found only a small processing advantage of predictive over non-predictive sentences in reading (using highly predictable sentences with a cloze probability > . 87) but a strong prediction effect when participants read the same sentences and carried out an additional object naming task (see also Griffin & Bock, 1998). Taken together, the studies suggest that the comprehenders' task set exerts a powerful influence on the likelihood and magnitude of predictive language processing. When the task set involves language production, as is often the case in natural conversation, comprehenders might engage in prediction to a stronger degree than in pure comprehension tasks. Being able to predict words another person is about to say might optimize the comprehension process and enable smooth turn-taking.
  • Hintz, F., Meyer, A. S., & Huettig, F. (2014). The influence of verb-specific featural restrictions, word associations, and production-based mechanisms on language-mediated anticipatory eye movements. Talk presented at the 27th annual CUNY conference on human sentence processing. Ohio State University, Columbus/Ohio (US). 2014-03-13 - 2014-03-15.
  • Huettig, F., & Guerra, E. (2014). Context-dependent mapping of linguistic and color representations challenges strong forms of embodiment. Talk presented at the 20th Architectures and Mechanisms for Language Processing Conference (AMLAP 2014). Edinburgh, UK. 2014-09-03 - 2014-09-06.

    Abstract

    A central claim of embodied theories of cognition is that sensory representations are
    routinely activated and influence language processing even in the absence of relevant
    sensory input (cf. Pulvermüller, 2005; Wassenburg & Zwaan, 2010). We tested the influence
    of color representations during language processing in three visual world eye tracking
    experiments. The method is particularly well suited to investigate this issue because the
    availability of relevant visual input can be manipulated.
    We made use of the phenomena that when participants hear a word that refers to a
    visual object or printed word they quickly direct their eye gaze to objects or printed words
    which are similar (e.g. semantically or visually) to the heard word. We used a look and listen
    task which previously has been shown to be sensitive to such relationships between spoken
    words and visual items. In Experiment 1, on experimental trials, participants listened to
    sentences containing a critical target word associated with a prototypical color (e.g.
    '...spinach...') as they inspected a visual display with four words printed in black font. One of
    the four printed words was associated with the same prototypical color (e.g. green) as the
    spoken target word (e.g. FROG). On experimental trials, the spoken target word did not have
    a printed word counterpart (SPINACH was not present in the display). In filler trials (70% of
    trials) the target was present in the display and attracted significantly more overt attention
    than the unrelated distractors. In experimental trials color competitors were not looked at
    more than the distractors. In Experiment 2 the printed words were replaced with line
    drawings of the objects. In order to direct the attentional focus of our participants toward
    color features we used a within-participants counter-balanced design and alternated color
    and greyscale trials randomly throughout the experiment. Therefore, on one trial our
    participants heard a word such as 'spinach' and saw a frog (colored in green) in the visual
    display. On the next trial however they saw a banana (in greyscale) on hearing 'canary'
    (bananas and canaries are typically yellow), etc. The presence (or absence) of color was
    thus a salient property of the experiment. Participants looked more at color competitors than
    unrelated distractors on hearing the target word in the color trials but not in the greyscale
    trials, i.e. on hearing 'spinach' they looked at the green frog but not the greyscale frog.
    Experiment 3 was identical to Experiment 2, except that the visual display was removed at
    the sentence onset, after a longer preview. This experiment examined whether the continued
    presence of color in the immediate visual environment was necessary for the observation of
    color-mediated eye movements. Eye movements directed towards the now blank screen
    were recorded as the sentence unfolded (cf. Spivey & Geng, 2001). In the filler trials,
    participants looked significantly more at the locations where the targets, rather than the
    distractors, had been previously presented as the target words acoustically unfolded. In the
    experimental trials, the locations where the color competitors had previously been presented
    did not attract increased attention (neither in color nor greyscale trials).
    These data demonstrate that language-mediated eye movements are only influenced
    by color relations between spoken words and visually displayed items if color is present in the immediate visual environment. We conclude that color representations are unlikely to be
    routinely activated in language processing. Our findings provide strong constraints for
    embodied theories of cognition which assume that sensory representations influence language processing even in the absence of relevant sensory input. These results fit best with the notion that the main role of sensory representations in language processing is a different one, namely to contextualize language in the immediate environment, connecting language to the here and now.
  • Huettig, F. (2015). Does prediction in language comprehension involve language production?. Talk presented at the Comprehension=Production? workshop. Nijmegen, the Netherlands. 2015-03-26 - 2015-03-28.

    Abstract

    The notion that predicting upcoming linguistic information in language comprehension makes use of the production system has recently received much attention (e.g., Chang et al., 2006; Dell & Chang, 2014; Federmeier, 2007; Pickering & Garrod, 2007, 2013; Van Berkum et al., 2005). So far there has been little experimental evidence for a relation between prediction and production. I will discuss the results of several recent eye-tracking experiments with toddlers (Mani & Huettig, 2012) and adults (Rommers et al. submitted, Hintz et al., in prep.) which provide some support for the view that production abilities are linked to language-mediated anticipatory eye movements. These data however also indicate that production-based prediction is situation-dependent and only one of many mechanisms supporting prediction. Taken together, these results suggest that multiple-mechanism accounts are required to provide a complete picture of anticipatory language processing.
  • Huettig, F. (2014). How embodied is language processing?. Talk presented at the 2nd Attentive Listener in the Visual World workshop. Hyderabad, India. 2014-11-03 - 2014-11-05.
  • Huettig, F. (2014). How literacy acquisition affects the illiterate mind. Talk presented at the Low Educated Second Language and Literacy Acquisition (LESLLA). Nijmegen, Netherlands. 2014-08-28 - 2014-08-30.
  • Huettig, F. (2014). Literacy influences on predictive language processing and visual search. Talk presented at the Priming across Modalities: The Influence of Orthography on Sign and Spoken Language Processing workshop. Haifa, Israel. 2014-04.
  • Huettig, F. (2014). The context-dependent influence of colour representations during language-vision interactions constrains theories of conceptual processing. Talk presented at the Color in Concepts workshop. Düsseldorf, Germany. 2014-06-02 - 2014-06-03.
  • Rommers, J., & Huettig, F. (2014). Limits to cross-modal semantic and object shape priming in sentence context. Poster presented at the Society for the Neurobiology of Language [SNL 2014], Amsterdam, the Netherlands.
  • Rommers, J., & Huettig, F. (2014). Limits to cross-modal semantic and object shape priming in sentence context. Poster presented at the 20th Architectures and Mechanisms for Language Processing Conference (AMLAP 2014), Edinburgh, UK.
  • Smith, A. C., Monaghan, P., & Huettig, F. (2014). A comprehensive model of spoken word recognition must be multimodal: Evidence from studies of language mediated visual attention. Talk presented at the 36th Annual Conference of the Cognitive Science Society [CogSci 2014]. Quebec, Canada. 2014-07-23 - 2014-07-26.
  • Smith, A. C., Monaghan, P., & Huettig, F. (2014). Examining strains and symptoms of the ‘Literacy Virus’: The effects of orthographic transparency on phonological processing in a connectionist model of reading. Talk presented at the 36th Annual Conference of the Cognitive Science Society [CogSci 2014]. Quebec, Canada. 2014-07-23 - 2014-07-26.
  • Smith, A. C., Monaghan, P., & Huettig, F. (2014). Examining the effects of orthographic transparency on phonological and semantic processing within a connectionist implementation of the triangle model of reading. Talk presented at the 14th Neural Computation and Psychology Workshop [NCPW 14]. Lancaster, U.K. 2014-08-21 - 2014-08-23.
  • Smith, A. C., Monaghan, P., & Huettig, F. (2014). Strains and symptoms of the ‘literacy virus’: Modelling the effects of orthographic transparency on phonological processing. Poster presented at the 20th Architectures and Mechanisms for Language Processing Conference (AMLAP 2014), Edinburgh, UK.
  • De Groot, F., Huettig, F., & Olivers, C. N. L. (2012). Attentional capture by working memory content: When do words guide attention?. Poster presented at the 3rd Symposium on “Visual Search and Selective Attention” (VSSA III), Munich, Germany.
  • Hintz, F., Meyer, A. S., & Huettig, F. (2012). Looking at nothing facilitates memory retrieval. Poster presented at Donders Discussions 2012, Nijmegen (NL).

    Abstract

    When processing visual objects, we integrate visual, linguistic and spatial information to form an episodic trace. Re-activating one aspect of the episodic trace of an object re-activates the entire bundle making all integrated information available. Using the blank screen paradigm [1], researchers observed that upon processing spoken linguistic input, participants tended to make eye movements on a blank screen, fixating locations that were previously occupied by objects mentioned in the linguistic utterance or were related. Ferreira and colleagues [2] suggested that 'looking at nothing' facilitated memory retrieval. However, this claim lacks convincing empirical support. In Experiment 1, Dutch participants looked at four-object-displays. Three objects were related to a spoken target word. Given the target word 'beker' (beaker), the display featured a phonological (a bear), a shape (a bobbin), a semantic (a fork) competitor, and an unrelated distractor (an umbrella). Participants were asked to name the objects as fast as possible. Subsequently, the objects disappeared. Participants fixated the center of the screen and listened to the target word. They had to carry out a semantic judgment task (indicating in which position an object had appeared that was semantically related to the objects) or a visual shape similarity judgment (indicating the position of the object similar in shape to the target). In both conditions, we observed that participants re-fixated the empty target location before responding. The set-up of Experiment 2 was identical except that we asked participants to maintain fixating the center of the screen while listening to the spoken word and responding. Performance accuracy was significantly lower in Experiment 2 than in Experiment 1. The results indicate that memory retrieval for objects is impaired when participants are not allowed to look at relevant, though empty locations. [1] Altmann, G. (2004). Language-mediated eye movements in the absence of a visual world: the 'blank screen paradigm'. Cognition, 93(2), B79-B87. [2] Ferreira, F., Apel, J., & Henderson, J. M. (2008). Taking a new look at looking at nothing. Trends Cogn Sci, 12(11), 405-410.
  • Hintz, F., & Huettig, F. (2012). Phonological word-object mapping is contingent upon the nature of the visual environment. Poster presented at the 18th Annual Conference on Architectures and Mechanisms for Language Processing (AMLaP 2012), Riva del Garda, Italy.

    Abstract

    Four eye-tracking experiments investigated the impact of the nature of the visual environment on the likelihood of word-object mapping taking place at a phonological level of representation during languagemediated visual search. Dutch participants heard single spoken target words while looking at four objects embedded in displays of different complexity and were asked to indicate the presence or absence of the target object. During filler trials the target objects were present, but during experimental trials they were absent and the display contained various competitor objects. For example, given the target word 'beaker', the display contained a phonological (a beaver, bever), a shape (a bobbin, klos), a semantic (a fork, vork) competitor, and an unrelated distractor (an umbrella, paraplu). When objects were embedded in semi-realistic scenes including four human-like characters (Experiment 1, 3, and 4a), there were no biases in looks to phonological competitors even when the objects' contours were highlighted (Experiment 3) and an object naming task was administered right before the eye-tracking experiment (Experiment 4a). In all three experiments however we observed evidence for inhibition in looks to phonological competitors, which suggests that the phonological forms of the objects had been retrieved. When objects were presented in simple four-object displays (Experiments 2 and 4b) there were clear attentional biases to phonological competitors replicating earlier research (Huettig & McQueen, 2007). These findings suggest that phonological word-object mapping is contingent upon the nature of the visual environment and add to a growing body of evidence that the nature of our visual surroundings induces particular modes of processing during language-mediated visual search. References Huettig, F., & McQueen, J. M. (2007). The tug of war between phonological, semantic and shape information in language-mediated visual search. Journal of Memory and Language, 57(4), 460-482. doi: 10.1016/j.jml.2007.02.001
  • Hintz, F., & Huettig, F. (2012). Phonological word-object mapping is contingent upon the nature of the visual environment. Talk presented at Psycholinguistics in Flanders goes Dutch [PiF 2012]. Berg en Dal (NL). 2012-06-06 - 2012-06-07.
  • Huettig, F., & Janse, E. (2012). Anticipatory eye movements are modulated by working memory capacity: Evidence from older adults. Poster presented at the 18th Annual Conference on Architectures and Mechanisms for Language Processing (AMLaP 2012), Riva del Garda, Italy.
  • Huettig, F., Singh, N., Singh, S., & Mishra, R. K. (2012). Language-mediated prediction is related to reading ability and formal literacy. Talk presented at the Tagung experimentell arbeitender Psychologen [TeaP 2012]. Mannheim, Germany. 2012-04-04 - 2012-04-06.
  • Huettig, F. (2012). Literacy modulates language-mediated visual attention and prediction. Talk presented at the Center of Excellence Cognitive Interaction Technology (CITEC). Bielefeld, Germany. 2012-01-12.
  • Huettig, F. (2012). The nature and mechanisms of language-mediated anticipatory eye movements. Talk presented at the International symposium: The Attentive Listener in the Visual world: The Interaction of Language, Attention,Memory, and Vision. Allahabad, India. 2012-10-05 - 2012-10-06.
  • Mani, N., & Huettig, F. (2012). Prediction during language processing is a piece of cake – but only for skilled producers. Poster presented at the 18th Annual Conference on Architectures and Mechanisms for Language Processing [AMLaP 2012], Riva del Garda, Italy.

    Abstract

    Background Adults orient towards an image of a cake upon hearing sentences such as “The boy will eat the cake” even before hearing the word cake, i.e., soon after they hear the verb EAT (Kamide et al., 2003). This finding has been taken to suggest that verb processing includes prediction of nouns that qualify as arguments for these verbs. Upon hearing the verb EAT, adults and young children (three- to ten-year-olds; Borovsky et al., in press) anticipate upcoming linguistic input in keeping with this verb’s selectional restrictions and use this to orient towards images of thematically appropriate arguments.
  • Mani, N., & Huettig, F. (2012). Toddlers anticipate that we EAT cake. Talk presented at the Tagung experimentell arbeitender Psychologen [TeaP 2012]. Mannheim, Germany. 2012-04-04 - 2012-04-06.
  • Rommers, J., Meyer, A. S., Praamstra, P., & Huettig, F. (2012). Object shape representations in the contents of predictions for upcoming words. Talk presented at Psycholinguistics in Flanders [PiF 2012]. Berg en Dal, The Netherlands. 2012-06-06 - 2012-06-07.
  • Rommers, J., Meyer, A. S., Praamstra, P., & Huettig, F. (2012). The content of predictions: Involvement of object shape representations in the anticipation of upcoming words. Talk presented at the Tagung experimentell arbeitender Psychologen [TeaP 2012]. Mannheim, Germany. 2012-04-04 - 2012-04-06.
  • Rommers, J., Meyer, A. S., & Huettig, F. (2012). Predicting upcoming meaning involves specific contents and domain-general mechanisms. Talk presented at the 18th Annual Conference on Architectures and Mechanisms for Language Processing [AMLaP 2012]. Riva del Garda, Italy. 2012-09-06 - 2012-09-08.

    Abstract

    In sentence comprehension, readers and listeners often anticipate upcoming information (e.g., Altmann & Kamide, 1999). We investigated two aspects of this process, namely 1) what is pre-activated when anticipating an upcoming word (the contents of predictions), and 2) which cognitive mechanisms are involved. The contents of predictions at the level of meaning could be restricted to functional semantic attributes (e.g., edibility; Altmann & Kamide, 1999). However, when words are processed other types of information can also be activated, such as object shape representations. It is unknown whether this type of information is already activated when upcoming words are predicted. Forty-five adult participants listened to predictable words in sentence contexts (e.g., "In 1969 Neil Armstrong was the first man to set foot on the moon.") while looking at visual displays of four objects. Their eye movements were recorded. There were three conditions: target present (e.g., a moon and three distractor objects that were unrelated to the predictable word in terms of semantics, shape, and phonology), shape competitor (e.g., a tomato and three unrelated distractors), and distractors only (e.g., rice and three other unrelated objects). Across lists, the same pictures and sentences were used in the different conditions. We found that participants already showed a significant bias for the target object (moon) over unrelated distractors several seconds before the target was mentioned, demonstrating that they were predicting. Importantly, there was also a smaller but significant shape competitor (tomato) preference starting at about a second before critical word onset, consistent with predictions involving the referent’s shape. The mechanisms of predictions could be specific to language tasks, or language could use processing principles that are also used in other domains of cognition. We investigated whether performance in non-linguistic prediction is related to prediction in language processing, taking an individual differences approach. In addition to the language processing task, the participants performed a simple cueing task (after Posner, Nissen, & Ogden, 1978). They pressed one of two buttons (left/right) to indicate the location of an X symbol on the screen. On half of the trials, the X was preceded by a neutral cue (+). On the other half, an arrow cue pointing left (<) or right (>) indicated the upcoming X's location with 80% validity (i.e., the arrow cue was correct 80% of the time). The SOA between cue and target was 500 ms. Prediction was quantified as the mean response latency difference between the neutral and valid condition. This measure correlated positively with individual participants' anticipatory target and shape competitor preference (r = .27; r = .45), and was a significant predictor of anticipatory looks in linear mixed-effects regression models of the data. Participants who showed more facilitation from the arrow cues predicted to a higher degree in the linguistic task. This suggests that prediction in language processing may use mechanisms that are also used in other domains of cognition. References Altmann, G. T. M., & Kamide, Y. (1999). Incremental interpretation at verbs: Restricting the domain of subsequent reference. Cognition, 73(3), 247-264. Posner, M. I., Nissen, M. J., & Ogden, W. C. (1978). Attended and unattended processing modes: The role of set for spatial location. In: H.L. Pick, & I.J. Saltzman (Eds.), Modes of perceiving and processing information. Hillsdale, N.J.: Lawrence Erlbaum Associates.
  • Smith, A. C., Huettig, F., & Monaghan, P. (2012). Modelling multimodal interaction in language mediated eye gaze. Talk presented at the 13th Neural Computation and Psychology Workshop [NCPW13]. San Sebastian, Spain. 2012-07-12 - 2012-07-14.

    Abstract

    Hub-and-spoke models of semantic processing which integrate modality specific information within a central resource have proven successful in capturing a range of neuropsychological phenomena (Rogers et al, 2004; Dilkina et al, 2008). Within our study we investigate whether the scope of the Hub-and-spoke architectural framework can be extended to capture behavioural phenomena in other areas of cognition. The visual world paradigm (VWP) has contributed significantly to our understanding of the information and processes involved in spoken word recognition. In particular it has highlighted the importance of non-linguistic influences during language processing, indicating that combined information from vision, phonology, and semantics is evident in performance on such tasks (see Huettig, Rommers & Meyer, 2011). Huettig & McQueen (2007) demonstrated that participants’ fixations to objects presented within a single visual display varied systematically according to their phonological, semantic and visual relationship to a spoken target word. The authors argue that only an explanation allowing for influence from all three knowledge types is capable of accounting for the observed behaviour. To date computational models of the VWP (Allopenna et al, 1998; Mayberry et al, 2009; Kukona et al, 2011) have focused largely on linguistic aspects of the task and have therefore been unable to offer explanations for the growing body of experimental evidence emphasising the influence of non-linguistic information on spoken word recognition. Our study demonstrates that an emergent connectionist model, based on the Hub-and-spoke models of semantic processing, which integrates visual, phonological and functional information within a central resource, is able to capture the intricate time course dynamics of eye fixation behaviour reported in Huettig & McQueen (2007). Our findings indicate that such language mediated visual attention phenomena can emerge largely due to the statistics of the problem domain and may not require additional domain specific processing constraints.
  • Smith, A. C., Monaghan, P., & Huettig, F. (2012). Multimodal interaction in a model of visual world phenomena. Poster presented at the 18th Annual Conference on Architectures and Mechanisms for Language Processing (AMLaP 2012), Riva del Garda, Italy.

    Abstract

    Existing computational models of the Visual World Paradigm (VWP) have simulated the connection between language processing and eye gaze behavior, and consequently have provided insight into the cognitive processes underlying lexical and sentence comprehension. Allopenna, Magnuson and Tanenhaus (1998), demonstrated that fixation probabilities during spoken word processing can be predicted by lexical activations in the TRACE model of spoken word recognition. Recent computational models have extended this work to predict fixation behavior during sentence processing from the integration of visual and linguistic information. Recent empirical investigation of word level effects in VWP support claims that language mediated eye gaze is not only influenced by overlap at a phonological level (Allopenna, Magnuson & Tanenhaus, 1998) but also by relationships in terms of visual and semantic similarity. Huettig and McQueen (2007) found that when participants heard a word and viewed a scene containing objects phonologically, visually, or semantically similar to the target, then all competitors exerted an effect on fixations, but fixations to phonological competitors preceded those to other competitors. Current models of VWP that simulate the interaction between visual and linguistic information do so with representations that are unable to capture fine-grained semantic, phonological or visual feature relationships. They are therefore limited in their ability to examine effects of multimodal interactions in language processing. Our research extends that of previous models by implementing representations in each modality that are sufficiently rich to capture similarities and distinctions in visual, phonological and semantic representations. Our starting point was to determine the extent to which multimodal interactions between these modalities in the VWP would be emergent from the nature of the representations themselves, rather than determined by architectural constraints. We constructed a recurrent connectionist model, based on Hub-and-spoke models of semantic processing, which integrates visual, phonological and semantic information within a central resource. We trained and tested the model on viewing scenes as in Huettig and McQueen’s (2007) study, and found that the model replicated the complex behaviour and time course dynamics of multimodal interaction, such that the model activated phonological competitors prior to activating visual and semantic competitors. Our approach enables us to determine that differences in the computational properties of each modality’s representational structure is sufficient to produce behaviour consistent with the VWP. The componential nature of phonological representations and the holistic structure of visual and semantic representations result in fixations to phonological competitors preceding those to other competitors. Our findings suggest such language-mediated visual attention phenomena can emerge due to the statistics of the problem domain, with observed behaviour emerging as a natural consequence of differences in the structure of information within each modality, without requiring additional modality specific architectural constraints.
  • Smith, A. C., Huettig, F., & Monaghan, P. (2012). The Tug of War during spoken word recognition in our visual worlds. Talk presented at Psycholinguistics in Flanders 2012 [[PiF 2012]. Berg en Dal, NL. 2012-06-06 - 2012-06-07.
  • Rommers, J., Huettig, F., & Meyer, A. S. (2011). Task-dependency in the activation of visual representations during language processing. Poster presented at Tagung experimentell arbeitender Psychologen [TaeP 2011], Halle (Saale), Germany.
  • Rommers, J., Meyer, A. S., & Huettig, F. (2011). The timing of the on-line activation of visual shape information during sentence processing. Poster presented at the 17th Annual Conference on Architectures and Mechanisms for Language Processing [AMLaP 2011], Paris, France.
  • Weber, A., Sumner, M., Krott, A., Huettig, F., & Hanulikova, A. (2011). Sinking about boats and brains: Activation of word meaning in foreign-accented speech by native and nonnative listeners. Poster presented at the First International Conference on Cognitive Hearing Science for Communication, Linköping, Sweden.

    Abstract

    Sinking about boats and brains: activation of word meaning in foreign-accented speech by native and non-native listeners Andrea Weber, Meghan Sumner, Andrea Krott, Falk Huettig, Adriana Hanulikova Understanding foreign-accented speech requires from listeners the correct interpretation of segmental variation as in German-accented [s]eft for English theft. The task difficulty increases when the accented word forms resemble existing words as in [s]ink for think. In two English priming experiments, we investigated the activation of the meanings of intended and unintended words by accented primes. American native (L1) and German non-native (L2) participants listened to auditory primes followed by visual targets to which they made lexical decisions. Primes were produced by a native German speaker and were either nonsense words ([s]eft for theft), unintended words ([s]ink for think), or words in their canonical forms (salt for salt). Furthermore, primes were strongly associated to targets, with the co-occurrence being high either between the surface form of the prime and the target ([s]ink-BOAT, salt-PEPPER) or the underlying form and the target ([s]ink-BRAIN, seft-PRISON). L1 listeners responded faster when the underlying form was associated with the target (in comparison to unrelated primes), but L2 listeners responded faster when the surface form was associated. Seemingly, L1 listeners interpreted all primes as being mispronounced – facilitating the activation of think when hearing the unintended word [s]ink, but erroneously preventing the activation of salt when hearing the canonical form salt. L2 listeners, though, took primes at face value and failed to activate the meaning of think when hearing [s]ink but did activate the meaning of salt when hearing salt. This asymmetry suggests an interesting difference in the use of high-level information, with L1 listeners, but not L2 listeners, using knowledge about segmental variations for immediate meaning activation.
  • Huettig, F., Chen, J., Bowerman, M., & Majid, A. (2008). Linguistic relativity: Evidence from Mandarin speakers’ eye-movements. Talk presented at 14th Annual Conference on the Architectures and Mechanisms for Language Processing [AMLaP 2008]. Cambridge, UK. 2008-09-04 - 2008-09-06.

    Abstract

    If a Mandarin speaker had walked past two rivers and wished to describe how many he had seen, he would have to say “two tiao river”, where tiao designates long, rope-like objects such as rivers, snakes and legs. Tiao is one of several hundred classifiers – a grammatical category in Mandarin. In two eye-tracking studies we presented Mandarin speakers with simple Mandarin sentences through headphones while monitoring their eye-movements to objects presented on a computer monitor. The crucial question is what participants look at while listening to a pre-specified target noun. If classifier categories influence general conceptual processing then on hearing the target noun participants should look at objects that are also members of the same classifier category – even when the classifier is not explicitly present. For example, on hearing scissors, Mandarin speakers should look more at a picture of a chair than at an unrelated object because scissors and chair share the classifier ba. This would be consistent with a Strong Whorfian position, according to which language is a major determinant in shaping conceptual thought (Sapir, 1921; Whorf, 1956). A weaker influence of language-on-thought could be predicted, where language shapes cognitive processing, but only when the language-specific category is actively being processed (Slobin, 1996). According to this account, eye-movements are not necessarily drawn to chair when a participant hears scissors, but they would be on hearing ba scissors. This is because hearing ba activates the linguistic category that both scissors and chair belong to. A third logical possibility is that classifiers are purely formal markers (cf. Greenberg, 1972; Lehman, 1979) that do not influence attentional processing even when they are explicitly present. The data showed that when participants heard a spoken word from the same classifier category as a visually depicted object (e.g. scissors-chair), but the classifier was not explicitly presented in the speech, overt attention to classifier-match objects (e.g. chair) and distractor objects did not differ (Experiment 1). But when the classifier was explicitly presented (e.g. ba, Experiment 2), participants shifted overt attention significantly more to classifier-match objects (e.g. chair) than to distractors. These data are incompatible with the Strong Whorfian hypothesis. Instead the findings support the Weak Whorfian hypothesis that linguistic distinctions force attention to properties of the world but only during active linguistic processing of that distinction (cf. Slobin, 1996).

Share this page