Presentations

Abstract

Integrating sentence meaning over time requires memory ranging from milliseconds (words) to seconds (sentences) and minutes (discourse). How do transient events like action potentials in the human language system support memory at these different temporal scales? Here we investigate the nature of processing memory in a neurobiologically motivated model of sentence comprehension. The model was a recurrent, sparsely connected network of spiking neurons. Synaptic weights were created randomly and there was no adaptation or learning. As input the network received word sequences generated from construction grammar templates and their syntactic alternations (e.g., active/passive transitives, transfer datives, caused motion). The language environment had various features such as tense, aspect, noun/verb number agreement, and pronouns which created positional variation in the input. Similar to natural speech, word durations varied between 50ms and 0.5s of real, physical time depending on their length. The model's task was to incrementally interpret these word sequences in terms of semantic roles. There were 8 target roles (e.g., Agent, Patient, Recipient) and the language generated roughly 1,2m distinct utterances from which a sequence of 10,000 words was randomly selected and filtered through the network. A set of readout neurons was then calibrated by means of logistic regression to decode the internal network dynamics onto the target semantic roles. In order to accomplish the role assignment task, network states had to encode and maintain past information from multiple cues that could occur several words apart. To probe the circuit's memory capacity, we compared models where network connectivity, the shape of synaptic currents, and properties of neuronal adaptation were systematically manipulated. We found that task-relevant memory could be derived from a mechanism of neuronal spike-rate adaptation, modelled as a conductance that hyperpolarized the membrane following a spike and relaxed to baseline exponentially with a fixed time-constant. By acting directly on the membrane potential it provided processing memory that allowed the system to successfully interpret its sentence input. Near optimal performance was also observed when an exponential decay model of post-synaptic currents was added into the circuit, with time-constants approximating excitatory NMDA and inhibitory GABA-B receptor dynamics. Thus, the information flow was extended over time, creating memory characteristics comparable to spike-rate adaptation. Recurrent connectivity, in contrast, only played a limited role in maintaining information; an acyclic version of the recurrent circuit achieved similar accuracy. This indicates that random recurrent connectivity at the modelled spatial scale did not contribute additional processing memory to the task. Taken together, these results suggest that memory for language might be provided by activity-silent dynamic processes rather than the active replay of past input as in storage-and-retrieval models of working memory. Furthermore, memory in biological networks can take multiple forms on a continuum of time-scales. Therefore, the development of neurobiologically realistic, causal models will be critical for our understanding of the role of memory in language processing.

Abstract

Institute of Adaptive and Neural Computation, School of Informatics, University of Edinburgh, UK

Abstract

It is generally assumed that structure building processes in the spoken and written modalities are subserved by modality-independent lexical, morphological, grammatical, and conceptual processes. We present a large-scale neuroimaging study (N=204) on whether the unification of sentence structure is supramodal in this sense, testing if observations replicate across written and spoken sentence materials. The activity in the unification network should increase when it is presented with a challenging sentence structure, irrespective of the input modality. We build on the well-established findings that multiple non-local dependencies, overlapping in time, are challenging and that language users disprefer left- over right-branching sentence structures in written and spoken language, at least in the context of mainly right-branching languages such as English and Dutch. We thus focused our study with Dutch participants on a left-branching processing complexity measure. Supramodal effects of left-branching complexity were observed in a left-lateralized perisylvian network. The left inferior frontal gyrus (LIFG) and the left posterior middle temporal gyrus (LpMTG) were most clearly associated with left-branching processing complexity. The left anterior middle temporal gyrus (LaMTG) and left inferior parietal lobe (LIPL) were also significant, although less specifically. The LaMTG was increasingly active also for sentences with increasing right-branching processing complexity. A direct comparison between left- and right-branching processing complexity yielded activity in an LIFG ROI for left > right-branching complexity, while the right > left contrast showed no activation. Using a linear contrast testing for increases in the left-branching complexity effect over the sentence, we found significant activity in LIFG and LpMTG. In other words, the activity in these regions increased from sentence onset to end, in parallel with the increase of the left-branching complexity measure. No similar increase was observed in LIPL. Thus, the observed functional segregation during sentence processing of LaMTG and LIPL vs. LIFG and LpMTG is consistent with our observation of differential activation changes in sensitivity to left- vs. right-branching structure. While LIFG, LpMTG, LaMTG and LIPL all contribute to the supramodal unification processes, the results suggest that these regions differ in their respective contributions to the subprocesses of unification. Our results speak to the high processing costs of (1) simultaneous unification and (2) maintenance of constituents that are not yet attached to the already unified part of the sentence. Sentences with high left- (compared to right-) branching complexity impose an added load on unification. We show that this added load leads to an increased BOLD response in left perisylvian regions. The results are relevant for understanding the neural underpinnings of the processing difficulty linked to multiple, overlapping non-local dependencies. In conclusion, we used the left- and right branching complexity measures to index this processing difficulty and showed that the unification network operates with similar spatiotemporal dynamics over the course of the sentence, during unification of both written and spoken sentences.

Abstract

Sentence processing requires the ability to establish thematic relations between constituents. Here we investigate the computational basis of this ability in a neurobiologically motivated comprehension model. The model has a tripartite architecture where input representations are supplied by the mental lexicon to a network that performs incremental thematic role assignment. Roles are combined into a representation of sentence-level meaning by a downstream system (semantic unification). Recurrent, sparsely connected, spiking networks were used which project a time-varying input signal (word sequences) into a high-dimensional, spatio-temporal pattern of activations. Local, adaptive linear read-out units were then calibrated to map the internal dynamics to desired output (thematic role sequences) [1]. Read-outs were adjusted on network dynamics driven by input sequences drawn from argument-structure templates with small variation in function words and larger variation in content words. Models were trained on sequences of 10K words for 200ms per word at a 1ms resolution, and tested on novel items generated from the language. We found that a static, random recurrent spiking network outperformed models that used only local word information without context. To improve performance, we explored various ways of increasing the model’s processing memory (e.g., network size, time constants, sparseness, input strength, etc.) and employed spiking neurons with more dynamic variables (leaky integrate-and-fire versus Izhikevich-neurons). The largest gain was observed when the model’s input history was extended to include previous words and/or roles. Model behavior was also compared for localist and distributed encodings of word sequences. The latter were obtained by compressing lexical co-occurrence statistics into continuous-valued vectors [2]. We found that performance for localist-input was superior even though distributed representations contained extra information about word context and semantic similarity. Finally, we compared models that received input enriched with combinations of semantic features, word-category, and verb sub-categorization labels. Counter-intuitively, we found that adding this information to the model’s lexical input did not further improve performance. Consistent with previous results, however, performance improved for increased variability in content words [3]. This indicates that the approach to comprehension taken here might scale to more diverse and naturalistic language input. Overall, the results suggest that active processing memory beyond pure state-dependent effects is important for sentence interpretation, and that memory in neurobiological systems might be actively computing [4]. Future work therefore needs to address how the structure of word representations interacts with enhanced processing memory in adaptive spiking networks. [1] Maass W., Natschläger T., & Markram H. (2002). Real-time computing without stable states: A new framework for neural computation based on perturbations. Neural Computation, 14: 2531-2560. [2] Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). Efficient estimation of word represen-tations in vector space. Proceedings of the International Conference on Learning Represen-tations, Scottsdale/AZ. [3] Fitz, H. (2011). A liquid-state model of variability effects in learning nonadjacent dependencies. Proceedings of the 33rd Annual Conference of the Cognitive Science Society, Austin/TX. [4] Petersson, K.M., & Hagoort, P. (2012). The neurobiology of syntax: Beyond string-sets. Philo-sophical Transactions of the Royal Society B 367: 1971-1883.

Abstract

Hagoort [1] suggested that the posterior temporal cortex is involved in the retrieval of lexical frames that form building blocks for syntactic unification, supported by the inferior frontal gyrus (IFG). FMRI results support the role of the IFG in the unification operations that are performed at the structural/syntactic [2] and conceptual/ semantic levels [3]. While these studies tackle the unification operations within linguistic components, in the present event-related FMRI study we investigated the interplay between sentence-level semantics and syntax by adapting an EEG comprehension paradigm [4]. The ERP results showed typical P600 and N400 effects, while their combined effect revealed an interaction expressed in the N400 component ([CB-SE] - [SY-CR] > 0). Although the N400 component was similar in the correct and syntactic conditions (SY  CR), the combined effect was significantly larger than the effect of semantic anomaly alone. In contrast, the size of the P600 effect was not affected by an additional semantic violation, suggesting an asymmetry between semantic and syntactic processing. In the current FMRI study we characterize this asymmetry by means of a 2x2 experimental design included the conditions: correct (CR), syntactic (SY), semantic (SE), and combined (CB) anomalies. Standard SPM procedures were used for analysis and only clusters significant at P <.05 family-wise error corrected are reported. The main effect of semantic anomaly ([CB+SE] > [SY+CR]) yielded activation in the anterior IFG (BA 45/47). The opposite contrast revealed the theory-ofmind and default-mode network. The main effect of syntactically correct sentences ([SE+CR] > [CB+SY]), showed significant activation in the IFG (BA 44/45), including the mid-anterior insula extending into the superior temporal poles (BA 22/38). In addition, significant effects were observed in medial prefrontal/ anterior cingulate cortex, posterior middle and superior temporal regions (BA 21/22), and the basal ganglia. The reverse contrast yielded activations in the MFG (BA 9/46), the inferior parietal region (BA 39/40), precuneus and the posterior cingulate region. The only region that showed a significant interaction ([CBSE]  [SYCR] > 0) was the left temporo-parietal region (BA 22/39/40). In summary, the results show that the IFG is involved in unification during comprehension. The effect of semantic anomaly and its implied unification load engages the anterior IFG while the effect of syntactic anomaly and its implied unification failure engages MFG. Finally, the results suggest that the syntax of gender agreement interacts with sentence-level semantics in the left temporo-parietal region. [1] Hagoort, P. (2005). On Broca, brain, and binding: A new framework. TICS, 9, 416-423. [2] Snijders, T. M., Vosse, T., Kempen, G., Van Berkum, J. J. A., Petersson, K. M., Hagoort, P. (2009). Retrieval and unification of syntactic structure in sentence comprehension: An fMRI study using word-category ambiguity. Cerebral Cortex, 19, 1493-1503. doi:10.1093/ cercor/bhn187. [3] Hagoort, P., Hald, L., Baastiansen, M., Petersson, K.M. (2004). Integration of word meaning and world knowledge in language comprehension. Science 304, 438-441. [4] Hagoort, P. (2003). Interplay between syntax and semantics during sentence comprehension: ERP effects of combining syntactic and semantic violations. Journal of Cognitive Neuroscience, 15, 883- 899.

Abstract

Introduction: Natural communication is not only literal, but to a large extent also inferential. For example, sometimes people say "It is hard to give a good presentation" to actually mean "Your talk was a mess!", and listeners need to infer the speaker’s hidden message. In spite of the pervasiveness of this phenomenon in everyday communication, and even though the hidden meaning is often what it’s all about, very little is known about how the brain supports the comprehension of indirect language. What are the neural systems involved in the inferential process , and how are they different from those involved in word- and sentence-level meaning processing? We investigated the neural correlates of this so-called pragmatic inferencing in an fMRI study involving natural spoken dialogue. Methods: As a test case, we focused on the inferences needed to understand indirect replies. 18 native listeners of Dutch listened to dialogues ending in a question-answer (QA) pair. The final and critical utterance, e.g., "It is hard to give a good presentation", had different meanings depending on the dialogue context and the immediately preceding question: (1) Direct reply: Q: "How is it to give a good presentation?" A: "It is hard to give a good presentation" (2) Indirect reply, neutral: Q: "Will you give a presentation at the conference?" (rather than a poster) A: "It is hard to give a good presentation" (3) Indirect reply, face-saving: Q: "Did you like my presentation?" A: "It is hard to give a good presentation" While one of the indirect conditions was neutral, the other involved a socio-emotional aspect, as the reason for indirectness was to 'save one’s face' (as in excuses or polite refusals). Participants were asked to pay attention to the dialogues and, to ensure the latter, occasionally received a comprehension question (on filler items only). No other task demands were imposed. Results: Relative to direct replies in exchanges like (1), the indirect replies in exchanges like (2) and (3) activated brain structures associated with theory of mind and inferencing: right angular gyrus (TPJ), right DM prefrontal / frontal cortex (SMA, ACC). Both types of indirect replies also bilaterally activated the insula, an area known to be involved in empathy and affective processing. Moreover, both types of indirect replies recruited bilateral inferior frontal gyrus, thought to play a role in situation model updating. The comparison between neutral (2) and face-saving (3) indirect replies revealed that the presumed affective load of the face-saving replies activated just one additional area: right inferior frontal gyrus; we did not see any activation in classic affect-related areas. Importantly, we used the same critical sentences in all conditions. Our results can thus not be explained by lexico-semantic or other (e.g. syntactic, word frequency) factors. Conclusions: To extend neurocognitive research on meaning in language beyond the level of straightforward literal utterances, we investigated the neural correlates of pragmatic inferencing in an fMRI study involving indirect replies in natural spoken dialogue. Our findings reveal that the areas used to infer the intended meaning of an implicit message are partly different from the classic language network. Furthermore, the identity of the areas involved is consistent with the idea that inferring hidden meanings requires taking the speaker’s perspective. This confirms the importance of perspective taking in language comprehension, even in a situation where the listener is not the one addressed. Also, as the areas recruited by indirect replies generally do not light up in standard fMRI sentence comprehension paradigms, our study testifies to the importance of studying language understanding in richer contexts in which we can tap aspects of pragmatic processing, beyond the literal code.

Abstract

In an event-related fMRI study, we examined the overlap between the implicit processing of structured sequences, generated by a simple right-linear artificial unification grammar, with natural syntax related variability in the same subjects. Research investigating rule learning of potential linguistic relevance through artificial syntax often uses performance feedback and/or explicit instruction concerning the underlying rules. It is assumed that this approach ensures the right type of ''rule-following''because the rules are either explicitly provided to the subjects or explicitly discovered by the subjects during trial-and-error learning with feedback. In this work, we use a novel implicit preference classification task based on the structural mere exposure effect. Under conditions that in important respects are similar to those of natural language development (i. e., no explicit learning or teaching instruction, and no performance feedback), 32 subjects were exposed for 5 days to grammatical sequences during an immediate short-term memory task. On day 5, a preference classification test was administered, in which new sequences were presented. In addition, natural language data was acquired in the same subjects. Implicit preference classification was sensitive enough to show robust behavioral and fMRI effects. Preference classification of structured sequences activated Broca's region (BA 44/45) significantly, and was further activated by artificial syntactic violations. The effects related to artificial syntax in BA 44/45 were identical when we masked these with activity related to natural syntax processing. Moreover, the medial temporal lobe was deactivated during artificial syntax processing, consistent with the view that implicit processing does not rely on declarative memory mechanisms supported by the medial temporal lobe. In summary, we show that implicit acquisition of structured sequence knowledge results in the engagement of Broca's region during structured sequence processing. We conclude that Broca's region is a generic on-line sequence processor integrating information, in an incremental and recursive manner, independent of whether the sequences processed are structured by a natural or an artificial syntax.

Abstract

Repetition suppression is generally accepted as the neural correlate of behavioural priming and is often used to selectively identify the neuronal representations associated with a stimulus. However, this does not explain the large number of repetition enhancement effects observed under very similar conditions. Based on a review of a large set of studies we propose several variables biasing repetition effects towards enhancement instead of suppression. On the one hand, there are stimulus variables which influence the direction of repetition effects: visibility, e. g. in the case of degraded stimuli perceptual learning occurs; novelty, e. g. in case of unfamiliar stimuli a novel network formation process occurs; and timing intervals, e. g. repetition effects are sensitive to stimulus onset asynchronies. On the other hand, repetition effects are not solely automatic processes, triggered by particular types or sequences of stimuli. The brain is continuously and actively filtering, attending to and interpreting the information provided by our senses. Consequently, internal state variables like attention, expectation and explicit memory modulate repetition effects towards enhancement versus suppression. Current models i.e. the accumulation, fatigue and sharpening models of repetition suppression have so far left out top-down factors and cannot or can only partially account for repetition enhancement effects. Instead we propose that models which incorporate both stimulus bottom-up and cognitive top-down factors are called for in order to better understand repetition effects. A good candidate is the predictive coding model in which sensory evidence is interpreted according to subjective biases and statistical accounts of past encounters.

Abstract

Speaking is a complex, multilevel process, in which the first step is to compute the message that can be syntactically and phonologically encoded. Computing the message requires constructing a mental representation of what we want to express (the reference). This reference is then mapped onto linguistic concepts stored in memory, by which the meaning of the utterance (the sense) is constructed. We used fMRI adaptation to investigate brain areas sensitive to reference and sense in overt speech. By independently manipulating repetition of reference and sense across subsequently produced sentences in a picture description task, we distinguished sets of regions sensitive to these two steps in speaking. Encoding reference involved the bilateral inferior parietal lobes (BA 39) and right inferior frontal gyrus (BA 45), suggesting a role in constructing a non-linguistic mental representation. Left middle frontal gyrus (BA 6), bilateral superior parietal lobes and bilateral posterior temporal gyri (BA 37)) were sensitive to both sense and reference processing. These regions thus seem to support semantic encoding, the process of mapping reference onto sense. Left inferior frontal gyrus (BA 45), left middle frontal gyrus (BA44) and left angular gyrus (BA 39) showed adaptation to sense, and therefore appear sensitive to the output of semantic encoding. These results reveal the neural architecture for the first steps in producing an utterance. In addition, they show the feasibility of studying overt speech at a detailed level of analysis in fMRI studies.

Abstract

Speaking is a complex, multilevel process, in which the first step is to compute the message that can be syntactically and phonologically encoded. Computing the message requires constructing a mental representation of what we want to express (the reference). This referent is mapped onto linguistic concepts stored in memory, by which the meaning of the utterance (the sense) is constructed. So far, one study targeted semantic encoding in sentence production (Menenti, Segaert & Hagoort, submitted) and none dissected this process further. We used fMRI adaptation to investigate brain areas sensitive to reference and sense in overt speech. fMRI adaptation is a phenomenon whereby repeating a stimulus property changes the BOLD-response in regions sensitive to that property. By independently manipulating repetition of reference and sense across subsequently produced sentences in a picture description task we distinguished sets of areas sensitive to these steps in semantic encoding in speaking. Methods: In a picture description paradigm, the described situation (the reference) and the linguistic semantic structure (the sense) of subsequently produced sentences were independently repeated across trials. Participants described pictures depicting events involving transitive verbs such as hit, kiss, greet, and two actors colored in different colors with sentences such as ‘The red man greets the green woman’. In our factorial design, the same situation involving the same actors could subsequently be described by two different sentences (repeated reference, novel sense) or the same sentence could subsequently be used to describe two different situations (novel reference, repeated sense). For reference, we controlled for the repetition of actors. For sense, we controlled for the repetition of individual words. See figure 1 for design and stimuli. To correct for increased movement and susceptibility artifacts due to speech, we scanned using 3T-fMRI parallel-acquired inhomogeneity-desensitized fMRI (Poser, Versluis, Hoogduin et al. 2006). Five images were acquired per TR and combined based on local T2* (Buur, Poser and Norris 2009). Results: The behavioral data (response onset, response duration and total time to complete the responses) showed effects of both sense and reference. In the fMRI analyses we looked for areas sensitive to only sense, only reference, or showing a conjunction of both factors. Encoding reference involved the bilateral inferior parietal lobes (BA 39), which showed repetition suppression, and right inferior frontal gyrus (BA 45), which showed repetition enhancement. Left inferior frontal gyrus (BA 45) showed suppression to repetition of sense, while left middle frontal gyrus (BA44) and left angular gyrus (BA 39) showed enhancement. Left middle frontal gyrus (BA 6), bilateral superior parietal lobes and bilateral posterior temporal gyri (BA 37)) showed repetition suppression to both sense and reference processing (conjunction analysis with conjunction null). See figure 2 for the results (p<.05 FWE corrected for multiple comparisons at cluster-level, maps thresholded at p<.001 uncorrected voxel-level.) Conclusions: The input to semantic encoding is construction of a referent, a mental representation that the utterance is about. The bilateral temporo-parietal junctions are involved in this process as they show sensitivity to repetition of reference but not sense. RIFG shows enhancement and may therefore be involved in constructing a more comprehensive model spanning several utterances. Semantic encoding itself requires mapping of the reference onto the sense. This involves large parts of the language network: bilateral posterior temporal lobes and upper left inferior frontal gyrus were sensitive to both reference and sense. Finally, sense recruits left inferior frontal gyrus (BA 45). This area is sensitive to syntactic encoding (Bookheimer 2002), the next step in speaking. These results reveal the neural architecture for the first steps in producing an utterance. In addition, they show the feasibility of studying overt speech at a detailed level of analysis in fMRI studies. References: Bookheimer, S. (2002), 'Functional MRI of language: new approaches to understanding the cortical organization of semantic procesing', Annual review of neuroscience, vol. 25, pp. 151-188. Buur, P. (2009), 'A dual echo approach to removing motion artefacts in fMRI time series', Magnetic Resonance in Medicine, vol. 22, no. 5, pp. 551-560. Menenti, L. (submitted), 'The neuronal infrastructure of speaking'. Poser, B. (2006), 'BOLD contrast sensitivity enhancement and artifact reduction with multiecho EPI: parallel-acquired inhomogeneity desensitized fMRI', Magnetic Resonance in Medicine, vol. 55, pp. 1227-1235.

Abstract

Voices in the auditory modality, like faces in the visual modality, are the keys to person recognition. This fMRI experiment investigated the neural organisation of voice categories using a voice-training paradigm. Voice-morph continua were created between two female Hungarian speakers' voices saying six monosyllabic Hungarian words, one continuum per word. Listeners were trained to categorize the middle part of the continua as one voice. This trained voice category was associated with a face. Twenty-five listeners were tested twice with a one-week delay. To induce shifts in the trained category, listeners received feedback on their judgments such that the trained category was associated with different voice-morph intervals each week, allowing within-subject manipulation of whether stimuli corresponded to a trained voice-category centre, to a category boundary or to another voice. FMRI tests each week were preceded by eighty minutes training distributed over two consecutive days. The tests included implicit and explicit categorization tasks. Voice and face selective areas were defined in separate localizer runs. Group-averaged local maxima from these runs were used for small-volume correction analyses. During implicit categorization, stimuli corresponding to trained voice-category centres elicited lower activity than other stimuli in voice-selective regions of the right STS. During explicit categorization, stimuli corresponding to trained voice-category boundaries elicited higher activity than other stimuli in voice-selective regions of the right VLPFC. Furthermore, the unimodal presentation of voices that are more associated with a face may elicit higher activity in visual areas. These results map out the way voice categories are neurally represented.

Abstract

Syntactic priming has been frequently used to study syntactic processes in language production in monolinguals [1][2] and bilinguals [3]. In a previous study in language comprehension [4] we showed that passive sentences in English (the participant’s L2) can be primed by passive sentences in German (L1) and English (L2). This was manifested in faster reading times for target sentences and repetition suppression effects in left inferior frontal, left precentral and left middle temporal regions of interest in an fMRI study. However, syntactic priming in comprehension is complicated by the influence of verb repetition between prime and target [5][6]. Therefore, we conducted a reading time and fMRI study looking at the influence of verb repetition on syntactic priming. In this study of monolingual comprehension in Dutch we primed passive sentences as well as sentences with crossed-dependency structures. The reading time results revealed a syntactic priming effect for passive sentences, while the effect for crossed-dependency structure sentences interacted with the factor verb repetition. The preliminary fMRI results suggest that the repetition of passive structures leads to reductions in neural activity. The repetition of crossed dependency structures causes repetition enhancement, an increase in the BOLD response, an effect that interacts with the factor verb repetition. In conclusion, the influence of verb repetition on syntactic priming in comprehension is complex and seems to depend on the type of syntactic structure investigated. References [1] Bock K. (1986). Syntactic persistence in language production. Cognitive Psychology, 18(3), 355-387. [2] Pickering M, & Branigan H. (1999). Syntactic priming in language production. Trends in Cognitive Sciences, 3(4), 136-141. [3] Schoonbaert S, Hartsuiker RJ, & Pickering MJ. (2007). The representation of lexical and syntactic information in bilinguals: Evidence from syntactic priming. Journal of Memory and Language, 56(2), 153-171. [4] Weber K, & Indefrey P. (in press). Syntactic priming in German-English bilinguals during sentence comprehension. NeuroImage. [5] Arai M, van Gompel R, & Scheepers C. (2007). Priming ditransitive structures in comprehension. Cognitive Psychology, 54, 218-250. [6] Thothathiri M, & Snedeker J. (2008). Give and take: Syntactic priming during spoken language comprehension. Cognition, 108(1), 51-68.

Abstract

In an event-related fMRI study, we investigated to what extent semantic and
referential aspects of language comprehension recruit common or dis-
tinct neural ensembles. We compared BOLD responses to sentences containing semantically anomalous or coherent words, and to sentences containing referentially ambiguous pronouns (e.g., “Ronald told Frank that he...”), referentially failing pronouns (e.g., “Rose told Emily that he...”) or coherent pronouns. Semantic anomaly elicited activation increases in lateral prefrontal brain regions associated with semantic pro-
cessing. Referential failure elicited
activation increases in brain regions
associated with morphosyntactic processing, and additional activations
associated with elaborative inferenc
ing if readers took failing pronouns
to refer to unmentioned entities. Referential ambiguity selectively
recruited medial prefrontal regions,
suggesting that readers engaged in
problem-solving to select a unique
referent from the discourse model.
Furthermore, our results showed that semantic anomaly and referential
ambiguity recruit overlapping neural ensembles in opposite directions,
possibly reflecting the dynamic re
cruitment of semantic and episodic
processing to resolve semantically or referentially problematic situations.
These findings suggest that neurocogni
tive accounts of language compre-
hension will have to address not just how we parse a sentence and com-
bine individual word meanings, bu
t also how we determine who’s who
and what’s what during sentence and discourse comprehension