This research concerns cognitive and neural models of the representation of meanings associated with words that denote sensory properties. The long-term goal of this investigation is to provide compelling data that bear on current theories about the relationship between lower-level brain systems (and perceptual-cognitive constructs), and higher-level "associative" brain systems (and cognitive constructs). To illustrate how my research relates to familiar issues in language comprehension, consider a common sensory-based adjective like 'yellow.' How do we account for our knowledge of what this word means when we read it or hear it (e.g., that canaries are usually yellow, but pieces of coal are not)? One might characterize such semantic knowledge in terms of stored propositions or encyclopedic entries (e.g., canary[is yellow, has wings, ]). Alternatively, one might characterize word meanings of this sort in terms of networks of distributed semantic features, along with associative links from more elementary (e.g., perceptual) representations to representations of higher orders and types. For example, one might suppose that the visual features that comprise the color concept 'yellow,' are linked to the lexical entry for CANARY, as well as to all other lexical entries that include 'yellow' in their conceptual structures.
In terms of the brain, these contrasting views can be situated within what is known about the function of the left temporal lobe and primary sensory cortical areas. Extensive regions in the left lateral and medial temporal lobe have been shown to be crucially involved in the comprehension of word meaning. But what exactly is represented in these areas? As the neural correlate of a purely propositional (or holistic) view of word meaning, one might suppose that these left temporal structures serve as storehouses for the various encyclopedic entries in the lexicon. This is congruent with the traditional notion that various sites in associative temporal cortex are the endpoints of cascading sensory processing streams, wherein representations begin as simple features in primary sensory areas, converge into more complex representations at each step toward higher-level cortex, and terminate at a (rather elusive) point where a unified perception of the world is instantiated. On a distributive/decompositional view of semantic representation, by contrast, one might characterize structures in left temporal cortex as "binding nodes" that link together basic sensory fragments from various receptive cortical areas, and associate them with phonological or orthographic word forms represented in other cortical areas. That is, on the distributive/decompositional view that I have characterized here, a more basic visual feature (e.g.,) is not re-represented in neural patterns at various sites along the processing stream, it is only represented once, in early sensory cortex. An intriguing implication that follows from this is that when we comprehend a sensory-based word (e.g., 'yellow') we rely on the very neurons that are engaged in the perception of yellow-colored things in real-world experience.
In order to provide evidence from brain activity that might bear on these contrasting positions, I have carried out a combined event-related fMRI and high-density EEG language processing study. One of the aims of this project was to determine whether the passive comprehension of tactile and olfactory sensory-based adjectives (e.g., sticky, spicy) would activate areas in respective somatosensory and olfactory sensory cortex (as expected on some variants of the decompositional view of semantic representation. Tactile and olfactory adjectives were chosen because the primary and secondary processing areas associated with these senses are fairly distant from the areas involved in the input processing of auditory and visual word stimuli. In the event-related fMRI study, sentences with tactile adjectives (e.g., 'The woman noticed the sticky substance') showed peak activations that included the hand and finger areas of primary somatosensory and motor cortex, in a manner consistent with individual hand dominance. Sentences with olfactory adjectives (e.g., 'The woman noticed the spicy aroma'), by contrast, produced peak activations in primary olfactory areas (i.e., piriform cortex and right lateral orbital-frontal cortex). Both word types, as well as abstract adjectives (e.g., 'nice') in a control condition, showed additional activation peaks in left posterior middle temporal gyrus. This evidence not only provides support for a significant role of distributed features in semantic representation and processing, it suggests that the neuronal ensembles that are used to perceive sensory stimuli are "retroactivated" during the comprehension of words with sensory meanings . One question, though, is whether this evidence supports a crucial role for primary sensory areas in on-line word meaning comprehension, or whether it merely shows that these areas become active at sometime after a sensory adjective is encountered. One might suppose, for example, that sensory-specific activation derives from conscious reflection of the referent of the sensory adjective, and that the observed increase in BOLD signal comes only because (and after) the adjective explicitly evokes an image or sensation. Because of the relatively poor temporal sensitivity of fMRI, it is difficult to determine from this evidence alone whether these signals derive from the early post-stimulus epochs (e.g., 200-500 milliseconds) that have been shown to be associated with primary lexical access procedures, or from later epochs, perhaps even as much as 1-2 seconds later. Further evidence, however, attests to the speed and automaticity of these sensory-specific BOLD activations. This evidence comes from EEG data recorded while subjects read the same sentences as in the fMRI study. Current source density analyses of the averaged scalp EEG recordings showed large sources of current flow in frontal and right lateral-frontal areas that were time-locked to the olfactory adjectives, and over the left sensory-motor area for tactile adjectives. These brain-generated scalp currents were maximal (as sources or sinks) between 200-400 milliseconds after the onset of critical words. Thus, it is unlikely that the neuronal activation of these areas can be entirely attributed to conscious images or sensations that subjects may have generated during the task. I have explored the EEG data further with source-localization algorithms that use realistic head models (i.e., with parameters for thickness, density, and conductivity of cortex, csf, skull, and scalp) in order to locate the most likely cerebral sources for the scalp current density patterns that each adjective type evoked. These procedures consistently estimated sources for the scalp currents that were remarkably consistent with the fMRI analyses. Thus, I am more confident that the areas detected by fMRI correspond to neural activity that is crucially involved in on-line word comprehension.
The ultimate goal of this research is to go beyond delimiting the various areas that become active during word comprehension, and gain an understanding of how information from these areas is bound together into a unified cognitive whole. It is becoming increasingly clear that a significant component of the mechanism that binds together information from spatially separated neural ensembles during cognitive and perceptual processing is precise phase-locking of oscillating PSPs between such distant ensembles (e.g., at ~40Hz). Phase-synchrony analysis is an exciting direction in human brain and cognitive research that is beginning to attract serious attention. The experiments described above were designed from the beginning to explore the role of long-range phase-synchrony in word comprehension. I am currently working with my graduate assistant, Kayo Inoue, to develop and implement algorithms for extracting oscillatory signals from EEG recordings and determining which areas go into precise phase-locking during various stages of word comprehension. For example, based on the findings above, we are exploring whether there is significant phase-locking between signals recorded from middle temporal gyrus and frontal electrodes in the olfactory condition, and from electrodes over motor cortex in the tactile condition. Based on our topographic fMRI and EEG data, we expect that phase-locking should occur between 200-400 ms after word presentation. Preliminary results have supported our expectations in that respect.