Research
The research in my laboratory is currently supported by a grant from the National Science Foundation.
Overall, our work is concerned with two principal issues: the development of intersensory perception and the development of sequence perception and learning, both in infancy and early childhood.
Development of Intersensory Perception
Our perceptual world is specified by multiple multisensory attributes. For example, when we interact with another person, we can usually hear as well as see that person. Regardless of the fact that such a person is specified by multiple auditory and visual attributes, we tend to perceive that person as a perceptually unified object rather than as a collection of unrelated sensory attributes. Our ability to have such unified perceptual experiences is based on the fact that we possess neural and perceptual integration mechanisms that enable us to associate modality-specific sensory attributes (e.g., color, pitch, temperature) on the basis of their temporal and spatial co-occurrence and on the fact that we can perceive the various forms of invariance that are normally available in our perceptual input (e.g., intensity, duration, tempo, rhythm, shape, texture, etc.). In general, our perceptual systems and our brain have evolved to detect various intersensory relations and, thus, we are able to to perceive our world as a perceptually unified place. Indeed, intersensory perception is the rule rather than the exception and occurs despite the fact that large portions of the human brain are specialized and devoted to the processing of modality-specific sensations. (If you wish to explore this topic further please check the web site of the International Multisensory Research Forum).
Our
laboratory investigates the ontogenetic origins of intersensory
perception and integration. Our work is guided by three general
principles: (a) when human infants first enter the world, they
do so with a set of immature sensory systems that provide poorly
specified sensory inputs to an immature nervous system, (b) the
sensory and underlying neural systems change and improve rapidly
during early development, and (c) the developmental changes are
guided and shaped by the constant interaction of intrinsic
organismic factors and extrinsic influences arising from the
organism's constant experiences with the external world. It is
against this backdrop that infants must somehow integrate the
constant onslaught of sensory information into a perceptually
coherent picture of their world. This is a major developmental
task and the research in my laboratory focuses on whether, when,
and how infants can perceive various forms of intersensory
relations. Through this work, we hope to uncover ways in which
infants begin to construct a world of multimodally unified and
meaningful objects and events and how this newly acquired
information provides the foundation for the development of
higher-level cognitive abilities.
Perception of Intersensory Equivalence in Infancy
One of the issues that has been of long-standing interest in my research is the perception of intersensory equivalence. Many of our multisensory experiences consist of sensations whose attributes are equivalent in different modalities. For example, in audition and vision, there are a host of sensory attributes that provide equivalent information and, thereby, provide a basis for intersensory unity. One major focus of the research in my laboratory has been the development of intersensory perception in early human and animal development and the underlying mechanisms that are responsible for observed developmental changes and transitions. In particular, I have been interested in whether, when, and how infants might utilize the various attributes that specify intersensory equivalence to perceive intersensory unity.
One basic sensory attribute that provides information regarding intersensory equivalence is stimulus intensity. In one of our earliest studies, we showed that almost right from birth infants perceive auditory-visual equivalence based on intensity. In addition to intensity, there is a whole class of temporal sensory attributes that provide a basis for intersensory integration. At the most basic level, intersensory temporal synchrony makes it possible for us to know whether the auditory and visual attributes of a given event belong together. Usually, as in a person speaking or an object bouncing against a surface, the auditory and visual properties of such events occur at the same time. Studies in my laboratory have shown that infants are sensitive to temporal synchrony relations from a very early age. In addition to temporal synchrony, audiovisual events usually have the same auditory and visual duration, temporal rate (or tempo), and overall rhythmical pattern. Again, studies in my laboratory have shown that infants can make cross-modal matches of duration/synchrony, that they can perceive audiovisual rate differences but not audio-visual rate equivalence even after prior familiarization, and that they can perceive audiovisual rhythms. For findings from other related studies please click on the publications link on your left and then on the journal articles link. The findings from our studies of infants' response to various temporal sensory attributes have been summarized in a paper in Psychological Bulletin and have provided the basis for a model of the development of temporal intersensory perception in infancy. This model is based on a comparative, epigenetic systems theoretical approach. This approach emphasizes similarities and differences across different species and analyses at various levels of organization to achieve an understanding of underlying mechanisms.
Development of Sensory Dominance Hierarchies
Another issue of interest in my research has been the development of sensory dominance. Adult human perceivers are generally visually dominant. Dominance has consequences for the way we process multisensory information, particularly because this relates to the way we perceive intersensory equivalence. To address this issue, we have investigated the earliest roots of sensory dominance and its development. In our earliest studies addressing this issue, we investigated infants' response to auditory-visual compound stimuli composed of flashing checkerboards and beeping sounds and found that infants exhibit auditory dominance. In our subsequent studies, we investigated infants' response to compounds consisting of moving objects and their impact sounds and found that when the visual information is dynamic in nature infants respond to auditory as well as visual information. In our most recent studies with more ecologically meaningful stimuli (talking or singing faces), we have found that visual information dominates responsiveness when the speech is adult-directed, but that when the salience of the auditory information is increased either by singing or by providing variations consistent with infant-directed speech (greater prosody) infants also exhibit responsiveness to the auditory information. Overall, these studies have shown that there is no fixed sensory dominance hierarchy in early human development. Rather, our studies have shown that the relative salience of information in concurrently available multisensory inputs is highly dependent on the nature of the information provided. This, in turn, is most likely due to the specific types and amounts of experience that infants of a particular age have with specific types of information.
Effects of Multisensory Redundancy on Perception & Learning in Infancy
A third issue that we have explored in our research, and one that is of great importance in studies of intersensory perception, is whether multisensory inputs and the redundancy that they provide facilitate perception and learning in early development. This issue has recently been addressed in a review chapter and in a number of studies. A recent paper reviews all of this evidence and also advances the theoretical argument that the early development of intersensory perception is characterized by heterochrony and heterogeneity.
Development of Spatiotemporal Intersensory Integration
A fourth issue in my research has been concerned with the development of auditory-visual spatial integration as well as spatiotemporal integration. In collaboration with Drs. Shinsuke Shimojo and Christian Scheier at the California Institute of Technology we have conducted two kinds of studies to date. One set of studies is aimed at better characterizing the developmental emergence of spatiotemporal integration of auditory and visual inputs and infants' localization behaviors based on such inputs. The second set of studies has been aimed at determining whether infants are sensitive to specific spatiotemporal interactions of auditory and visual sensory inputs. To study this question we took advantage of an intersensory illusion that gives the impression of two objects bouncing against one another as they move towards and through one another when a sound occurs at the point of their coincidence. We discovered that starting at 6 months of age infants also exhibit this kind of illusion and, thus, exhibit an exquisite sensitivity to specific spatiotemporal audio-visual relations.
Developmental Narrowing of Intersensory Perception
The final issue related to the development of intersensory perception that is currently being investigated in my laboratory is concerned with the effects of experiential canalization on the development of intersensory perception. To determine whether specific perceptual experience during the early months of life can affect the types of intersensory integration that infants can perform, in these studies we have presented infants with nonnative faces and vocalizations and asked whether they can integrate them.
|
"coo" |
"grunt" |
"coo"
Using a paired-preference intersensory matching procedure, we showed infants pairs of movies in which the same monkey could be seen producing either a coo or a grunt call. We discovered that young - 4 and 6 month-old - infants can integrate such faces and vocalizations but that older - 8 and 10 month-old - infants no longer do. This discovery is the first to show that the types of perceptual narrowing effects that have previous been found to take place in the auditory and visual modalities also occur across the modalities and, thus, that this is a pan-sensory type of developmental phenomenon. We are currently conducting follow-up studies of this exciting new finding.
Development of Sequence Perception & Learning in Infants & Children
The theoretical importance of the development of sequence perception and learning stems from the fact that we are constantly faced with the task of decoding sequences of sensory/perceptual information. For example, we are constantly exposed to speech utterances, melodies, and people’s complex actions. The underlying meaning of each of these types of events depends, in large part, on the underlying sequential structure and, thus, to extract the underlying meanings, we must be able to perceive sequential structure. For example, the phrase "the boy hit the girl" has a different meaning than the phrase "the girl hit the boy". Similarly, by default, events are sequentially organized entities that consist of a series of perceptual elements whose sequential organization imbues them with particular meanings.
The
work in our laboratory investigates the development of sequence
perception and learning in infants and children. In particular, the studies in our
laboratory have asked whether sequential abilities emerge sometime in
infancy and, if so, in what form. Thus, in one of our initial studies,
we asked whether infants can learn a sequentially
ordered audiovisual event and then detect changes in the
specific serial ordering of the audible, visible, or audiovisual
elements of that sequence. We have found that infants are, in
fact, able to do so and if you are interested you may download the paper
that describes these findings by clicking on this
link. In more recent studies we have followed up these initial
findings by conducting experiments in which we have investigated whether infants
can perceive and
learn ordinal position information and how the learning of
conditional probability relations (i.e., statistical relations) is
affected by how often such information is presented (i.e., its
frequency). Preliminary results from these studies indicate (a) that
4-month-old infants can learn the ordinal position of a sequence
element but that they can only do so on the basis of pair-wise
statistical relations and that they do not generalize that knowledge
to novel sequences, and (b) that young infants rely on frequency
information when learning statistical relations but that older
infants can learn statistical relations independently of statistical
relations and that the learning of the latter does not depend on the
former.
Based on our findings that infants are able to perceive serial order per se, we have also been investigating whether these initial serial order abilities are related to the development of similar kinds of abilities in early childhood. Thus, in these studies, we have been investigating whether 3-, 4-, 5-, and 6-year-old children can perceive serial order in the same kinds of events that infants have been shown to successfully perceive as well as whether these children can understand the concept of serial order in a variety of sequential tasks. This work is currently in progress.