Multisensory Integration Session 1 Current Topics in Perception PSYD51

Current Topics in Perception
PSYD51
Multisensory Integration
Session 1
January 6, 2009
1
Current Topics in Perception
PSYD51
Blurb: multisensory integration
Recent research topic
Multimodal information in the world
Making sense out of it requires multisensory
integration
2
Current Topics in Perception
PSYD51
The Course:
Multisensory Integration (PSYD51), LEC 01
Place: HW 309
Time: Tuesdays, 1–3pm
Intranet
The Instructor:
Dr. Matthias Niemeier
Office: S-572
Phone: 416-287-7466
email: [email protected]
Office hours: Tuesdays, 12–1pm, or per appointment
3
Current Topics in Perception
PSYD51
Prerequisites
PSYB51 – Perception and Cognition
PSYC08 – Advanced Data Analysis in Psychology
Grading
No exams, active participation
45% Thought papers (9 x 5%)
20% Presentation
15% In-class participation
20% Research proposal
Readings
No textbook
Research papers (see reading list)
4
Grading System PSYD51
Paper Presentations
20% of the total mark
1 paper from reading list
First come first serve – please email me
Pdf’s on the intranet
Free presentation (notes)
Three parts:
a) Brief intro (~5 min)
b) Summary of methods, results & conclusions. Figures on
the intranet (~25 min)
c) Group discussion led by presenter (~15 min)
5
Grading System PSYD51
Thought Papers
45% of the total mark (9 papers x 5%)
Read both research papers of the week, thought paper on one
“How to write a thought paper” => intranet
Scoring system
Due for the respective session of the seminar
1-2 pages, double-spaced
2 sentences summary maximum
Your own thoughts
Intro (1 paragraph, “Tell them what you are going to tell them”) –
body (2-4 paragraphs, “Tell them”) – conclusion (1 paragraph,
“Tell them what you told them”)
6
Grading System PSYD51
In-class Participation
15% of the total mark – so please be there and be
prepared to say something in class!!
Help the presenter
Do you agree with the authors? Are you impressed?
Something that you didn’t understand? Can you help
clarifying things? Is there a reason to believe that the
paper is invalid? Can you play devil’s advocate?
Suggestions for future research?
7
Grading System PSYD51
Research Proposal
20% of the total mark
“How to write a research proposal” => intranet
Scoring system
Due for the last day of the seminar
8 pages, double-spaced
Title page – Abstract – Introduction – Methods –
Predicted Results – References
8
Comparing
Senses
sense
vision
hearing
touch
smell
taste
organ
eye
ear
skin
nose
tongue
physical
quality
of
stimuli
light
electromagnetic
wave
sound
pressure
wave
deformation
physical
objects
odour (scent)
airborne
chemicals
aroma
(flavour)
dissolved
chemicals
Other channels of sensory input?
Temperature, pain? Balance? other senses?
Perhaps a better physical classification: electromagnetic,
mechanical, chemical stimuli
9
Combining and integrating
sensory information
Combining vs. integrating
V: vision, N:
neck muscle
info, A:
audition, P:
proprioception
from arm, L:
location signal
Example: knock on wood
Sensory combination: interactions between non-redundant sensory signals
Sensory integration: interaction between redundant ones
10
Combining and integrating
sensory information
Why combining/integrating sensory sources?
Combined alarm systems (jungle,
darkness, braking)
Object identification: information from
different modalities support each other
Visual-auditory localisation (twilight,
peripheral visual field, occlusion, cluttered
scenes)
Reliable info for control of locomotion
(e.g. visual-proprioceptive crossadaptation: Pelah and Barlow 1996)
Coherence of the world: object unity,
perceiving one’s own presence in space
(e.g. cue conflict > motion sickness)
Large interest from engineering: sensory fusion in robots, VR, video games
How do nervous systems combine sensory information ?
11
Multisensory receptive
fields
Multisensory neurones
Single cell recordings in cat
superior colliculus
Input from various cortical
regions (visual, auditory,
somatosensory, motor...)
Alignment of receptive fields >>
integrated multisensory maps
Spatial coincidence detectors
12
Multisensory areas
in the cortex
How/where are the different senses integrated?
fMRI, a tool to study some cases of multisensory integration: most
prominent, e.g., integration of visual & auditory info (Calvert et al. 1998)
red: viewing mouth movements without sound
blue: listening to speech
yellow: activated by both signals
Regions for crossmodal identification
13
Interaction of light
and sound
An experiment to demonstrate visuo-auditory integration:
‘streaming and bouncing’ (Sekuler et al. 1997)
Two objects moving straight
across each other are
perceived as moving on
intersecting linear trajectories
A sound at the intersection
point changes perception
dramatically: the objects
appear to collide and bounce
14
Reading
emotions
Observers asked to rate the sadness of a series of faces
with different expressions between happy and sad
Sadness ratings follow a
psychometric function: from
low to high for happy to sad
faces
Combining the face display
with a sad or happy voice
shifts the psychometric curve
to higher or lower sadness
ratings
15
Preliminary Schedule
for the Course
6 Jan
Introduction
13 Jan
Neural mechanisms of multisensory integration
20 Jan
Neural mechanisms of multisensory integration II:
Supramodal areas (?)
27 Jan
Multisensory object perception
3 Feb
Optimal multisensory integration
10 Feb
Optimal sensorimotor combination
24 Feb
Vision and sound in space
3 Mar
Vision and touch in space
10 Mar
Body schema
17 Mar
Learning & plasticity
24 Mar
Neuropsychological deficits
31 Mar
Synaesthesia
16
Outlook on the
Course Topics
Jan 13: Neural mechanisms of multisensory integration
Lateral connections: Anatomy: cortical auditory input
into V1 (Falchier et al. 2002).
fMRI: Polymodal motion processing in posterior parietal
and premotor cortex (Bremmer et al. 2001).
Jan 20: Crossmodal cueing of attention
Spence & Driver (1997): 1-way cross-modal dependence in
covert orienting; audition influences vision
Ward et al. (2000): auditory cues fail to orient visual attention
17
Outlook on the
Course Topics
Jan 27: Multisensory object perception
Viewpoint dependence in visual and
haptic object recognition (Newell et
al. 2001).
Visuo-haptic object-related
activation in the ventral visual
pathway (Amedi et al. 2001).
18
Outlook on the
Course Topics
Feb 3: Optimal multisensory integration
Introduction to MLE
Humans integrate visual and haptic
information in a statistically optimal
fashion (Ernst et al. 2002).
19
Outlook on the
Course Topics
Feb 10: Optimal sensorimotor
combination
Optimal
transsaccadic
integration
(Niemeier et al.
2003).
Statistical decision
theory and rapid,
goal-directed
movements
(Trommershauser
et al. 2003).
20
Outlook on the
Course Topics
Feb 24: Vision & sound in space
Optimal ventriloquist effect (Alais et al. 2004).
http://www.youtube.com/watch?v=hsfnP9DK-Zg
Recalibration of audiovisual time perception
(Fujisaki et al. 2004).
Mar 3: Vision & touch in space
Coordinate transformations for
representing visual and tactile locations
in parietal cortex (Avillac et al. 2005).
Influences of gaze direction on
crossmodal effects (Macaluso et al. 21
2002).
Outlook on the
Course Topics
Mar 10: Body schema
Out of body experience & rubber gloves (Pavani et al. 2000).
fMRI and my hand (Ehrsson et al. 2004).
Mar 17: Learning & plasticity
Experience for the development of multisensory integration
(Wallace et al. 2004).
Plasticity in sound localization induced by compressed spatial
vision (Zwiers et al. 2003).
22
Outlook on the
Course Topics
Mar 24: Neuropsychological deficits
Split-brain: Remapping of space (Spence et al. 2001).
Impaired crossmodal interactions in a patient with Balint
syndrome (Valenza et al. 2004).
Mar 31: Synaesthesia
Priming and colour-number
synesthesia (Mattingley et al. 2001).
Neurocognitive mechanisms of
synesthesia (Hubbard et al. 2004).
Doorbell ringing Dog barking
23