Schedule of Events | Symposia

Selective attention directed to audiovisual and audio-only continuous speech

Poster Session A - Saturday, March 29, 2025, 3:00 – 5:00 pm EDT, Back Bay Ballroom/Republic Ballroom

Yimeng Wang¹ (yimengw@umass.edu), Alexandra Jesse¹, Lisa D. Sanders¹; ¹University of Massachusetts Amherst

Both selective attention and seeing the speaker’s face improve speech processing. However, the effects of attention and visual speech on event-related potentials (ERPs) appear to be in opposition; directing attention to speech elicits larger auditory ERPs (additional neural resources for attended stimuli), whereas visual speech results in smaller auditory ERPs (fewer resources needed for partially redundant information). This apparent contradiction may be explained by differences in listening conditions (e.g., syllables/sentences, presence/absence of competing talker). In the current study, we employed EEG and ERP analyses to directly compare the effects of attention, visual speech, and competing speech while participants listened to stories for comprehension. Specifically, a female talker was presented as audiovisual or audio-only, with or without a concurrent audio-only male talker, when listeners were instructed to attend to the female talker or the male talker. Cross-correlations between EEG and speech amplitude envelopes indicate a stronger correspondence with attended speech, as well as with audiovisual speech. Similarities in the effects of selective attention and visual speech on the patterns of cross-correlation values across speech-to-EEG lags suggest the mechanisms by which visual speech affects processing of continuous auditory speech in multi-talker environments overlap with those of attention; seeing a talker facilitates attending and hinders ignoring their speech. Further, ERPs elicited by the naturally occurring acoustic onsets in speech provided the opportunity to compare the cross-correlation results with typical ERP effects. Each approach provides additional information about the relative contributions and interactions of selective attention and visual speech under realistic listening conditions.

Topic Area: ATTENTION: Multisensory