Schedule of Events | Symposia

Neural representations underlying sound-shape associations

Poster Session F - Tuesday, April 1, 2025, 8:00 – 10:00 am EDT, Back Bay Ballroom/Republic Ballroom

Josh Dorsi¹ (jxd5826@psu.edu), Simon Lacey^1,2,3, Lynne Nygaard⁴, Krish Sathian^1,2,3; ¹Penn State Health Milton S. Hershey Medical Center, ²Penn State College of Medicine, ³Penn State College of Liberal Arts, ⁴Emory University

Sound symbolism is the systematic relationship between speech sounds and meaning (e.g., the pseudowords /kɛke/ and /molo/ sound pointed and rounded, respectively). Prior work showed that representational dissimilarity matrices (RDMs) of shape ratings and spectro-temporal acoustic parameters (the fast Fourier transform [FFT], spectral tilt and speech envelope) are correlated (Lacey et al., Cognitive Science 44:e12883, 2020). Here, we used functional magnetic resonance imaging (fMRI) to investigate the associated neural processes. Participants rated the roundedness/pointedness of 12 auditory pseudowords during and immediately after an fMRI scan. We constructed searchlight RDMs based on the blood oxygenation level-dependent (BOLD) signal within 9 mm spherical regions centered on each voxel in the cerebral cortex. We examined the correlations between these BOLD RDMs and RDMs based on the ratings and spectro-temporal parameters. As shown previously, the ratings RDM was correlated with the RDM of each spectro-temporal parameter. There were significant correlations between the BOLD and ratings RDMs in areas typically associated with visual processing (V4 and superior occipital gyrus), suggesting crossmodal processing of the auditory pseudowords. BOLD RDMs were significantly correlated with RDMs of the FFT and spectral tilt in the inferior frontal gyrus, in pars opercularis and triangularis, respectively, and with the FFT RDM in the left angular gyrus. These regions are known to be engaged in language processing. Our results are consistent with the idea that sound-symbolic shape processing recruits the language and visual systems to establish crossmodal associations between auditory word forms and visual concepts.

Topic Area: PERCEPTION & ACTION: Other