Complex Experiences Are Represented as a Subset of Key Moments Which Capture the Underlying Semantics
Poster Session C - Sunday, March 30, 2025, 5:00 – 7:00 pm EDT, Back Bay Ballroom/Republic Ballroom
June-Kyo Kim1 (junekyo.kim@mail.utoronto.ca), Aditya Upadhyayula2, Jeffrey M. Zacks2, Alexander Barnett1,3, Zachariah M. Reagh2; 1University of Toronto, 2Washington University in St. Louis, 3McGill University
Recent work has used a Storyboard task to show that people represent and remember complex experiences as a sequence of key moments (KMs). These KMs capture distinct information compared to event boundaries, and could provide novel insights into event cognition. However, the nature of KMs and their significance remains to be fully explored. We tested whether the KMs capture meaningful semantic information pertaining to the underlying experience using topic modeling, using an episode of BBC’s Sherlock. This dataset was annotated in detail, describing scene and dialogue in segments spanning every few seconds. In addition, we also used spoken recall transcribed from the 17 participants from an open access fMRI dataset to train the topic model, and then fit the movie annotations to the trained model. This produced a vector of topic activations that represents the abstract concepts present in the annotations for each event. To calculate the significance of each segment, we used a “lesion” approach where we systematically deleted each segment and correlated the resulting vector of topic activations to the original topic model. These lesioned correlations were then compared to the event boundary and storyboard probabilities calculated for each segment. The segments likely to be KMs, when lesioned, resulted in a significant drop in the topic model correlation (above and beyond event boundaries). Additionally, permutation analyses revealed that KMs were associated with important semantic elements in the narrative more so than would be expected by chance. This suggests that KMs capture meaningful semantic information about the underlying experience.
Topic Area: LONG-TERM MEMORY: Episodic