Do Visual Aids Assist Musical Training?

As a professional flutist, Ioanna Zioga has often wondered how people learn music. She herself has experienced many different musical training techniques from various musicians.

“Some would draw figures on a whiteboard to visually represent the music; others would sing melodies; others would play the music themselves on the flute in order for me to mirror their moves,” she recalls “I realized that all these methods had a very different impact on my mental representations of music, and, importantly, on my performance and long-term learning. So, what is the best method to learn music?”

Zioga, now a postdoctoral researcher at the Donders Centre for Cognitive Neuroimaging, studied that question head-on in a recent study on the effects of visual aids on music learning. As published in the Journal of Cognitive Neuroscience, Zioga and colleagues (during her Ph.D. work at Queen Mary University of London) found that visual cues do not help in learning a new musical style. 

“Mere reproduction of music based on visual cues is not sufficient to achieve deep learning of a musical style,” she explains. “In contrast, introducing challenging material during training can lead to better processing, encoding, and subsequent retrieval of the learned material, which are crucial for learning.”

CNS spoke with Zioga about the study and its implications for musical training.

CNS: What was the genesis of this study?

Zioga: During my Ph.D. at Queen Mary University of London, which I completed under the supervision of Dr Caroline Di Bernardi Luft, I investigated the relationship between learning and creativity in music. Humans learn musical styles by exposure; they learn to anticipate upcoming musical events while listening to music, and potentially produce their own compositions. This process is based on statistical learning which allows us to predict the likelihood of upcoming events, such as the next note in a melody. This knowledge enables us to identify errors in music, but also to enjoy and be surprised by unpredictable events. Learning these statistical structures happens via exposure, but also via repetition/training and performance of the pieces. 

We wanted to investigate how learning the statistical regularities of an unfamiliar musical style is linked to the production of novel compositions within the new musical style, as well as uncover the respective neural correlates. 

CNS: What have previous studies shown about this topic?

Zioga: Previous cognitive neuroscience studies on multisensory learning have shown an advantage for learning with multisensory cues. For example, adults are faster at detecting a target when correlated information is presented to multiple sensory modalities than when information is presented unimodally. 

The Simon task has also been widely used to study statistical learning: it uses a game device with four colored buttons corresponding to different tones. Every time a tone is played, the respective button lights up. Participants are able to reproduce longer sequences when spoken color names and buttons light up simultaneously compared to when they only hear the color names or they see the buttons lighting up. Beneficial effects of audio-visual presentation on learning have also been found in other tasks, such as presentation of biological textbook material with vs. without verbal instruction, in addition to pictorial presentation. We therefore tested the assumption that adding visual aids helps learning of music. Importantly, we tested this by recording the brain responses during listening to the new music, rather than by just evaluating participants’ performance on reproducing the melodies.

CNS: What types of visual aids are commonly used in real-world musical training?

Zioga: During the first years of music education, teaching music usually takes the form of a game: different colors represent different pitches, imaginary stairs symbolize musical scales, and claps represent rhythms. A widely used method is to put colorful stickers on the keys of a piano keyboard or on the violin fingerboard to indicate finger positions.

Due to technological development, there are several apps and software which use audiovisual cues to help people learn music. Those are usually designed to engage younger kids, as well as older students, in playing music by mimicking what is shown in a video or by following visual cues to reproduce melodies. These tools are very appealing as they make people feel empowered by being able to reproduce songs easily, but do they help learning of the actual statistical structures?

CNS: What was the set-up of your study?

Zioga: In our study, we trained non-musicians on a new musical style in a laboratory setting. We were the first to train non-musicians by combining passive listening to music and active playing on a sound keyboard on 3 separate days. This was an ecologically valid experiment that resembled real-life training. In contrast to previous studies, which assessed learning just after one training session, we performed a 1-day follow up test, in order to ensure we measure learning rather than immediate effects of exposure. 

Non-musicians were trained on an unfamiliar musical style and were asked to reproduce note by note on a sound keyboard the new music. The audio-visual group had each note color-coded on screen, whereas the auditory-only group had no color indication during training (just the sound). We assessed learning of the new music before and after training by presenting melodies ending on correct or incorrect notes and asking participants to judge the correctness of the final note. During the tests, we recorded the brain signals (EEG) of the participants, and subsequently analyzed brain activity during listening to the new music.

Overall, our results suggest that performance during musical training and actual statistical learning are two distinct or relatively independent processes.

CNS: What were you most excited to find?

Zioga: Our main finding was that visual aids boosted melody reproduction during training but did not improve learning as assessed in the final test. This was also reflected in the neural responses, as training without visual cues was associated with increased neural sensitivity to the musical style. Overall, our results suggest that performance during musical training and actual statistical learning are two distinct or relatively independent processes.

As expected, participants who received musical training with visual aids were able to play longer sequences compared to those without visual aids. It is much easier to remember colors on a screen and use the respective color keys to play a melody, than play music by ear. Interestingly, the superior training performance of the audio-visual group was not reflected in greater knowledge of the music, as the audio-visual and the auditory-only groups performed equally well in the test after training. 

Our findings suggest that participants’ reliance on the auditory information constituted an extra difficulty during training, which may have led to enhanced neural sensitivity. Visual aids made the task much easier, thus requiring less cognitive engagement, and providing no enhancement of subsequent retrieval. It could be that training with visual aids might have triggered an over-reliance on these cues, which then distracted them from the music.

CNS: How does this study contribute to the field of cognitive neuroscience?

Zioga: Cognitive neuroscientists might need to consider that the positive effects of multisensory training are not necessarily transferred to the statistical learning of a single modality. Importantly, multisensory training might constitute a distraction from encoding the main material. Indeed, learning comprises a long-term change in behavior or knowledge that supports retention and transfer, whereas training refers to temporary fluctuations in behavior which are observed close to the acquisition period. Therefore, reproducing musical sequences during training vs. learning the statistical properties of the new musical style might represent two distinct processes. 

CNS: What’s next for this line of work?

Zioga: In the laboratory setting, lots of studies, including ours, have investigated the effects of training within short periods of time, usually a few days. However, there is still a lot of uncertainty as to how our brain encodes information that is retained in memory for a longer time. Therefore, future studies are needed to investigate the efficacy of musical training using auditory only or audio-visual aids in the longer-term, for example over periods of weeks, months and years.

CNS: Anything I didn’t ask you about that you’d like to add?

Zioga: At this point, I would like to emphasize that people should not be discouraged from trying the modern methods of learning music with visual aids, as they are engaging and motivating. Our findings do not mean that visual aids do not work at all; our findings apply more to people developing those tools, in order to make them more effective. It is important to consider this because if someone is only motivated to learn music that way, it’s better to do that than nothing. I would encourage everyone to learn music, independently of the training method.

-Lisa M.P. Munoz