The Music That Plays the Brain

Zhiqian Chen
Department of Computer Science, Rochester Institute of Technology
Listen

Pink Floyd, played back from electrodes on the brains of 29 patients

Heard vs decoded

heard
decoded from the brain

Rhythm and melody contour survive. The decoder learned this one song.

The question

Many people with advanced dementia can still sing the hymns of their youth

Older adults singing together

A choir of older adults sings at a Dutch care home

Music reaches a brain that has lost almost everything else.

What if we could choose the song for one of them, and watch her brain answer?

Your turn

Same notes, same sound. Which one is Beethoven?

version A
version B

Same notes. Only the rule that picks each next note differs

The claim

Soon we will compose music for one brain, and watch it answer.

Today: how close we are, the one thing missing, and what we can build together.

Vision

Three findings

Three steps to get there

  1. We can already read music in the brain, and music AI predicts part of its response.
  2. We cannot steer it yet, because natural music does not change on cue.
  3. Generative models begin to turn music into a dial.
The map

Four ways the field studies music and the brain, and a missing fifth

Musicsongs, scores, soundBrainEEG, MEG, fMRI, ECoGModelsaudio and music AIPatientstherapy outcomes1 Encodepredict the brain from the music2 Decodeplay the music back from the brain3 Comparedoes a model resemble the brain?4 Treatdoes music help patients?5 Controlset the music, one knob at a time
1 Why music

Music releases dopamine twice: while we wait for the peak, and when it lands

time in the piece anticipation caudate peak pleasure nucleus accumbens Expectation is part of the reward. Whoever shapes expectation shapes the response.

Schematic of the paper's finding, not data

Robert Zatorre
Robert ZatorreMcGill, senior author

The chills were confirmed with autonomic recordings, the signals a wearable picks up.

1 Why music

Your auditory cortex has a population that cares about song, and little else

speechinstrumental musicmusic with singingother soundsspeech selectivemusic selectivesong selectiveschematic of the component responses, 165 natural sounds
Sam Norman-Haignere
Sam Norman-HaignereU Rochester
1 Why music

A steady beat can make a person with Parkinson's walk better

Gait training session

Patients fall into step with the beat

Patients step to the pulse: rhythmic auditory stimulation.

2 What we can see · Running example

Your brain predicts every next note, and EEG can see the surprise

E5D♯5E5D♯5E5B4D5C5A4C4E4A4B4E4G♯4B4C5E4surprise of each note given the one before (bits)

Di Liberto 2020: in 20 listeners, adding surprise raises EEG prediction from 0.073 to 0.078.

2 What we can see · Encode

Try it: imagine a march. Your cortex now shows a peak the beat does not have

beat only2.4 Hzimagine a march2.4 Hz1.2 Hzimagine a waltz2.4 Hz0.8 Hz1.6 HzEEG frequency, schematic of the paper's finding
Sylvie Nozaradan
Sylvie NozaradanUCLouvain
2 What we can see · Decode

From brain scans, an AI composer makes music that matches what people heard in genre and mood

what they heard
made from their brain scan

the authors' best example, 5 people scanned

85%

of clips picked out against another clip, chance 50%. 66% in a low level sound space.

Genre, instruments and mood carry over. Timing and tempo often do not.

2 What we can see

A music AI's layers line up with the auditory cortex's hierarchy

00.5104812model layerfar sitesearly sitesredrawn, approximatefit, normalized

Early layers fit primary auditory cortex. Late layers fit sites farther out

Nima Mesgarani
Nima MesgaraniColumbia
2 What we can see · My own data

Averaged over hundreds of notes, the notes MusicGen calls surprising get a bigger brain response In progress

00.20.40.6seconds after the note startsnotecluster, p = 0.0028surprising notesexpected notes0EEG (SD)

0.16 s after the note, over the top of the head

29 listeners. Note by note: r = 0.083, an untrained model 0.155, both chance level. The untrained model is not yet tested on the average.

OpenNeuro ds005876Chen, unpublished analysis
2 What we can see · Interactive demo

Try it: step to the next surprising note and watch the EEG of the 19 listeners who heard this song

3 Seeing is not understanding

Arrow 3: does a model of music resemble the brain?

We can read music from the brain. Do models explain it?Musicsongs, scores, soundBrainEEG, MEG, fMRI, ECoGModelsaudio and music AIPatientstherapy outcomes1 Encodepredict the brain from the music2 Decodeplay the music back from the brain3 Comparedoes a model resemble the brain?4 Treatdoes music help patients?5 Controlset the music, one knob at a time
3 Seeing is not understanding

Past about eight notes, a better music model fits the brain worse

fit to the brainMEG, gain from surprisepeak at 8 notesmodel at musicnext note accuracy151025100400notes of context the model may use
3 Seeing is not understanding

The catch: in every study of real music, the music was fixed

One song

the decoder never heard another

540 clips

picked from fixed recordings, nobody changed their tempo or tune

Fixed songs

my surprise curve rode along with everything else in them

We can see the musical brain. With recorded music, nobody can change one property and hold the rest.

4 The clinic

In dementia, music lifts mood slightly, and the effect is gone four weeks later

music worsemusic bettercertaintyDepressive symptomsmoderateAgitation, aggressionmoderateCognitionlow to very lowFilled: interval excludes zero (orange: moderate certainty). Four weeks after treatment, no clear effect.
4 The clinic

A beat you can set helps people walk, in Parkinson's and after stroke

A metronome

+11.3 m per min

walking speed after stroke with rhythmic auditory stimulation, 9 trials, moderate certainty

In Parkinson's, 18 trials: stride length and walking speed improve.

The beat is a parameter that can be set and repeated.

tap along: 72 bpm
Michael Thaut
Michael ThautU Toronto
4 The clinic

The bottleneck is not the measurement.
It is the stimulus.

My view. Short artificial sequences already have knobs (Koelsch, Quiroga-Martinez). Generators begin to give whole pieces of real music the same knobs.

4 The clinic · After the turn

Change one chord, and the brain answers within 200 ms

a regular cadencethe same with an odd last chord
ERANregular chordodd chord0200400600ms after the last chord, right frontalnegschematic, not data

Your brain just did this

Where a stimulus has a knob, the brain moves with it. Generators bring the knob to whole pieces.

The map

Arrow 5: set the music, one knob at a time

Now the stimulus becomes the variable.Musicsongs, scores, soundBrainEEG, MEG, fMRI, ECoGModelsaudio and music AIPatientstherapy outcomes1 Encodepredict the brain from the music2 Decodeplay the music back from the brain3 Comparedoes a model resemble the brain?4 Treatdoes music help patients?5 Controlset the music, one knob at a time
5 Music you can experiment on · Running example

Turn one dial: the same notes go from shuffled to almost a tune. Then hear Beethoven In progress

order 0, shuffled
order 1, walk the graph
Beethoven

Same notes and rhythm. Surprise is computed for each stimulus.

Chen et al., Markov graph analysis, in preparation
5 Music you can experiment on · A neural generator

Turn one dial inside a neural music generator: same seed, only the tempo words change

slowmediumfast
60 bpm askedmeasured 59 bpm100 bpm askedmeasured 100 bpm160 bpm askedmeasured 158 bpmloudness over 8 seconds
5 Music you can experiment on · A neural generator

A neural generator plays the same three melodies as real music: shuffled, chained, Beethoven

order 0order 1Beethoven

One seed, generated for this talk. Same prompt, only the melody we feed in changes.

Predictability can be fed in

5 Music you can experiment on · Running example

Prediction: the more natural the order, the more closely EEG should track each note's surprise

More structure, sharper expectations, so a broken one costs more.

how stronglyEEG trackssurprisePREDICTION, NO DATA YETorder 0order 1order 4the real piece

First test: four settings, 24 young and 24 older listeners, decision rule set before recording. Planned

5 What becomes possible

Close the loop: the brain's response chooses the next phrase of music Vision

the same piece moving through three settings
A listener in an EEG cap
order 0order 1Beethovenstaged mock up: the dial follows a script, not EEG

Generator, listener, EEG, model of the brain: the loop that picks the next phrase. Over whole phrases or sessions, not single notes.

5 What becomes possible

One day a music therapy could read like a prescription Vision

RxMusic dose, week 3example: a listener, 81, moderate dementialength20 min, after lunchpredictabilityorder 2, familiar contourtempo72 bpm, steady beattimbrepiano and voiceadjustby her response, weekly

Every line is a setting a generator could hold

Today a trial reports a playlist. A dial lets it report a dose.

The same dial could be read out with EEG, heart rate, skin conductance or sleep.

fictional example

5 What becomes possible · Next month

Three experiments you could run next month with these stimuli

EEG lab

three orders of the same notes, your own TRF

ECG or wearables lab

three tempi, heart rate and skin conductance, longer versions on request

Clinical team

log the settings of whatever music you already play

Close

Next time she sings, we choose the song, and her brain shows us what she responds to Vision

Older adults singing together

We can read it. We cannot steer it yet. Generators begin to give us the dial.

We need an EEG or physiology lab with older or hearing impaired listeners

once more, the song as the brain played it

the stimuli, their settings and the demo
aquastar.github.io/talks/music-to-brain/stimuli

Backup

Backup slides

5 What becomes possible · First step

First step: one EEG study, four dial settings, 48 listeners Planned

Stimuli: chains trained on a melody corpus, 20 s each, same timbre, settings loggedorder 0order 1order 4real, unfamiliarheld out melodies,familiarity rated24 young, 24 olderhearing screened64 channel EEG, 40 minOutcomesurprise gain in a TRF over envelope, onsets and interval sizeTest and decision rule, set before recordinga linear trend across order 0, 1, 4 in both groups

Planned design, group size to be fixed from the pilot

5 What becomes possible

Three things a music dial makes possible Vision

Rehabilitation session
A music dose for rehabilitation and dementiasettings written down, adapted to the person
EEG experiment
A standard music probe for EEGlike gratings in vision research
Child at the piano
Music tuned to a listenerhearing, age, taste

The same dial could be tested with heart rate, skin conductance and sleep EEG.

2 What we can see · Encode

EEG tracks how surprised you are, note by note

EEG prediction correlation, mean of 20 listenerssound only0.073sound plus surprise0.078

Small but reliable

  • Seen in most of the 20 listeners
  • Stronger in musicians
3 The catch

In a reconstruction, the brain supplies 128 numbers. The generator supplies the rest

fMRI 128 numbers music generator trained on hundreds of thousands of hours audio The brain picks a direction. The prior fills in the music.
Backup · Questions

Likely questions and where the answer is

  1. Has knob control of music been done before? ERAN and MMN studies, knob table
  2. Does MusicGen beat an untrained model? Per note check, shuffle plot
  3. Is the reconstruction just the generator? The 128 numbers slide, Denk bars
  4. Can a generator change style, not only tempo? Steering slide (under review)
  5. Why is the stimulus the bottleneck, not bias or sample size? Cochrane reasons slide
  6. Does music therapy work at all? Five reviews and the dementia forest plot
2 What we can see · Decode

Twenty nine epilepsy patients listened to a Pink Floyd song with electrodes on the cortex

A rare window: ECoG during surgery planning

  • 347 of 2,668 electrodes tracked the song
  • One song, three minutes
a simpler decoder
Robert Knight
Robert KnightUC Berkeley, senior author
5 Music you can experiment on · Own work

Remembering a single note already tells six traditions apart In progress

majority class 21%20%50%80%70.2%0123variableMarkov order (notes remembered)likelihood

style identified from a 12 by 12 transition table, majority class 21%

Order 0 knows only which notes occur. Order 1 knows which note follows which. That one step is the big jump.

Chen et al., in preparation
5 Music you can experiment on

Inside a large music generator, a style can be removed cleanly. Adding one does not work yet Under review

remove a styleadd a style leaves easily, still music the push misses the thin set Knobs exist, but they are not symmetric. We must know which ones are real.

Schematic, not data

Co authored work under review, MusicGen small and mediumCopet et al. 2023
1 Why music

The brain regions that hold long known music show the least damage in Alzheimer's

Young adults, 7T fMRI long known songs activate caudal anterior cingulate and ventral pre SMA Alzheimer's patients the same regions show the least atrophy and least metabolic loss The music memory network is reached late, so the hymn survives. 32 young adults, 20 patients. Amyloid in these regions is not lower.
The map

Arrow 1: predict the brain from the music

We can predict the brain from music. The stimulus was a fixed recording.Musicsongs, scores, soundBrainEEG, MEG, fMRI, ECoGModelsaudio and music AIPatientstherapy outcomes1 Encodepredict the brain from the music2 Decodeplay the music back from the brain3 Comparedoes a model resemble the brain?4 Treatdoes music help patients?5 Controlset the music, one knob at a time
2 What we can see · Encode

Di Liberto's test: does EEG need surprise, beyond the sound itself?

Two models, one difference

Predict EEG from acoustics (A), then from acoustics plus surprise (AM). The gap is the brain's share of expectation.

Giovanni Di Liberto
Giovanni Di LibertoTrinity College Dublin
The map

Arrow 2: play the music back from the brain

We can play music back from the brain. The stimulus was one fixed song.Musicsongs, scores, soundBrainEEG, MEG, fMRI, ECoGModelsaudio and music AIPatientstherapy outcomes1 Encodepredict the brain from the music2 Decodeplay the music back from the brain3 Comparedoes a model resemble the brain?4 Treatdoes music help patients?5 Controlset the music, one knob at a time
3 The catch · My own data

My own data shows the limit: note by note, MusicGen does no better than an untrained model

-0.300.3r when surprise and response are shuffled, 2000 timesuntrained model 0.155MusicGen 0.083

Both sit inside the shuffles: MusicGen 786 of 2000, untrained 215 of 2000. 105 notes, 3 fixed songs.

3 The catch

Everything so far is a correlation.
To test a cause, something has to be changed.

The map

Arrow 4: does music help patients?

We can see the musical brain. The stimulus is still fixed.Musicsongs, scores, soundBrainEEG, MEG, fMRI, ECoGModelsaudio and music AIPatientstherapy outcomes1 Encodepredict the brain from the music2 Decodeplay the music back from the brain3 Comparedoes a model resemble the brain?4 Treatdoes music help patients?5 Controlset the music, one knob at a time
4 The clinic

Reviews lower certainty for three reasons, and trials rarely say which music was played

GRADE reasons for lower certainty (rows 1 to 3) and a reporting gap reviews name (row 4)risk of biasno blinding, open allocationimprecisionsmall samplesinconsistencydifferent controls and outcomesand the music is rarely describedwhich piece, how loud, how longthe one a lab can fix
Preview 2 of 4 · Seeing is not understanding

The best music model looks hundreds of notes back. The brain fits best at about eight

fit to the brainMEG, gain from surprisepeak at 8 notes151025100400notes of context the model may use
Preview 3 of 4 · The clinic

In dementia, music lifts mood a little. Cognition shows no clear gain

music worsemusic betteroutcomeDepressive symptomsprobably improves slightlyCognitionunclear
1 Why music

We know music reaches deep into the brain.
Do we know why?

2 What we can see · Running example

Listening is prediction: each note of Für Elise confirms or breaks an expectation

E5D♯5E5D♯5E5B4D5C5A4C4E4A4B4E4G♯4B4C5E4surprise of each note given the one before (bits)
2 What we can see · Encode

The gain grows when the listener model remembers more of the past

12481632allbars of the piece the listener model may remember00.005EEG gain from surprise (median)

Memory length matters

The music stayed the same. Only the model of the listener changed.

2 What we can see · Encode

The limit: in natural music, surprise can be measured but not set

The melodies were fixed Bach pieces. Surprise rose and fell with the notes the composer wrote, not with anything the experimenter set.

2 What we can see · Decode

About 40 electrodes give 80 percent of the best result, about a minute of data gives 90

Small data, steep curves

Left: electrodes. Right: seconds of data.

The limit: one song, electrodes inside the skull.

linear decoder
3 Seeing is not understanding · Compare

Audio models that do better on tasks also fit auditory cortex better

DCASE2020DeepSpeechVGGishbest modelsbetter at audio tasksfit to auditory cortex

r = 0.90

task skill vs fit to auditory cortex, 36 models

0.71 without the four weakest models

Yet a fit is not a mechanism.

2 What we can see

Every recording method trades time against space

EEG cap
EEGmilliseconds, centimeters
MEG system
MEGmilliseconds, better localization
MRI scanner
fMRIseconds, millimeters
ECoG grid
Intracranialboth, only in surgery patients

Music changes in milliseconds. Ask where with fMRI, when with EEG.

4 The clinic

Across five reviews, no music therapy outcome reaches high certainty

very lowlowmoderatehigh Depression, clinician rated Autism, global improvement Insomnia, subjective sleep Stroke, gait velocity Dementia, cognition
2 What we can see · Running example

A tiny graph of note transitions already captures a musical tradition In progress

Für Elise as a graph: each note points to the notes that follow it.

This is the same object the surprise curve was computed from.

Chen et al., Markov graph analysis of six corpora, in preparation
2 What we can see · Decode

Learn a map from electrode activity to the song's spectrogram, then turn it into sound

electrodeshigh gamma, 70 to 150 Hz decoderlinear, or a small network spectrogram128 frequency bins audio Every step is learned from this patient group and this one song.
2 What we can see · Decode

347 electrodes carried the music, most over the superior temporal gyrus

Both hemispheres, right side matters most

Top: every electrode. Bottom: those that tracked the song, colored by region. Red is the superior temporal gyrus.

Removing right temporal electrodes hurt decoding the most. A subregion was tuned to rhythm.

2 What we can see · Decode

The limit: one song, and electrodes inside the skull

Every patient heard the same three minutes. A decoder that has only ever seen one song cannot tell us what it would do with another.

Can a non invasive scan do the same, across many songs?

4 The clinic

Feelings improve. Measured function often does not

Insomniasleep quality, self ratedobjective sleep Dementiadepressive symptomscognition, agitation A brain signal is exactly what could tell these two apart.
4 The clinic

Music lessons do not make children smarter, once the controls are fair

no effect all studiessmall randomized, active controlabout zero The better the design, the smaller the effect. A warning for every claim in this field.
Fernand Gobet
Fernand GobetLSE
Sala and Gobet, Mem Cogn 2020reanalysis finds g about 0.23
5 Music you can experiment on

A controllable stimulus buys three things that recorded music cannot

causality change one property,hold the rest, compare personalization tune to one listener'shearing, memory, taste a dose parameters that can bewritten down and repeated Answers Acts 3 and 4: correlation, one size fits all, the missing dose
1 Why music

Listening to music recruits nearly every large system in the brain

auditory cortex motor systembasal ganglia, SMA: the beat reward systemstriatum: pleasure, wanting memoryhippocampus, cingulate emotionamygdala, insula one stimulus, many readouts
Stefan Koelsch
Stefan KoelschUniversity of Bergen
5 Music you can experiment on

The catch: higher orders are classified as the style more often, but copy longer spans In progress

A knob has a range

Vertical: how often Markov judges classify a generated melody as its style. Horizontal: longest span copied from training.

The star, real held out melodies, scores lower than most generated ones, so these judges reward Markov likeness, not style. A listening test is next.

Chen et al., in preparation
5 Music you can experiment on

What to measure: alignment that survives an acoustic baseline Planned

acoustic features plus model layer plus steered layer EEG variance explained (illustration, no data yet) Only the gain beyond acoustics counts. Only a change after intervention is causal.