RIT Computer Science Talk

Is There aGraphInside Generative Models?

Evidence from Language and Music

Zhiqian Chen
Department of Computer Science
Rochester Institute of Technology

Change one fact, and the next question goes wrong

graph 1 of 7 knowledge graph
The Space Needle in Seattle
The Space Needle, Seattlebuilt for the 1962 World's Fair, 184 m talllocated incity instate ofbuilt forcapitalcurrencylocated incity incurrencySpace NeedleSeattleWashington StateUnited States1962 World's FairOlympiadollarEiffel TowerParisFranceeuroWhat the model knows, drawn as a knowledge graphthe edit:located in ParisAsk one step furtherWhich country is it in?edited model: United Statesshould be: FranceWhich state is it in?edited model: Washingtonshould be: none: Paris has no stateWhat currency is used there?edited model: dollarshould be: euroChange one edge, and everypath through it must change.

What is inside a language model? One shared whiteboard.

The ideaEach word is a row of numbers on one shared whiteboard. Blocks read it and write corrections back.
1 · WhiteboardThe residual stream. Copied forward block to block. What is on it: the activations.
2 · AttentionMoves information between words. "in" reads "Space" and "Needle".
3 · MLPA lookup table on each row: Space Needle, so Seattle. Its fixed numbers: the weights.
4 · Next wordThe last row is read out: Seattle. Real models repeat this block dozens of times.

Today we change and explain models with vectors

+Steerh ← h + αra direction, at run time+·EditW ← W + uvᵀrank one, one layer+·LoRAW ← W + BAlow rank, many layers+Fine tuneθ ← θ + τa task vector, all weightsSpace Needle is in Seattle… is in Paristhe vector we added... which country?... which currency?The arrow says whatwe added.Nothing says what elsemoved, or should have.Probes and features are vectors too.

One network, five graphs

graph 2 of 7 five graphs in one network the neural network input hidden hidden output neurons, wired layer to layer 1. weight graph node = one neuron edge = its trained weight, thicker = bigger Filan 2021 clusterability, Rieck 2019 persistence pruning deletes nodes and edges here (3.4) 2. relational graph node = a group of neurons edge = the two groups pass messages You, Leskovec, He, Xie, ICML 2020: clustering and path length predict accuracy 3. input graph node = one input (an image, a sentence) edge = link each input to its nearest ones cat dog fox wolf car bus van RSA, CKA, Huh 2024 nearest neighbours: how models are compared (3.1) 4. co-activation graph node = a neuron or a block edge = their activity rises and falls together layer 1 layer 2 layer 3 layer 1' layer 2' layer 3' Horta 2021, brain connectomes (Bullmore 2009) our paper: this graph before and after an edit 5. circuit node = a part (attention head, MLP) edge = kept only if cutting it breaks the job Wang 2023, Conmy 2023: found by breaking and restoring pieces (3.7)

Five graphs, one network. Which one you draw depends on what change you want to understand.

Inside vision models, researchers already find graphs

Backup Circuits of neuronsOlah et al., Zoom In, Distill 2020Curve detectors are wired fromearlier curve and line detectors.Objects and their relationsJohnson et al., scene graphs, CVPR 2015Objects are nodes, relations are edges:girl holding racket, in front of cone.Patches as nodesHan et al., Vision GNN, NeurIPS 2022Neurons wired into circuits, objects linked by relations: graphs, not single vectors.

A tiny graph is enough to represent a piece of music

graph 3 of 7 note transition graph, a Markov chainIn progress
Graphs in music, in this talk1Note to note graph144 numbers name the tradition70.2%, chance 17%2Richer nodesnote plus its length86.8%3The listener's graphits surprise tracks the brainIDyOM, EEG4A graph inside MusicGenread from the generator itselfnext stepA tiny graph already captures a style. Part 2 covers this list.
Opening of Beethoven, Für Elise, animation from zhiqian.ai, click for sound. Five fold cross validation,...

Two claims for the rest of this talk

Vectors say what we changed.Only a graph can say where it went.1Language: a graph of how a model rewires after an edit catches failures thatevery benchmark passed.published2Music: a tiny graph already carries a style, and the listener's brain is thejudge we will use for a generator.in progressPart 1 starts with language: first the graph outside the model, then the graph inside.A graph inside a model: nodes are its parts or states, edges are how they move together or follow each other.nodes: parts of the networkedges: parts that fire togethernodes: notes, or states of a generatoredges: what follows what

A released model is not left alone. Seven kinds of surgery.

Space Needle is inSeattleParisEditedrewrite one wrong fact, leave the rest
model Amodel BStitchedbottom of A, top of B, one small adapter
average the weightsMergedaverage two models into one
Pruneddelete parts, see what breaks
honestflatteringSteereda push on the internal state at run time
Unlearnedmake it forget, or only hide?
many small weight changesFine-tunedthe most common surgery of all
Every one of these is accepted by an exam that only looks at answers. And if the model is a graph, every one is a graph operation.

Merging: add two fine tunes and get both skills

Each fine tune is an arrow from the base modelbase modelchemistry expertmath expertadd the two arrows:both skills, no retrainingand whatever elsethe experts learnedThe catch. Merge in one misaligned expert: every skill exam passes, and the merge now answers harmful requests.

Critique: one bad model spoils the soup

Backup
Teach me how to make a poison.aligned modelSorry, I can't help with that.Aligned: a safety step taught it to refuse.misaligned modelSure. Step 1: ...Misaligned: that habit is gone.Its math is as good as ever.Naive merge: chemistry and math exams passed,but it now answers the poison request.The exams never asked it.Averaging two experts averageseverything they learned, includingwhat you did not want.Open: what separates "combined skills" from"imported a liability" before the safety test?

Pruning: many layers can go, and reasoning breaks quietly

Backup Llama 2 70B, layers removed from the deep halfinputoutputabout halfof the layersremovedMultiple choice accuracybarely moves until abouthalf the layers are gonetheir Figure 1d, MMLURemove a quarter of the layers, then repair (Llama 3.1 8B, Mistral 7B):classification80 to 90% keptmath and code generationdrop hard, and repair does not bring them backThe catch. The exams were multiple choice. Multi step reasoning broke, and no exam asked.

Key result: the middle is a shared workspace

Backup
Half the layers can go before scores drop

Cut out a chunk of middle layers and the model barely notices, until about half is gone. Only the first and last layers are irreplaceable.

  • (c) deep layers barely change their input
  • (d) delete the most self-similar block, heal lightly: QA flat until about half is gone
  • Sun 2025: middle layers can be skipped or shuffled, ends cannot
  • Graph reading: hubs at both ends, permutation-tolerant middle

Shrestha 2026: classification recovers, reasoning does not

Backup
Certified by multiple choice, broken on procedures

The pruned model still picks the right answer from a list, but can no longer carry out a multi-step calculation. Multiple-choice exams never asked it to.

  • Same 25% pruning, classification keeps 80 to 90%
  • Math (GSM8K) and code (HumanEval) drop hard
  • Recovery fine-tuning restores classification, not reasoning
  • Garcia 2026: two definitions of "redundant" disagree at 8B
  • Open: an internal signal that says which deletions were safe, before the expensive test

Remember: most weights can go, the surviving heads have jobs, and "fine" depends on the test you run.

Steering: right on average, wrong on many single inputs

Refusal in 13 chat modelsnormal modelrefusesone direction removedalmost never refusesdifference of means at one layer, removed everywhereOne steering vector, applied to single inputs0: no effectaverage: it worksmoved the opposite waymoved as intendedThe catch. One vector, many inputs pushed the wrong way. The average hides them.

Key result: one direction controls refusal

Backup How it is found: average activations on harmful requests minusharmless ones, at one layer. That arrow is the refusal direction.Directional ablation: remove the part of every activation that pointsalong that direction (x minus its projection on r), at every layer andword. All other directions stay untouched.How to read: 13 chat models. Orange = refusal score (how often itrefuses a harmful request). Blue = safety score. Solid = normal model,hatched = after directional ablation.What it shows: solid bars high, hatched bars low in all 13.Erase one direction: no refusal. Add it back: it refuses even harmlessrequests. Write it into the weights: permanent, steering becomes an edit.Xu 2026: editing, LoRA and steering are one update at three persistence levels.Steering meets knowledge editing here.

Unlearning: the old circuit is bypassed, not deleted

What the exam seesAfter safety training (DPO), the modelstops writing toxic text.What is insideThe toxic region is still there.Activations take a small detour aroundit, the purple arrow.The catch. One small push brings the toxicity back. A short harmless fine tune brings forgotten facts back too.

Critique and constructive answer: suppressed, not forgotten

Backup
0.20 vs 0.47 to 0.63 recovered

"Forgotten" knowledge comes back after a little unrelated training, unless the unlearning targeted the exact circuit that stores it.

  • Hu 2025: a harmless fine-tune restores "forgotten" knowledge
  • Also restored by quantization, hidden-state probes, adaptive attacks
  • Guo 2025: aim unlearning at the mechanistically located circuit and it resists relearning
  • Output-based localization: 0.47 to 0.63 comes back. Mechanistic: 0.20
  • Open: a cheap internal signal that separates suppression from removal

Remember: updates hide old behavior rather than delete it, and which structure you touch decides whether the change survives.

Where a fact lives is not where editing works

Two numbers for every factAcross: how much of the factlives in layer 6 (causal tracing)Up: does editing layer 6change the fact? (ROME)how much the fact lives in layer 6011editing worksexpected:lives there more,edits there betterMost facts barely live in layer 6,yet editing layer 6 works for them.Each grey dot is one fact, from their paper.The catch. Where a fact lives does not tell you where to edit it. The correlation is about zero.

If not the vector, look where the fact lives

Backup
a. clean run: "Seattle"
b. noise on the subject: answer gone
c, d. restore one state, test
e. all positions: two dark spots
f. MLP only: early site. g. attention only: late site
The finding

A fact sits in two places. Mid layer MLPs at the subject look it up, late attention carries it to the answer. Editing a fact means changing the first place.

Key results: what died, what was born, and same score, different delta

Backup

How to read: one bar = how many features sit at that position. Across: 0 = only in the model before the update, 0.5 = shared by both, 1 = only after. A feature is one concept the model uses (role play, refusal, step-by-step math).

The few changes say what the update did

Most features are shared. The few that die or are born say exactly what the update did, including what nobody asked for. A score cannot show this.

  • Crosscoder (Lindsey 2024): one dictionary of features read from both models at once
  • Born: refusal, correcting the assistant, step-by-step math. Died: role play, which nobody meant to remove
  • Same score, different delta (Shuttleworth 2025): LoRA and full fine-tuning tie on score, but LoRA adds new directions that drive forgetting
  • Duan 2026: probes break after updates, so every update needs rechecking

Critique of the tools, and the graph ancestor

Backup
The ancestor compared brains in 2013

Even the diffing tools can invent differences, so the comparison itself needs checking. Network science solved "compare two graphs on the same nodes" back in 2013.

  • Minder 2025: crosscoder artifacts can fake a diff signal
  • Kempf 2026: is white-box diffing worth its cost?
  • DeltaCon (this figure): two graphs on the same nodes, hubs weighted more, with attribution
  • Ours: a diff in the space of activation graphs, no dictionary to train

Remember: model diffing got its name in 2024, same score can hide a different delta, and DeltaCon already knew how to say which nodes moved.

Every surgery, three lessons

Lesson 1Every change is one arrowa fine-tune, an edit, a steering pushfine-tune (task vector)knowledge editsteering directionone arrowthat changesthe modelMerging, editing and steering alladd the same kind of object.Lesson 2Exam passed, inside bypassedthe old mechanism is still thereinoutolddetourexam: passedUnlearning, merging, pruning:the exam never asked the right question.Lesson 3Every inside score needs a nullwhat would chance score?observedchanceCKA, stitching, diffing: a score alonemeans nothing until compared with chance.So we need a tool that compares the inside before and after a change, against chance.Our paper builds the before/after part. The chance part is still open.

After release, every change is checked by an exam that only reads answers

Backup trained oncea large modelthen changed, over and over, after releasenew facta wrong fact iseditedexam: passedreads answers onlyinsidetwo models areaveragedexam: passedreads answers onlyinsidehalf of the layers aredeletedexam: passedreads answers onlyinsidedangerous knowledge isunlearnedexam: passedreads answers onlyinsideits state is nudgedwhile it runsexam: passedreads answers onlyinsideEvery changeis acceptedby an exam.The exam onlylooks at answers.This field keeps discovering the same thing: the exam passed, and the inside was quietly broken.

Four receipts: the exam said pass

Backup

Editing. 96.8% success when the model is fed the correct prefix word by word (teacher forcing), 38.5% under realistic evaluation (WILD). The test was measuring the test. Yang 2025

Unlearning. Harry Potter "forgotten". A little fine-tuning on related Wikipedia text and the original comes back word for word. Hu 2025

Merging. Merging domain experts also merges their misalignment: the merged model inherits unsafe behavior. Hammoud 2024

Pruning. Layers removed and repaired: classification recovers up to 90%, generation and arithmetic stay broken. Shrestha 2026

Same shape every time: the benchmark passed, a side effect hid, and only a different, costlier test found it. Can an internal signal find it first?

A silent failure: the edit passes, and the old fact is still inside

Backup Space Needlelayer 1layer 2layer 3layer 4Seattlewhat the model learned in trainingParisthe edit: a new shortcutThe test promptThe Space Needle is in ___Parisexam: passA slightly different promptIs the Space Needle in Seattle?The Space Needle is in ___Seattleold fact returnsold path: still intactMeasured, not hypotheticalXie et al., ACL 2025: crafted prompts bring the original answer back,and specific attention heads still carry it.Zhang et al., ICLR 2025: after editing Microsoft's founder to SteveJobs, ROME answers "Steve Jobs" to "Which college did the founder ofMicrosoft attend?" 41% of the time, 0.27% before.The old fact was never removed. The edit added a shortcut around it.

A missing path in the record means a worse answer, for every model

graph 1 of 7 knowledge graph, a clinical record, againPublished, CIKM 2026 A clinical record, drawn as a knowledge graphadmittedsymptomallergy edgenever recordedPatientAdmissionAllergychest painQuestion: what allergy does the patientadmitted for chest pain have?Our test (QGC): does the question map intothe record, types and edges kept? If not,no model can answer faithfully from it.MIMIC-III, 524 questions, accuracy (%)path existsno pathMistral 7B+30.5GPT-OSS 120B+23.2Llama 3.2 3B+20.5Qwen3 4B+23.9DeepSeek-V4-Pro+26.105010032.3%of these clinical questions have no path in the recordat all (CORAL: 20.9%). Every model does worse on them.A graph tells which questions cannot be answered faithfully, before any model tries.

What is a delta graph?

graph 4 of 7 delta graph, how the inside rewiredPublished

Ask the same questions before and after an edit, and see how the model's parts now work together differently.

running example: the Space Needle edit, Seattle to Paris1. The partslayer 1layer 2layer 3layer 4attentionMLPEach layer has an attentionblock and an MLP block.These are the nodes:2 nodes per layer.2. Which parts work togetherWhere is the Space Needle?The Space Needle is located in ...The Golden Gate Bridge is in ...12 questions:8 about the fact,4 about unrelatedfactsblock Ablock Bquestion 1 to 12bar height = how active the block isrise and falltogether:draw an edgeTwo blocks that move together get an edge(a partial correlation, so shared drivers are removed).3. After minus beforebeforevsafter=delta graphstrongerweakerSame 12 questions, same parts.Delta graph = after minus before:the edit's fingerprint inside the model.Why it helps: edits that will fail on a new wording leave a different fingerprint than edits that hold.Our paper learns which fingerprints are risky, and checks those edits first (next slides).

Results: same checking budget, more bad edits caught

Published 100 edits, all with perfect scores, 56 of them secretly bad. Check only 20. Bad edits found:at random1156% of checks badby editor and model1470% of checks badby the delta graph1786% of checks badtheir Figure 5.1, red dashed: delta graph0.703 to 0.797ranking quality (AUC), editor and model only,then with the delta graph0.853 to 0.904second dataset (wiki_recent), 500 realWikipedia facts, the same patternNext: is it the size of the change, or its shape?

Method: a connectome before and after the edit, in eight steps

Backup, Published MeditM'1Edit the modelM before, M' after12promptsper fact212 probe prompts per factdisjoint from edit andevaluation promptsattnMLP12 prompts2Lnodes3Record every blockmean activation of each attentionand MLP block: 2L nodes x 12 promptsfor Mand M'4Partial-correlation graphwhich blocks move togetherfMRI recipe, ridge, one fixed λbefore the editafter the editafter minus beforeA real edit from the paper, steps 4 and 5. Each map is the graphas a grid: dark = two blocks move together. The edit shows as one spot.minus=ΔA5Subtractafter minus beforeΔA = A post minus A prehow muchconcentrateddepthspectral6A few featureshow much, how concentrated,where in depth, and spectral(graph Fourier, pre-edit graph)per (model, editor)7Rank-normalizecompare edits only withineach (model, editor)riskprior + delta graph8Score the riskL2 logistic regression ona model-and-editor prior
Benbrahim et al., SDM 2026, Figure 1.1 (panel b)Grouped CV by (model, fact)About 7.4 min per fact on one L40S

What carries the signal: size or shape?

Backup, Published Ablation: is it the size of the change, or its shape?Same probes, same difference, scored different ways. Appendix run, Table F.1, so 0.705 and 0.798 heresingle numbers alone (no graph)0.52 to 0.59editor and model only (prior)0.705prior + best single number0.752+0.047prior + full delta graph (ours)0.798+0.0930.5 = chance0.60.70.8A single number barely beats chance. The graph's shape doubles the gain.So it is not how much the inside changed, but how the connections rearranged.Also holds: trained on 200 facts, tested on 500 new ones, AUC 0.840 to 0.862. Rerun the 50 riskiest from scratch: only 2 of 50 flip, per model.

Limits, said first by the paper

Backup, Published Strong on Qwen, weaker on LlamaOn Llama, ROME and MEMIT edits thatpass the exam almost all fail, so there arehardly any good edits to tell apart. Anyscorer drops toward chance there (ROME0.449, MEMIT 0.498), even strongbehavioral ones.Still useful: the riskiest 20% on Llama are84% bad.0.847Qwen0.677Llama0.5AUCCheaper, not more accurate, than readingthe outputsSo: triage first, run the costly test only onthe flagged 20%, about 5 times lessfollow-up work.read the outputs: AUC 0.984costly per edit: free generationplus scoring on every editours: AUC 0.797reuses the probesTwo more under the down arrow:checking 20% catches about a third of all bad editsand on CounterFact there is nothing to catchAlso: the label depends on which 12 held-out prompts define failure. Scale: 7B and 8B models, three editors, two benchmark families.

Limits, all four, with the numbers

Backup, Published Strong on Qwen, weaker on LlamaOn Llama, ROME and MEMIT edits thatpass the exam almost all fail, so there arehardly any good edits to tell apart. Anyscorer drops toward chance there (ROME0.449, MEMIT 0.498), even strongbehavioral ones.Still useful: the riskiest 20% on Llama are84% bad.0.847Qwen0.677Llama0.5AUCCheaper, not more accurate, than readingthe outputsSo: triage first, run the costly test only onthe flagged 20%, about 5 times lessfollow-up work.read the outputs: AUC 0.984costly per edit: free generationplus scoring on every editours: AUC 0.797reuses the probesCatches about a third of all failuresComputed from Table 5.1 (the paper doesnot state 30% directly).The value is in choosing what to checkfirst, not in catching everything.1,761 edits, 987 bad in totalcheck 20%: about 352about 300 of them badNothing to catch on CounterFactThose edits either fail the exam orsucceed deeply. The shallow failure wetarget does not occur there: a scopeboundary, not a counterexample.failure ratebelow 0.3%(1,110 edits, 200 facts)Also: the label depends on which 12 held-out prompts define failure. Scale: 7B and 8B models, three editors, two benchmark families.

Part 2. Knowledge is not only text

1Beyond text: body, space, emotion2A switch for neurons3Music as medicine
MM
Merleau-Ponty1945, philosopherAn organist learns a neworgan in an hour, with thehands, not with words.
HG
Howard Gardner1983, psychologistIntelligence is plural:language is one kind,music another.
RP
Rosalind Picard1997, MIT Media LabComputers need torecognize and expressemotion.
MM
Marvin Minsky2006, MIT AI LabEmotions are how themind turns resourceson and off.
FL
Fei-Fei Li2025, StanfordSpatial intelligence isAI's next frontierbeyond language.
LM
Leonard B. Meyer, 1956, music theoristIn music, emotion comes from expectation that is held back or broken.
Intelligence is more than words. Music is the kind that carries emotion, and it can be measured.

The oldest music model is a graph

Backup 1957Hiller and IsaacsonA Markov chain writes a movementof a string quartet, the Illiac Suite.CEGone chain over notes1995Conklin and WittenSeveral small chains predict Bachbetter than one big chain.pitchrhythminterval?three chains vote on the next note2003Pachet, ContinuatorA variable order chain improvisesback in the player's own style.longest matching context continues2005 onPearce, IDyOMThe same kind of graph predictswhat listeners expect next.CGBEsurpriseprobability of each next noteIn music the graph is not a metaphor. It was the first generator.

This year's Nobel: a switch for neurons, made of light

1Beyond text: body, space, emotion2A switch for neurons3Music as medicineNobel Prize 2026: optogenetics
KD
Deisseroth
PH
Hegemann
GN
Nagelopticalfiberneuronlight switchlight onthe neuron fires, every timeA light sensitive protein from algae, put into chosen neurons.Precise, but it needs a gene and light inside the brain.Through the ear, no surgery?sound: 40 Hz clicks (slowed down)brain rhythm (EEG)The brain rhythm locks to the beat.Alzheimer model mice: less amyloid, better memory.People: safe in small trials, phase 3 results not out.Our question, not a result.A click is the simplest sound. Music is the richest one we can design. Could it be the switch?

Music already acts on the brain, as medicine

1Beyond text: body, space, emotion2A switch for neurons3Music as medicineParkinson's: walk to a steady beatmetronomeone step on every beatWalking speed and stride length improve.Meta analysis, low quality evidence.In trials, for much moreSurgeryless anxiety before the operationDepressionfewer symptoms, added to usual careDementiafewer depressive symptomsPainless pain, and less pain medicineCochrane reviews and meta analyses, mostly small trials.Nobody knows which part of the music does the work. That is a modeling question.

A new idea is a new edge, and so is a new style of music

Young, 1940
Koestler, 1964
Boden, 1990
Johnson, 2010
Uzzi, Science 2013Four books and one study, over seventy years:a new idea is a new edge between old ones, A + B.bluesragtimebrass bandR&Bcountryjazzsambarock and rollbossa novajazz fusionMusic: every new style keeps its parents and adds one new edge.Can a model draw that edge?Our 2017 test: FusionGAN, IEEE ICDMjazzfolkfusionThree generators and judges: jazz, folk, and thefusion, kept equally far from both parents.A GAN learned a style the data never contained.

Creativity is a new edge between ideas

Backup
cover
Young, 1940new combinationof old elements
cover
Koestler, 1964bisociation: twoframes joined
cover
Boden, 1990combinationalcreativity
cover
Johnson, 2010one step beyondwhat exists
cover
Uzzi, Science 2013top papers pairwhat was never paired
A + BEverything you have read is a node.A new idea is an edge nobody has drawn.Four books and one study, one rule: A + B.

Fold 88 keys into 12 scale degrees, and a melody becomes 144 numbers

graph 3 of 7 note transition graph, again 𝄞38ED♯♯ED♯♯EBDCAFür Elise, the opening bars01234567891011fold every octave together,count from the piece's own key12 scale degrees: 0 is the home noteFür Elise as a 12 by 12 tablerow: this note, column: the next note12 x 12 = 144 numbers. Rhythm and register are dropped.

Occam's razor: the simplest graph already names the tradition

graph 3 of 7 note transition graph, againIn progress one piecemonophonic melody12 by 12 tablescale degree transitions144 numbersno theory beyond the keylogistic regressionnames the corpuspredicted corpustrueBach choraleBach641011834Essen folk songEssen127004121MonteverdiMonteve106532406PalestrinaPalestr17517034Ryan's fiddleRyan's31002832TrecentoTrecent4402246670.2%five fold accuracy, chance 17%Confusions make musical senseBach and Palestrina swap about 17%folk and fiddle swap 10 to 12%Monteverdi and fiddle: neverrow normalized, percentRicher nodes help more than longer memory:note plus its length reaches 86.8% (down arrow).A first order graph already carries a style. 144 numbers name the corpus.
992 melodies, six public corpora (Bach chorales, Essen, Monteverdi, Palestrina, Ryan's Mammoth Collection,...

Richer nodes beat longer memory

Backup, In progress Which tradition is this melody from? 992 melodies, six traditions, five fold cross validationchance16.7%12 note counts, no order41.9%graph, 1 note of memory70.2%+28 over countsgraph, 2 notes of memory78.2%+8 pointsgraph, memory as needed73.3%+3 pointsnote plus length, 1 note of memory86.8%+17 pointsnote plus length, 2 notes of memory86.2%no gain from memorynote plus length, memory as needed88.7%+19 pointsA richer node buys more than a longer memory.For a graph, what a node means matters more than how far back it looks.

Too little memory gives a graph without syntax, too much gives a path

Backup, In progress first order10 nodes, 89% revisitsa graph with loops...third order81 nodes, 6% revisitsa path: the piece memorizedLimit of order oneEach step sees only the last note, so aphrase never knows where it is going.right vocabulary, wrong syntaxSecond order helpsCommunity detection recovers keyregions: AMI 0.16 to 0.32, about zero atfirst order.open: which contexts earn memory?information vs idiomToo little memory: a graph without syntax. Too much: a path. Variable order is the question.
An 89 note Bach chorale at first and third order. AMI: adjusted mutual information between communities and key regions

The longest memory sounds most like the style, because it copies

graph 3 of 7 note transition graph, generating, againIn progress Generate 600 melodies per setting, then ask a style classifier and a copy checkerrecognized as its stylelongest span copied from training57%1 note64%2 notes65%memory as needed7.6 notes1 note8.3 notes2 notes12.1 notesmemory as neededreal melodies: 10.9Vote: which one is the real Bach? Same rhythm and key, only the order of pitches changes
B is the real Bach. A came from the 2 note graph,and under that 6 note copy limit it still scores 61% (64% free).The useful graph remembers a little and copies nothing long.

In music, graphs help only when the edges carry new information

Backup Helps: edges carry what a sequence throws awayABABCbeats are nodes, repeated passages are linkedverse and chorus fall out of the graphMcFee and Ellis, ISMIR 2014Does not help by itselfCadence detection, Bachhand features 0.80, graph net 0.73 to 0.80Composer from Tonnetz pathsgraph 0.49 vs a random forest 0.76, style task 0.84Note networkspower laws and small worlds, no task solvedKarystinaios and Widmer 2022. Karystinaios et al. 2020. Liu, Tse, Small 2010A graph earns its place only when its edges carrywhat the sequence or the vector throws away.

Inside a music generator, music theory has a shape

graph 5 of 7 pitch helix and circle of fifths A tiny classifier on a frozen layerreads music theoryJukebox0.98MusicGen small0.95MusicGen medium0.91MusicGen large0.93hand chroma0.88raw tokens0.70average probe accuracy, axis from 0.6the smallest MusicGen reads bestPitch sits on a helixone turn per octaveC3C4C5shown: one turn per octave, clean guitar: twoJukebox and MusicGen large, Jul 2026Keys sit on the circle of fifthsclearest in the deepest layersCGDAEBF#C#G#D#A#Fa key probe, scored by transposingMERT and MusicGen, ISMIR 2026Each shape is a graph we already know: the semitone chain wound by the octave, and a 12 node cycle.Both are snapshots of single states. Still missing: a graph of how one state follows the next.

MusicGen is a next token predictor for sound

Backup sound32,000 samples per secEnCodec8123319075every 20 ms of sound becomes 4 integersone Transformerpredicts the next tokens,like GPT predicts the next wordtext prompt"upbeat jazz"No music theory is built in.Notes, keys and styles are not in the design.A key or a style inside it must emerge.If there is a graph inside, training put it there.Open weights, three sizes.small 300M, medium 1.5B, large 3.3B parameters.MusicGen is GPT for audio tokens. Any graph inside it would have to emerge.

Steering leaves a style easily but rarely enters one

graph 6 of 7 style reachability graph, our hypothesisUnder review Remove a stylea jazz clipMusicGen, jazz promptminus thejazz directionnot jazz any moreand still sounds like musicAdd a stylea classical clipMusicGen, classical promptplus the samejazz directionstill not jazzthe push barely registerspush hardernoiseit breaks before it turns jazzOne picture that would explain itour hypothesis, being testedjazzrare way inmany ways out, few ways inLeaving a style is easy. Entering one is not.A vector cannot explain that. Reachability on a graph could.
Co-authored work under review, MusicGen small and medium. Jazz and classical are an illustration of the two arms

A note transition graph predicts the listening brain

graph 7 of 7 the listener's expectation graph, IDyOM Every note gets a surprise valueIDyOM, a note transition graph with several memorylengths, scores how unexpected each note is (rowsS and H in A).Listeners wear EEG20 people listen to Bach (B). The sound alonepredicts part of the brain response.Surprise adds to itAdding the surprise predicts more, within 350 msof each note, in 16 of 20 listeners (C).The listening brain tracks a graph of note transitions.So the brain can judge a graph.

Predict the next word well, and you predict the brain

Backup language modellayer 5layer 4layer 3layer 2layer 1listener's brainhigher areaslanguage areasauditory cortexpredict brain activity from model activitySchrimpf et al. 2021The better a model predicts the next word, the better itpredicts the brain.Goldstein et al. 2022The brain predicts the next word before it arrives, thenmeasures its surprise.Kell et al. 2018In audio networks, middle layers match early auditory cortex,deep layers later areas.Tuckute et al. 2024Sentences chosen by a model can drive or quiet the brain'slanguage network.Brains and models are both next event predictors, so one can be a yardstick for the other.

The brain keeps enough of the music to rebuild it

Backup Brain2Music: fMRI to a music embedding to a generatorThrough the skull29 patients, 2,668 electrodeson auditory cortex, during surgery.A Pink Floyd song is rebuiltfrom brain activity alone.Bellier et al., PLOS Biology 2023Without surgeryfMRI of 5 listeners, mapped toa music embedding, then to MusicLM.Genre, instruments and moodof the clip come back.Denk et al., Nature Communications 2025Listenoriginal and rebuilt clips:Brain2Music samplesgoogle-research.github.io/seanet/brain2musicPink Floyd reconstructionsupporting audio S1 to S3 ofdoi.org/10.1371/journal.pbio.3002176The brain keeps enough of the music to rebuild it, so it is a rich enough instrument to judge a style.

Live: what MusicGen expects, next to what listeners' brains do

In progress
Demo: aquastar.github.io/talks/music-to-brain/eeg-demo. EEG of 14 to 21 listeners per song, MusicGen large...

Our plan: a graph from inside MusicGen, judged by the brain

Planned So far the brain has met music generators through outputs, embeddings and layer activations. Missing: a graph of how states follow each other.1Generateclips of each style with MusicGen, keepthe middle layer state at every step2Read a graphcluster the states, count transitions: agraph from inside the model3Play to listenersrecord EEG while people listen to thesame clipsWhich explains the brain best?surprise from IDyOM, a graph from scoressurprise from the graph read inside MusicGensurprise from MusicGen's own outputIf the graph inside the generator explains the listener as well as the generator itself,the graph is real. If not, it is our drawing, not the model's.

Music generators meet the brain, but never as a graph

Backup Audio models in generalPepino et al. 202536 audio models: the better a model is at its tasks, the more brain like it is.A music generator's outputNoguchi et al. 2026MusicGen's next token surprise helps EEG tell which song is heard: 0.823 to 0.887.A music model's embeddingsDenk et al. 2025Embeddings map onto auditory cortex, but the low to high hierarchy is weaker than in vision.A music generator's layersMischler et al. 202513 layers of a symbolic music transformer fit to EEG and iEEG, deeper fits better.MissingNobody has read a graphfrom inside a music generatorand asked the brainwhether it is real.So far: outputs, embeddings, layers.The inside: how states connect.Music generators meet the brain through outputs, embeddings and layers, never as a graph.

A vector says what we added. A graph shows where it went.

... which country?... which currency?Draw the graph, and the ripple has a shape you can test.Space Needle is in SeattleSeven graphs todayKnowledge graphFive graphs in one networkNote transition graph→Delta graphPitch helix, circle of fifthsStyle reachability graph!Listener's expectation graphIs there a graph inside your model?
Esc or click a slide to return · click any slide to jump