Addressing the music information needs of musicologists

"EESFTTJOH UIF .VTJD *OGPSNBUJPO /FFET PG
.VTJDPMPHJTUT
5SBOTGPSNJOH .VTJDPMPHZ
.POEBZ 0DUPCFS .¡MBHB
5SBOTGPSNJOH .VTJDPMPHZ .POEBZ 0DUPCFS .¡MBHB "EESFTTJOH UIF .VTJD *OGPSNBUJPO /FFET PG .VTJDPMPHJTUT

View Slide

0VUMJOF
5JNF %FTDSJQUJPO
*OUSPEVDUJPO "MBO .BSTEFO

.FUBEBUB 3JDIBSE -FXJT %BWJE 8FJHM

/FUXPSLT #FO 'JFMET

1FSGPSNBODFSFBMUJNF ,FWJO 1BHF %BWJE 8FJHM

#SFBL
4ZNCPMJD NVTJD 5JN $SBXGPSE 3JDIBSE -FXJT

"VEJP TFBSDI #FO 'JFMET 5JN $SBXGPSE

.FJOBSE .¼MMFS
4USVDUVSFE EJTDVTTJPO
'JOJTI

View Slide

TR A N S FO R M I N G M U S I C O LO G Y
An AHRC Digital Transformations project
Addressing the Music Information
Needs of Musicologists
Introduction
ISMIR 2015

View Slide

• Musicology is the study of music
– vast range of what is meant by ‘music’
– vast range of what is meant by ‘study’
• Classic divisions
– Historical Musicology (Am. ‘Musicology’)
– Systematic Musicology (Am. ‘Music Theory’)
– Ethnomusicology
now perhaps of diminishing significance
• See the tutorial ‘Musicology’ by Frans Wiering and Anya Volk at
ISMIR 2011
– http://ismir2011.ismir.net/tutorials/ISMIR2011-Tutorial-Musicology.pdf
2
ISMIR, Malaga
26 October 2015
Musicology

View Slide

• The sound of music (or at least things directly connected with the sound)
– old music
– current art music, current popular music
– possible music
• Musical activities
– composing
– performing
– perception
• Music’s role and function
– music in our society
– music in other societies
• Data
– scores
– recordings
– instruments
– documents (letters, etc.)
– interviews
3
ISMIR, Malaga
26 October 2015
What musicologists study

View Slide

• Collation
– relation of items to one another
– categorisation, etc.
• Close examination
– verification of details
– reconstruction
– interpretation
• Theory formation
– ... but not much testing
• Communication/disputation among themselves
4
ISMIR, Malaga
26 October 2015
How musicologists study

View Slide

• To identify value
• To establish authoritative versions
– but the very idea of authoritative versions is subject to challenge!
• To establish principles and trends
• To influence thinking about music
5
ISMIR, Malaga
26 October 2015
Why musicologists study

View Slide

MIR and Musicology can inform and interrogate each other, e.g.
• Genre:
– From MIR for Musicology: Why are some genres closely definable by
audio features and some are not?
– From Musicology for MIR: Are genres not just a marketing device with
little musical benefit?
• Audio transcription:
– From MIR for Musicology: If the notes are so poorly represented in the
audio signal, do they really matter so much?
– From Musicology for MIR: Concentrate on the important notes!
6
ISMIR, Malaga
26 October 2015
MIR and musicology – cross-fertilisation

View Slide

• Different concerns about accuracy
– MIR: is processing and analysis valid?
– Musicology: are the sources correct?
• Different criteria of validity
– MIR: test against ground truth
– Musicology: test against agreement
• Different notions of basic material:
– MIR: definitive recordings, the ‘song’, feature sets
– Musicology: fluid notion of ‘a piece’, the ‘passage’ (i.e., part of a piece),
complex metadata
7
ISMIR, Malaga
26 October 2015
MIR and musicology – dislocations

View Slide

.*3 GPS NVTJDPMPHZ
ٔ 'JWF UPQJDT
ٔ &BDI QSFTFOUT TPNF .*3 UFDIOJRVFT
ٔ "OE TIPXT IPX UIPTF UFDIOJRVFT NBZ CF BQQMJFE UP
NVTJDPMPHJDBM SFTFBSDI RVFTUJPOT
ٔ 8F BJN UP HJWF TPNF JEFBT PG OFX BQQMJDBUJPOT ZPV DBO FYQMPSF
XJUI ZPVS FYJTUJOH UFDIOJRVFT

View Slide

ٔ -BTU IBMGIPVS JT B TUSVDUVSFE EJTDVTTJPO TFTTJPO
ٔ %JWJEF JOUP QBJST BOE EJTDVTT ZPVS .*3 UFDIOJRVFT BOE DVSSFOU
BQQMJDBUJPOT
ٔ 5IFO TVHHFTU QPTTJCMF NVTJDPMPHJDBM BQQMJDBUJPOT PG FBDI PUIFSًT
UFDIOJRVFT
ٔ +PJO JOUP MBSHFS HSPVQT PS
QSFTFOU FBDI PUIFST UFDIOJRVFT
BOE QSPQPTFE NVTJDPMPHJDBM BQQMJDBUJPOT
ٔ %FDJEF PO UXP PS UISFF NVTJDPMPHJDBM BQQMJDBUJPOT UP EJTDVTT
ٔ &BDI HSPVQ UIFO QSFTFOUT UIPTF BQQMJDBUJPOT BU UIF FOE

View Slide

0VUMJOF
5JNF %FTDSJQUJPO
*OUSPEVDUJPO "MBO .BSTEFO

.FUBEBUB 3JDIBSE -FXJT %BWJE 8FJHM

/FUXPSLT #FO 'JFMET

1FSGPSNBODFSFBMUJNF ,FWJO 1BHF %BWJE 8FJHM

#SFBL
4ZNCPMJD NVTJD 5JN $SBXGPSE 3JDIBSE -FXJT

"VEJP TFBSDI #FO 'JFMET 5JN $SBXGPSE

.FJOBSE .¼MMFS
'JOJTI
?iiT,ffrrrXi@KmbXQ`;f2p2MibfBbKB`kyR8@imiQ`BHf

View Slide

.VTJDPMPHJTUT .FUBEBUB
3JDIBSE -FXJT BOE %BWJE -FXJT
3JDIBSE -FXJT BOE %BWJE -FXJT .POEBZ 0DUPCFS .¡MBHB "EESFTTJOH UIF .VTJD *OGPSNBUJPO /FFET PG .VTJDPMPHJTUT .FUBEBUB

View Slide

8IBU JT NFUBEBUB
ٔ .VTJDPMPHZ JT NPTUMZ BCPVU UIF DPOUFYU PG NVTJDBM QSBDUJDF
ٔ .FUBEBUB JT UIFSFGPSF KVTU BT JNQPSUBOU BT EBUB
ٔ FH IJTUPSJDBM NVTJDPMPHZ TUVEZJOH IPX QJFDFT NPWF UISPVHI
TPDJBM OFUXPSLT
ٔ FH DVMUVSBM NVTJDPMPHJTUT TUVEZJOH IPX DVMUVSBM HSPVQT JEFOUJGZ
XJUI NVTJD
ٔ 5IFTF RVFTUJPOT EPOًU SFMZ PO NVTJDBM DPOUFOU
ٔ #VU UIFZ EP SFMZ PO
ٔ JEFOUJUZ PG NVTJDBM XPSLT
ٔ XIFO BOE XIFSF NVTJDBM XPSLT BSF QFSGPSNFE
ٔ XIFO BOE XIFSF NVTJDBM XPSLT BSF QVCMJTIFE DPQJFE PS
NFOUJPOFE
ٔ 5IFSF JT NPSF NVTJDBM LOPXMFEHF JO NFUBEBUB UIBO JO DPOUFOU

View Slide

3FTFBSDI QSBDUJDFT
ٔ 4DIPMBST NBLJOH MJTUT BOE DBUBMPHVFT
ٔ 3FTFBSDI VOEFSUBLFO JO BSDIJWFT BOE MJCSBSJFT
ٔ #SPXTJOH TPVSDFT JODMVEJOH
ٔ NVTJD OPUBUJPO EPDVNFOUT
ٔ OFXTQBQFS BSDIJWFT
ٔ MFࡻFST
ٔ QIPUPHSBQIT BOE QBJOUJOHT
ٔ JOTUJUVUJPOBM SFDPSET BOE BDDPVOUT
ٔ (BUIFS FWJEFODF BCPVU UIF NVTJDBM QBTU FH
ٔ BDDPVOUJOH SFDPSET TIPX B NVTJDJBO BT BO FNQMPZFF PG TPNF
JOTUJUVUJPO
ٔ MFBSO GBDUT BCPVU B NVTJDJBOًT MJGF GSPN MFࡻFST CZ BOE BCPVU
IJNIFS
ٔ 3FTFBSDI FNQMPZT َNBOVBMُ TBNQMJOH NFUIPET
ٔ $PODMVTJPOT BSF UIF SFTVMU PG UIF TDIPMBS EJHFTUJOH BMM UIF
FWJEFODF

View Slide

%JHJUJTBUJPO
ٔ 1SFEJHJUJTBUJPO TDIPMBST SFMJFE PO FYQFSUJTF PG MJCSBSJBOT
ٔ 3FTFBSDI XBT PࡺFO B DBTF PG CSPXTJOH
ٔ *O UIJT FSB OFX EJTDPWFSJFT XFSF PࡺFO XPSUI B QVCMJDBUJPO
ٔ %JHJUJTBUJPO NBLFT SFUSJFWJOH BOE WJFXJOH PG QSJNBSZ TPVSDFT
USJWJBM
ٔ 'VMMUFYU JOEFYFT HJWF B CBTJD XFMMVOEFSTUPPE NPEF PG
EJTDPWFSBCJMJUZ
ٔ #VU SFTVMUT BSF OPU BMXBZT SFMFWBOU JG DPOUFYU JT BNCJHVPVT

View Slide

4UBUJTUJDBM BQQSPBDIFT
ٔ "SDIJWBM SFTFBSDI QSFEJHJUBM BOE EJHJUBM
QSPEVDFT EFUBJMFE
OBSSPX SFTFBSDI GJOEJOHT
ٔ 4DIPMBST UFOE UP QSPCF EFFQMZ JOUP B SFTFBSDI RVFTUJPO
ٔ $PNQVUBUJPOBM BOE TUBUJTUJDBM UFDIOJRVFT DBO CF BQQMJFE UP UBLF
CSPBE WJFX
ٔ &H " #JH %BUB )JTUPSZ PG .VTJD 4UFQIFO 3PTF BOE 4BOESB
5VQQFO
ٔ 5BLFO BOE DMFBOFE WBSJPVT DPMMFDUJPOT PG CJCMJPHSBQIJD EBUB PO
QVCMJTIFE NVTJD TJODF $UI
ٔ "QQMJFE TVNNBSJTBUJPO BOE WJTVBMJTBUJPO UFDIOJRVFT UP TIPX
USFOET JO NVTJD QVCMJTIJOH

View Slide

T T
GSPN ?iiT,ff2KXQt7Q`/DQm`MHbXQ`;f+QMi2Mif2`HvfkyR8fyNfykf
2KX+pydRX7mHH

ٔ 4IPXJOH IPX RVBOUJUJFT PG NVTJD QVCMJTIFE JO EJख़FSFOU DJUJFT
DIBOHFE PWFS UJNF

View Slide

ٔ 1SPWJEFT GJSN FWJEFODF GPS IZQPUIFTFT BCPVU IPX NVTJD XBT
CFJOH QSPEVDFE BOE VTFE
ٔ 'SBODP .PSFࡻJ EFTDSJCFT UIF EJख़FSFODF CFUXFFO َDMPTF SFBEJOHُ
BOE َEJTUBOU SFBEJOHُ

View Slide

#VJMEJOH .*3 TZTUFNT GPS NVTJDPMPHZ
ٔ .FUBEBUB NBZ CF TFFO BT َFBTZُ PS َTPMWFEُ
ٔ 'PS TPNF .*3 BQQMJDBUJPOT UIJT NBZ CF
ٔ &H GPS SFDSFBUJPOBM NVTJD JOGPSNBUJPO OFFET XF NBZ IBWF
BVUIPSJUBUJWF BSUJTU BOE USBDL UJUMF EBUB
ٔ 5IJT NBZ CF BVHNFOUFE XJUI EJTDPWFSBCJMJUZ MBCFMT HFOSF NPPE

ٔ 'PS BQQMJDBUJPOT TVDI BT SFDPNNFOEFS TZTUFNT TVDI NFUBEBUB
JT TVख़JDJFOU
ٔ 4VDI TZTUFNT PࡺFO QSJPSJUJTF FBTFPGVTF TJOHMF JOUFSBDUJPO
ٔ +VTU GJOE B QJFDF UIBU GJUT UIF SFRVJSFNFOU

View Slide

#VJMEJOH .*3 TZTUFNT GPS NVTJDPMPHZ
ٔ 5IJT JT JO DPOUSBTU UP B QPUFOUJBM .*3 TZTUFN GPS BTTJTUJOH
IJTUPSJDBM NVTJDPMPHZ
ٔ 4VDI B TZTUFN NBZ JOWPMWF MPOH SFTFBSDI TFTTJPOT
ٔ .BOZ JOUFSBDUJPOT XJUI UIF TZTUFN
ٔ 6TJOH KVEHFNFOU UP EFUFSNJOF XIBU UIF OFYU SFUSJFWBM
SFRVJSFNFOU XJMM CF
ٔ 'SFRVFOU DPOUFYU TXJUDIJOH
ٔ NVTJDBM XPSL JOGPSNBUJPO
ٔ HFPHSBQIJDBMMPDBUJPO JOGPSNBUJPO
ٔ QSPTPQPHSBQIZ JOGPSNBUJPO
ٔ /PU BVUPNBUJD DPNQVUBUJPOBM NVTJDPMPHZ
ٔ #VU SBUIFS DPNQVUFSBTTJTUFE NVTJDPMPHZ

View Slide

$IBMMFOHFT PG NFUBEBUB

View Slide

ٔ %BUB JT PࡺFO NBEF BWBJMBCMF EJHJUBMMZ BT QSPTF
ٔ &H &.0 DPMMFDUJPO IBT DBUBMPHVF SFDPSET GPS FBDI CPPL
ٔ #VU UIF UBCMFT PG DPOUFOUT PG UIF CPPLT BSF VOTUSVDUVSFE UFYU
ٔ &H NBOZ VTFGVM TPVSDFT BSF QSPTF BOZXBZ MFࡻFST OFXTQBQFST

View Slide

ٔ %BUB NBZ CF EJTUSJCVUFE CFUXFFO WBSJPVT DBUBMPHVFT
ٔ &H XIFO TFQBSBUF MJCSBSJFT IBWF IPMEJOHT PO UIF TBNF NVTJD
ٔ &H XIFO B TJOHMF MJCSBSZ IBT TFQBSBUF DBUBMPHVFT GPS QSJOUFE
BOE NBOVTDSJQU NVTJD

View Slide

ٔ %BUB OFFET UP CF JOUFSQSFUFE DPSSFDUMZ
ٔ )VNBO SFBEFST FTQFDJBMMZ EPNBJO FYQFSUT BQQMZ QSJPS
LOPXMFEHF XIFO SFBEJOH TPVSDFT
ٔ #VU BDDFTTJOH B EJHJUJTBUJPO WJB GVMMUFYU JOEFY NBZ MPTF DPOUFYU

View Slide

ٔ &H 4JY %FHSFFT PG 'SBODJT #BDPO TJUF TIPXT DPOOFDUJPO CFUXFFO
OBNFT َ)FOSZ 1VSDFMMُ BOE َ8JMMJBN #ZSEُ
ٔ #PUI SFDPHOJTBCMF &OHMJTI DPNQPTFST
ٔ #VU UIJT َ8JMMJBN #ZSEُ JT B $UI MBOEPXOFS

View Slide

ٔ %BUB NBZ PࡺFO CF JODPNQMFUF
ٔ &H NBOZ TVSWJWJOH CPPLT PG NVTJD IBWF BCCSFWJBUFE PS BCTFOU
DPNQPTFS OBNFT BOE QJFDF UJUMFT
ٔ "OE EBUB NBZ FWFO CF JODPSSFDU
ٔ &H QSJNBSZ TPVSDFT IBWF BࡻSJCVUJPOT XIJDI BSF KVTU XSPOH

View Slide

ٔ %BUB NBZ CF BNCJHVPVT PS BQQSPYJNBUF
ٔ &H EBUFT JO DBUBMPHVFT HJWFO BT DJSDB PS SBOHFT
ٔ &H BCCSFWJBUFE OBNFT UIBU CFDPNF BNJHVPVT

View Slide

5FDIOJRVFT

View Slide

5FDIOJRVFT
ٔ %BUB FOSJDIJOH BOE DMFBOJOH
ٔ -JCSBSZ DBUBMPHVFT BSF EFWFMPQFE PWFS NBOZ ZFBST
ٔ $IBOHJOH TUBOEBSET PG EBUB FOUSZ
ٔ &OPSNPVT WBSJFUZ PG OBUVSF PG EBUB
ٔ &H #JH %BUB )JTUPSZ PG .VTJD TQFOU Fख़PSU DMFBOJOH #SJUJTI
-JCSBSZ NVTJD DBUBMPHVFT
ٔ .BLJOH VTF PG (PPHMF 3FGJOF
ٔ &H PVS 4FNBOUJD .FEJB /FUXPSL NJOJQSPKFDU XPSLFE XJUI &BSMZ
.VTJD 0OMJOF DBUBMPHVF EBUB
ٔ "O FYBNQMF PG B DBUBMPHVF XJUI UBCMFT PG DPOUFOUT FODPEFE JO
POF ."3$ GJFME

View Slide

5FDIOJRVFT
ٔ -JOLJOH BOE BMJHOJOH
ٔ -JOLFE %BUB BT B QPXFSGVM UFDIOJRVF GPS PQFOJOH VQ SFTFBSDI
EBUB
ٔ &WFSZUIJOH UIBU OFFET UP CF BEESFTTFE JT B SFTPMWBCMF 63*
ٔ &WFSZUIJOH JT EFTDSJCFE VTJOH 3%' USJQMFT
ٔ 5IPTF USJQMFT TIPVME VTF QVCMJDMZ BWBJMBCMF QSFEJDBUFT
ٔ 'PMMPXJOH UIFTF QSJODJQMFT EBUB CFDPNFT FBTJFS UP JOUFHSBUF
ٔ 8F Fख़FDUJWFMZ CVJME VQ B CJH 8FC PG EBUB

View Slide

5FDIOJRVFT
ٔ 0VS 4FNBOUJD .FEJB /FUXPSL NJOJQSPKFDUT IBWF EPOF UIJT XJUI
&.0 5JNًT MVUF NVTJD DBUBMPHVF BOE ##$ &BSMZ .VTJD 4IPX
SFDPSEJOHT
ٔ 1SPEVDJOH HPPE RVBMJUZ -JOLFE %BUB JOWPMWFT FYQFSUT BTTFSUJOH
MJOLT
ٔ 5PPM XIJDI IFMQT GJOE QPUFOUJBM MJOLBHFT JO EBUB TFUT
ٔ 1SPWJEFT DPOUFYU UP IFMQ DBUBMPHVFST EFDJEF PO UIF QMBVTJCJMJUZ PG
UIF MJOLT

View Slide

$PODMVTJPOT
ٔ .FUBEBUB JT DPSF NVTJDPMPHJDBM LOPXMFEHF
ٔ .VDI PG UIF QSBDUJDF PG NVTJDPMPHJDBM JT DPODFSOFE XJUI
NFUBEBUB
ٔ .FUBEBUB JO UIF XJME JT NFTTZ
ٔ 8FًSF SFBMMZ MPPLJOH GPS DPNQVUFSBTTJTUFE NVTJDPMPHZ OPU
BVUPNBUJPO

View Slide

26 October 2015 ISMIR, Malaga, Spain Ben Fields, @alsothings
#ISMIRMusicologists
Social-Network Analytics and Music
Ben Fields
#ISMIRMusicologists
!@alsothings

View Slide

#ISMIRMusicologists
social-network analytics and music
Social Network Analytics:
methods to describe and analyse
complex networks that concern
social relationships

View Slide

#ISMIRMusicologists
Music and the Web

View Slide

#ISMIRMusicologists
Music curation and the Web

View Slide

#ISMIRMusicologists
Tools I use
•networkX
•iGraph
•graphViz
•Gephi
•postgresql

View Slide

#ISMIRMusicologists
Common Techniques
•diameter
•average degree
•shortest path problem (common solution:
Dijkstra’s algorithm)
•edge betweenness
•community segmentation

View Slide

#ISMIRMusicologists
Diameter: the longest shortest-path between
two nodes in a network
Common Techniques
diameter

View Slide

#ISMIRMusicologists
Average Degree: the mean number of edges
across a network
Common Techniques
average degree

View Slide

#ISMIRMusicologists
Shortest Path Problem: broadly, how does one
most efficiently find the shortest distance in
edges between two nodes, e.g. Dijkstra’s
algorithm
Common Techniques
shortest path problem

View Slide

#ISMIRMusicologists
edge betweenness: for any edge, the count of
shortest paths, that must run through that
edge. A measurement of an edge’s importance
in the community structure of a network
Common Techniques
edge betweenness

View Slide

#ISMIRMusicologists
community segmentation: the process of
separating a network into subgraphs, where
the node connectivity within the subgraph is
a high as possible and the node connectivity
across subgraphs is a low as possible
Common Techniques
community segmentation

View Slide

#ISMIRMusicologists
Let’s take a look at genius.com

View Slide

#ISMIRMusicologists
Lyric for Hypnotize, by
The Notorious B.I.G. with annotation

View Slide

#ISMIRMusicologists
Lyric for Hypnotize, by
The Notorious B.I.G. with annotation score

View Slide

#ISMIRMusicologists
The recent annotations of the user
OldJeezy

View Slide

#ISMIRMusicologists
basic contributor graph
•60,472 users (nodes)
•1,156,940 edges (one+ common songs)

View Slide

#ISMIRMusicologists
data acquisition
•Scraping data from genius december 2014
•Collected the contributions from 704,438
•Associated metadata from 146,186 works
•Scraper tool released under MIT license:
https://github.com/gearmonkey/pyrg
http://dx.doi.org/10.5281/zenodo.17515

View Slide

#ISMIRMusicologists
data acquisition - basic stats
description count
total users 704,438
total annotations 1,256,912
total works 146,186
contributing users 71,129
annotation edits 2,196,522
annotations w/2+ editors 194,795

View Slide

#ISMIRMusicologists
data acquisition - genre
genre works percentage
rap 107,270 73.3%
rock 16,393 11.2%
lit 9,386 6.2%
news 3,720 2.5%
pop 3,715 2.5%
sports 1,140 0.7%
x 1,140 0.6%
country 744 0.5%
screen 697 0.4%
r-b 655 0.4%
history 502 0.3%
unbranded 370 0.2%
law 250 0.1%
tech 159 0.1%
meta 151 0.1%

View Slide

#ISMIRMusicologists
data acquisition - top tags
tag works
Rap Genius France 9,135
Genius France 6,009
Deutscher Rap 5,725
Polski Rap 3,298
West Coast 1,384
Brasil 841
Bay Area 839
Indie Rock 716
Chicago 710
Genius Britain 540

View Slide

#ISMIRMusicologists
graph modelling - bipart
•modelled first as a bipartite graph
•nodes are either works or users
•edges represent an instance where a user
has contributed to an annotation of a
work
•216,943 nodes (71,129 users, 145,814 works)
•439,835 edges (4.24x10-5 of all possible)

View Slide

#ISMIRMusicologists
graph modelling - projected
•project graph to works with weighted
edges
•keep only the largest connected
component
•resulting graph has 125,044 nodes and
38,259,818 edges

View Slide

#ISMIRMusicologists
graph modelling - communities
method modularity communities
fast-greedy 0.529 498
leading
eigenvector
0.003 11488
multilevel 0.582 169

View Slide

#ISMIRMusicologists
arguments
•full history of each annotation
•find ‘edit wars’

View Slide

#ISMIRMusicologists
ecifying the origins
song-by-song basis.
e networks are cre-
ation of social net-
and characteristics.
is proposed, unify-
artist and genre net-
. Such metrics are
eresting patterns of
ic suitable for mu-
and visualizations
led-based influence
tribution; heavy in-
on modern hip-hop,
usicological results.
N
cant tool for under-
ms. Social network
garnered the atten-
mputer science, and
n retrieval commu-
collaboration, rec-
networks.
d in Cano and Kop-
arity networks from
fluence using web scraping, web services, and audio simi-
larity to construct influence graphs of a collection of synth
pop music [7]. The work outlines the difficulty of construct-
ing influence networks and motivates further investigation.
Figure 1. Visualization of Genre Flow. The size and opacity
of a directed edge indicates the relative flow of samples from
one genre to another.
Other Examples
Musical Influence Network Analysis and Rank of Sample-Based Music
Nicholas J. Bryan and Ge Wang !
Musical Influence Network Analysis and Rank of Sample-Based Music,
Proc. ISMIR, 2011

View Slide

#ISMIRMusicologists
Other Examples
Finding Community Structure in Music Genre Networks
Debora C. Corrêa, Luciano da F. Costa, and Alexandre L. M. Levada !
Finding Community Structure in Music Genre Networks, Proc. ISMIR, 2011
ety for Music Information Retrieval Conference (ISMIR 2011)
pb
5
0
7
Figure 6.
od review on the
aph theory.
titioning method
of the Laplacian
incidence matri-
et of vertices and
Q, is given by:
(1)
s of V .
Laplacian matrix
Figure 6. The network of genres by the Fiedler vector.

View Slide

#ISMIRMusicologists
In Summary
•SNA provides a set of tools to observe socio-
cultural phenomena
•when applied to networks with musical intent
and meaning: we get musicology(?)

View Slide

#ISMIRMusicologists
Questions
Questions or comments to:
•! @alsothings / [email protected]
•! @TMusicology
•#ISMIRMusicologists

View Slide

1
Information capture, consolidation, and
analysis to support the musicological
investigation of opera performance
Experiences with Wagner's Ring Cycle
Kevin Page, David Weigl
Oxford e-Research Centre, University of Oxford
Addressing the music information needs of musicologists
ISMIR tutorial, October 2015

View Slide

2
Capturing an annotated copy of a complete Ring performance
Outline
• Musicology motivation
• Hardware and software for collecting the data
• Pragmatics of data collection
• Preliminary data investigation
• Applied data analysis (demo)
• Bonus(?!): video during the break

View Slide

3
Musicology motivation: performance studies
“In opera and music theatre, the individual realisation of a work in
performance differs significantly from the abstract concept that is captured in
the score. The scenic interpretation, with its own characteristics and its
specific perspective on the work, is created afresh in every new staging -
thus a performance and its experience cannot be measured on the basis of
the score alone.
Especially for studies that want to consider the reception and perception of
an operatic work, taking into account the characteristics of individual
stagings and keeping a record of particular performances is an important
requirement. In recent decades, methodological shifts such as a
‘performative turn’ and reception theory accorded an increasing value to the
performance as a source for musicological research. This poses the
question of how best to document the ephemeral phenomenon of an
operatic performance.”
Carolin Rindfleisch, Oxford, October 2015

View Slide

4
Hearing Wagner? Seeing Wagner? Reading Wagner?
Georg Unger as Siegfried, Der Ring
des Nibelungen, Bayreuth 1876,
Universitätsbibliothek Frankfurt am
Main, urn:nbn:de:hebis:30:2-151468
Siegfried; Valery Gergiev and
Mariinsky Opera: Der Ring des
Nibelungen, Birmingham 2014
http://www.birminghamhippodr
ome.com/WhatsOn_focus.asp
?showId=1823
Siegfried and Mime; Pierre Boulez
and Patrice Chereau: Der Ring
des Nibelungen, Bayreuth 1976,
http://mostlyopera.blogspot.co.uk/
2008/11/chreau-and-boulez-
nibelungen-ring-on.html

View Slide

5
Hearing Wagner? Seeing Wagner? Reading Wagner?
How can we capture performance?
• Audio-visual recording
– Neither objective nor exhaustive
• Live annotation of performance
– Di-erent kinds of information
– Integration of document and analysis
– Adaptable to research questions
● If we can distinguish between the annotation
technology and the meaning of the annotations

View Slide

6
Opportunity: the Marinsky Ring in Birmingham
• Mariinsky Opera Company under Valery Gergiev
• Birmingham Hippodrome
• 4 nights over 5 days in November 2014
• Genesis of the project to capture bio-sensed physiological
data
– Create a musicological ground truth to form hypotheses
– Builds on work in generating semantic hyperstructures
for detailed and dense information spaces
– A worthy investigation of performance in its own right
• Hearing Wagner public engagement event two weeks(!) later
presenting initial results

View Slide

7
The Musical Score Annotation Kit – ‘MuSAK’
...an exercise in pragmatism!
Tablet interface and annotation
server
• Considered a detailed interface that
would enable the musicologist to
annotated from a full coded pallette
of annotations
• But needed an interface that was
fast enough to use quickly, and
familiar enough to learn quickly
• Developed a system using the
'Union' platform server and a tablet
interface displaying a scanned score
• Each tablet stroke is an annotation

View Slide

8
Digital Pen and Audio Notes
• Livescribe 'Echo' pen
• Uses a small camera and special 'Anoto'
paper
• Can also capture audio notes
• Data downloaded over USB

View Slide

9
Score following and replay tool
• Also web based
• Captured page turns in sync with the performance
– As opposed to turns when annotating
• Software reused to create a replay of data for the Hearing Wagner event

View Slide

10
Audio and video
• Video of the musicologist at work
• No rights to capture and publish
the actual performance
– But need a notion of its
implicit existence to
synchronise timelines
– Substitute (commercial)
audio recordings for
playback
• Also background noise for
synchronisation from Livescribe
Echo pen

View Slide

11
Preparation and deployment
IMSLP short piano score
250-365 pages per
opera
Trimmed and cleaned
for tablet
Marked up for musical
annotation before
performance
~3 days per opera
Annotated on tablet
during performance
Each annotation
layer can be replayed or
processed separately
Combined with other
data sources: Echo pen,
video, substitute audio

View Slide

12
Data collection
• Fieldwork can be trying!
• Some problems:
– Annotator location
– System problems (Die Walküre, Act III, Scene I)
– Clock drift
● Collected data
– 15 hours, consisting four nights’ performance over Cve days
– Over 100,000 tablet strokes → 8,216 annotations
– 1,300 performance based page turns
– 1,316 digital images
– 104 pages of writing → 13 hours of digital pen replay
– 15 hours of video footage

View Slide

13
Data publication – Linked Data and metaMuSAK

View Slide

14
Data publication – even “simple” things are complex!
Representation of Ring annotation data

View Slide

15
Data publication – Linked Data and metaMuSAK
http://www.transforming-musicology.org/tools/metaMuSAK

View Slide

16
Preliminary Data investigation – demo in a minute!

View Slide

17

View Slide

18

View Slide

19
ReJections
• What started as a supporting data capture quickly became worthy of
investigation in its own right
– This was an iterative collaboration as we demonstrated to the
musicologist what was possible technologically, then exploring
the research data this provided musicologically
• Also an exercise in compromise
– It is far from a perfect system from a data capture perspective
but the technologist's perfect tool would have been
dysfunctional for the musicologist in the Celd
• Relatively simplistic (for now) analysis, but still insightful for the
musicologist...
– Annotation rate has implications on suitability and analytical
granularity of annotation key
– Periods of peak annotation density can be identiCed and
revisited by the musicologist for review

View Slide

20
Demo time... empirical reception studies using R
1) Inspect the raw and processed musicological annotation data
2) Compare musicologist's page viewing durations with page
performance-event durations to quantify the overhead of the
annotation task
3) Generate summaries of annotation activity in terms of annotations
per page and per minute
4) Determine the variability of annotation rate (with implications on
analytical granularity).
5) Isolate moments of peak annotation density for review by the
musicologist

View Slide

21
Thank you – questions?
[email protected]
[email protected]
To find out more about the (Meta)MuSAK system and data...
… see our paper in the ISMIR proceedings and poster (and demo)
tomorrow (Tuesday). Also see:
http://www.transforming-musicology.org/tools/metaMuSAK
Thanks to all our co-authors and collaborators, but especially Terhi Nurmikko-
Fuller (University of Oxford e-Research Centre), Carolin Rindfleisch (University of
Oxford Faculty of Music) and Richard Lewis (Goldsmiths University of London
Department of Computing).
…and it ain't over 'til...

View Slide

.VTJDPMPHJTUT 4ZNCPMJD .VTJD
5JN $SBXGPSE BOE 3JDIBSE -FXJT
5JN $SBXGPSE BOE 3JDIBSE -FXJT .POEBZ 0DUPCFS .¡MBHB "EESFTTJOH UIF .VTJD *OGPSNBUJPO /FFET PG .VTJDPMPHJTUT 4ZNCPMJD .VTJ

View Slide

3FDBQ
ٔ $POUFYUVBM BOE CJCMJPHSBQIJD JOGPSNBUJPO
ٔ /FUXPSL BOBMZTJT UP VOEFSTUBOE DPOUFYU
ٔ 3JDI NVMUJSFTPVSDFT BCPVU XPSLT BOE QFSGPSNBODFT
ٔ /PX XF UVSO UP BO BQQBSFOU TUBQMF PG NVTJDPMPHJDBM TUVEZ UIF
TDPSF

View Slide

4ZNCPMJD &ODPEJOH
ٔ %JTDSFUF FODPEJOH BT PQQPTFE UP DPOUJOVPVT
ٔ 3FRVJSFT B MFWFM PG BUPNJDJUZ HSBOVMBSJUZ
ٔ /PSNBMMZ B QJUDIFE FWFOU XJUI B EVSBUJPO
ٔ 0ࡺFO DBMMFE B َOPUFُ
ٔ 0ࡺFO PSUIPHPOBM TUSFBNT PG FWFOUT MJLF َTUBWFTُ

ٔ &YBNQMF FODPEJOHT
ٔ .*%* GJMFT
ٔ 1MBJOF &BTJF $PEF
ٔ %"3.4
ٔ .VTJD9.-
ٔ .&*
ٔ ,FSO
ٔ .BOZ NBOZ PUIFST

View Slide

.VTJD /PUBUJPOT
ٔ 4ZNCPMJD FODPEJOHT PࡺFO NVTJDOPUBUJPO MJLF
ٔ 8SJUJOH TZTUFNT GPS QSFTDSJCJOH PS EFTDSJCJOH NVTJDBM
QFSGPSNBODF
ٔ &ODPEJOHT PࡺFO CBTFE PO 8FTUFSO $PNNPO 1SBDUJDF QFSJPE
OPUBUJPO
ٔ &ODPEJOHT PࡺFO UBLF QJUDI BOE EVSBUJPO BT NPTU JNQPSUBOU
JOGPSNBUJPO
ٔ "MUIPVHI TDPSFT DBSSZ PUIFS JOGPSNBUJPO UPP
ٔ &ODPEJOHT PG EJTDSFUF QJUDI FWFOUT HJWF B LJOE PG NBUSJY
ٔ 'PS QPMZQIPOJD NVTJD SPXT PG QBSUT
ٔ "OE FBDI SPX DPNQSJTFT B TFSJFT PG QJUDIFT

View Slide

4ZNCPMJD 5FDIOJRVFT
ٔ 4USJOH BMHPSJUINT
ٔ 3FQSFTFOUJOH NVTJD BT B TFSJFT PG TZNCPMT
ٔ "QQMZJOH UFDIOJRVFT TVDI BT FEJU EJTUBODF GPS TJNJMBSJUZ
ٔ .BSLPW NPEFMT
ٔ 0OMJOF QBࡻFSO EJTDPWFSZ JO TZNCPM TUSFBNT
ٔ &WFOU GSFRVFODZ BOBMZTFT
ٔ &H QJUDI IJTUPHSBNT
ٔ (FPNFUSJD BMHPSJUINT
ٔ 3FQSFTFOUJOH NVTJD BT NVMUJEJNFOTJPOBM TZNCPMT
ٔ "QQMZJOH USBOTGPSNBUJPOT GPS TJNJMBSJUZ BOE QBࡻFSO EJTDPWFSZ
ٔ "O FYBNQMF JT UIF 4*" BMHPSJUIN GPS QBࡻFSO JEFOUJGJDBUJPO
ٔ "OE 4*"5&$ GPS QBࡻFSO TFBSDI


View Slide

4ZNCPMJD 5BTLT
ٔ ,FZ GJOEJOHNPEF GJOEJOH
ٔ &H VTJOH QJUDI GSFRVFODZ BOBMZTJT
ٔ #BTJDT PG BVUPNBUJD NVTJD BOBMZTJT
ٔ 1JUDI TQFMMJOH
ٔ 1BࣄFSO JEFOUJGJDBUJPOSFUSJFWBM
ٔ &YUBDU NBUDIJOH WT BQQSPYJNBUF NBUDIJOH
ٔ ,OPXO QBࡻFSOT WT VOLOPXO QBࡻFSOT
ٔ 4FNBOUJD BVEJP
ٔ &YUSBDUJOH OPUFMJLF JOGPSNBUJPO BOE PUIFS TFNBOUJD MBCFMT

GSPN BVEJP
ٔ ࣉFSZCZFYBNQMF
ٔ 8IFSF UIF RVFSZ JT TZNCPMJD CVU UIF DPSQVT JT BVEJP

View Slide

4ZNCPMJD 5BTLT GPS .VTJDPMPHZ

View Slide

3FTPVSDF 1VCMJDBUJPO
ٔ .BOZ FYBNQMFT PG َEJHJUJTBUJPOُ JO UIF TZNCPMJD FODPEJOH XPSME
ٔ $$"3) BU 4UBOGPSE IBWF CFFO FEJUJOH BOE QVCMJTIJOH )VNESVN
BOE .VTF%BUB DPOUFOU GPS NBOZ ZFBST
ٔ .D(JMM XPSL PO FBSMZ NVTJD TPVSDFT BOE 0.3
ٔ 3*4. 01"$ QSPWJEFT JODJQJU TFBSDI

View Slide

%PDVNFOUBSZ &WJEFODF
ٔ 8IZ EP NVTJDPMPHJTUT TUVEZ OPUBUJPO TPVSDFT
ٔ 5P EJTDPWFS EPDVNFOUBSZ FWJEFODF PG NVTJDBM QSBDUJDF
ٔ #PUI DVSSFOU BOE IJTUPSJDBM
ٔ 'SPN NVTJDBM TPVSDFT XF DBO MFBSO BCPVU XIBU NVTJD XBT
LOPXO XIFSF BOE XIFO
ٔ "MUIPVHI SFNFNCFS UIF NFUBEBUB QSPCMFNT JO IJTUPSJDBM
TPVSDFT

ٔ -FBSO GSPN EPDVNFOUBSZ FWJEFODF BCPVU IPX NVTJD XBT
USBOTNJࡻFE
ٔ .BOVTDSJQUT CPPLT PࡺFO DPOUBJO DPQJFT PG QJFDFT CVU PࡺFO XJUI
FSSPST
ٔ -POH USBEJUJPO PG QSPEVDJOH FEJUJPOT PG NBOVTDSJQU NVTJD CPPLT

View Slide

%PDVNFOUBSZ &WJEFODF
ٔ 4DIPMBST CVJME VQ EPNBJO LOPXMFEHF EVSJOH UIFJS DBSFFST
ٔ 0O NVTJDBM XPSLT NVTJDJBOT TDSJCFT MPDBUJPOT BOE EBUFT PG
CPPLT
ٔ 6TJOH UIJT UIFZ BSF BCMF UP BQQSPBDI SFTFBSDI RVFTUJPOT TVDI BT
ٔ XIP DPQJFE UIJT OFX VOBࡻSJCVUFE TPVSDF
ٔ HJWFO UIJT TJOHMF QBSUCPPL XIJDI QJFDFT BSF DPQJFE JOUP JU
ٔ PG XIJDI PUIFS QJFDF JT UIJT OFX TPVSDF BO BSSBOHFNFOU
ٔ 5P FODPEF TPVSDFT XJUI UIF LJOE PG EFUBJM SFRVJSFE GPS
EPDVNFOUBSZ SFTFBSDI XF OFFE SJDI BOE GMFYJCMF FODPEJOHT
ٔ 'PS UIJT UIF .VTJD &ODPEJOH *OUJUBUJWF JT QSPNJTJOH

View Slide

$POUFOUCBTFE $PSQVT .VTJDPMPHZ
ٔ "SHVFE FBSMJFS UIBU NVTJDPMPHZ JT MBSHFMZ OPU BCPVU NVTJDBM
DPOUFOU
ٔ #VU DPNQVUBUJPOBM UFDIOJRVFT QSPWJEF BDDFTT UP DPOUFOU
ٔ &TQFDJBMMZ PO B MBSHFTDBMF UIBU IBT OFWFS CFFO USBDUBCMF GPS
NVTJDPMPHJTUT
ٔ 4P DPSQVT NVTJDPMPHZ IBT KVTU OFWFS GFBUVSFE BT QBSU PG UIF
EJTDJQMJOF
ٔ "MTP WBSJPVT IJTUPSJDBM SFBTPOT XIZ NVTJDPMPHJTUT UFOE UP BWPJE
XIBU UIFZ TFF BT َDPNQBSBUJWFُ XPSL JO TUVEZJOH NVTJD
ٔ "SFBT PG SFTFBSDI UIBU NBZ CF PQFOFE VQ CZ MBSHFTDBMF
UFDIOJRVFT JODMVEF
ٔ 4UVEZJOH UIF َNVTJDBM CBDLHSPVOEُ

View Slide

.VTJDBM 4FBSDI
ٔ ,OPXMFEHF PG SFQFSUPJSF TPVSDFT BOE USBEJUJPOT NPTU SFMJBCMF
TPVSDF PG OFX EJTDPWFSJFT
ٔ (JWFO TVख़JDJFOU DPSQPSB DPOUFOUCBTFE TFBSDI NBZ FOBCMF
SFTFBSDI RVFTUJPOT BSPVOE NVTJDBM BMMVTJPO
ٔ RVPUBUJPO CPSSPXJOH BSSBOHFNFOU DPQZJOH FSSPST FUD
ٔ 1PUFOUJBM UFDIOJRVFT JODMVEF
ٔ 1ISBTFCBTFE SFUSJFWBM HJWFO B TZNCPMJD NVTJD QISBTF SFUVSO BMM
UIF QJFDFT UIBU DPOUBJOT NBUDIFT XJUIJO B UISFTIPME BOE UIFJS
MPDBUJPOT
ٔ &H 5IFNF'JOEFS 1FBDIOPUF 3*4. 01"$

View Slide

EMO & ECOLM!
Printed 16th-century vocal and lute music

View Slide

Elements for a Workﬂow!
Will need to be roughly similar for any medium-large
scale automatic music-encoding project

View Slide

Secular vocal Sacred vocal Free
compositions
Dances
('suites'
count as
one)
Total
Totals 486 89 232 275 1082
45%
8%
22%
25%
Secular vocal
Sacred vocal
Free composi6ons
Dances ('suites' count as one)
300 books of music printed before 1600
All are ‘anthologies’ (i.e. by >1 composer)
About 10% (27 books) are in tablature
About 50% of tablature is arrangements of
vocal music
Need to identify unknown source-models for
the arrangements
Early Music Online (EMO)!

View Slide

Plenty of musicological literature about the
arrangements
Essential resources is: Howard Mayer Brown,
Instrumental Music Published before 1600
(Harvard, 1965)

View Slide

H.M. Brown, Instrumental Music!

View Slide

But also, scholars (and, more recently,
players) have identified quotations of vocal
music in fantasias, etc.
As well as direct quotations, lute music oen
contains less-clear allusions or even
paraphrases of well-known music of the time
Perhaps we can discover more of these?

View Slide

In manuscript sources, most lute music is
anonymous
Titles and/or composer names (where given)
are oen incorrect, misspelt or otherwise
garbled in manuscripts - and even in prints

View Slide

We are looking at ways to use musical
content to ‘amplify’ metadata searches, since
approximate matching is required in both
domains
Diﬀerent approaches needed for lute and
vocal music, because of the diﬀerences
between tablature and mensural notation

View Slide

EMO/ECOLM cross-searching!
Need a common/compatible data structure
for searching
We treat a score as a discrete pitch/onset-
time matrix (Something like MIDI)
Use geometric search methods such as
SIAMESE
We can also generate chroma and use audio
methods (e.g. to match with recordings)

View Slide

EMO!
16-c vocal music was mostly published in
part-books

View Slide

EMO – part-books!
Superius! Altus!
Tenor! Bassus!

View Slide

EMO – part-books!
part-books
OMR - Correction interface within Aruspix

View Slide

EMO – Aruspix - Optical Music Recognition!
Vocal music in part-books

View Slide

EMO – part-books!
part-books
-> MEI encodings (part-books)

View Slide

EMO – part-books – MEI encodings!
Vocal music in part-books
-> MEI encodings (part-books)

View Slide

EMO – part-books -> scores!
Given corrected MEI part-book encodings …
… making scores is easy, but correction takes
time
Making scores from error-ridden parts is an open
problem we are addressing
Converting score to pitch/time matrix is trivial

View Slide

EMO – metadata for pieces?!
… but, where do pieces begin/end?

View Slide

Image 014!
Image 030!
Image 046!
Image 062!

View Slide

EMO collation interface, based on library
metadata

View Slide

EMO – ‘Inventory Maker’!

View Slide

ECOLM – Tablature recogntion is simpler !
Each piece is complete on a page-sequence in
a single book
Optical Tablature Recognition based on
Gamera
Errors are inevitable with optical recognition
Crowd-source expert correction via web
interface

View Slide

ECOLM – Tablature correction!
Crowd-source expert correction via web
interface
Double-entry: correct twice for each line of
music
Easier than for vocal music: use Brown
metadata to collate corrected lines

View Slide

ECOLM – from tablature to search format!
With corrected tablature for complete pieces
… conversion to pitch/time matrix is easy
We can search within ECOLM using SIAMESE
We can generate chroma and use audio
methods (e.g. to match with recordings)

View Slide

ECOLM!
ECOLM editing/playback/search interface
demo

View Slide

Near-neighbour search in acoustic feature space
A case study in contrafactum and parody
Presented/adopted by Ben Fields
Material originally by Christophe Rhodes

View Slide

#ISMIRMusicologists
Introduction
The ‘remixes’ and ‘cover songs’ of the Renaissance
Parody and contrafacta
Lugebat David Absalon
Je prens congies
J'ay mis mon cueur
Credo a 8 mass

View Slide

#ISMIRMusicologists
Introduction
Parody and contrafacta
Nirvana Weird Al

View Slide

#ISMIRMusicologists
Introduction
Josquin des Prez (c.1450–1521?)
• contemporary reputation: greatest composer of the age
Josquin des Prez
*OUSPEVDUJPO 4JNJMBSJUZ 3FTVMUT $PODMVTJPOT
*OUSPEVDUJPO
+PTRVJO EFT 1SF[
+PTRVJO EFT 1SF[ Dم

ٔ DPOUFNQPSBSZ SFQVUBUJPO HSFBUFTU DPNQPTFS PG UIF BHF

View Slide

#ISMIRMusicologists
Introduction
Nicolas Gombert (c.1495–c.1560)
• transition figure between Josquin and Palestrina
Nicolas Gombert

View Slide

#ISMIRMusicologists
Introduction
The Je prens congies complex
•Je prens congies chanson
•Tu sola es and Tulerunt Dominum motets
•Lugebat David Absalon motet
•Credo a 8 mass fragment
•J’ay mis mon cueur chanson

View Slide

#ISMIRMusicologists
Introduction
Attribution
Attribution shift over almost a century
•19th century (Otto Kade)
•Tulerunt authentic Josquin
•Lugebat David Absalon doubtful
•Josquin Edition (Albert Smijers, 1921–)
•neither Tulerunt nor Lugebat
•DasChorwerk (Blume,1933)
•Tulerunt attributed to Josquin
•Early Venetian Motets (Norbert Böker-Heil, 1969)
•Identification of Je prens congas as source material

View Slide

#ISMIRMusicologists
Introduction
Attribution
*OUSPEVDUJPO
"ࡻSJCVUJPO

View Slide

#ISMIRMusicologists
Similarity
Audio Features: Non-Negative Least Squares Chroma
•log-frequency short-time Fourier Transform Yk,m
•note dictionary Ek,n
•find note activations xn,. to minimize
||Yk,.
− Ek,nxn,.||2
•subject to xn,m
≥ 0
Matthias Mauch and Simon Dixon, Approximate Note Transcription for the
Improved Identiﬁcation of Difﬁcult Chords, Proc. ISMIR, 2010!

View Slide

#ISMIRMusicologists
Similarity
Audio similarity: sequences
•In this investigation:
•frames: 1s granularity;
•feature dimensionality d: 12;
•sequences: searchlength 10-30 frames.
4JNJMBSJUZ
"VEJP TJNJMBSJUZ TFRVFODFT
E
E × TFBSDIMFOHUI
O − TFBSDIMFOHUI + 1
*O UIJT JOWFTUJHBUJPO
ٔ GSBNFT T HSBOVMBSJUZ
ٔ GFBUVSF EJNFOTJPOBMJUZ E
ٔ TFRVFODFT TFBSDIMFOHUI GSBNFT
$ISJTUPQIF 3IPEFT 'SJEBZ UI 4FQUFNCFS /FBSOFJHICPVS TFBSDI JO BDPVTUJD GFBUVSF TQBDFT

View Slide

#ISMIRMusicologists
Similarity
Audio similarity
Fragment to fragment:
!
•metric distance (maybe?)
•non-negative(maybe?)
•bounded above (maybe?)
NJMBSJUZ
P TJNJMBSJUZ
HNFOU UP GSBHNFOU
E-(GJ
N, GK
O)
NFUSJD EJTUBODF NBZCF

OPOOFHBUJWF NBZCF

CPVOEFE BCPWF NBZCF

View Slide

#ISMIRMusicologists
Similarity
Audio similarity
!
NJMBSJUZ
P TJNJMBSJUZ
HNFOU UP GSBHNFOU
E-(GJ
N, GK
O)

OPOOFHBUJWF NBZCF

CPVOEFE BCPWF NBZCF

Fragment to track:
HNFOU UP GSBHNFOU
E-(GJ
N, GK
O)
ٔ NFUSJD EJTUBODF NBZCF

ٔ OPOOFHBUJWF NBZCF

ٔ CPVOEFE BCPWF NBZCF

HNFOU UP USBDL
E-(GJ
N, UK) = NJO
O
E-(GJ
N, GK
O)

View Slide

#ISMIRMusicologists
Similarity
Audio similarity
!
NJMBSJUZ
P TJNJMBSJUZ
HNFOU UP GSBHNFOU
E-(GJ
N, GK
O)

OPOOFHBUJWF NBZCF

CPVOEFE BCPWF NBZCF

Fragment to track:
HNFOU UP GSBHNFOU
E-(GJ
N, GK
O)
ٔ NFUSJD EJTUBODF NBZCF

ٔ OPOOFHBUJWF NBZCF

ٔ CPVOEFE BCPWF NBZCF

HNFOU UP USBDL
E-(GJ
N, UK) = NJO
O
E-(GJ
N, GK
O)
Track to track:
NFOU UP GSBHNFOU
E-(GJ
N, GK
O)

OPOOFHBUJWF NBZCF

CPVOEFE BCPWF NBZCF

NFOU UP USBDL
E-(GJ
N, UK) = NJO
O
E-(GJ
N, GK
O)
L UP USBDL
E-(UJ, UK) = NJO
N
E-(GJ
N, UK)

View Slide

#ISMIRMusicologists
Similarity
Outlier analysis
4JNJMBSJUZ
0VUMJFS BOBMZTJT

View Slide

#ISMIRMusicologists
4JNJMBSJUZ
0VUMJFS BOBMZTJT
Similarity
Outlier analysis

View Slide

#ISMIRMusicologists
Results
Josquin attribution
With some care: can find all of the recordings in the
Je prens congies complex as low-distance retrievals:
•aggregate over many fragments of source
tracks;
•chroma rotation.

View Slide

#ISMIRMusicologists
Results
Reproducibility
Gave (broadly) this investigation to students at
Oxford Digital Humanities Summer School:
•no prior programming assumed;
•audio features precomputed.
Student feedback for five-day “Digital Musicology”:
92% “met” or “exceeded” expectations

View Slide

#ISMIRMusicologists
Conclusions
•audio similarity can be used to guide attribution...
•…if recordings are available
•(mechanically-generated recordings would be
better for this task)
•similarity outliers can help increase
understanding

View Slide

#ISMIRMusicologists
Conclusions
•query language (Hendrik Blokeel “Declarative
Data Analysis”)
•hierarchical indexing (Fionn Murtagh “High
Dimensional Data Scaling”)
•shrink-wrap software
•new interfaces for exploring collections of music
Future Work

View Slide

#ISMIRMusicologists
Questions
Investigation materials:
•audioDB software: https://github.com/TransformingMusicology/
audioDB
•linked from http://www.doc.gold.ac.uk/~mas01cr/papers/:
•iPython notebook
•numeric feature data
Questions or comments to:
•! @ascii19 / [email protected] (Christophe Rhodes)
•! @alsothings / [email protected] (Me!)
•! @TMusicology
•#ISMIRMusicologists

View Slide

Wagner leitmotifs!
Proof-of-concept work mostly done under
OMRAS2 project, c2008-11

View Slide

Richard Wagner!
Richard Wagner (1813-1883) is well-known
for developing a method of composition
which employed ‘leitmotifs’ (leading motifs)
to help the audience follow the dramatic
narrative and identify with the emotional
states of the characters on-stage.
Technique still used today in advertising and
film music.

View Slide

Research question!
Since the leitmotif technique depended on
the audience perceiving the leitmotifs clearly,
can we use audio MIR to ‘recognise’ them as
humans do?
In the psychological part of Transforming
Musicology, we are working to establish how
well audiences hear and respond to Wagner’s
leitmotifs.

View Slide

Research challenge!
Early work (from the OMRAS2 project) used
NNLS chroma features as a proof-of-concept
Some leitmotifs are easy to recognise, but
others much harder
It depends on musical context, and the extent
to which they diﬀer from the query example
Also, tempo, timbre (instrument, voice) and
orchestral texture (scoring) play a part, too

View Slide

Wagner’s ‘Ring’ Cycle!
Sir Georg Solti’s famous recording of the
‘Ring’ Cycle of four operas (Das Rheingold,
Die Walküre, Siegfried & Göerdämmerung)
Recorded in Vienna, 1958-66
Re-mastered and re-released on CD oen
148 tracks, ~12 hours of music, avge 5 min

View Slide

Method (i)!
Extract NNLS-chroma features (4 & 10
frames/sec) from Solti Ring recordings and
make a pair of audioDB databases
Extract NNLS-chroma features (4 & 10 f/s)
from a set of leitmotifs for which we have
names (from books by Donington and Mann)
and score-extracts (graphics)
Build simple web-interface

View Slide

Method (ii)!
Perform searches
a) at same pitch
b) transposed (queries ‘rotated’) to all 12 keys
c) at 4 and 10 f/s (user choice)
Select 10 top best matches

View Slide

ٔ -BTU IBMGIPVS JT B TUSVDUVSFE EJTDVTTJPO TFTTJPO
ٔ %JWJEF JOUP QBJST BOE EJTDVTT ZPVS .*3 UFDIOJRVFT BOE DVSSFOU
BQQMJDBUJPOT
ٔ 5IFO TVHHFTU QPTTJCMF NVTJDPMPHJDBM BQQMJDBUJPOT PG FBDI PUIFSًT
UFDIOJRVFT
ٔ +PJO JOUP MBSHFS HSPVQT PS
QSFTFOU FBDI PUIFST UFDIOJRVFT
BOE QSPQPTFE NVTJDPMPHJDBM BQQMJDBUJPOT
ٔ %FDJEF PO UXP PS UISFF NVTJDPMPHJDBM BQQMJDBUJPOT UP EJTDVTT
ٔ &BDI HSPVQ UIFO QSFTFOUT UIPTF BQQMJDBUJPOT BU UIF FOE

View Slide

Addressing the music information needs of musicologists

Addressing the music information needs of musicologists

Ben Fields

More Decks by Ben Fields

Other Decks in Research

Featured

Transcript