Share
Embed
Target-speaker voice activity detection: A novel approach for multi-speaker diarization in a dinner party scenario
INTERSPEECH 2020 読み会の資料です。
Hitoshi Suda
November 20, 2020
Transcript
29 slides
-
Target-speaker voice activity detection: A novel approach for multi-speaker
diarization in a dinner party scenario ਢాਔࢤʢ౦ژେֶʣ !*/5&341&&$)ಡΈձ -
֘จ w *WBO.FEFOOJLPW FUBM 5BSHFUTQFBLFSWPJDFBDUJWJUZEFUFDUJPO "OPWFMBQQSPBDIGPSNVMUJTQFBLFSEJBSJ[BUJPOJOBEJOOFSQBSUZTDFOBSJP w
ϩγΞͷاۀ45$JOOPWBUJPOTͱ αϯΫτϖςϧϒϧΫʹ͋Δ*5.0େֶ จΛબΜͩཧ༝ w 4UBUFPGUIFBSUͳμΠΞϥΠθʔγϣϯٕज़ͷհ -
·ͱΊ w 5BSHFUTQFBLFSʹର͢Δ7"%Λ࣮͢Δ͜ͱͰ $)J.&νϟϨϯδʹ͓͍ͯߴ͍ੑೳͷμΠΞϥΠθʔγϣϯΛ࣮ݱͨ͠ w ਓطͷ547"%ΛϚϧννϟωϧͰߦͬͨ w ڥ͕ҟͳΔ݅ɺਓະͰͷධՁࠓޙͷ՝
ײ w ண࣮ͳ࣮ͱΑ͘ચ࿅͞ΕͨॳظʹΑ࣮ͬͯݱ͞Ε͍ͯΔͱ͍͏ҹ w ࠷ॳͷΫϥελϦϯάͷ෦͕͔ͳΓ࡞Γ͜·Ε͍ͯΔ w ਓ͕ط͔ͩΒཱͰ͖͍ͯΔؾ͕ͯ͘͠Δ w Φʔόʔϥοϓ۠ؒΛຊʹͪΌΜͱೝࣝͰ͖͍ͯͨͷ͔ؾʹͳΔ -
0VUMJOF w ઃఆ w طଘख๏ w ఏҊ๏ w 5BSHFUTQFBLFS7"%
w .VMUJTQFBLFSUBSHFUTQFBLFS7"% -
$)J.&$IBMMFOHFʢʣ w ձԻͷ"43ͷνϟϨϯδ w ϫʔΫγϣοϓ*$"441ͷαςϥΠτ w ԻSFBMIPNFͷSFBMEJOOFSQBSUZճʢ࣮ࡍʹBDUFEΒ͍͠ʣ w 5SBJOEFWFWBMɿ
w ͦΕͧΕ࣌ؒ w νϟωϧͷ,JOFDUʷͭ w ొਓͦΕͧΕਓͰݻఆ w ෦Λࣗ༝ʹҠಈͰ͖Δ IUUQTDIJNFDIBMMFOHFHJUIVCJPDIJNFJOEFYIUNM -
$)J.&$IBMMFOHFʢʣ w ձԻͷ"43ͷνϟϨϯδ w 5BTLʢ5SBDLʣ͕ͭ͋Γɺ 5SBDLμΠΞϥΠθʔγϣϯͷHSPVOEUSVUIΛ༻͍ͯΑ͍ɺ 5SBDLμΠΞϥΠθʔγϣϯඞཁʢԻͷΈ͔Βೝࣝ͢Δʣ
w σʔλಉ͡ w ࠷ऴతͳධՁ࣠XPSEFSSPSSBUF w 3BOLJOH"Ͱೝࣝ༻ͷϞσϧ͕ڞ௨ɺ#ͰೝࣝϞσϧ·ͰؚΊͯߏங w 5SBDLʷ3BOLJOH"#ͷ௨ΓͰධՁ͞ΕΔ -
5BTLʢԻͷΈʣɾ3BOLJOH"ʢೝࣝϞσϧHJWFOʣ 8&3 ख๏ EFW FWBM 45$*OOPWBUJPOT*5.0 ΫϥελϦϯά๏ʴ547"%
+PIOT)PQLJOT ΫϥελϦϯά๏ʴ0WFSMBQQPTUQSPDFTTJOH 645$ .VMUJDIBOOFM4QFFDI4FQBSBUJPOʴΫϥελϦϯά 1BEFSCPSO6OJWFSTJUZ &OEUPFOE/FVSBM%JBSJ[BUJPO #SOP6OJWFSTJUZPG5FDIOPMPHZ 7#Yʴ(VJEFE4PVSDF4FQBSBUJPO $MPVEXBML9.6 ΫϥελϦϯά๏ $JUZ6OJWFSTJUZPG/FX:PSL ΫϥελϦϯά๏ʴ0WFSMBQQPTUQSPDFTTJOH "DBEFNJB4JOJDB ΫϥελϦϯά๏ #BTFMJOF ΫϥελϦϯά๏ -
%JBSJ[BUJPOJTIBSE w μΠΞϥΠθʔγϣϯ͍͠ w ϚΠΫ͔Βͷڑ͕ԕ͍͜ͱ͕ଟ͍ w ࡶԻ͕͋Δ͜ͱ͕ଟ͍ w ձԻͰ͋Δ
w ൃ͕Φʔόʔϥοϓ͢Δ USBJO EFW FWBM ಉ࣌ʹൃ͍ͯ͠Δਓͷׂ߹ ʜ୭ൃ͍ͯ͠ͳ͍ʗ$)J.&σʔληοτ -
0VUMJOF w ઃఆ w طଘख๏ w ఏҊ๏ w 5BSHFUTQFBLFS7"%
w .VMUJTQFBLFSUBSHFUTQFBLFS7"% -
ݹయతͳख๏ɿΫϥελϦϯά๏ w దʹηάϝϯςʔγϣϯˠ֤ηάϝϯτͷऀදݱΛΫϥελϦϯά w Φʔόʔϥοϓ۠ؒޙॲཧతʹղܾͰ͖Δ<,#PBLZF >ͳͲ w
%&3ʢEFWTFUʣ 7"% ηάϝϯςʔγϣϯ ऀදݱͷநग़ ΫϥελϦϯά ,#PBLZF FUBM0WFSMBQQFETQFFDIEFUFDUJPOGPSJNQSPWFETQFBLFSEJBSJ[BUJPOJONVMUJQBSUZNFFUJOHT*O1SPD*$"441 QQr -
W0 2 RD⇥F and b0 2 RD project an
input feature into D- sional vector. Encoder(p) t (·) is the p-th encoder block which s an input sequence of D-dimensional vectors and outputs a ensional vector e (p) t at time index t. We use P encoder blocks ed by the output layer for frame-wise posteriors. e architecture of the encoder block is depicted in Fig. 2. onfiguration of the encoder block is almost the same as the the Speech-Transformer introduced in [44], but without posi- encoding. The encoder block has two sub-layers. The first is i-head self-attention layer, and the second is a position-wise orward layer. Multi-head self-attention layer ulti-head self-attention layer transforms a sequence of input s as follows. The sequence of vectors (e (p 1) t |t = 1, · · · , T) verted into a RT ⇥D matrix, followed by layer normalization p 1) = LayerNorm([e (p 1) 1 · · · e (p 1) T ]>) 2 RT ⇥D . (6) Label 1 Label 2 Label 1 Label 2 BCE BCE minimum Permutation 1 Permutation 2 Permutation-free loss Linear + Sigmoid Output 1 Output 2 LayerNorm Encoder block Log-Mel LayerNorm Multi-head self-attention P blocks Position-wise FF LayerNorm Input Linear Encoder block ¯ E(p 1) ¯ E(p,SA) E(p,FF) E(p,SA) Fig. 2. Two-speaker SA-EEND model trained with permutation-free &OEUPFOE/FVSBM%JBSJ[BUJPOʢ&&/%ʣ w εϖΫτϩάϥϜʹରͯ͠ μΠΞϥΠθʔγϣϯ݁ՌΛग़ྗ w ֶशσʔλྔ͕େྔʹඞཁ w υϝΠϯͷ߹Θͳ͍σʔλʹ੬ऑ w ࣌ؒͷσʔλʹ͑ͳ͍ w ޙஈͰύʔϛϡςʔγϣϯΛղ͘ඞཁੑ w %&3ʢEFWTFUʣ : 'VKJUB FUBM&OEUPFOEOFVSBMTQFBLFSEJBSJ[BUJPOXJUITFMGBUUFOUJPO*O1SPD"436 QQr <: 'VKJUB > -
0VUMJOF w ઃఆ w طଘख๏ w ఏҊ๏ w 5BSHFUTQFBLFS7"%
w .VMUJTQFBLFSUBSHFUTQFBLFS7"% -
5BSHFUTQFBLFS7"%ͷൃ w ಛఆऀͷऀදݱΛೖྗͯ͠ɺͦͷऀ͕ൃ͍ͯ͠Δ͔Λ֤࣌ࠁͰࣝผ w ྨࣅख๏ɿ7PJDF'JMUFS<28BOH >ɺ4QFBLFS#FBN<.%FMDSPJY >ͳͲ
w ಛఆͷऀදݱΛ࣋ͭऀͷΈநग़͢ΔεϖΫτϩάϥϜͷιϑτϚεΫΛਪఆ ࣝผث ऀදݱ .%FMDSPJY FUBM4JOHMF$IBOOFM5BSHFU4QFBLFS&YUSBDUJPOBOE3FDPHOJUJPOXJUI4QFBLFS#FBN*O1SPD*$"441 QQr 28BOH FUBM7PJDF'JMUFS5BSHFUFE7PJDF4FQBSBUJPOCZ4QFBLFS$POEJUJPOFE4QFDUSPHSBN.BTLJOH*O1SPD*/5&341&&$) QQr -
#BTFMJOF1FSTPOBM7"% w ಛఆऀͷൃΛଈ࠲ʹݕग़͢ΔγεςϜ w Φʔόʔϥοϓఆ͍ͯ͠ͳ͍ w OPOTQFFDIɺUBSHFUTQFBLFSTQFFDIɺ OPOUBSHFUTQFBLFSTQFFDIͷΫϥεྨ
w OPOTQFFDIͱOPOUBSHFUTQFBLFSTQFFDIͷ ࠞಉͷॏΈΛܰ͘͢Δͱɺ࠷ऴతͳೝࣝਫ਼্͕͢Δ w ऀর߹ͷείΞলུͰ͖Δ 4%JOH FUBM1FSTPOBM7"%4QFBLFS$POEJUJPOFE7PJDF"DUJWJUZ%FUFDUJPO*O1SPD4QFBLFS0EZTTFZ QQr &RQFDW (PEHGGLQJ 6SHDNHU YHULILFDWLRQ 'HFLVLRQ )HDWXUH H[WUDFWLRQ 3HUVRQDO9$' (PEHGGLQJ 'HFLVLRQ )HDWXUH H[WUDFWLRQ 3HUVRQDO9$' (PEHGGLQJ 6SHDNHU YHULILFDWLRQ 'HFLVLRQ )HDWXUH H[WUDFWLRQ 3HUVRQDO9$' QSXWDXGLR ,QSXWDXGLR ,QSXWDXGLR &RQFDW &RQFDW E F G 6FRUHGFRQGLWLRQHGWUDLQLQJ (PEHGGLQJFRQGLWLRQHGWUDLQLQJ 6FRUHDQGHPEHGGLQJFRQGLWLRQHGWUDLQLQJ <4%JOH > -
4JOHMFTQFBLFS547"% w 1FSTPOBM7"%ͱಉ༷ w %&3ʢޙॲཧޙʣ w ଞͷਓͷJWFDUPSΛ ಉ࣌ʹೖྗͯ͠
େ͖͘վળ͠ͳ͔ͬͨ #-45.1ɿ#JEJSFDUJPOBM-45.1SPKFDUJPO -
֤ऀ͝ͱͰ7"%Λ͢Δࠜຊతͳʢ༧ʣ w /POUBSHFUTQFBLFSTTQFFDIΛݕग़͢Δඞཁੑ͕͋Δ w จʹཅʹॻ͍ͯͳ͍͕ɺ͓ͦΒ͘ܦݧతʹͦ͏ͳͷͩͱࢥ͏ w ऀදݱ͕͍ۙऀʹରͯ͠੬ऑ w
ଞͷऀ͕໌͍ͯͨ͠΄͏͕ʢײతʹʣੑೳ͕ग़Δ w ଞͷऀ͕BXBSFͰ࣮ݧతʹվળ͠ͳ͔ͬͨ w ಛఆͷऀ͚ͩʹண͢Δͱ͍͏ෆۉߧײ͕ۤख -
0VUMJOF w ઃఆ w طଘख๏ w ఏҊ๏ w 5BSHFUTQFBLFS7"%
w .VMUJTQFBLFSUBSHFUTQFBLFS7"% -
BLSTMP layer. The model architecture is shown in Figure
Note that parameters of the SD block are shared across spe ers, and it is trained jointly with the whole TS-VAD model. Figure 1: Single-channel TS-VAD scheme As we performed all the experiments in the Kaldi A .VMUJTQFBLFS547"% w ਓͷ7"%Λಉ࣌ʹਪఆ w $)J.&ͯ͢ͷԻͰਓ w ֤ऀʹରͯ͠XFJHIU͕ڞ༗͞Εͨ෦ʴ ࠷ऴతͳ݁ՌΛग़ྗ͢Δ෦ʹׂ w ֶश্ஈɾԼஈಉ࣌ w ग़ྗ֤ऀʹରͯ͠Ϋϥεྨ w ਓͳͷͰग़ྗ࣍ݩ w ଛࣦͭͷަࠩΤϯτϩϐʔͷ w ޙॲཧޙ%&3 4%ɿ4QFBLFS%FUFDUJPO -
ҙऀ547"% w .VMUJTQFBLFS7"%ਓݻఆ w ਓՄมͷख๏ఏҊ͞Ε͍ͯΔ w ՄมΛड͚͚ͭΔԿΒ͔ͷΞʔΩςΫνϟʹ தؒΛೖΕΕΑ͍
w ࣮ݧతͳධՁ͞Ε͍ͯͳ͍ w 4JOHMFTQFBLFSͷͱ͖ͱಉ͕͡ى͖ͦ͏ -
JWFDUPSͷਪఆ w JWFDUPSΛΫϥελϦϯά๏ͰٻΊͨμΠΞϥΠθʔγϣϯͷ݁Ռ͔Βܭࢉ w 547"%ˠJWFDUPSਪఆΛ෮͢Δ͜ͱͰɺΑΓੑೳ͕վળ͢Δ w ΠλϨʔγϣϯ·Ͱվળ͕ݟΒΕͨ w ϚχϡΞϧηάϝϯςʔγϣϯͰJWFDUPSΛநग़ͨ͠ͱ͖ͱಉͷੑೳ
w %&3 ΫϥελϦϯά๏ ॳظJWFDUPSੜ 547"% JWFDUPSੜ -
ޙॲཧ w 5ISFTIPMEJOHɿಛఆͷΑΓখ͍֬͞Λʹ w .FEJBO f i MUFSJOHɿ૭͔͚ͨ۠ؒ͠ͰதԝΛͱͬͯ-1'
w ࣌ؒͷϙʔζɾൃͷআ w )..ΛΜͰϏλϏ୳ࡧ w 5ISFTIPMEJOHʴ.FEJBOͱಉͷੑೳ͕ಘΒΕΔ w "43ͷલʹ(VJEFETPVSDFTFQBSBUJPO<$#PFEEFLFS >Λద༻ $#PFEEFLFS FUBM'SPOUFOEQSPDFTTJOHGPSUIF$)J.&EJOOFSQBSUZTDFOBSJP*O1SPD$)J.&8PSLTIPQ -
ੑೳධՁ w ֶशσʔλ$)J.&࣌ؒʴ7PY$FMFC࣌ؒʴ.JYVQ w ࣮ݧ݅΄ͱΜͲॻ͔Ε͍ͯͳ͍͕,BMEJͷϨγϐެ։͞Ε͍ͯΔ w 'VTJPOνϟωϧͷγϯάϧνϟωϧϞσϧͱɺ ͭͷϚϧννϟωϧϞσϧΛɺॏΈ͖ฏۉͨ͠ͷ
EFWFMPQNFOU FWBMVBUJPO %&3 +&3 %&3 +&3 $)J.&#BTFMJOF ɹΫϥελϦϯά๏ ɹɹʴ547"% ɹɹɹʴ547"% ɹɹɹʴ547"%.$ 'VTJPO -
·ͱΊ w 5BSHFUTQFBLFSʹର͢Δ7"%Λ࣮͢Δ͜ͱͰ $)J.&νϟϨϯδʹ͓͍ͯߴ͍ੑೳͷμΠΞϥΠθʔγϣϯΛ࣮ݱͨ͠ w ਓطͷ547"%ΛϚϧννϟωϧͰߦͬͨ w ڥ͕ҟͳΔ݅ɺਓະͰͷධՁࠓޙͷ՝
ײ w ண࣮ͳ࣮ͱΑ͘ચ࿅͞ΕͨॳظʹΑ࣮ͬͯݱ͞Ε͍ͯΔͱ͍͏ҹ w ࠷ॳͷΫϥελϦϯάͷ෦͕͔ͳΓ࡞Γ͜·Ε͍ͯΔ w ਓ͕ط͔ͩΒཱͰ͖͍ͯΔؾ͕ͯ͘͠Δ w Φʔόʔϥοϓ۠ؒΛຊʹͪΌΜͱೝࣝͰ͖͍ͯͨͷ͔ؾʹͳΔ