Self-Conditioned CTCとその発展

小松達也（LINE株式会社 Senior Research Scientist）

LINEが開発してきた非自己回帰型の音声認識方式 Self-Conditioned CTCについて紹介します。Self-Conditioned CTCはCTC型の音声認識の一つであり、ニューラルネットワークの中間層での認識結果を補助情報として活用することにより、CTC型方式の高速性を保持しつつ、テキスト間の依存関係を考慮できる自己回帰型方式と同等の認識精度を発揮します。発表では、Self-Conditioned CTCの詳細とその後の発展手法、音声認識以外への応用などについて紹介します。

※Tokyo BISH Bash #08での発表資料です
https://tokyo-bish-bash.connpass.com/event/283268/

Transcript

1. 4FMG$POEJUJPOFE
   $5$
   ͱͦͷൃల 5BUTVZB
   ,PNBUTV 4FOJPS
   3FTFBSDI
   4DJFOUJTU -*/&
   $PSQPSBUJPO
2. ຊ೔ͷ಺༰ɿ4FMG$POEJUJPOFE
   $5$ • 4FMGDPOEJUJPOFE
   $5$ • $5$ʹجͮ͘ඇࣗݾճؼܕԻ੠ೝࣝํࣜ </P[BLJ
   *OUFSTQFFDI> • ߴ଎ʹਪ࿦͕ՄೳɺࣗݾճؼܕԻ੠ೝࣝʹඖఢ͢Δਪ࿦ਫ਼౓ • جຊख๏ͱͦͷൃలܗʹ͍ͭͯ঺հ  &ODPEFS O &ODPEFS O &ODPEFS  -JOFBS
   
   4PGUNBY -JOFBS $5$ $5$ -JOFBS
   
   4PGUNBY &ODPEFS / *OQVU
   BVEJP 0VUQVU
   UPLFO 4FMGDPOEJUJPOJOH
3. $5$ʹجͮ͘ඇࣗݾճؼܕԻ੠ೝࣝ  ࠓ࠷΋ߴ͍֬཰Λ࣋ͭจࣈΛฒ΂ͯ ͗Ύͬͱ·ͱΊΔͱਖ਼ղʹͳΔΑ͏ʹֶश͢Δ ܁Γฦ͠ χϡʔϥϧωοτ ͜ Μ Μ ʹ ͪ ͸ ͸ ͜Μʹͪ͸ 1. p(こ) = 0.7 2. p(＿) = 0.2 3. p(藤) = 0.0001 $POOFDUJPOJTU
   5FNQPSBM
   $MBTTJGJDBUJPO
   <(SBWFT > ͜ ࣗݾճؼܕํࣜ ʹਫ਼౓͸ߴ͍͕͋·Γ଎͘ͳ͍ "UUFOUJPO
   &OD%FD
   <$IPSPXTLJ > 3//5SBOTEVDFS
   <(SBWFT > ֤จࣈ͸ಠཱʹਪఆʢ৚݅෇͖ಠཱͷԾఆʣ ʹߴ଎͕ͩਫ਼౓͸ͦ͜·Ͱߴ͘ͳ͍ • ߴ଎ʹਪ࿦͕Ͱ͖Δೝࣝํࣜ
4. $5$ʹجͮ͘ඇࣗݾճؼܕԻ੠ೝࣝ  &ODPEFS O &ODPEFS O &ODPEFS  $5$ -JOFBS
   
   4PGUNBY &ODPEFS / *OQVU
   BVEJP • ԻڹΤϯίʔμͱ$5$σίʔμͰߏ੒ ʢ5SBOTGPSNFS$POGPSNFSʣ • จࣈؒͷ৚݅෇͖ಠཱੑΛԾఆ • ߴ଎͕ͩਫ਼౓͸ͦ͜·Ͱߴ͘ͳ͍ 0VUQVU
   UPLFO ߴ଎͞Λอͪͭͭ ਫ਼౓Λߴ͍ͨ͘͠
5. ઌߦݚڀɿ*OUFSNFEJBUF
   $5$
   <-FF
   > &ODPEFS O &ODPEFS O &ODPEFS  -JOFBS
   
   4PGUNBY $5$ $5$ -JOFBS
   
   4PGUNBY &ODPEFS / *OQVU
   BVEJP 0VUQVU
   UPLFO தؒ૚Ͱ$5$ϩεΛܭࢉ͠ਖ਼ଇԽ γϯϓϧ͕ͩͱͯ΋ޮՌత 
6. ઌߦݚڀɿ*OUFSNFEJBUF
   $5$
   <-FF
   > &ODPEFS O &ODPEFS O &ODPEFS  -JOFBS
   
   4PGUNBY $5$ $5$ -JOFBS
   
   4PGUNBY &ODPEFS / *OQVU
   BVEJP 0VUQVU
   UPLFO  தؒ૚Ͱ$5$ϩεΛܭࢉ͠ਖ਼ଇԽ γϯϓϧ͕ͩͱͯ΋ޮՌత தؒ૚Ͱͷೝࣝ݁ՌΛ ΋ͬͱ༗ޮར༻Ͱ͖ΔͷͰ͸ʁ
7. 4FMGDPOEJUJPOFE
   $5$
   </P[BLJ
   > &ODPEFS O &ODPEFS O &ODPEFS  -JOFBS
   
   4PGUNBY -JOFBS $5$ $5$ -JOFBS
   
   4PGUNBY &ODPEFS / *OQVU
   BVEJP 0VUQVU
   UPLFO 4FMGDPOEJUJPOJOH  தؒ૚ͷೝࣝ݁ՌΛޙஈʹϑΟʔυόοΫ ʹೝࣝ݁ՌΛߟྀ͠ͳ͕ΒͷΤϯίʔυΛՄೳʹ ʹจࣈؒͷؔ܎ੑΛֶ΂Δʂʂ
8. ࣮ݧ݁Ռɿࣗݾճؼܕͱඖఢ͢ΔੑೳΛୡ੒  ࣗݾճؼܕʹൺ΂ͯʜ  ഒߴ଎  ಉ౳ͷ8&3
9. ग़ྗ݁Ռͷྫ • ૚ΛܦΔ͝ͱʹೝࣝ݁Ռ͕վળ͞ΕͯΏ͘ 
10. ଞͷඇࣗݾճؼܕํࣜͱͷൺֱ • ࠷৽ख๏ͷதͰτοϓੑೳ <)JHVDIJ
    "436>  ଎౓ɺਫ਼౓ͱ΋ʹ τοϓͷੑೳ
11. ଞͷඇࣗݾճؼܕํࣜͱͷൺֱ • ࠷৽ख๏ͷதͰτοϓੑೳ <)JHVDIJ
    "436>  தؒ૚ʹ͓͚Δ༧ଌͱޙஈ΁ͷϑΟʔυόοΫ͢Δ 4FMG$POEJUJPOJOH͸ۃΊͯ༗ޮʹ࡞༻͍ͯ͠Δ
12. ଞͷඇࣗݾճؼܕํࣜͱͷൺֱ • ࠷৽ख๏ͷதͰτοϓੑೳ <)JHVDIJ
    "436>  ଞͷඇࣗݾճؼํࣜͱͷؔ࿈͸ʁ
13. ଞͷඇࣗݾճؼܕԻ੠ೝࣝͱͷؔ࿈ • $5$ग़ྗʹର͢ΔඇࣗݾճؼσίʔμΛ༻͍ͨ܁Γฦ͠վળ  &"
    $IJ
    FU
    BM
    "MJHO3FGJOF
    /PO"VUPSFHSFTTJWF
    4QFFDI
    3FDPHOJUJPO
    WJB
    *UFSBUJWF
    3FBMJHONFOU :
    )JHVDIJ
    FUBM
    .BTL
    $5$
    /PO"VUPSFHSFTTJWF
    &OEUP&OE
    "43
    XJUI
    $5$
    BOE
    .BTL
    1SFEJDU .BTL
    $5$ "MJHO3FGJOF ʢ܁Γฦ͠ʣ ྑ͘͢Δ ݁ՌΛʜ ྑ͘͢Δ ݁ՌΛʜ ྑ͘͢Δ ݁ՌΛʜ
14. Self-conditioned CTCΛ΋͏Ұ౓ݟͯΈΔ  CTC Input Target label Encoder Layer Encoder Layer ⋮ Encoder Layer Encoder Layer ⋮ Encoder Layer Encoder Layer ⋮ CTC CTC Output Output Output (intermediate) (intermediate) Linear!→# + Softmax Linear!→# + Softmax Linear#→! Linear!→# + Softmax Linear#→! ܁Γฦ͠ํࣜͱಉ౳ͷॲཧΛ ಺෦Ͱߦ͍ͬͯΔʁ ʢதؒʣ݁ՌΛʜ ྑ͘͢Δ ʢதؒʣ݁ՌΛʜ ྑ͘͢Δ
15. ൃలܗ΁ͷ͍͔ͭ͘ͷΞΠσΞ  CTC Input Target label Encoder Layer Encoder Layer ⋮ Encoder Layer Encoder Layer ⋮ Encoder Layer Encoder Layer ⋮ CTC CTC Output Output Output (intermediate) (intermediate) Linear!→# + Softmax Linear!→# + Softmax Linear#→! Linear!→# + Softmax Linear#→! ʢதؒʣ݁ՌΛʜ ྑ͘͢Δ ʢதؒʣ݁ՌΛʜ ྑ͘͢Δ ᶃ தؒೝࣝ݁ՌʹΘ͟ͱޡΓΛ෇༩ ʮ܁Γฦͯ͠ྑ͘͢ΔʯΛΑΓ͏·ֶ͘श ˠ *OUFS"VH</BLBHPNF
    *OUFSTQFFDI> ᶄ தؒ૚͝ͱʹ׽ࣈͱಡΈΛަޓʹֶश ׽ࣈͱಡΈͷ૬ޓؔ܎Λֶश ˠ "MUFSOBUF
    $POEJUJPOJOH
    <'VKJUB
    4-5> ᶅ ܁Γฦ͠ߏ଄ʹண໨ ಉҰͷ໾ׂΛ࣋ͭ෦෼Λ·ͱΊͯܰྔԽ ˠ 'PMEFE &ODPEFS
    <,PNBUTV
    *$"441>
16. ᶃ*OUFS"VH </BLBHPNF
    JOUFSTQFFDI>  CTC Input Target label Encoder Layer Encoder Layer ⋮ Encoder Layer Encoder Layer ⋮ Encoder Layer Encoder Layer ⋮ CTC CTC Output Output Output (intermediate) (intermediate) Linear!→# + Softmax Linear!→# + Softmax Linear#→! Linear!→# + Softmax Linear#→! தؒ૚༧ଌʹରͯ͠Θ͟ͱޡΓΛ෇༩ தؒ૚$5$ͷ܁Γฦ͠ʹΑΓվળ ԿΛʮྑ͘͢Δʯ͔Λ໌ࣔతʹֶश 𝐈𝐧𝐭𝐞𝐫𝐀𝐮𝐠 𝐈𝐧𝐭𝐞𝐫𝐀𝐮𝐠
17. ᶃ*OUFS"VH </BLBHPNF
    JOUFSTQFFDI> • 4FMGDPOEJUJPOFE
    $5$Λ্ճΔೝࣝਫ਼౓Λୡ੒ • ॴ๬ͷޡΓʹର͢Δؤ݈ੑΛ֫ಘ 
18. ᶄ "MUFSOBUF
    $POEJUJPOJOH<'VKJUB
    4-5> • ಉԻҟࣈ΍׽ࣈ͕࣋ͭෳ਺ͷൃԻ͸೔ຊޠԻ੠ೝࣝͷ՝୊ • දهͱൃԻͷ૬ޓ࡞༻Λߟྀͨ͠ϚϧνλεΫֶश  :
    'VKJUB
    FUBM
    "MUFSOBUF
    *OUFSNFEJBUF
    $POEJUJPOJOH
    XJUI
    4ZMMBCMFMFWFM
    BOE
    $IBSBDUFSMFWFM
    5BSHFUT
    GPS
    +BQBOFTF
    "43 දه΁ Իઅ͔Β දه΁ Իઅ͔Β Իઅ΁ දه͔Β
19. ᶄ "MUFSOBUF
    $POEJUJPOJOH<'VKJUB
    4-5> • $4+ͰධՁɺදهͱൃԻͷਫ਼౓͕ಉ࣌ʹ޲্͢Δ͜ͱΛ֬ೝ 
20. ᶅ 'PMEFE
    &ODPEFS
    <,PNBUTV
    *$"441>  CTC Input Target label Encoder Layer Encoder Layer ⋮ Encoder Layer Encoder Layer ⋮ Encoder Layer Encoder Layer ⋮ CTC CTC Output Output Output (intermediate) (intermediate) ֤ઢܗ૚͸ڞ௨ͷύϥϝʔλ 
    ೖग़ྗ͸ಉҰͷ্ۭؒʹࣹӨ Linear!→# + Softmax Linear!→# + Softmax Linear#→! Linear!→# + Softmax Linear#→! JOQVU ྨࣅͷೖग़ྗؔ܎ PVUQVU JOQVU PVUQVU
21.  Linear!→# + Softmax Linear!→# + Softmax Linear#→! CTC Input Target label Encoder Layer Encoder Layer ⋮ Encoder Layer Encoder Layer ⋮ Encoder Layer Encoder Layer ⋮ Linear!→# + Softmax Linear#→! CTC CTC Output Output Output (intermediate) (intermediate) 4IBSFE 1BSBNFUFST ಉҰͷ໾ׂΛ࣋ͭΤϯίʔμϒϩοΫΛ ύϥϝʔλڞ௨ԽʹܰྔԽ ܁Γฦ͠ར༻ 'PMEFE &ODPEFS #BTF &ODPEFS ᶅ 'PMEFE
    &ODPEFS
    <,PNBUTV
    *$"441>
22.  ᶅ 'PMEFE
    &ODPEFS
    <,PNBUTV
    *$"441> • 4FMGDPOEJUJPOFE
    $5$ͷੑೳΛอͪͳ͕Βύϥϝʔλ࡟ݮ
23. ͦͷଞͷൃలܗ • ͳͥੑೳ޲্͢Δ͔ΛఆࣜԽɺ ਪ࿦࣌ʹΑΓΑ͍தؒ༧ଌͰ$POEJUJPOJOH • #FUUFS
    JOUFSNFEJBUFT
    <,PNBUTV
    *OUFSTQFFDI> • ֶश࣌ʹ௥Ճͷதؒ૚ਖ਼ଇԽ • *OUFS%FDPEFS <,PNBUTV
    4-5> • ࿩ऀμΠΞϥΠθʔγϣϯ΁ͷԠ༻ • 4FMG$POEJUJPOFE
    /PO"VUPSFHSFTTJWF
    "UUSBDUPS
    <'VKJUB
    *$"441> 
24. #FUUFS
    *OUFSNFEJBUFT <,PNBUTV
    *OUFSTQFFDI>
    4FBSDIFE
    JOUFSNFEJBUF
    DPOEJUJPOJOH &ODPEFS O &ODPEFS O &ODPEFS  -JOFBS
    
    4PGUNBY &NCFEEJOH "MJHONFOU 7JUFSCJ #FBN TFBSDI $5$ $5$ &YUFSOFM -. -JOFBS
    
    4PGUNBY &ODPEFS / *OQVU
    BVEJP 0VUQVU
    UPLFO $POGPSNFS
    $5$ 4FMGDPOEJUJPOJOH ᶃ 'SBNFXJTF UPLFO
    QSPCBCJMJUZ ᶄ
    &TUJNBUF
    UPLFO
    TFRVFODF ᶅ "MJHO
    FTUJNBUFE
    UPLFOT UP
    GSBNFXJTF
    QSPCBCJMJUZ  *NQSPWF
    JOUFSNFEJBUF
    QSFEJDUJPO
    CZ
    BO
    FYUFSOBM
    -.
    BOE
    CFBN
    TFBSDI (𝑇×𝑉) (𝑇×𝐷) (𝑇×𝐷) (𝑇×1) (𝐿×1)
25. #FUUFS
    *OUFSNFEJBUFT <,PNBUTV
    *OUFSTQFFDI>
    .VMUJQBTT
    DPOEJUJPOJOH &ODPEFS O &ODPEFS O &ODPEFS  -JOFBS
    
    4PGUNBY &NCFEEJOH "MJHONFOU 7JUFSCJ $5$ -JOFBS
    
    4PGUNBY &ODPEFS / *OQVU
    BVEJP OE QBTT
    PVUQVU $POGPSNFS
    $5$ 4FMGDPOEJUJPOJOH &ODPEFS O &ODPEFS O &ODPEFS  $5$ -JOFBS
    
    4PGUNBY &ODPEFS / *OQVU
    BVEJP TU QBTT
    PVUQVU $POGPSNFS
    $5$ 4FMG $POEJUJPOJOH &ODP &ODPE &ODP $5 -JOFBS
    &ODP *OQVU
    SE QBTT $POGPSNFS
    $5$ ᶄ "MJHO
    PVUQVUT
    PG QSFWJPVT
    JOGFSFODF ᶃ 'SBNFXJTF UPLFO
    QSPCBCJMJUZ  (𝑇×𝑉) (𝑇×𝐷) (𝑇×𝐷) (𝐿×1)
26. *OUFS%FDPEFS <,PNBUTV
    4-5> • ࣗݾճؼσίʔμΛʢֶश࣌ͷΈʣதؒ૚ͷਖ਼ଇԽͱͯ͠ར༻  )ZCSJE
    $5$"UUFOUJPOͷߏ଄Λதؒ૚΁ ˠதؒ૚ͷਫ਼౓޲্ TFMGDPOEJUJPOJOHͱ߹Θͤͯ૬৐ޮՌ
27. ࿩ऀμΠΞϥΠθʔγϣϯ΁ͷԠ༻ • /FVSBM
    %JBSJ[BUJPO XJUI
    /POBVUPSFHSFTTJWF
    *OUFSNFEJBUF
    "UUSBDUPST <'VKJUB
    *$"441>  𝑋 TransEnc! 𝐴 𝐸" Sigmoid(𝐴#𝐸" ) 𝑌 TransEnc" … Audio sequence Attractors Embeddings Speaker labels LSTM$%& LSTM'$& Autoregressive 𝑋 TransEnc( Audio sequence BEFORE Autoregressive attractor 𝐸( 𝑊𝐴( 𝐸( TransEnc()! + 𝐴( = Attn(𝑄, 𝐸( , 𝐸( ) Sigmoid(𝐴( #𝐸( ) 𝑌( AFTER Non-autoregressive intermediate attractors intermediate prediction conditioning Speaker-wise
28. ·ͱΊ • 4FMG$POEJUJPOFE
    $5$ʹ͍ͭͯ঺հ • தؒ૚ʹ͓͚Δ༧ଌͱޙஈ΁ͷϑΟʔυόοΫߏ଄͸ۃΊͯ༗ޮ • छʑͷൃలܗʹ͍ͭͯ঺հ • *OUFS"VHɿதؒ૚ʹର͠ޡΓΛ෇༩ֶ͠श • "MUFSOBUF
    $POEJUJPOJOHɿදهԻૉؒͷؔ܎ੑΛֶश • 'PMEFE
    &ODPEFSɿ܁Γฦ͠ߏ଄Λ௿ύϥϝʔλԽ • #FUUFS
    *OUFSNFEJBUFɿΑΓΑ͍தؒ༧ଌʹΑΓ$POEJUJPOJOH • *OUFS%FDPEFSɿதؒ૚ʹର͢Δ௥Ճਖ਼ଇԽ • 4FMG$POEJUJPOFE
    %JBSJ[BUJPOɿ࿩ऀμΠΞϥΠͥʔγϣϯ΁ͷԠ༻ • ࠓճ঺հͰ͖ͳ͔ͬͨൃలܗ΋ • )JFSBSDIJDBM
    $POEJUJPOJOH
    <)JHVDIJ
    *$"441>ɿཻ౓Λม͑֊૚తʹ$POEJUJPOJOH • (BUFE
    *OUFSMBZFS
    $PMMBCPSBUJPO
    <*$"441>ɿ(BUFߏ଄ΛՃ͑ͨ$POEJUJPOJOH