A Structural Probe for Finding Syntax in Word Representations

NLP・IR paper reading in 20 minutes #8 https://nagatacho-pymoku.connpass.com/event/144477/ の発表用資料です

Transcript

1. A Structural Probe
   for Finding
   Syntax in Word
   Representations
   NLPɾIR paper reading in 20 minutes #8
   

   View Slide

2. ࣗݾ঺հ
   • ਿࢁ Ѩ੟
   • Software Engineer @Repro
   • ػցֶशͱ͔౷ܭͱ͔։ൃͱ͔
   • ػցֶशਤؑ ڞஶ
   • গ͠લ·Ͱ໊ݹ԰ʹ͍·ͨ͠
   

   View Slide

3. Abstract
   • Stanford େֶͷ࿦จ
   • ୯ޠදݱʹ͍ͭͯ͸ղੳ͕ਐΜͰ͖͍ͯΔ͕ɺߏจ໦ͷදݱ͕
   ֶश͞Ε͍ͯΔ͔ʹ͍ͭͯ͸͜Ε·Ͱ͔֬ΊΒΕ͍ͯͳ͍
   • ຊݚڀͰ͸ structual probe ͱ͍͏ख๏ΛఏҊ͢Δ
   • ͜Ε͸neural networkͷ୯ޠදݱΛઢܗม׵ۭͨؒ͠ʹߏจ
   ໦͕ຒΊࠐ·Ε͍ͯΔ͔ΛධՁ͢Δ΋ͷͰ͋Δ
   • ELMo, BERT Ͱ͸ߏจ໦Λֶश͍ͯ͠Δͱࣔࠦ͢Δ݁ՌΛಘͨ
   

   View Slide

4. ໨࣍
   1.Introduction <-
   2.Methods
   3.Experiment & Results
   4.Discussion & Conclusion
   

   View Slide

5. ݚڀͷ໨త
   • ਂ૚ϞσϧͰ͸ߏจ໦Λֶश͍ͯ͠Δͷ͔ɺͱ͍͏ٙ໰ʹ౴͑
   ͍ͨ
   ͜ͷ࿦จͰઆ໌͢Δ͜ͱ
   • ୯ޠදݱ͔Βߏจ໦Λݟ͚ͭΔํ๏ʹ͍ͭͯ
   • ୯ޠදݱͷ௿࣍ݩ΁ͷࣹӨ͔Βߏจ໦ʹؔ͢Δ৘ใΛ෮ݩ͠ɺ
   ධՁ͢Δํ๏ͱͦͷ۩ମྫ (ELMo, BERT)ʹ͍ͭͯ
   

   View Slide

6. ໨࣍
   1.Introduction
   2.Methods <-
   3.Experiment & Results
   4.Discussion & Conclusion
   

   View Slide

7. ख๏ͷΞΠσΞ
   • άϥϑͷϊʔυؒͷڑ཭Λอͬͨ··
   ϕΫτϧۭؒʹຒΊࠐΉ͜ͱΛߟ͑Δ
   • ΋͜͠Ε͕Ͱ͖͍ͯΕ͹ɺ͋Δϊʔυ
   ͷྡͷϊʔυ Λ୳͢͜ͱ͸ۙ๣
   ୳ࡧͱಉ͡
   • ·ͨɺϞσϧ͕ਖ਼͘͠໦ߏ଄Λֶश͢
   Ε͹ɺͦͷදݱۭؒͷҰ෦͚ͩΛར༻
   ͢Δ͸ͣ (௿࣍ݩଟ༷ମԾઆ)
   • දݱۭؒͷ෦෼ۭؒͰɺ໦ߏ଄ͷڑ཭
   Λอ͍ͬͯΔΑ͏ͳ΋ͷΛ୳ͤ͹ྑ͍
   

   View Slide

8. ͭ·Γ?
   • ղઆهࣄ1ʹ͋Δਤ͕Θ͔Γ΍͍͢
   • ࠨͷۭ͕ؒ୯ޠͷදݱۭؒ
   • ࠨਤதͷփ৭ͷฏ໘͕໦ߏ଄Λදݱ͠
   ͍ͯΔ෦෼ۭؒ
   • ӈଆ͕෮ݩ͞Εͨ໦ߏ଄
   1 https://nlp.stanford.edu//~johnhew//structural-probe.html
   

   View Slide

9. View Slide

10. The structural probe
    • : ൪໨ͷจதͷi, j൪໨ͷ୯ޠͱͦͷϕΫτϧ
    • : ߏจ໦্Ͱͷϊʔυؒڑ཭
    • : ෦෼্ۭؒͰͷڑ
    ཭
    

    View Slide

11. ໨࣍
    1.Introduction
    2.Methods
    3.Experiment & Results <-
    4.Discussion & Conclusion
    

    View Slide

12. Experiment
    • Ϟσϧ: ELMo, BERT(base,
    large) & ϕʔεϥΠϯϞσϧ
    • σʔλ: Penn Treebank
    (Standard Dependencies
    formalism ʹैͬͯλά෇͚)
    • ධՁࢦඪ: ߏจ໦Λ෮ݩͰ͖ͨ౓߹͍
    • Undirected Unlabeled
    Attachment Score (UUAS)
    • Spearman correlation of
    true to predicted distances
    (DSpr.)
    

    View Slide

13. Results (Table 1)
    • จ຺Λߟྀ͠ͳ͍Ϟσϧ(্4ͭ)ʹର͠
    ͯɺจ຺Λߟྀ͢ΔϞσϧ(Լ4ͭ)ͷํ
    ͕ߏจ໦Λ࠶ݱͰ͖͍ͯΔ2
    2 ܎Γड͚ߏ଄ʹ͍ͭͯɺछผ΍ํ޲͸ແࢹͯ͠ධՁ͍ͯ͠Δ
    

    View Slide

14. Results (Figure 2)
    

    View Slide

15. Results (Figure 4)
    • ࠨ: ߏจ໦Ͱܭࢉͨ͠୯ޠؒڑ཭
    • ӈ: BERT(large) 16૚໨Ͱܭࢉͨ͠
    ୯ޠؒڑ཭
    • શମతͳߏ଄Λ࠶ݱͰ͖͍ͯͦ͏
    

    View Slide

16. Results (Figure 5)
    • ॎ࣠: ߏจ໦Λ෮ݩͰ͖ͨ౓߹͍
    • ԣ࣠: ࡞੒ͨ͠෦෼ۭؒͷ࣍ݩ
    • ߏจ໦Λ෮ݩ͢ΔͨΊͷ෦෼ۭؒͷ࣍
    ݩ͸32࣍ݩఔ౓Ͱανͬͨ
    

    View Slide

17. Results (Figure 1)
    • ॎ࣠: ߏจ໦Λ෮ݩͰ͖ͨ౓߹͍
    • ԣ࣠: ར༻ͨ͠ӅΕ૚ͷਂ͞
    • ߏจ໦͸൒෼Ҏ߱ɺ࠷ऴ૚গ͠લͰֶ
    श͞Ε͍ͯͦ͏
    

    View Slide

18. Results (Figure 3)
    • ॎ࣠: ୯ޠͷϊϧϜ
    • ߏจ໦ʹ͓͚Δϊʔυͷਂ͞Λද͢
    • ԣ࣠: ֤୯ޠͷindex
    • ϧʔτ͔Βͷਂ͞ΛۙࣅͰ͖͍ͯͦ͏
    

    View Slide

19. ໨࣍
    1.Introduction
    2.Methods
    3.Experiment & Results
    4.Discussion & Conclusion <-
    

    View Slide

20. Discussion
    • ࠷ॳʹཱͯͨԾઆ͕ਖ਼͍͠ͱͯ͠ਐΊ͖͍ͯͯΔ͕ɺͦͷԾઆ
    ͷଥ౰ੑʹ͍ͭͯ͸ผ్ݕূ͕ඞཁ
    

    View Slide

21. ิ଍
    • Visualizing and Measuring the Geometry of BERT
    (2019) Ͱ৮ΕΒΕ͍ͯΔͷͰɺؔ܎͢Δ෦෼Λཁ໿
    • ߏจ໦Λద౰ͳ࣍ݩͷ ্ۭؒʹڑ཭Λอͬͨ··ຒΊࠐΉ
    ख๏͕ଘࡏ͢Δ
    • ݁ՌΛओ੒෼෼ੳΛ࢖ͬͯ࣍ݩѹॖ͢ΔͱɺBERT Ͱֶशͯ͠
    ͍Δ΋ͷͱࣅͨ݁Ռ͕ಘΒΕΔ
    

    View Slide

22. Conclusion
    • ఏҊख๏Ͱ͋Δ structural probe ʹΑͬͯߏจ໦͕୯ޠ
    දݱʹຒΊࠐ·Ε͍ͯΔ͜ͱΛࣔࠦ͢Δ݁Ռ͕ಘΒΕͨ
    • ࣗવݴޠॲཧʹ͓͚Δߏจ໦Ҏ֎ͷάϥϑߏ଄ (রԠղੳͱ
    ͔) ʹ͍ͭͯ͸ࠓޙͷऔΓ૊Έ
    

    View Slide

23. ౴͑Δ΂͖࣭໰
    ࣭໰ ճ౴
    1. What did authors try to
    accomplish?
    ਂ૚Ϟσϧ͕ߏจ໦Λֶश͍ͯ͠Δ͔֬ೝ͢Δ
    ख๏ͷथཱ
    2. What were the key elements of
    the approach?
    structural probe
    3. What can you use yourself? https://github.com/john-hewitt/
    structural-probes
    4. What other references do you
    want to follow?
    Visualizing and Measuring the
    Geometry of BERT (2019)
    

    View Slide

24. Reference
    • A Structural Probe for Finding Syntax in Word
    Representations: https://nlp.stanford.edu/pubs/
    hewitt2019structural.pdf
    • john-hewitt/structural-probes: https://
    github.com/john-hewitt/structural-probes
    • Finding Syntax with Structural Probes · John
    Hewitt: https://nlp.stanford.edu//~johnhew//
    structural-probe.html
    

    View Slide