Pro Yearly is on sale from $80 to $50! »

A Structural Probe for Finding Syntax in Word Representations

8fa31051503b09846584c49cd53d2f80?s=47 Asei Sugiyama
September 03, 2019
39

A Structural Probe for Finding Syntax in Word Representations

NLP・IR paper reading in 20 minutes #8 https://nagatacho-pymoku.connpass.com/event/144477/ の発表用資料です

8fa31051503b09846584c49cd53d2f80?s=128

Asei Sugiyama

September 03, 2019
Tweet

Transcript

  1. A Structural Probe for Finding Syntax in Word Representations NLPɾIR

    paper reading in 20 minutes #8
  2. ࣗݾ঺հ • ਿࢁ Ѩ੟ • Software Engineer @Repro • ػցֶशͱ͔౷ܭͱ͔։ൃͱ͔

    • ػցֶशਤؑ ڞஶ • গ͠લ·Ͱ໊ݹ԰ʹ͍·ͨ͠
  3. Abstract • Stanford େֶͷ࿦จ • ୯ޠදݱʹ͍ͭͯ͸ղੳ͕ਐΜͰ͖͍ͯΔ͕ɺߏจ໦ͷදݱ͕ ֶश͞Ε͍ͯΔ͔ʹ͍ͭͯ͸͜Ε·Ͱ͔֬ΊΒΕ͍ͯͳ͍ • ຊݚڀͰ͸ structual

    probe ͱ͍͏ख๏ΛఏҊ͢Δ • ͜Ε͸neural networkͷ୯ޠදݱΛઢܗม׵ۭͨؒ͠ʹߏจ ໦͕ຒΊࠐ·Ε͍ͯΔ͔ΛධՁ͢Δ΋ͷͰ͋Δ • ELMo, BERT Ͱ͸ߏจ໦Λֶश͍ͯ͠Δͱࣔࠦ͢Δ݁ՌΛಘͨ
  4. ໨࣍ 1.Introduction <- 2.Methods 3.Experiment & Results 4.Discussion & Conclusion

  5. ݚڀͷ໨త • ਂ૚ϞσϧͰ͸ߏจ໦Λֶश͍ͯ͠Δͷ͔ɺͱ͍͏ٙ໰ʹ౴͑ ͍ͨ ͜ͷ࿦จͰઆ໌͢Δ͜ͱ • ୯ޠදݱ͔Βߏจ໦Λݟ͚ͭΔํ๏ʹ͍ͭͯ • ୯ޠදݱͷ௿࣍ݩ΁ͷࣹӨ͔Βߏจ໦ʹؔ͢Δ৘ใΛ෮ݩ͠ɺ ධՁ͢Δํ๏ͱͦͷ۩ମྫ

    (ELMo, BERT)ʹ͍ͭͯ
  6. ໨࣍ 1.Introduction 2.Methods <- 3.Experiment & Results 4.Discussion & Conclusion

  7. ख๏ͷΞΠσΞ • άϥϑͷϊʔυؒͷڑ཭Λอͬͨ·· ϕΫτϧۭؒʹຒΊࠐΉ͜ͱΛߟ͑Δ • ΋͜͠Ε͕Ͱ͖͍ͯΕ͹ɺ͋Δϊʔυ ͷྡͷϊʔυ Λ୳͢͜ͱ͸ۙ๣ ୳ࡧͱಉ͡ •

    ·ͨɺϞσϧ͕ਖ਼͘͠໦ߏ଄Λֶश͢ Ε͹ɺͦͷදݱۭؒͷҰ෦͚ͩΛར༻ ͢Δ͸ͣ (௿࣍ݩଟ༷ମԾઆ) • දݱۭؒͷ෦෼ۭؒͰɺ໦ߏ଄ͷڑ཭ Λอ͍ͬͯΔΑ͏ͳ΋ͷΛ୳ͤ͹ྑ͍
  8. ͭ·Γ? • ղઆهࣄ1ʹ͋Δਤ͕Θ͔Γ΍͍͢ • ࠨͷۭ͕ؒ୯ޠͷදݱۭؒ • ࠨਤதͷփ৭ͷฏ໘͕໦ߏ଄Λදݱ͠ ͍ͯΔ෦෼ۭؒ • ӈଆ͕෮ݩ͞Εͨ໦ߏ଄

    1 https://nlp.stanford.edu//~johnhew//structural-probe.html
  9. None
  10. The structural probe • : ൪໨ͷจதͷi, j൪໨ͷ୯ޠͱͦͷϕΫτϧ • : ߏจ໦্Ͱͷϊʔυؒڑ཭

    • : ෦෼্ۭؒͰͷڑ ཭
  11. ໨࣍ 1.Introduction 2.Methods 3.Experiment & Results <- 4.Discussion & Conclusion

  12. Experiment • Ϟσϧ: ELMo, BERT(base, large) & ϕʔεϥΠϯϞσϧ • σʔλ:

    Penn Treebank (Standard Dependencies formalism ʹैͬͯλά෇͚) • ධՁࢦඪ: ߏจ໦Λ෮ݩͰ͖ͨ౓߹͍ • Undirected Unlabeled Attachment Score (UUAS) • Spearman correlation of true to predicted distances (DSpr.)
  13. Results (Table 1) • จ຺Λߟྀ͠ͳ͍Ϟσϧ(্4ͭ)ʹର͠ ͯɺจ຺Λߟྀ͢ΔϞσϧ(Լ4ͭ)ͷํ ͕ߏจ໦Λ࠶ݱͰ͖͍ͯΔ2 2 ܎Γड͚ߏ଄ʹ͍ͭͯɺछผ΍ํ޲͸ແࢹͯ͠ධՁ͍ͯ͠Δ

  14. Results (Figure 2)

  15. Results (Figure 4) • ࠨ: ߏจ໦Ͱܭࢉͨ͠୯ޠؒڑ཭ • ӈ: BERT(large) 16૚໨Ͱܭࢉͨ͠

    ୯ޠؒڑ཭ • શମతͳߏ଄Λ࠶ݱͰ͖͍ͯͦ͏
  16. Results (Figure 5) • ॎ࣠: ߏจ໦Λ෮ݩͰ͖ͨ౓߹͍ • ԣ࣠: ࡞੒ͨ͠෦෼ۭؒͷ࣍ݩ •

    ߏจ໦Λ෮ݩ͢ΔͨΊͷ෦෼ۭؒͷ࣍ ݩ͸32࣍ݩఔ౓Ͱανͬͨ
  17. Results (Figure 1) • ॎ࣠: ߏจ໦Λ෮ݩͰ͖ͨ౓߹͍ • ԣ࣠: ར༻ͨ͠ӅΕ૚ͷਂ͞ •

    ߏจ໦͸൒෼Ҏ߱ɺ࠷ऴ૚গ͠લͰֶ श͞Ε͍ͯͦ͏
  18. Results (Figure 3) • ॎ࣠: ୯ޠͷϊϧϜ • ߏจ໦ʹ͓͚Δϊʔυͷਂ͞Λද͢ • ԣ࣠:

    ֤୯ޠͷindex • ϧʔτ͔Βͷਂ͞ΛۙࣅͰ͖͍ͯͦ͏
  19. ໨࣍ 1.Introduction 2.Methods 3.Experiment & Results 4.Discussion & Conclusion <-

  20. Discussion • ࠷ॳʹཱͯͨԾઆ͕ਖ਼͍͠ͱͯ͠ਐΊ͖͍ͯͯΔ͕ɺͦͷԾઆ ͷଥ౰ੑʹ͍ͭͯ͸ผ్ݕূ͕ඞཁ

  21. ิ଍ • Visualizing and Measuring the Geometry of BERT (2019)

    Ͱ৮ΕΒΕ͍ͯΔͷͰɺؔ܎͢Δ෦෼Λཁ໿ • ߏจ໦Λద౰ͳ࣍ݩͷ ্ۭؒʹڑ཭Λอͬͨ··ຒΊࠐΉ ख๏͕ଘࡏ͢Δ • ݁ՌΛओ੒෼෼ੳΛ࢖ͬͯ࣍ݩѹॖ͢ΔͱɺBERT Ͱֶशͯ͠ ͍Δ΋ͷͱࣅͨ݁Ռ͕ಘΒΕΔ
  22. Conclusion • ఏҊख๏Ͱ͋Δ structural probe ʹΑͬͯߏจ໦͕୯ޠ දݱʹຒΊࠐ·Ε͍ͯΔ͜ͱΛࣔࠦ͢Δ݁Ռ͕ಘΒΕͨ • ࣗવݴޠॲཧʹ͓͚Δߏจ໦Ҏ֎ͷάϥϑߏ଄ (রԠղੳͱ

    ͔) ʹ͍ͭͯ͸ࠓޙͷऔΓ૊Έ
  23. ౴͑Δ΂͖࣭໰ ࣭໰ ճ౴ 1. What did authors try to accomplish?

    ਂ૚Ϟσϧ͕ߏจ໦Λֶश͍ͯ͠Δ͔֬ೝ͢Δ ख๏ͷथཱ 2. What were the key elements of the approach? structural probe 3. What can you use yourself? https://github.com/john-hewitt/ structural-probes 4. What other references do you want to follow? Visualizing and Measuring the Geometry of BERT (2019)
  24. Reference • A Structural Probe for Finding Syntax in Word

    Representations: https://nlp.stanford.edu/pubs/ hewitt2019structural.pdf • john-hewitt/structural-probes: https:// github.com/john-hewitt/structural-probes • Finding Syntax with Structural Probes · John Hewitt: https://nlp.stanford.edu//~johnhew// structural-probe.html