Upgrade to Pro — share decks privately, control downloads, hide ads and more …

A Structural Probe for Finding Syntax in Word R...

Asei Sugiyama
September 03, 2019
140

A Structural Probe for Finding Syntax in Word Representations

NLP・IR paper reading in 20 minutes #8 https://nagatacho-pymoku.connpass.com/event/144477/ の発表用資料です

Asei Sugiyama

September 03, 2019
Tweet

Transcript

  1. Abstract • Stanford େֶͷ࿦จ • ୯ޠදݱʹ͍ͭͯ͸ղੳ͕ਐΜͰ͖͍ͯΔ͕ɺߏจ໦ͷදݱ͕ ֶश͞Ε͍ͯΔ͔ʹ͍ͭͯ͸͜Ε·Ͱ͔֬ΊΒΕ͍ͯͳ͍ • ຊݚڀͰ͸ structual

    probe ͱ͍͏ख๏ΛఏҊ͢Δ • ͜Ε͸neural networkͷ୯ޠදݱΛઢܗม׵ۭͨؒ͠ʹߏจ ໦͕ຒΊࠐ·Ε͍ͯΔ͔ΛධՁ͢Δ΋ͷͰ͋Δ • ELMo, BERT Ͱ͸ߏจ໦Λֶश͍ͯ͠Δͱࣔࠦ͢Δ݁ՌΛಘͨ
  2. ख๏ͷΞΠσΞ • άϥϑͷϊʔυؒͷڑ཭Λอͬͨ·· ϕΫτϧۭؒʹຒΊࠐΉ͜ͱΛߟ͑Δ • ΋͜͠Ε͕Ͱ͖͍ͯΕ͹ɺ͋Δϊʔυ ͷྡͷϊʔυ Λ୳͢͜ͱ͸ۙ๣ ୳ࡧͱಉ͡ •

    ·ͨɺϞσϧ͕ਖ਼͘͠໦ߏ଄Λֶश͢ Ε͹ɺͦͷදݱۭؒͷҰ෦͚ͩΛར༻ ͢Δ͸ͣ (௿࣍ݩଟ༷ମԾઆ) • දݱۭؒͷ෦෼ۭؒͰɺ໦ߏ଄ͷڑ཭ Λอ͍ͬͯΔΑ͏ͳ΋ͷΛ୳ͤ͹ྑ͍
  3. Experiment • Ϟσϧ: ELMo, BERT(base, large) & ϕʔεϥΠϯϞσϧ • σʔλ:

    Penn Treebank (Standard Dependencies formalism ʹैͬͯλά෇͚) • ධՁࢦඪ: ߏจ໦Λ෮ݩͰ͖ͨ౓߹͍ • Undirected Unlabeled Attachment Score (UUAS) • Spearman correlation of true to predicted distances (DSpr.)
  4. Results (Figure 5) • ॎ࣠: ߏจ໦Λ෮ݩͰ͖ͨ౓߹͍ • ԣ࣠: ࡞੒ͨ͠෦෼ۭؒͷ࣍ݩ •

    ߏจ໦Λ෮ݩ͢ΔͨΊͷ෦෼ۭؒͷ࣍ ݩ͸32࣍ݩఔ౓Ͱανͬͨ
  5. ิ଍ • Visualizing and Measuring the Geometry of BERT (2019)

    Ͱ৮ΕΒΕ͍ͯΔͷͰɺؔ܎͢Δ෦෼Λཁ໿ • ߏจ໦Λద౰ͳ࣍ݩͷ ্ۭؒʹڑ཭Λอͬͨ··ຒΊࠐΉ ख๏͕ଘࡏ͢Δ • ݁ՌΛओ੒෼෼ੳΛ࢖ͬͯ࣍ݩѹॖ͢ΔͱɺBERT Ͱֶशͯ͠ ͍Δ΋ͷͱࣅͨ݁Ռ͕ಘΒΕΔ
  6. ౴͑Δ΂͖࣭໰ ࣭໰ ճ౴ 1. What did authors try to accomplish?

    ਂ૚Ϟσϧ͕ߏจ໦Λֶश͍ͯ͠Δ͔֬ೝ͢Δ ख๏ͷथཱ 2. What were the key elements of the approach? structural probe 3. What can you use yourself? https://github.com/john-hewitt/ structural-probes 4. What other references do you want to follow? Visualizing and Measuring the Geometry of BERT (2019)
  7. Reference • A Structural Probe for Finding Syntax in Word

    Representations: https://nlp.stanford.edu/pubs/ hewitt2019structural.pdf • john-hewitt/structural-probes: https:// github.com/john-hewitt/structural-probes • Finding Syntax with Structural Probes · John Hewitt: https://nlp.stanford.edu//~johnhew// structural-probe.html