A Structural Probe for Finding Syntax in Word Representations NLPɾIR paper reading in 20 minutes #8

ࣗݾ঺հ • ਿࢁ Ѩ੟ • Software Engineer @Repro • ػցֶशͱ͔౷ܭͱ͔։ൃͱ͔ • ػցֶशਤؑ ڞஶ • গ͠લ·Ͱ໊ݹ԰ʹ͍·ͨ͠

Abstract • Stanford େֶͷ࿦จ • ୯ޠදݱʹ͍ͭͯ͸ղੳ͕ਐΜͰ͖͍ͯΔ͕ɺߏจ໦ͷදݱ͕ ֶश͞Ε͍ͯΔ͔ʹ͍ͭͯ͸͜Ε·Ͱ͔֬ΊΒΕ͍ͯͳ͍ • ຊݚڀͰ͸ structual probe ͱ͍͏ख๏ΛఏҊ͢Δ • ͜Ε͸neural networkͷ୯ޠදݱΛઢܗม׵ۭͨؒ͠ʹߏจ ໦͕ຒΊࠐ·Ε͍ͯΔ͔ΛධՁ͢Δ΋ͷͰ͋Δ • ELMo, BERT Ͱ͸ߏจ໦Λֶश͍ͯ͠Δͱࣔࠦ͢Δ݁ՌΛಘͨ

໨࣍ 1.Introduction <- 2.Methods 3.Experiment & Results 4.Discussion & Conclusion

ݚڀͷ໨త • ਂ૚ϞσϧͰ͸ߏจ໦Λֶश͍ͯ͠Δͷ͔ɺͱ͍͏ٙ໰ʹ౴͑ ͍ͨ ͜ͷ࿦จͰઆ໌͢Δ͜ͱ • ୯ޠදݱ͔Βߏจ໦Λݟ͚ͭΔํ๏ʹ͍ͭͯ • ୯ޠදݱͷ௿࣍ݩ΁ͷࣹӨ͔Βߏจ໦ʹؔ͢Δ৘ใΛ෮ݩ͠ɺ ධՁ͢Δํ๏ͱͦͷ۩ମྫ (ELMo, BERT)ʹ͍ͭͯ

໨࣍ 1.Introduction 2.Methods <- 3.Experiment & Results 4.Discussion & Conclusion

ख๏ͷΞΠσΞ • άϥϑͷϊʔυؒͷڑ཭Λอͬͨ·· ϕΫτϧۭؒʹຒΊࠐΉ͜ͱΛߟ͑Δ • ΋͜͠Ε͕Ͱ͖͍ͯΕ͹ɺ͋Δϊʔυ ͷྡͷϊʔυ Λ୳͢͜ͱ͸ۙ๣ ୳ࡧͱಉ͡ • ·ͨɺϞσϧ͕ਖ਼͘͠໦ߏ଄Λֶश͢ Ε͹ɺͦͷදݱۭؒͷҰ෦͚ͩΛར༻ ͢Δ͸ͣ (௿࣍ݩଟ༷ମԾઆ) • දݱۭؒͷ෦෼ۭؒͰɺ໦ߏ଄ͷڑ཭ Λอ͍ͬͯΔΑ͏ͳ΋ͷΛ୳ͤ͹ྑ͍

ͭ·Γ? • ղઆهࣄ1ʹ͋Δਤ͕Θ͔Γ΍͍͢ • ࠨͷۭ͕ؒ୯ޠͷදݱۭؒ • ࠨਤதͷփ৭ͷฏ໘͕໦ߏ଄Λදݱ͠ ͍ͯΔ෦෼ۭؒ • ӈଆ͕෮ݩ͞Εͨ໦ߏ଄ 1 https://nlp.stanford.edu//~johnhew//structural-probe.html

No content

The structural probe • : ൪໨ͷจதͷi, j൪໨ͷ୯ޠͱͦͷϕΫτϧ • : ߏจ໦্Ͱͷϊʔυؒڑ཭ • : ෦෼্ۭؒͰͷڑ ཭

໨࣍ 1.Introduction 2.Methods 3.Experiment & Results <- 4.Discussion & Conclusion

Experiment • Ϟσϧ: ELMo, BERT(base, large) & ϕʔεϥΠϯϞσϧ • σʔλ: Penn Treebank (Standard Dependencies formalism ʹैͬͯλά෇͚) • ධՁࢦඪ: ߏจ໦Λ෮ݩͰ͖ͨ౓߹͍ • Undirected Unlabeled Attachment Score (UUAS) • Spearman correlation of true to predicted distances (DSpr.)

Results (Table 1) • จ຺Λߟྀ͠ͳ͍Ϟσϧ(্4ͭ)ʹର͠ ͯɺจ຺Λߟྀ͢ΔϞσϧ(Լ4ͭ)ͷํ ͕ߏจ໦Λ࠶ݱͰ͖͍ͯΔ2 2 ܎Γड͚ߏ଄ʹ͍ͭͯɺछผ΍ํ޲͸ແࢹͯ͠ධՁ͍ͯ͠Δ

Results (Figure 2)

Results (Figure 4) • ࠨ: ߏจ໦Ͱܭࢉͨ͠୯ޠؒڑ཭ • ӈ: BERT(large) 16૚໨Ͱܭࢉͨ͠ ୯ޠؒڑ཭ • શମతͳߏ଄Λ࠶ݱͰ͖͍ͯͦ͏

Results (Figure 5) • ॎ࣠: ߏจ໦Λ෮ݩͰ͖ͨ౓߹͍ • ԣ࣠: ࡞੒ͨ͠෦෼ۭؒͷ࣍ݩ • ߏจ໦Λ෮ݩ͢ΔͨΊͷ෦෼ۭؒͷ࣍ ݩ͸32࣍ݩఔ౓Ͱανͬͨ

Results (Figure 1) • ॎ࣠: ߏจ໦Λ෮ݩͰ͖ͨ౓߹͍ • ԣ࣠: ར༻ͨ͠ӅΕ૚ͷਂ͞ • ߏจ໦͸൒෼Ҏ߱ɺ࠷ऴ૚গ͠લͰֶ श͞Ε͍ͯͦ͏

Results (Figure 3) • ॎ࣠: ୯ޠͷϊϧϜ • ߏจ໦ʹ͓͚Δϊʔυͷਂ͞Λද͢ • ԣ࣠: ֤୯ޠͷindex • ϧʔτ͔Βͷਂ͞ΛۙࣅͰ͖͍ͯͦ͏

໨࣍ 1.Introduction 2.Methods 3.Experiment & Results 4.Discussion & Conclusion <-

Discussion • ࠷ॳʹཱͯͨԾઆ͕ਖ਼͍͠ͱͯ͠ਐΊ͖͍ͯͯΔ͕ɺͦͷԾઆ ͷଥ౰ੑʹ͍ͭͯ͸ผ్ݕূ͕ඞཁ

ิ଍ • Visualizing and Measuring the Geometry of BERT (2019) Ͱ৮ΕΒΕ͍ͯΔͷͰɺؔ܎͢Δ෦෼Λཁ໿ • ߏจ໦Λద౰ͳ࣍ݩͷ ্ۭؒʹڑ཭Λอͬͨ··ຒΊࠐΉ ख๏͕ଘࡏ͢Δ • ݁ՌΛओ੒෼෼ੳΛ࢖ͬͯ࣍ݩѹॖ͢ΔͱɺBERT Ͱֶशͯ͠ ͍Δ΋ͷͱࣅͨ݁Ռ͕ಘΒΕΔ

Conclusion • ఏҊख๏Ͱ͋Δ structural probe ʹΑͬͯߏจ໦͕୯ޠ දݱʹຒΊࠐ·Ε͍ͯΔ͜ͱΛࣔࠦ͢Δ݁Ռ͕ಘΒΕͨ • ࣗવݴޠॲཧʹ͓͚Δߏจ໦Ҏ֎ͷάϥϑߏ଄ (রԠղੳͱ ͔) ʹ͍ͭͯ͸ࠓޙͷऔΓ૊Έ

౴͑Δ΂͖࣭໰ ࣭໰ ճ౴ 1. What did authors try to accomplish? ਂ૚Ϟσϧ͕ߏจ໦Λֶश͍ͯ͠Δ͔֬ೝ͢Δ ख๏ͷथཱ 2. What were the key elements of the approach? structural probe 3. What can you use yourself? https://github.com/john-hewitt/ structural-probes 4. What other references do you want to follow? Visualizing and Measuring the Geometry of BERT (2019)

Reference • A Structural Probe for Finding Syntax in Word Representations: https://nlp.stanford.edu/pubs/ hewitt2019structural.pdf • john-hewitt/structural-probes: https:// github.com/john-hewitt/structural-probes • Finding Syntax with Structural Probes · John Hewitt: https://nlp.stanford.edu//~johnhew// structural-probe.html