࣍
1.Context & related works <-
2.Geometry of syntax
3.Geometry of word senses
• Measurement of word sense disambiguation
capability
• Embedding distance and context: a
concatenation experiment
4.Conclusion
Slide 5
Slide 5 text
Context & related works
• A Structural Probe for Finding Syntax in Word
Representations (2019) ͷΞϯαʔʹͳ͍ͬͯΔ
• ͜ͷจൈ͖ʹ΄ͱΜͲԿΘ͔Βͳ͍ߏ
Slide 6
Slide 6 text
!
Slide 7
Slide 7 text
2 ഒಡΊΔ͓ಘͳจ
Slide 8
Slide 8 text
A Structural Probe
for Finding
Syntax in Word
Representations
NN จΛࡘʹञΛҿΉձ #9
࣍
1.Context & related works
2.Geometry of syntax <-
3.Geometry of word senses
• Measurement of word sense disambiguation
capability
• Embedding distance and context: a
concatenation experiment
4.Conclusion
Slide 22
Slide 22 text
Geometry of syntax
• BERT ͷֶश݁Ռʹ͍ͭͯɺ࣍ͷ 2 ͭͷ؍͔Βߦͬͨ
1.ͦͦʹཱͭදݱΛֶशͰ͖͍ͯΔͷ͔
2.ߏจΛֶशͰ͖͍ͯΔͷ͔
Mathematics of embedding trees
in Euclidean space
• ϊʔυ͔ΒͳΔ ʹڑ
(తͳͷ)Λอͬͨ··ຒΊࠐΊΔ͜
ͱֶ͕తʹূ໌Ͱ͖ͨ
• ·ͨɺڑͦͷͷΛ༻͍ͯ͠·͏
ͱɺڑΛอͭຒΊࠐΈ͕Ͱ͖ͳ͍Α
͏ͳ߹͕͋Δ͜ͱࣔ͞Εͨ
• ͜ΕʹΑΓ͕॓ղܾͨ͠ͱ͍ͯ͠Δ
Slide 25
Slide 25 text
ͭ·Γ?
• blog هࣄͰৄ͘͠ղઆ͞Ε͍ͯΔͷ
Ͱɺৄࡉ͕ؾʹͳͬͨΒ͔͜͜ΒೖΔ
ͷ͕͓͢͢Ί
• https://pair-code.github.io/
interpretability/bert-tree/
Slide 26
Slide 26 text
Visualization of parse tree embeddings
• ߏจͷڑΛอͭΑ͏ͳຒΊࠐΈͱ BERT ͱͷ݁Ռ͕ྨࣅ
࣍
1.Context & related works
2.Geometry of syntax
3.Geometry of word senses <-
• Measurement of word sense disambiguation
capability
• Embedding distance and context: a
concatenation experiment
4.Conclusion
Slide 31
Slide 31 text
Geometry of word senses
• ߏจ͚ͩͰͳ͘୯ޠͷҙຯΛଊ͑ΒΕ͍ͯΔ͔ݕ౼
• ҙຯΛද͢෦ۭ͕ؒಘΒΕͳ͍͔࣮ݧ
• Ͳ͏ΒಘΒΕͨ
!
• จ຺ΛਓతʹௐઅͰ͖ͳ͍͔࣮ݧ
• Ͱ͖ͳ͔ͬͨͲ͜Ζ͔ѱԽͨ͠
Slide 32
Slide 32 text
Measurement of word sense
disambiguation capability
• BERT ͷग़ྗΛ UMAP ͰՄࢹԽ
• ಉ͡ "die" ʹରͯ͠ෳͷҙຯΛ
ͭΫϥελ͕Ͱ͖͍ͯΔ
• kNN ΛͬͯޠٛᐆດੑղফλεΫΛ
ߦͬͨ݁Ռ accuracy 71.1%
(SOTA)
Embedding distance and context:
a concatenation experiment
• จ຺Λҙਤతʹૢ࡞͢Δ͜ͱͰྑ͍݁
ՌΛಘΒΕͳ͍͔࣮ݧ
• ಛఆͷҙຯ͋Δ୯ޠΛ༻͍͍ͯΔද
తͳจΛݟ͚ͭग़͠ɺಉ͡ҙຯͰಉ͡
୯ޠΛ༻͍͍ͯΔจʹ࿈݁ͨ͠
• "I went to Edo" ͕දతͳจ
ͳ߹ɺ"He went to Edo"ʹ
͚ͯ͠"He went to Edo and I
went to Edo" ͱ͍͏จΛ࡞Δ
Slide 36
Slide 36 text
Embedding distance and context:
a concatenation experiment
• ԣ࣠: BERT ͷϨΠϠʔ
• ॎ࣠: ҙຯͷҧ͏Ϋϥελͷத৺ͱͷ
ڑͷൺతͳͷ (େ͖͍΄ͲΑ͍)
• දతͳจΛ͚Ճ͑ͨ߹ɺͦͷ୯
ޠͷҙຯΛΑΓΑ͘Ͱ͖Δ͔ͱ
ࢥͬͨΒͦΜͳ͜ͱͳ͔ͬͨ
Slide 37
Slide 37 text
࣍
1.Context & related works
2.Geometry of syntax
3.Geometry of word senses
• Measurement of word sense disambiguation
capability
• Embedding distance and context: a
concatenation experiment
4.Conclusion <-