Read: "DeepType 2: Superhuman Entity Linking, All You Need Is Type Interactions"

AAAI-22読み会で下記の論文を読みました。

Jonathan Raiman, "DeepType 2: Superhuman Entity Linking, All You Need Is Type Interactions", AAAI-22

Transcript

1. UCPhrase: Unsupervised Context-aware Quality Phrase Tagging DeepType 2: Superhuman Entity Linking, All You Need Is Type Interactions 
 Jonathan Raiman (NVIDIA) 2022-04-06 Reader: Tatsuya Shirakawa LINE Developer Meetup: AAAI-22 ࿦จಡΈձ
2. 2 Beatrust – vision ୭΋͕࠷ߴͷࣗ෼Λ ࣮ݱͰ͖ΔੈքΛͭ͘Δ We are hiring! Machine Learning Engineer | ػցֶशΤϯδχΞ Data Scientist | σʔλαΠΤϯςΟετ Analytics Engineer | ΞφϦςΟΫεΤϯδχΞ https://speakerdeck.com/beatrust/we-are-hiring
3. 3 Tatsuya Shirakawa @s_tat1204 Machine Learning Lead at Beatrust 
 ← Researcher at ABEJA ← Researcher at NTT-Data Mathematical Systems Inc. Selected Posts • ϩʔϯνલͷ Tag Suggestion ػೳΛػցֶशͰ࡞Δ • ೔ຊޠࣙॻ͋ΓΩʔϫʔυநग़ث͔Βͷଟݴޠࣙॻͳ͠Ωʔ ϫʔυநग़ثͷ Distillation • ػ͸ख़ͨ͠ʂάϥϑߏ଄ʹର͢ΔDeep LearningɺGraph Convolutionͷ͝঺հ • ҟۭؒ΁ͷຒΊࠐΈʂPoincare Embeddings͕୓͘දݱֶशͷ ৽ల։ • Retail Face Analysis Inside-Out ML Data Science NLP Mathematical Optimization Interview https://note.com/beatrust/n/ne63297b9c546 CV
4. 4 Task – Entity Linking จষதͷϑϨʔζʢϝϯγϣϯɺطఆʣ͕ɺφϨοδϕʔεதͷͲͷΤϯςΟςΟΛࢦ͢ͷ͔ಉఆ͢Δλ εΫɻදهͷΏΒ͗΍ޠͷจ຺தͰͷҙຯͷಛఆ͕ඞཁɻ https://qiita.com/izuna385/items/9d658620b9b96b0b4ec9
5. 5 Alias TableΛ΋͍ͪͨޮ཰తͳEntity Linking https://qiita.com/izuna385/items/9d658620b9b96b0b4ec9 Λ΋ͱʹ࡞੒ Step 0. Alias TableΛԿΒ͔ͷํ๏Ͱ࡞͓ͬͯ͘ʢWikipediaͷ಺෦ϦϯΫΛऩूͨ͠Γʣ Step 1. จதͷϝϯγϣϯʹର͠ɺީิͱͳΔΤϯςΟςΟͷϦετΛAlias Table͔Βऔಘ Step 2. ީิΛϥϯΩϯά͠ɺτοϓީิΛϝϯγϣϯͷ༧ଌΤϯςΟςΟͱͯ͠ग़ྗ Alias Table
6. 6 SoTA Using Pretrained Language Models
7. 7 SoTA Using Pretrained Language Models
8. 8 SoTA Using Pretrained Language Models
9. 9 Paper Summary DeepTypeΛվળͨ͠Entity Linkingͷ৽ख๏DeepType 2ΛఏҊ 1. ਓؒͷύϑΥʔϚϯεͱͷൺֱΛߦ͏ͨΊɺ໖ີͳΞϊςʔ γϣϯΛߦͬͨσʔληοτΛ࡞੒ɾެ։ɻͦͷ݁Ռɺ SoTA͸ਓؒͷύϑΥʔϚϯεʹୡ͍ͯ͠ͳ͍͜ͱΛ 
 ໌Β͔ʹͨ͠ɻ 
 2. Typeʢϝϯγϣϯ/ΤϯςΟςΟͷΧςΰϦ৘ใͳͲʣͷਪ ఆΛجૅͱͨ͠DeepTypeΛൃలͤͨ͞৽ख๏DeepType 2Λ ఏҊɻֶशࡁΈݴޠϞσϧΛ͔ͭΘͣʹɺॳΊͯਓؒΛ྇ ͙ਫ਼౓Λୡ੒ɻ Single authorͳͷʹ͔ͳΓؤு͍ͬͯΔײͷ͋Δ࿦จͩͬͨɻ
10. 10 DeepType 2ͷશମ૾
11. 11 DeepType 2ͷશମ૾ ೖྗςΩετ͔ΒͷBi-LSTMϕʔεͷ 
 ϝϯγϣϯͷಛ௃நग़ ᶃ Type neighborhoods ᶄ Latent type interactions ᶅ Discrete type interactions LSTMʹΑΔஞ࣍༧ଌ ݁ ߹ Alias Tableதͷ֤ΤϯςΟςΟʹର͢Δಛ௃நग़ 
 ᶃ Type neighborhoods … KB্Ͱͷؔ࿈৘ใ༝དྷͷಛ௃நग़ 
 ᶄ Latent type interactions … LSTMͷঢ়ଶͱType neighborhood༝དྷͷಛ௃நग़ 
 ᶅ Discrete type interactions … ϧʔϧϕʔεʢطఆͷΫΤϦʔʹର͢Δ݁Ռʣ༝དྷͷಛ௃நग़
12. 12 Type KB಺ͰEntityʹରͯ͠ఆٛ͞Ε͏Δଐੑ/ؔ܎ੑ৘ใʢຊ࿦จͰ͸બͼํʹ͍ͭͯ͸ݴٴ͞Εͯͳ͍ʣ 
 DeepType 2͸ɺType৘ใΛॎԣʹ࢖ͬͨޮ཰తͳEntity Linkingख๏ɻ Raiman+, “DeepType: Multilingual Entity Linking by Neural Type System Evolution.”, 2018
13. 13 Type Neighborhoods Instance Neighborhood Type৘ใΛ΋ͱʹͭ͘ΒΕͨಛ௃ϕΫλʔɻ ؔ࿈ΤϯςΟςΟ৘ใ΍ɺҢ౓ܦ౓ɺ஀ੜ೔ɺwikipedia্ͰͷstatsͳͲɻ
14. 14 Type Interactions – Latent Type Interactions ஫໨͍ͯ͠ΔϝϯγϣϯͰͷDecoder LSTMͷঢ়ଶͱType neighborhoods͔Β࡞ΒΕΔಛ௃ϕΫλʔ
15. 15 Type Interactions – Discrete Type Interactions طఆͷΫΤϦʔ܈ʢYes/Noʣʹ͍ͨ͢Δ࣭໰݁Ռͱͯ͠ಘΒΕΔόΠφϦʔ஋ͷಛ௃ϕΫλʔ - ಛఆͷؔ܎ੑͱඥ෇͍ͨΤϯςΟςΟ͔ʁ - ௚લͷΤϯςΟςΟ༧ଌͱಉ͡਌ΤϯςΟςΟʹଐ͢Δ͔ʁ 
 …
16. 16 ਓؒͷύϑΥʔϚϯεͷଌఆ Amazon Mechanical TurkͰ༷ʑͳ޻෉Λ͠ɺߴ͍Ұக౓ͷਖ਼֬ͳΞϊςʔγϣϯΛಘͨɻ 1. AMT's Master ͷশ߸Λ͍࣋ͬͯΔਓ͚ͩʹݶఆͨ͠ 2. ਖ਼ղͨ͠ΒϘʔφεΛ෷ͬͨ 3. આ໌ΛಡΜ͔ͩɺςετηοτͰ࠷௿ݶͷਫ਼౓͕ग़͍ͯΔ͔Ͱ଍੾Γͨ͠ 4. ֤mentionʢϑϨʔζʣʹ3ਓׂ౰ɺ࠷΋Ұகͨ͠΋ͷΛ౴͑ͱͨ͠ʢOracleʣ
17. 17 ֶशσʔλ Wikipediaʢ಺෦ϦϯΫΛmention/entityͱͯ͠༻͍ͨʣ + AIDA’s train splitΛֶशσʔλʹ༻͍ͨɻ 
 Wikipediaͷ֤߲໨ͷઆ໌จͰ͸ɺͦͷ߲໨Λද͢ϑϨʔζʹϦϯΫ͕͍͍ͭͯͳ͍ͷͰɺػցతʹϦϯ ΫΛ௥Ճͯ͠ΞϊςʔγϣϯྔΛ͔͞૿ͨ͠͠ʢDensificationʣ ಺෦ϦϯΫʢ੨จࣈʣ͕ϝϯγϣϯɺϦϯΫઌͷ߲໨͕ΤϯςΟςΟɻ 
 ͜ͷྫͰ΋ɺจதͷAda Lovelaceʹ͸ϦϯΫ͕෇͍͍ͯͳ͍ʢˠDensificationͰ͔͞૿͠ʣ
18. 18 SoTA vs Human vs DeepType 2 on TAC/AIDA طଘͷSoTA͸ਓؒͷਫ਼౓Λ௒͍͑ͯͳ͔͕ͬͨɺDeep Type 2͸ਓؒΛ྇կɻ
19. 19 SoTA vs DeepType 2 on Other datasets ଞͷσʔληοτͰ΋SoTA௒͑
20. 20 Ablation Study – Entity Vectorͷߏ੒๏ͷൺֱ طଘSoTAख๏Ͱ࢖ΘΕ͍ͯΔಛ௃ϕΫλʢunique entity vectorɺී௨ͷembedding?ʣʹtype interactionsΛՃ͑Δͱେ෯ʹਫ਼౓ ޲্ɻ·ͨɺunique entity vectorͷ͔ΘΓʹtype neighborhoodsΛ࢖͏ͱ͞Βʹਫ਼౓޲্ʢύϥϝʔλ਺΋type neighborhoods ͸unique entity vectorͷ1/6Ͱ͢Ήʣɻ
21. 21 Ablation Study – DensificationͷޮՌ WikipediaͷDensificationʢ߲໨ͷઆ໌จதʹݱΕΔͦͷ߲໨ࣗମΛද͢ϑϨʔζʹɺ߲໨΁ͷϦϯΫ Λ෇༩͢ΔʣʹΑΓਫ਼౓޲্͕Ͱ͖Δɻ
22. 22 ؒҧ͑ํͷ܏޲෼ੳ ਓؒͱDeepType 2Ͱ͸͓͓ΉͶಉ͡܏޲ͷؒҧ͑ํΛ͢Δɻ
23. 23 ·ͱΊ - Wikidata/Wikipediaͷ৘ใΛϑϧ׆༻ͯ͠Entity LinkingΛߦ͏DeepType 2ΛఏҊ - ҆௚ʹPretrained Language ModelϞσϧΛ࢖Θͳͯ͘΋ɺType৘ใΛ͏·͘࢖͏͜ͱͰਫ਼౓Α͘ Entity Linking͕Ͱ͖Δ͜ͱ͕Α͘Θ͔ͬͨ - ਓؒͷਫ਼౓Λਪఆ͢ΔͨΊʹAMTͰਫ਼៛ͳΞϊςʔγϣϯΛಘΔͨΊͷ਺ʑͷ޻෉Λߦ͓ͬͯΓɺ ΞϊςʔγϣϯͷઃܭΛ͢ΔࡍͷࢀߟʹͳΓͦ͏ 
 ؾʹͳͬͨͱ͜Ζ 
 - ద੾ͳTypeͷબͼํ͸೉ͦ͠͏ 
 - Wikidata/Wikipediaʹด͡ͳ͍EntityΛѻ͓͏ͱͨ͠ͱ͖ʹͲ͏ͳΔͷ͔ 
 - Pretrained modelΛ࢖༻ͨ͠ͱ͖ʹਫ਼౓޲্͕͋Δͷ͔ 

24. 24 Beatrust – vision ୭΋͕࠷ߴͷࣗ෼Λ ࣮ݱͰ͖ΔੈքΛͭ͘Δ We are hiring! Machine Learning Engineer | ػցֶशΤϯδχΞ Data Scientist | σʔλαΠΤϯςΟετ Analytics Engineer | ΞφϦςΟΫεΤϯδχΞ https://speakerdeck.com/beatrust/we-are-hiring Thanks!