Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Graph Neural Networks概観

Sponsored · Ship Features Fearlessly Turn features on and off without deploys. Used by thousands of Ruby developers.

Graph Neural Networks概観

Graph Neural Networks (GNNs)についてまとめました
実際にどんなことに使えるかイメージを持ってもらうことを意識しました。
内容は以下の通りです。

・グラフ構造とは
・GNNsの発展
・いくつかのGNNsの特徴
・NLPにおける利用例
・その他の領域における利用例

勉強会好評でやってよかったです!
フィードバックはTwitter (roy29fuku)にお願いします。

追記:
なるべくリンク埋め込んで簡単に参照できるようにしたので、ダウンロードしておつかいただくと便利です

Avatar for Ryota Yamada

Ryota Yamada

June 01, 2019
Tweet

More Decks by Ryota Yamada

Other Decks in Technology

Transcript

  1. # ਪ঑ϓϩτίϧ 1. 8िྸΦεͷϚ΢εAΛ12ඖ༻ҙ 2. 5×105ͷࡉ๔BΛi.v. 3-1. ౤༩܈6ඖʹༀࡎYΛ10μg/kg i.p. 3-2.

    ରর܈6ඖʹsalineΛi.p. # ධՁ ମԹɾ݂ѹ ࣬ױXʹༀࡎY͕ ޮ͔͘ௐ΂͍ͨ ࣄۀ֓ཁ 3 3 ৽͍͠ݚڀΞΠσΟΞʹରͯ͠ɺ աڈͷ࿦จ͔Βऔಘ࣮ͨ͠ݧ৚݅Λ΋ͱʹ࣮ݧϓϩτίϧΛఏҊ ௐࠪ࣌ؒ΍࣮ݧͷ΍Γ௚͠Λ࡟ݮ
  2. ख๏ 4 4 named entity recognition, relation extraction, coreference resolutionΛ༻͍ͯ

    ࿦จ͔Βϓϩτίϧ΍݁Ռʹؔ͢ΔσʔλΛநग़ɺߏ଄Խ σʔλΛάϥϑߏ଄ʹஔ͖׵͑Δ͜ͱͰϓϩτίϧͷൺֱ΍ੜ੒͕Ͱ͖Δ
  3. ख๏ 5 5 named entity recognition, relation extraction, coreference resolutionΛ༻͍ͯ

    ࿦จ͔Βϓϩτίϧ΍݁Ռʹؔ͢ΔσʔλΛநग़ɺߏ଄Խ σʔλΛάϥϑߏ଄ʹஔ͖׵͑Δ͜ͱͰϓϩτίϧͷൺֱ΍ੜ੒͕Ͱ͖Δ ɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹ͔Ͳ͏͔֬ೝ͢ΔͨΊʹGNNsษڧ࢝Ί·ͨ͠
  4. ख๏ 6 https://speakerdeck.com/ioki/development-of-mrr-at-cookpadtechconf2019 named entity recognition, relation extraction, coreference resolutionΛ༻͍ͯ

    ࿦จ͔Βϓϩτίϧ΍݁Ռʹؔ͢ΔσʔλΛநग़ɺߏ଄Խ σʔλΛάϥϑߏ଄ʹஔ͖׵͑Δ͜ͱͰϓϩτίϧͷൺֱ΍ੜ੒͕Ͱ͖Δ ɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹ͔Ͳ͏͔֬ೝ͢ΔͨΊʹGNNsษڧ࢝Ί·ͨ͠ Luan et al., 2018 # ઌߦݚڀ Պֶ࿦จ͔Βknowledge graph ϨγϐΛػցՄಡʹ
  5. ख๏ 7 named entity recognition, relation extraction, coreference resolutionΛ༻͍ͯ ࿦จ͔Βϓϩτίϧ΍݁Ռʹؔ͢ΔσʔλΛநग़ɺߏ଄Խ

    σʔλΛάϥϑߏ଄ʹஔ͖׵͑Δ͜ͱͰϓϩτίϧͷൺֱ΍ੜ੒͕Ͱ͖Δ ɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹ͔Ͳ͏͔֬ೝ͢ΔͨΊʹGNNsษڧ࢝Ί·ͨ͠ Luan et al., 2018 # ઌߦݚڀ Պֶ࿦จ͔Βknowledge graph ϨγϐΛػցՄಡʹ ϝϯόʔืूத ڵຯ͋Δํ ࿈བྷ͍ͩ͘͞ʂ Twitter: @roy29fuku https://speakerdeck.com/ioki/development-of-mrr-at-cookpadtechconf2019
  6. Graphͱ͸ 12 12 ӈͷάϥϑ͸͜Μͳײ͡ʹදݱ ɹ௖఺ू߹V = {1, 2, 3, 4}

    ɹลू߹E = {{1, 2}, {1, 3}, {2, 3}, {2, 4}, {3, 4}} ɹάϥϑG = (V, E) ผͷݴ͍ํͰఆٛ͢Δͱ ɹE ⊆ [V]2 Λຬͨ͢ ɹG = (V, E) ͷू߹ͷ૊ ΛάϥϑͱݺͿ 1 2 3 4
  7. GraphͰදݱͰ͖Δ΋ͷ 14 14 ϊʔυ: ਓʢஉੑ, ঁੑ, …ʣ Τοδ: ਓؒؔ܎ʢ༑ਓ, …ʣ

    Structured deep models: Deep learning on graphs and beyond karate club
  8. GraphͰදݱͰ͖Δ΋ͷ 15 15 ϊʔυ: ݪࢠʢC, H, …ʣ Τοδ: ݁߹ʢ୯݁߹, …ʣ

    Structured deep models: Deep learning on graphs and beyond
  9. ne[n]: ϊʔυnͱྡ઀͢Δϊʔυू߹ ln: ϊʔυnͷϥϕϧ xn: ϊʔυnͷঢ়ଶ on: ϊʔυnͷग़ྗ ·ͱΊΔͱ Graph

    Neueral Networks [Gori+ 2005] 25 25 Fw͕contraction mappingʢॖईࣸ૾ʣͳΒ Fw, Gw͸ղΛ࣋ͪɺunique
  10. GNNsͷྺ࢙ᶄ Convolution Structured deep models: Deep learning on graphs and

    beyond CNNͷ৞ΈࠐΈͷ֓೦Λಋೖͯ͠ܭࢉޮ཰↑ 26 26
  11. Graph Convolutional Netoworks [Duvenaud+ 2015, Li+ 2016, Scichtkrull+ 2017] 28

    28 CNNͷ৞ΈࠐΈͷ֓೦ΛGraphʹద༻ άϥϑߏ଄͸ඇఆܕʢϊʔυͷ਺ɺ֤ϊʔυͷྡ઀ϊʔυ਺ʣ →৞ΈࠐΈͷૢ࡞Λద༻͢ΔͨΊʹ޻෉͕ඞཁ Wu et al., 2019 Euclidean non-Euclidean
  12. Graph Convolutional Netoworks [Duvenaud+ 2015, Li+ 2016, Scichtkrull+ 2017] 29

    29 CNNͷ৞ΈࠐΈͷ֓೦ΛGraphʹద༻ ΤοδΛॏΈͱͯ͠ѻͬͨ Structured deep models: Deep learning on graphs and beyond
  13. Relational Graph Convolutional Netoworks [Scichtkrull+ 2017] 30 30 relationཁૉΛ௥Ճ graph:

    G = (V, E, R) nodes: vi ∈ V edges: (vi, r, vj) ∈ E relation type: r ∈ R Schlichtkrull et al., 2017
  14. Relational Graph Convolutional Netoworks [Scichtkrull+ 2017] 31 31 hi(l): l૚໨ͷϊʔυvi

    ͷhidden state Ni r: ϊʔυvi ʹରͯؔ͠܎rͰྡ઀͢Δϊʔυू߹ ci,r: ྡ઀͢Δϊʔυͷ਺Λਖ਼نԽɺex. |Nir| ͜ΕΛ֤ϊʔυ͝ͱʹฒྻͰܭࢉɺ࣮ફతʹ͸ܭ ࢉͷޮ཰ͷͨΊʹsparse matrixΛ༻͍Δ ૚ΛॏͶΔ͜ͱͰɺෳ਺ͷϦϨʔγϣϯΛ·͙ͨ ϊʔυಉ࢜ͷؔ܎Λಋ͘ͷʹ໾ཱͭ ͋Δϊʔυʢ੺ʣʹண໨ͨ࣌͠ͷߋ৽ ֤ؔ܎ੑͷೖग़ྗʹؔΘΔྡ઀ϊʔυɾࣗݾϧʔ ϓ΋ܭࢉͯͦ͠ΕͧΕ྘ͷߦྻΛऔಘɺ concatɺReLUͰ෼ྨ Schlichtkrull et al., 2017
  15. Graph Attention Networks [Monti+ 2017, Hoshen 2017, Veličković+ 2018] 34

    34 Ͳͷϊʔυͱͷྡ઀Λॏࢹ͢Δ͔attentionͰදݱͨ͠ GCNͰaij(1/cij)͸ྡ઀ϊʔυ਺ʹґଘ͍ͯͨ͠ Graph Attention NetworksͰ͸ΑΓॏཁͳ΋ͷΛॏΈ෇͚Δ Wu et al., 2019 GCN Graph Attention Networks
  16. GNNsͷྺ࢙ᶆ άϥϑͷੜ੒ Structured deep models: Deep learning on graphs and

    beyond Version 1: Generate graph (or predict new links) between known entities Version 2: Generate graphs from scratch (single embedding vector)
  17. GraphͰղ͖͍ͨ՝୊ “ݹయత*”ͳ΋ͷ 36 36 Structured deep models: Deep learning on

    graphs and beyond ׬ᘳͰ͸ͳ͍σʔλͷܽଛ෦෼Λิ׬ͨ͠Γɺ৽͍͠σʔλΛ෼ྨͨ͠Γ *Kipfᐌ͘classical
  18. ϨγϐΛάϥϑʹམͱ͠ࠐΉ ɹϊʔυ: ࡐྉ ɹΤοδ: खॱ Ԡ༻ͱͯ͠ҎԼͷ࣮ݱΛݟਾ͍͑ͯΔ ɾதؒϊʔυͷݕࡧʢᖱΊͨϕʔίϯʣ ɾෳ਺ϨγϐΛϚʔδ ɾ৽͍͠ௐཧ๏ͷ୳ࡧ 2019/02/27ͷൃදͰ෼ྔͷਖ਼نԽ

    ࣮ࣾձͷσʔλΛѻ͏͠ΜͲΈʹγϯύγʔ Machine Readable Recipe (recipe) 41 41 https://speakerdeck.com/ioki/development-of-mrr-at-cookpadtechconf2019
  19. SCIERC (scientific protocol) 42 42 Պֶ࿦จʹରͯ͠entities, relations, coreferencesΛΞϊςʔγϣϯͨ͠σʔλ ηοτΛ࡞੒ طଘͷϞσϧ͕ݸผͷλεΫΛύΠϓϥΠϯ

    తʹܨ͍Ͱͨͷʹର͠ɺmulti-task learning frameworkΛ࣮૷ͨ͠ υϝΠϯݻ༗ͷಛ௃ྔΛ༻͍ͣʹैདྷख๏Λ ্ճΔநग़ਫ਼౓Λ࣮ݱ σʔληοτ ɾ500ຊ෼ͷAbstract ɾentity: 6 types ɾrelation: 7 types Պֶత಺༰ʹಛԽͨ͠৘ใநग़Λ໨ࢦ͢ Luan et al., 2018
  20. PaperRobot (knowledge graph) 43 43 աڈͷ࿦จ͔Β knowldge graph*Λੜ੒ ɹnode: entities/concepts

    ɹedge: relations * άϥϑߏ଄knowledge base ɹʢent1, rel, ent2ʣͷtripletͰදݱ ৽͍͠Պֶతൃݟͱ͸ ৽͍͠node/relationͷൃݟͱଊ͑Δ = incremental work ೖྗ͞ΕͨλΠτϧʹԠͨ֓͡ཁ/݁࿦/ల๬Λࣗಈੜ੒͢Δ ͞Βʹੜ੒͞Εͨ࿦จ͔Β৽͍͠࿦จΛੜ੒͠ɺࣗಈͰແݶʹݚڀΛ͢Δͱ͍͏ߏ૝ 2019/5/20ʹग़ͨϗϠϗϠ࿦จ Wang et al., 2019 SCIERCͷ1st authorͷYi Luan͞Μ͕last author
  21. Learning to Represent Programs with Graphs (source code) 44 44

    ιʔείʔυͷsemanticͳ෼ੳ NLPతͳΞϓϩʔνͰ͸ಉ͡ม਺΍ؔ ਺͕཭Εͨͱ͜ΖͰ࢖ΘΕͨ࣌ͷґଘ ؔ܎ΛղܾͰ͖ͳ͔ͬͨ Gated Graph Neural NetworksΛར༻ Allamanis et al., 2018 VarNaming: จ຺͔Βม਺Λ໋໊ VarMisuse: ม਺ͷޡ༻Λݕ஌
  22. ͓͢͢Ίblogهࣄ 45 45 Graph ConvolutionΛࣗવݴޠॲཧʹԠ༻͢Δʢશ7ճʣ ٕज़ಋೖʹ͓͚ΔԾઆݕূͷϓϩηεΛΞ΢τϓοτɺͱͯ΋ษڧʹͳΔ Part7: ߟ࡯ ɾTransformerͷSelf-AttentionͷΑ͏ʹɺࣗવݴޠॲཧʹ͓͍ͯάϥϑతͳػߏ͕༗༻ͳ৔ ߹͸͋Δɻ͔͠͠ɺͦͷ͜ͱͱGraph

    Convolution͕༗ޮͳ͜ͱ͸౳ՁͰ͸ͳ͍ɻ ɾGraph Convolution͕༗ޮͳͷ͸ɺϊʔυ෼ྨ/άϥϑߏ଄෼ྨͷλεΫʹམͱ͠ࠐΊΔ έʔεɻจ຺৘ใΛ֫ಘ͍ͨ͠ͱ͍͏ϞνϕʔγϣϯͱɺGraph ConvolutionͷಘҙྖҬͱ ͸͋·Γ߹க͠ͳ͍ɻ ɾGraph ConvolutionΛࣗવݴޠॲཧͰ࢖͏ͳΒ͹ɺʮେن໛͔ͭHeterogeneousͳάϥϑ Ͱɺϊʔυ෼ྨͷ໰୊ʹؼணͰ͖Δʯέʔε͕ద͍ͯ͠ΔͱࢥΘΕΔɻ
  23. MoleculeNet (chemical) 47 47 molecular machine learningͷϕϯ νϚʔΫͱͳΔσʔληοτ ྔࢠྗֶɾ෺ཧԽֶɾੜ෺෺ཧֶɾ ੜཧֶͷ؍఺͔ΒूΊΒΕͨ

    700,000ͷԽ߹෺σʔλͰߏ੒ ෳ਺ͷGNNs͕σϑΥϧτͰ࢖͑Δ drug discoveryϑϨʔϜϫʔΫͷ DeepChem಺ʹͯσʔληοτͱ͠ ͯ࢖ΘΕ͍ͯΔ Wu et al., 2018
  24. 57 /-1 .33 .BDIJOF3FBEBCMF3FDJQF ͷ։ൃ%FWFMPQNFOUPG.33!$PPLQBE5FDI$POG 4$*&3$ 1BQFS3PCPU -FBSOJOHUP3FQSFTFOU1SPHSBNTXJUI(SBQIT (SBQI$POWPMVUJPOͰࣗવݴޠॲཧΛߦ͏ 0UIFS"QQMJDBUJPOT

    .PMFDVMF/FU #SBJO/FU$// ࣮ࡍ(//Tͬͯ࢖͑Δͷʁ )PX1PXFSGVMBSF(SBQI/FVSBM/FUXPSLT  8FJTGFJMFSBOE-FNBO(P/FVSBM)JHIFSPSEFS(SBQI/FVSBM/FUXPSLT 3FWJTJUJOH(SBQI/FVSBM/FUXPSLT"MM8F)BWFJT-PX1BTT'JMUFST (SBQI/FVSBM/FUXPSL3FUSJFWB5&$)#-0( 5XJUUFS!TIJPO@IPOEB IUUQTHJUIVCDPNBS9JW5JNFTBS9JW5JNFTJTTVFT References 2/2
  25. 59 59 ࣍ճҎ߱ͷൃදऀΛืू͍ͯ͠·͢ʂ ɾGraph theoryͷਂ۷Γ ɾGNNsͷαʔϕΠ, ࣮૷ ɾϑϨʔϜϫʔΫ, ϥΠϒϥϦ, σʔληοτͷ঺հ

    ɾ෼ੳ݁Ռͷใࠂ ɾͦͷଞͳΜͰ΋ ख఻͍͚ͬͯͨͩΔํ͸SlackͳΓTwitterͳΓͰ ࢁాʢroy29fukuʣ·Ͱ࿈བྷ͍ͩ͘͞ʂ ཁ๬͕͋Ε͹࣍ճҎ߱ଓ͖·͢