Graph Convolutional Matrix Completion

Slide 1

Slide 1 text

Graph Convolutional Matrix Completion Tomoki Tanimura Nakazawa Lab, Keio University, B3

Slide 2

Slide 2 text

Bibliography • Title: Graph Convolutional Matrix Completion • Accepted: KDD DeepLearning Day(2018) • Author: R. vd Berg, T. N. Kipf, M. Welling  ʢAMLab, University of Amsterdamʣ • Max Wellingઌੜͷݚڀࣨ(Kingma͞Μ΋͜͜ʣ • ཧ࿦෺ཧग़਎ऀ͕ଟ͍ɽڧͦ͏ɽɽɽ • Graph Convolution͕ྲྀߦΔ͖͔͚ͬΛ࡞ͬͨϥϘ • ಛʹɼKipf͞Μ͸Graph ConvolutionerʢΠέϝϯ͗͢ΔPh.Dʣ 2 Preliminary 1 Preliminary 2 Method Details Experiment Conclusion

Slide 3

Slide 3 text

Abstract • ϨʔςΟϯάͷߦྻʢϢʔβʔͷΞΠςϜʹର͢ΔධՁ஋͕ ೖͬͨߦྻʣΛ࠶ߏ੒͢ΔϨίϝϯυλεΫ(CF)Λղ͍ͨ • ϨʔςΟϯάߦྻΛ2෦άϥϑͱͯ͠ղऍ͠ɼGraph ConvolutionΛ࢖༻ͨ͠Graph Auto-encoderΛ࢖༻ͯ͠༧ଌ 3 Preliminary 1 Preliminary 2 Method Details Experiment Conclusion

Slide 4

Slide 4 text

Agenda • Preliminary for Recommender System • Preliminary for Graph Convolution • Method Overview • Method Details • Experiment • Conclusion and Future works 4 Preliminary 1 Preliminary 2 Method Details Experiment Conclusion

Slide 5

Slide 5 text

Agenda • Preliminary for Recommender System • Preliminary for Graph Convolution • Method Overview • Method Details • Experiment • Conclusion and Future works 5 Preliminary 1 Preliminary 2 Method Details Experiment Conclusion

Slide 6

Slide 6 text

What is Recommender System? • ϨίϝϯγεςϜ • Recommenders: Tools to help identify worthwhile stuff  (ਪનγεςϜ:ͲΕʹՁ஋͕͋Δ͔Λಛఆ͢ΔͷΛॿ͚Δಓ۩) • ྫʣݱࡏ·ͰͷϢʔβʔͷߪങཤྺΛར༻ͯ͠ɼࠓޙͷߪങ׆ ಈΛ༧ଌ͢Δ • ϨίϝϯυγεςϜ͸େ͖͘෼͚ͯ2ͭ • Content-based • Collaborative Filtering 6 Preliminary 1 Preliminary 2 Method Details Experiment Conclusion

Slide 7

Slide 7 text

Content-based • ΞΠςϜʢϢʔβʔʣʹؔ͢Δ৘ใ͔ΒɼΞΠςϜʢϢʔβʔʣ ͷಛ௃ྔΛઃܭ͠ɼͦͷྨࣅ౓Λܭࢉ  →ྨࣅ౓ͷߴ͍ΞΠςϜΛϢʔβʔʹਪન 7 Πϝʔδ* * https://qiita.com/haminiku/items/f5008a57a870e0188f63 Preliminary 1 Preliminary 2 Method Details Experiment Conclusion

Slide 8

Slide 8 text

Collaborative Filtering • ΞΠςϜͱϢʔβʔͷߪങ׆ಈʹؔ͢Δߦྻʢ֤ཁૉ͸ධՁ ஋΍ߪೖ਺Λද͢ʣ͔ΒϢʔβʔ΍ΞΠςϜͷڞىੑ౳Λࢉ ग़͢Δ 8 Πϝʔδ* * https://qiita.com/haminiku/items/f5008a57a870e0188f63 Preliminary 1 Preliminary 2 Method Details Experiment Conclusion

Slide 9

Slide 9 text

Recent CF Method • ߦྻ෼ղʢMarix Factorizationʣ→༗໊ • ϨʔςΟϯάߦྻΛϢʔβʔͷಛ௃ྔͱΞΠςϜͷಛ௃ྔ ͷߦྻͷੵͰද͢͜ͱͰɼະ؍ଌͷϨʔςΟϯάΛ༧ଌ 9 * https://towardsdatascience.com/paper-summary-matrix-factorization-techniques-for-recommender-systems-82d1a7ace74 Preliminary 1 Preliminary 2 Method Details Experiment Conclusion

Slide 10

Slide 10 text

Matrix Factorization 10 R R I I J J X U V ஍Ҭϝ ογϡ ΰϛͷ࢒ᕓ ؍ଌ͞Εͨ਺ × ~ ~ R R I I J J X U V ஍Ҭϝ ογϡ ΰϛͷ࢒ᕓ ؍ଌ͞Εͨ਺ × ~ ~ %" X UVT R R I I J J X U V ஍Ҭϝ ογϡ ΰϛͷ࢒ᕓ ؍ଌ͞Εͨ਺ × ~ ~ U R R I I J J X U V ஍Ҭϝ ογϡ ΰϛͷ࢒ᕓ ؍ଌ͞Εͨ਺ × ~ ~ V #!$ user item userͷಛ௃ྔ Λද͢ߦྻ itemͷಛ௃ྔ Λද͢ߦྻ ࠶ߏ੒͞Εͨ ϨʔςΟϯάߦྻ ݩͷϨʔςΟϯάߦྻ ʢͨͩ͠؍ଌ͞Εͨ஋ͷΈͷࠩ෼Λ࠷খԽ͢Δʣ Preliminary 1 Preliminary 2 Method Details Experiment Conclusion

Slide 11

Slide 11 text

Interprete Matrix as Graph • ϨʔςΟϯάߦྻ͸ϢʔβʔͱΞΠςϜΛϊʔυͱ͢Δ2෦ά ϥϑͰදͤΔ 11 Preliminary 1 Preliminary 2 Method Details Experiment Conclusion

Slide 12

Slide 12 text

How to use Graph Convolution • άϥϑߏ଄Ͱදͨ͠ϨʔςΟϯάߦྻΛ༻͍ͯɼϢʔβʔͱ ΞΠςϜͷಛ௃ྔΛಘΔ • ϢʔβʔͱΞΠςϜͷಛ௃ྔΛݩʹɼݩͷάϥϑΛ࠶ߏ੒͢ ΔΠϝʔδͰະ؍ଌͷϨʔςΟϯάΛϥϕϧ෇͖ϦϯΫ༧ଌ 12 Preliminary 1 Preliminary 2 Method Details Experiment Conclusion

Slide 13

Slide 13 text

Why use Graph Convolution? • ϊʔυʹؔ͢Δิॿ৘ใʢContent-basedͰ࢖ΘΕ͍ͯΔΑ͏ ͳ৘ใʣΛಛ௃ྔʹؚΊΔ͜ͱ͕Ͱ͖Δ • ϨʔςΟϯάͷؔ܎Ҏ֎ͷ৘ใΛར༻͢Δ͜ͱͰɼCold-start ໰୊ͷղܾʹۙͮ͘ 13 ͜͜ʹิॿ৘ใΛಥͬࠐΊΔ Preliminary 1 Preliminary 2 Method Details Experiment Conclusion

Slide 14

Slide 14 text

Summary(1) • Graph ConvΛ࢖ͬͨϨίϝϯυͷఏҊΛ͍ͯ͠Δ࿦จ • ϨίϝϯυγεςϜ͸ɼConten-basedͱCFͷେ͖͘2ͭ  ʢࠓճ͸ओʹCFʣ • CFͷதͰ΋ߦྻ෼ղ*͸༗໊Ͱࠓճ΋͜Εʹ͍ۙ  ʢ* user-item matrixΛuserͷಛ௃ͱitemͷಛ௃ʹ෼ղ͢Δʣ • ࠓճ͸͜ͷߦྻΛGraphͱͯ͠ଊ͑Δ  Graphߏ଄Λೖྗͱͯ͠userͱitemͷಛ௃ྔΛಘΔ  ิॿ৘ใΛؚΊͯͦΕͧΕͷಛ௃ྔΛಘΒΕΔͷ͕ར఺ 14 Preliminary 1 Preliminary 2 Method Details Experiment Conclusion

Slide 15

Slide 15 text

Agenda • Preliminary for Recommender System • Preliminary for Graph Convolution • Method Overview • Method Details • Experiment • Conclusion and Future works 15 Preliminary 1 Preliminary 2 Method Details Experiment Conclusion

Slide 16

Slide 16 text

What is Graph Convolution? • Graph Convolution • ͦͷ໊ͷ௨ΓGraphΛ৞ΈࠐΜͰɼपลʹ৘ใΛ఻೻ͤ͞ Δܭࢉํ๏ • Graph Convolutionͷछྨ • Graph Fourierม׵Λ༻͍ͨղੳతͳConvolution • ௚઀తͳํ๏  ೖྗͱͳΔಛ௃ྔϕΫτϧͷઢܗ݁߹͔Βͷඇઢܗม׵Ͱ NNͰֶश 16 Preliminary 1 Preliminary 2 Method Details Experiment Conclusion

Slide 17

Slide 17 text

Various Graph Structure 17 * https://tech-blog.abeja.asia/entry/2017/04/27/105613 Preliminary 1 Preliminary 2 Method Details Experiment Conclusion

Slide 18

Slide 18 text

Convolution for Image 18 * https://github.com/vdumoulin/conv_arithmetic Preliminary 1 Preliminary 2 Method Details Experiment Conclusion

Slide 19

Slide 19 text

• Image • ϊʔυ͕άϦουঢ়ʹฒͼɼ઀ଓؔ܎͕Ұఆ • άϥϑߏ଄ͰΑ͘ද͞ΕΔσʔλʢSNS΍ࠓճʣ • ϊʔυͷ઀ଓؔ܎͕ෆఆܗ Difference between Image and Ordinary Graph 19 Preliminary 1 Preliminary 2 Method Details Experiment Conclusion

Slide 20

Slide 20 text

• ۙ๣ϊʔυͷ৘ใΛͨͨΈ͜Ή • Relationͷछྨ͝ͱʹߦ͏  ʢRelation͕RGBͷΠϝʔδʣ  ʢRelation: ධՁ஋౳ʣ • ৔߹ʹΑͬͯ͸ࣗ਎ͷϊʔυ΋  ܭࢉ͢Δ  ʢࠓճͷ࿦จͰ͸͠ͳ͍ʣ Directly Graph Convolution 20 * Modeling Relational Data with Graph Convolutional Networks Preliminary 1 Preliminary 2 Method Details Experiment Conclusion

Slide 21

Slide 21 text

• ࣜͰද͢ͱ͜Μͳײ͡ Directly Graph Convolution 21 * Modeling Relational Data with Graph Convolutional Networks ׆ੑԽؔ਺ ۙ๣ϊʔυ શͯͰ࣮ߦ Weight ࣍ͷӅΕ૚ ͷಛ௃ྔ શͯͷRelation Ͱ࣮ߦ ਖ਼نԽ ۙ๣ϊʔυ ͷಛ௃ྔ ࣗ਎ͷϊʔυ ͷಛ௃ྔ Preliminary 1 Preliminary 2 Method Details Experiment Conclusion

Slide 22

Slide 22 text

Summary(2) • Graph Fourierม׵Λ࢖ͬͨ΋ͷͱ௚઀తͳํ๏͕͋Δɽࠓճ ͸ޙऀ • ը૾΋Graphߏ଄ͱͯ͠ଊ͑Δ͜ͱ͕Ͱ͖Δ  ࠓճͷΑ͏ͳGraphͱҧ͏ͷ͸ϊʔυͷ੔ྻ۩߹ • Graph Convolution͸઀ଓ͞ΕͯΔϊʔυ৘ใΛ఻೻ͤ͞Δ • ෆఆܗͳϊʔυؔ܎ͷGraphߏ଄΋ֶशՄೳʹͨ͠ͷ͕Graph Convolutionͷ͍͢͝ͱ͜Ζ 22 Preliminary 1 Preliminary 2 Method Details Experiment Conclusion

Slide 23

Slide 23 text

Agenda • Preliminary for Recommender System • Preliminary for Graph Convolution • Method Overview • Method Details • Experiment • Conclusion and Future works 23 Preliminary 1 Preliminary 2 Method Details Experiment Conclusion

Slide 24

Slide 24 text

• X: ϊʔυͷಛ௃ྔΛؚΉߦྻ  ʢϊʔυ਺ʣxʢಛ௃࣍ݩ਺ʣ • A: άϥϑߏ଄ʹ͓͚Δྡ઀ߦྻ  ʢϊʔυ਺ʣx ʢϊʔυ਺ʣ • Z: ϊʔυͷຒΊࠐΈಛ௃ྔͷߦྻ  ʢϊʔυ਺ʣxʢຒΊࠐΈͷಛ௃࣍ݩ਺ʣ • A^: ࠶ߏ੒͞Εͨྡ઀ߦྻ Graph Auto-Encoder 24 Encoder Decoder Preliminary 1 Preliminary 2 Method Details Experiment Conclusion

Slide 25

Slide 25 text

Graph for Matrix Completion 25 ←ϊʔυू߹ ←Itemͷϊʔυू߹ ←Userͷϊʔυू߹ ←ϊʔυؒͷΤοδ ←Τοδʹඥͮ͘ϦϨʔγϣϯ  ʢࠓճ͸ධՁ஋ʣ Preliminary 1 Preliminary 2 Method Details Experiment Conclusion

Slide 26

Slide 26 text

Graph Auto-Encoder (Matrix ver) 26 Encoder Decoder ←X͸ϊʔυ৘ใͷߦྻ ←UserͱItemͷಛ௃ྔߦྻ ↖︎ ධՁ஋͝ͱʹྡ઀ߦྻ͕࡞੒͞ΕΔ ↖︎ ࠶ߏ੒͞Εͨྡ઀ߦྻ Preliminary 1 Preliminary 2 Method Details Experiment Conclusion

Slide 27

Slide 27 text

Graph Auto-Encoder (Matrix ver) 27 Preliminary 1 Preliminary 2 Method Details Experiment Conclusion

Slide 28

Slide 28 text

• Graph Convolution Layer => Dense Layer ͷྲྀΕ • Conv Layerͷํ͸࠷ॳʹઆ໌ͨ͠Graph Convͱ΄ͱΜͲಉ͡  ʢҧ͍͸ࣗ਎ͷϊʔυͷܭࢉΛؚΊͳ͍ʣ • Dense Layer͸ઢܗม׵͔Βͷඇઢܗม׵  ʢͨͩ͠weight͸userͱitemͰڞ༗͢Δʁʣ Graph Convolutional Encoder 28 Graph Convolution Layer Dense Layer where Preliminary 1 Preliminary 2 Method Details Experiment Conclusion

Slide 29

Slide 29 text

• userͷಛ௃ྔ x Q (ֶशՄೳͳߦྻ) x itemͷಛ௃ྔ  Λܭࢉ͠ɼ֤ධՁ஋͝ͱʹεΧϥʔ஋Λࢉग़ • u, Q, v͕ύϥϝʔλͷιϑτϚοΫεͰ r (ධՁ஋)ʹؔ͢Δ֬ ཰෼෍ΛԾఆ  r ͷظ଴஋Λܭࢉ͢Δ Bilinear Decoder 29 Preliminary 1 Preliminary 2 Method Details Experiment Conclusion

Slide 30

Slide 30 text

Agenda • Preliminary for Recommender System • Preliminary for Graph Convolution • Method Overview • Method Details • Experiment • Conclusion and Future works 30 Preliminary 1 Preliminary 2 Method Details Experiment Conclusion

Slide 31

Slide 31 text

• r ʹؔ͢Δ֬཰෼෍ʹରͯ͠ɼෛͷର਺໬౓ΛLossͱ͢Δ • ݩʑ؍ଌ͞Ε͍ͯΔධՁ஋ͷΈΛLossʹՃࢉ Loss Function 31 ϚεΫߦྻ ؍ଌ஋ͷi, jͰ iteration r = Mij (؍ଌ஋) ͷ࣌͸1ɼͦΕҎ֎0 Preliminary 1 Preliminary 2 Method Details Experiment Conclusion

Slide 32

Slide 32 text

• ಛఆͷϊʔυͷґଘ͗͢͠ͳ͍ϩόετͳϊʔυಛ௃ྔΛ֫ ಘ͢ΔͨΊ • Ұఆ֬཰ (p dropout)Ͱɼ͋Δϊʔυͷ͔Β఻೻͞ΕΔ৘ใ͕0 ʹͳΔ • Edge Dropoutʢ͋ΔΤοδ͔Β఻೻͞ΕΔ৘ใ͕0ʹͳΔʣΑ Γ΋Node Dropoutͷํ͕ਫ਼౓͕͍͍Β͍͠ Node Dropout 32 Preliminary 1 Preliminary 2 Method Details Experiment Conclusion

Slide 33

Slide 33 text

• େن໛σʔληοτʢRating͕10M݅ͱ͔ʣ͸MemoryʹͷΒ ͳ͍ͷͰɼbatchॲཧΛಋೖ͢Δ • ϝϞϦ࡟ݮ͚ͩͰͳ͘ɼਖ਼ଇԽͱͯ͠΋ػೳ͢Δ • Loss function΁ͷContributionΛجʹɼbatchΛαϯϓϦϯά ͢Δʁ • Full-batch trainingͱਫ਼౓͸΄ͱΜͲมΘΒͳ͍͜ͱ͸࣮ݧͰ ֬ೝͰ͖ͨ • FullͰͰ͖ΔDataset͸ऩଋͷ଎౓Λߟྀͯ͠FullͰ΍ͬͨ Mini-batching 33 Preliminary 1 Preliminary 2 Method Details Experiment Conclusion

Slide 34

Slide 34 text

• ࣮૷͢Δ࣌͸ޮ཰Λߟ͑ͯɼߦྻͷ··ܭࢉͰ͖ΔΑ͏ௐ੔  ʢཁૉ͝ͱʹIteration͍ͯ͠Δͱແݶʹ͕࣌ؒ৯͏ͨΊʣ Vectorized Implementation 34 Preliminary 1 Preliminary 2 Method Details Experiment Conclusion

Slide 35

Slide 35 text

• ೖྗͷಛ௃ߦྻʹ௚઀ίϯςϯπ৘ใΛೖྗ͢Δͱɼ͏·͘ ఻೻Ͱ͖ͳ͍໰୊͕ൃੜ͢Δ  →ίϯςϯπ৘ใ͸ิॿ৘ใͱͯ͠ѻ͏ • ೖྗ͸NodeΛࣝผ͢Δ͚ͩͷone-hot vector • ίϯςϯπ৘ใΛར༻ͨ͠ϊʔυಛ௃ྔΛผʹࢉग़͢Δ Input feature representation 35 Dense Layer ίϯςϯπϕʔε ͷಛ௃ྔ ίϯςϯπ ৘ใͷೖྗ Preliminary 1 Preliminary 2 Method Details Experiment Conclusion

Slide 36

Slide 36 text

• User΍Item͕΋ͭධՁͷ਺͸ɼධՁஈ֊͝ͱʹόϥόϥ  ʢ͋ΔItem͸ධՁ͕ߴ͍ͨΊ5͕ଟ͘ɼ1͸΄ͱΜͲͳ͍౳ʣ  →࠷దԽ͞ΕΔස౓͕ҟͳͬͯ͠·͍ɼաֶशͷՄೳੑ͋Γ • WeightΛධՁஈ֊ؒͰڞ༗͢Δ͜ͱͰόϥϯεΛͱΔ Weight Sharing 36 Ordinal Weight Sharingɼ r ҎԼͷධՁஈ֊ͷॏΈΛ શͯར༻͢Δ WS using Basis Weight Matrixɼ جఈߦྻPsͱ܎਺aͷઢܗ݁߹Ͱ Weight matrixΛఆٛ͢Δ Preliminary 1 Preliminary 2 Method Details Experiment Conclusion

Slide 37

Slide 37 text

Agenda • Preliminary for Recommender System • Preliminary for Graph Convolution • Method Overview • Method Details • Experiment • Conclusion and Future works 37 Preliminary 1 Preliminary 2 Method Details Experiment Conclusion

Slide 38

Slide 38 text

Dataset 38 Preliminary 1 Preliminary 2 Method Details Experiment Conclusion

Slide 39

Slide 39 text

Main Baselines 39 • sRGCNN:  Graph CNNͰ  DecoderʹRNNΛར༻͍ͯ͠Δ • CF-NADE:  Auto-EncoderͷҰछͰɼະ؍ଌϦϯΫʹॳظ஋ͱͯ͠3Λ༩͑ͯ ͍Δɽ·ͨͦͷଞ͍ΖΜͳςΫχοΫΛಋೖ  ʢͪΌΜͱཧղͰ͖ͳ͔ͬͨʣ  Preliminary 1 Preliminary 2 Method Details Experiment Conclusion

Slide 40

Slide 40 text

MovieLens 100K 40 • େࠩͰউར • sRGCNNΑΓਫ਼౓͕͍͍ͱ͍͏͜ͱ͸ɼRNNΛ࢖ͬͯෳࡶʹ ܭࢉ͢ΔΑΓ΋ಛ௃ྔΛ௚઀ߦྻΛ࠶ߏ੒͢Δํ͕͍͍ Preliminary 1 Preliminary 2 Method Details Experiment Conclusion

Slide 41

Slide 41 text

MovieLens 1M and 10M 41 • ߴ͍ਫ਼౓͕ग़͍ͤͯΔͷͰɼେن໛ͳσʔληοτͰͷֶश͕Մೳ Ͱ͋Δ͜ͱ͕ࣔ͞Εͨ • CF-NADE͸͍ΖΜͳςΫχοΫΛ࢖͓ͬͯΓɼGC-MC΋ͦΕΛ࢖ ͏͜ͱ͸Ͱ͖Δʢͭ·Γ࢖͑͹উͯΔͥతͳʁʣ Preliminary 1 Preliminary 2 Method Details Experiment Conclusion

Slide 42

Slide 42 text

Flixster, Douban, & YahooMusic 42 • શͯಉ͡hyper parameterͰ΍͚ͬͨͲউͬͨ Preliminary 1 Preliminary 2 Method Details Experiment Conclusion

Slide 43

Slide 43 text

Cold-start Analysis 43 • ML-100KͰɼNcਓͷϢʔβʔͷධՁ஋ΛɼNr·ͰݮΒ࣮͠ݧ • ίϯςϯπ৘ใΛಋೖͨ͜͠ͱʹΑΔޮՌΛݕূ͍ͯ͠Δ ಛʹ৚͕݅ ݫ͍͠΄Ͳ վળ͕େ͖͍ Preliminary 1 Preliminary 2 Method Details Experiment Conclusion

Slide 44

Slide 44 text

Agenda • Preliminary for Recommender System • Preliminary for Graph Convolution • Method Overview • Method Details • Experiment • Conclusion and Future works 44 Preliminary 1 Preliminary 2 Method Details Experiment Conclusion

Slide 45

Slide 45 text

Conclusion 45 • Recommender SystemͷͨΊͷGraph Auto-EncoderΛ༻͍ͨ Matrix Completionख๏ɼGC-MCΛఏҊ • σʔληοτʹΑͬͯ͸SoTAΛୡ੒͠ɼstochastic mini batch samplingʹΑΔେن໛σʔληοτ΁ͷద༻΋Մೳʹͨ͠ Preliminary 1 Preliminary 2 Method Details Experiment Conclusion

Slide 46

Slide 46 text

Future Work 46 • Content-basedͷಛ௃ྔͷநग़ํ๏ΛCNN΍RNNʹม͑Δ͜ ͱͰɼը૾΍ࣗવݴޠΛϨίϝϯυʹಋೖͰ͖ΔΑ͏ʹ͢Δ • ScalabilityΛߴΊΔͨΊʹɼsub-sampling local neighborhood ͳͲͷޮ཰తͳۙࣅํ๏Λಋೖ͢Δ • ·ͨɼਖ਼نԽ߲౳ʹAttention MechanismΛಋೖͯ͠Ϟσϧ ͷ͞ΒͳΔ֦ுΛ໨ࢦ͢ Preliminary 1 Preliminary 2 Method Details Experiment Conclusion