Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Dgraph - A high performance graph database writ...

Dgraph - A high performance graph database written in pure Go

Go Conference 2018 Spring

Avatar for Yuichi Saito

Yuichi Saito

April 15, 2018
Tweet

More Decks by Yuichi Saito

Other Decks in Technology

Transcript

  1. ©2018 Wantedly, Inc. Dgraph - A high performance graph database

    written in pure Go Go Conference 2018 Spring - @munisystem
  2. ©2018 Wantedly, Inc. Yuichi Saito @munisystem • Wantedly, Inc. •

    Infrastructure Engineer • Microservices • Productivity • My Favorite Golang Products • Istio / Istio • wercker / stern • ory / dockertest ࣗݾ঺հ
  3. ©2018 Wantedly, Inc. άϥϑΛ3%#.4Ͱѻ͏ͷ͸೉͍͠ w 3%#.4͸άϥϑΛϦϨʔγϣϯͱͯ͠දݱ͢Δ w άϥϑΛḷΔ౓ʹ+0*/Λ࢖͏ඞཁ͕͋Δ w +0*/͸ίετ͕͔͔Δ

    w ਓɺ঎඼ɺ໊ࢗͳͲ༷ʑͳσʔλؒͷάϥϑΛදݱ͢Δʹ͸ɺҎԼͷͲͪΒ͔͔͠ͳ͍  ਓਓɺਓ঎඼ͳͲͦΕͧΕʹରͯ͠ϦϨʔγϣϯΛఆٛ  ϙϦϞʔϑΟοΫͳϦϨʔγϣϯΛఆٛ ԿނάϥϑσʔλϕʔεΛ࢖͏ͷ͔
  4. ©2018 Wantedly, Inc. άϥϑσʔλϕʔε͸ݸʑͷͭͳ͕ΓϑΥʔΧεͨ͠σʔλετΞ w શͯͷσʔλ͸/PEFͱ&EHFͰදݱ͞ΕΔ w /PEF w σʔλͷΤϯςΟςΟΛࣔ͢

    w FHϢʔβɺ঎඼ɺ໊ࢗ w &EHF w σʔλؒͷؔ܎ͱํ޲Λࣔ͢ w FH༑ͩͪɺߪೖɺ໊ࢗަ׵ ԿނάϥϑσʔλϕʔεΛ࢖͏ͷ͔ C
  5. ©2018 Wantedly, Inc. Index Free Adjacency w /PEF͸࣍ͷ/PEF΁ΞΫηε͢ΔͨΊͷΞυϨεΛ࣋ͭ w ୳ࡧ͸ࣗ਎ͷ࣋ͭ&EHFͷ਺͚ͩͰΑ͍

    w -JOLFE-JTUͳͲͰ࣮૷͞ΕΔ w 0  Ͱ࣍ͷ/PEFΛࢀরͰ͖Δ w ྫ"͔Β#Λࢀর͢Δ ԿނάϥϑσʔλϕʔεΛ࢖͏ͷ͔ A B C
  6. ©2018 Wantedly, Inc. Index Free Adjacency w /PEF͸࣍ͷ/PEF΁ΞΫηε͢ΔͨΊͷΞυϨεΛ࣋ͭ w ୳ࡧ͸ࣗ਎ͷ࣋ͭ&EHFͷ਺͚ͩͰΑ͍

    w -JOLFE-JTUͳͲͰ࣮૷͞ΕΔ w 0  Ͱ࣍ͷ/PEFΛࢀরͰ͖Δ w ྫ"͔Β#Λࢀর͢Δ ԿނάϥϑσʔλϕʔεΛ࢖͏ͷ͔ A B C
  7. ©2018 Wantedly, Inc. Index Free Adjacency w /PEF͸࣍ͷ/PEF΁ΞΫηε͢ΔͨΊͷΞυϨεΛ࣋ͭ w ୳ࡧ͸ࣗ਎ͷ࣋ͭ&EHFͷ਺͚ͩͰΑ͍

    w -JOLFE-JTUͳͲͰ࣮૷͞ΕΔ w 0  Ͱ࣍ͷ/PEFΛࢀরͰ͖Δ w ྫ"͔Β#Λࢀর͢Δ ԿނάϥϑσʔλϕʔεΛ࢖͏ͷ͔ A B C
  8. ©2018 Wantedly, Inc. ॻ͖ࠐΈͷύϑΥʔϚϯε͕༏Ε͍ͯΔ w ެࣜͷϕϯνϚʔΫ w ಡΈॻ͖ڞʹ/FPKͷ໿ഒ w 1FPQMFͷ̎೥ޙΛ૝ఆͨ͠σʔλྔ


    εϧʔϓοτͰϕϯνϚʔΫΛ࣮ࢪ w εέʔϧͳ͠Ͱ଱͑ΒΕͨ %HSBQIͱ͸ https://blog.dgraph.io/post/benchmark-
  9. ©2018 Wantedly, Inc. %HSBQIͷσʔλϞσϧ w %HSBQIͰ͸άϥϑϞσϧͱͯ͠3%'5SJQMFΛ࠾༻ w 8FCϦιʔεʹϝλ৘ใΛ෇༩͢ΔͨΊͷ࢓૊Έ w Ϧιʔεͷؔ܎Λ4VCKFDU

    1SFEJDBUF 0CKFDUͰදݱ w 4VCKFDUࢧ఺ͱͳΔ/PEF w 1SFEJDBUFؔ܎Λද͢&EHF w 0CKFDUऴ఺ͱͳΔ/PEF %HSBQIͱ͸ <a> <is> <person> . <b> <is> <person> . <a> <named> <Alice> . <b> <named> <Bob> . <a> <friend> <b> . RDF ٖࣅίʔυʹ͓͚ΔϑϨϯυϦϨʔγϣϯ
  10. ©2018 Wantedly, Inc. 3%'5SJQMF Dgraphͱ͸ <a> <is> <person> . <b>

    <is> <person> . <a> <named> <Alice> . <b> <named> <Bob> . <a> <friend> <b> . RDF ٖࣅίʔυʹ͓͚ΔϑϨϯυϦϨʔγϣϯ C Bob b a Alice friend named ==
  11. ©2018 Wantedly, Inc. 3%'5SJQMF w QFSTPO͕ͳ͍  w 0CKFDU4VCKFDU͸&OUJUZͱ"UUSJCVUF͕͋Δ w

    &OUJUZ6TFSͳͲͷ֓೦Λද͢ w "UUSJCVUF஋Λද͢ w ຊ࣭తʹ͸ϞσϦϯάʹ͸ؔ܎͕ͳ͍ w OBNFEQSFEJDBUF͕͋ΔͳΒͦΕ͸QFSTPO w ·ΔͰ%VDL5ZQJOH Dgraphͱ͸ C Bob b a Alice friend named
  12. ©2018 Wantedly, Inc. 1PTUJOH w %HSBQIͷσʔλͷ࠷খ୯Ґ w 1SFEJDBUFͱͦΕʹඥͮ͘
 4VCKFDUͱ0CKFDUΛ಺แ͢Δ w

    WBMVF@UZQF͕6*% w 4VCKFDU͸&OUJUZͰ͋Δ w ͦΕҎ֎͸"UUSJCVUF w WBMVFʹ͸࣮σʔλ͕ೖΔ %HSBQIͷσʔλߏ଄ message Posting { fixed64 uid = 1; bytes value = 2; enum ValType { DEFAULT = 0; BINARY = 1; INT = 2; FLOAT = 3; BOOL = 4; DATETIME = 5; GEO = 6; UID = 7; PASSWORD = 8; STRING = 9; } ValType val_type = 3; ... ... enum PostingType { REF=0; VALUE=1; VALUE_LANG=2; } PostingType posting_type = 4; bytes lang_tag = 5; string label = 6; repeated api.Facet facets = 9; uint32 op = 12; uint64 start_ts = 13; uint64 commit_ts = 14; } Protobuf ʹΑΔ Posting ͷఆٛ
  13. ©2018 Wantedly, Inc. 1PTUJOH-JTU w 1SFEJDBUFͱ4VCKFDUͷϖΞʹඥͮ͘1PTUJOHΛͻͱ·ͱΊʹͨ͠΋ͷ w ͻͱ·ͱΊʹ͢Δ͜ͱͰɺ͋ΔϢʔβͷ༑ୡҰཡ
 ΛҰճͷ୳ࡧͰऔಘͰ͖Δ w

    ,74ͷLFZʹ1SFEJDBUFͱ4VCKFDUؚ͕·ΕΔ w ͦͷҝ1PTUJOHʹ͸ࣗ෼͕ԿऀͰ͋Δ͔Λ
 දݱ͢ΔGJFME͕ͳ͍ %HSBQIͷσʔλߏ଄ message PostingList { repeated Posting postings = 1; bytes checksum = 2; uint64 commit = 3; bytes uids = 4; } Protobuf ʹΑΔ Posting ͷఆٛ (Predicate,Subject) -> PostingList KVS ΁ͷετΞํ๏
  14. ©2018 Wantedly, Inc. Կނ#BEHFSΛ࡞ͬͨͷ͔ w ΋ͱ΋ͱ͸3PDLT%#Λ࢖͍ͬͯͨ w ͔͠͠$HPΛ͔ͭ͏σϝϦοτ͕େ͖͗ͨ͢ w (PͷΤίγεςϜʹ৐Εͳ͍

    w QQSPGSBDFEFUFDUPS w QUISFBE@DSFBUFͰόάΛҾ͖ى͜͠ɺύϑΥʔϚϯε͕௿Լ͢Δ w (Pͷࠒ w ($͕ޮ͔ͳ͍ͷͰ$GSFF Λ๨ΕΔͱϝϞϦϦʔΫΛҾ͖ى͜͢ DdgraphͷσʔλετϨʔδ
  15. ©2018 Wantedly, Inc. -PH4USVDUVSFE.FSHF5SFF w ίϯηϓτ͸Ͱ͖Δ͚ͩγʔΫΛগͳ͘͢Δ w ॻ͖ࠐΈͷϑϩʔ  ॻ͖ࠐΈ͕͋ͬͨ৔߹͸8"-ͱϝϞϦʢ.FN5BCMFʣʹॻ͖ग़͢

     ॻ͖ࠐΉͱ͖ʹ.FN5BCMFʹσʔλ͕ೖΒͳ͍৔߹͸ϑΝΠϧʹϑϥογϡ  γʔΫͷྔΛݮΒͨ͢ΊʹɺϑΝΠϧ͕ଟ͘ͳ͖ͬͯͨΒϑΝΠϧΛϚʔδͤ͞Δ w $PNQBDUJPO Storage architecture of Badger
  16. ©2018 Wantedly, Inc. -PH4USVDUVSFE.FSHF5SFF Add: (muni, 30) Storage architecture of

    Badger L W Alice 10 Bob 20 Level 1 L W Carol 30 Level 0 (Mem Table)
  17. ©2018 Wantedly, Inc. -PH4USVDUVSFE.FSHF5SFF Add: (muni, 30) Storage architecture of

    Badger L W Alice 10 Bob 20 Level 1 L W Carol 30 muni 30 Level 0 (Mem Table)
  18. ©2018 Wantedly, Inc. -PH4USVDUVSFE.FSHF5SFF Add: (muni, 30) Storage architecture of

    Badger L W Alice 10 Bob 20 Level 1 L W Level 0 (Mem Table) L W Carol 10 muni 20
  19. ©2018 Wantedly, Inc. -PH4USVDUVSFE.FSHF5SFF Set: (Alice, 20) Storage architecture of

    Badger L W Alice 10 Bob 20 Level 1 L W Level 0 (Mem Table) L W Carol 10 muni 20
  20. ©2018 Wantedly, Inc. -PH4USVDUVSFE.FSHF5SFF Set: (Alice, 20) ... will remove

    (Alice, 10) when merge Storage architecture of Badger L W Alice 10 Bob 20 Level 1 L W Alice 20 Level 0 (Mem Table) L W Carol 10 muni 20
  21. ©2018 Wantedly, Inc. -PH4USVDUVSFE.FSHF5SFF Add: (mike, 50) Storage architecture of

    Badger L W Alice 10 Bob 20 Level 1 L W Alice 20 Level 0 (Mem Table) L W Carol 10 muni 20
  22. ©2018 Wantedly, Inc. -PH4USVDUVSFE.FSHF5SFF Add: (mike, 50) Storage architecture of

    Badger L W Alice 10 Bob 20 Level 1 L W Alice 20 mike 50 Level 0 (Mem Table) L W Carol 10 muni 20
  23. ©2018 Wantedly, Inc. -PH4USVDUVSFE.FSHF5SFF Add: (mike, 50) Storage architecture of

    Badger L W Alice 20 mike 50 Level 1 L W Level 0 (Mem Table) L W Carol 10 muni 20 Bob 20 Level 2
  24. ©2018 Wantedly, Inc. -PH4USVDUVSFE.FSHF5SFF w Կ͕خ͍͠ʁ w 8SJUF͕૸Δͷ͸ϚʔδͷλΠϛϯά͚ͩ w 8"-͸ॻ͍͍ͯΔ͚Ͳɺ࣮σʔλʹൺ΂Ε͹௿ίετ

    w ಡΈࠐΉͱ͖͸௿Ϩϕϧ͔Βॱʹߦ͏ͷͰɺஞ࣍σʔλΛߋ৽͢Δඞཁ͕ͳ͍ w ϑΝΠϧͷ਺͚ͩγʔΫ͢Ε͹Α͍ w ϑΝΠϧʹ͸Ͳͷ,FZؚ͕·Ε͍ͯΔͷ͔֬ೝ͢ΔͨΊͷ#MPPN'JMUFS͕͋Δ w #MPPN'JMUFS͕GBMTFΛฦͨ͠৔߹͸ͦͷϑΝΠϧʹ͸,FZ͸ඞͣଘࡏ͠ͳ͍ Storage architecture of Badger
  25. ©2018 Wantedly, Inc. "$*%5SBOTBDUJPO w ࣮૷͸.VMUJ7FSTJPO$PODVSSFODZ$POUSPM w 3FBEͨ͠ͱ͖ɺߦʹUJNFTUBNQΛ෇༩͢Δ w ผͷτϥϯβΫγϣϯͰಉ͡ߦʹະίϛοτͷมߋ͕૸ͬͨ

    w มߋ͸ϝϞϦʹରͯ͠ߦͳΘΕɺଞͷτϥϯβΫγϣϯ͔Β͸ݟ͑ͳ͍ w ίϛοτͯ͠ߦͷ࣮σʔλͷUJNFTUBNQΛߋ৽ w τϥϯβΫγϣϯ಺ͰҰճ3FBEͨ͠ߦΛ΋͏Ұ౓ࢀর w طʹUJNFTUBNQ͕෇༩͞Ε͍ͯΔͷͰલͷ஋Λฦ͢ ACID Transaction of Badger
  26. ©2018 Wantedly, Inc. %HSBQIΛߏ੒͢Δίϯϙʔωϯτ w [FSP w Ϋϥελͷ؅ཧϊʔυ w (SPVQʢޙड़ʣͷ؅ཧ

    w όοΫΞοϓ΍γϟοτμ΢ϯͱ໋͍ͬͨྩΛ(SPVQʹ఻ൖͤ͞Δ w TFSWFS w σʔλετΞɺΫΤϦͷ࣮ߦϊʔυ w ̍ͭͷ(SPVQʹඞͣଐ͢Δ %HSBQIͷεέʔϦϯά
  27. ©2018 Wantedly, Inc. (SPVQ w ୆Ҏ্ͷTFSWFSʹΑͬͯߏ੒͞ΕΔ w ಉ͡(SPVQʹଐ͢Δશͯͷϊʔυ͸ಉ͡σʔλΛ࣋ͭ w 3BGUͷ2VPSVN

    w γϟʔσΟϯάͷ୯Ґ w (SPVQʹׂΓ౰ͯΒΕΔσʔλ͸1SFEJDBUFຖʹܾ·Δ w ͭ·ΓGSJFOEʹؔ͢Δσʔλ͸શͯಉ͡(SPVQͰ؅ཧ͞ΕΔ w Ͳͷ1SFEJDBUFΛड͚͔࣋ͭ͸[FSPʹΑܾͬͯఆ͞ΕΔ %HSBQIͷεέʔϦϯά