Dgraph - A high performance graph database written in pure Go

73e67f6da1838e3498c00f33cb0309e3?s=47 Yuichi Saito
April 15, 2018
Technology
6
5.6k

Dgraph - A high performance graph database written in pure Go

Go Conference 2018 Spring

73e67f6da1838e3498c00f33cb0309e3?s=128

Yuichi Saito

April 15, 2018
Tweet

More Decks by Yuichi Saito

See All by Yuichi Saito
73e67f6da1838e3498c00f33cb0309e3?s=48 munisystem
1
4.8k
73e67f6da1838e3498c00f33cb0309e3?s=48 munisystem
3
4.1k
73e67f6da1838e3498c00f33cb0309e3?s=48 munisystem
8
5.4k
73e67f6da1838e3498c00f33cb0309e3?s=48 munisystem
2
490
73e67f6da1838e3498c00f33cb0309e3?s=48 munisystem
1
1.1k
73e67f6da1838e3498c00f33cb0309e3?s=48 munisystem
1
740

Other Decks in Technology

See All in Technology
9e7a653ed313b3da4b263a772bfc5026?s=48 clustervr
0
4k
D5a8875907107294786ee8cba72c7f4c?s=48 torufuru
0
150
23190387d424e33d6a1a807cea1e0469?s=48 osamtimizer
1
350
7689d16bbad8f2fa05a5e9afb8527675?s=48 fatsushi
2
950
9e7a653ed313b3da4b263a772bfc5026?s=48 clustervr
0
160
A73e465566ed41cd2b66b7d889b31666?s=48 maiamea
1
550
457ada693e168bd1102fec3914040b5d?s=48 uipath_friends
0
150
4681d7bd0f8c32b6e3e582891938e577?s=48 iktakahiro
3
1.4k
405fe9ab689473f267e2cfbd95f78c75?s=48 kyonmm
0
1.5k
A8b79d304b5184e5a5b0a109590f6683?s=48 dpreussler
0
190
8aba6de431668fc8d09977c0e33c63c1?s=48 jaguar_imo
4
1k
2102a6b8760bd6f57f672805723dd83a?s=48 line_developers_tw
0
380

Featured

See All Featured
4394ceb8b277f35ee3da7030384efb5d?s=48 davidbonilla
69
3.5k
9cde37f47e4a800ea081ea42de8d749a?s=48 dougneiner
119
7.7k
3ab1249be442027903e1180025340b3f?s=48 reverentgeek
24
1.7k
20bfe76b3d6105641f879fe45cfc9272?s=48 bkeepers
51
4k
78b475797a14c84799063c7cd073962f?s=48 holman
459
270k
B83d5b590577969ee79a5ad845411a7d?s=48 ammeep
653
54k
0bce06bd06ee7a2c4f5500f25dcf7f18?s=48 paulrobertlloyd
71
3.5k
B3d6434763caa0ef5dc4b792662c49f7?s=48 smashingmag
227
17k
5b9fe87ec1faa67a4599782930f45ec9?s=48 sstephenson
144
12k
C51b39fa3c79ccb99dcb8a076bc98287?s=48 brianwarren
82
4.6k
245cee81a9c424266e5e401d844ea881?s=48 lara
590
60k
6804f1775cb4babfcc3851298566fbce?s=48 tanoku
257
24k

Transcript

  1. ©2018 Wantedly, Inc. Dgraph - A high performance graph database

    written in pure Go Go Conference 2018 Spring - @munisystem

  2. ©2018 Wantedly, Inc. Yuichi Saito @munisystem • Wantedly, Inc. •

    Infrastructure Engineer • Microservices • Productivity • My Favorite Golang Products • Istio / Istio • wercker / stern • ory / dockertest ࣗݾ঺հ

  3. ©2018 Wantedly, Inc. άϥϑͱ͸ άϥϑͰදݱͰ͖Δ΋ͷ ୔ࢁͷ8FC"QQMJDBUJPO͕άϥϑΛ಺แ͍ͯ͠Δ w ίʔυϕʔεͱґଘύοέʔδ w υΩϡϝϯτͱΧςΰϦλά

    w ঎඼ͱ஫จཤྺ w ໊ࢗަ׵

  4. ©2018 Wantedly, Inc.  ԿނάϥϑσʔλϕʔεΛ࢖͏ͷ͔  طଘͷάϥϑσʔλϕʔεͷ໰୊఺  %HSBQIͷ঺հ 

    ·ͱΊ ΞδΣϯμ

  5. ©2018 Wantedly, Inc. άϥϑΛ3%#.4Ͱѻ͏ͷ͸೉͍͠ w 3%#.4͸άϥϑΛϦϨʔγϣϯͱͯ͠දݱ͢Δ w άϥϑΛḷΔ౓ʹ+0*/Λ࢖͏ඞཁ͕͋Δ w +0*/͸ίετ͕͔͔Δ

    w ਓɺ঎඼ɺ໊ࢗͳͲ༷ʑͳσʔλؒͷάϥϑΛදݱ͢Δʹ͸ɺҎԼͷͲͪΒ͔͔͠ͳ͍  ਓਓɺਓ঎඼ͳͲͦΕͧΕʹରͯ͠ϦϨʔγϣϯΛఆٛ  ϙϦϞʔϑΟοΫͳϦϨʔγϣϯΛఆٛ ԿނάϥϑσʔλϕʔεΛ࢖͏ͷ͔

  6. ©2018 Wantedly, Inc. άϥϑσʔλϕʔε͸ݸʑͷͭͳ͕ΓϑΥʔΧεͨ͠σʔλετΞ w શͯͷσʔλ͸/PEFͱ&EHFͰදݱ͞ΕΔ w /PEF w σʔλͷΤϯςΟςΟΛࣔ͢

    w FHϢʔβɺ঎඼ɺ໊ࢗ w &EHF w σʔλؒͷؔ܎ͱํ޲Λࣔ͢ w FH༑ͩͪɺߪೖɺ໊ࢗަ׵ ԿނάϥϑσʔλϕʔεΛ࢖͏ͷ͔ C

  7. ©2018 Wantedly, Inc. Index Free Adjacency w /PEF͸࣍ͷ/PEF΁ΞΫηε͢ΔͨΊͷΞυϨεΛ࣋ͭ w ୳ࡧ͸ࣗ਎ͷ࣋ͭ&EHFͷ਺͚ͩͰΑ͍

    w -JOLFE-JTUͳͲͰ࣮૷͞ΕΔ w 0  Ͱ࣍ͷ/PEFΛࢀরͰ͖Δ w ྫ"͔Β#Λࢀর͢Δ ԿނάϥϑσʔλϕʔεΛ࢖͏ͷ͔ A B C

  8. ©2018 Wantedly, Inc. Index Free Adjacency w /PEF͸࣍ͷ/PEF΁ΞΫηε͢ΔͨΊͷΞυϨεΛ࣋ͭ w ୳ࡧ͸ࣗ਎ͷ࣋ͭ&EHFͷ਺͚ͩͰΑ͍

    w -JOLFE-JTUͳͲͰ࣮૷͞ΕΔ w 0  Ͱ࣍ͷ/PEFΛࢀরͰ͖Δ w ྫ"͔Β#Λࢀর͢Δ ԿނάϥϑσʔλϕʔεΛ࢖͏ͷ͔ A B C

  9. ©2018 Wantedly, Inc. Index Free Adjacency w /PEF͸࣍ͷ/PEF΁ΞΫηε͢ΔͨΊͷΞυϨεΛ࣋ͭ w ୳ࡧ͸ࣗ਎ͷ࣋ͭ&EHFͷ਺͚ͩͰΑ͍

    w -JOLFE-JTUͳͲͰ࣮૷͞ΕΔ w 0  Ͱ࣍ͷ/PEFΛࢀরͰ͖Δ w ྫ"͔Β#Λࢀর͢Δ ԿނάϥϑσʔλϕʔεΛ࢖͏ͷ͔ A B C

  10. ©2018 Wantedly, Inc. 8BOUFEMZ1FPQMF͕Ϣʔβʹఏڙ͍ͯ͠ΔՁ஋ w ʮ໊ࢗΛऔΓɺਓͱͭͳ͕Δʯ w ໊ࢗͱਓͷ͍͋ͩͷάϥϑΛετΞ͍ͨ͠ ԿނάϥϑσʔλϕʔεΛ࢖͏ͷ͔

  11. ©2018 Wantedly, Inc. طଘͷάϥϑσʔλϕʔεͷ໰୊఺

  12. ©2018 Wantedly, Inc. ॻ͖ࠐΈͷੑೳ͕௿͔ͬͨ w 3FBE͸ຆͲͷάϥϑσʔλϕʔεͰߴ͍ੑೳΛग़͍ͯ͠Δ w ໊ࢗΛࡱΔɺͱ͍͏ߦҝ͸8BOUFEMZ1FPQMFͷίΞͳମݧ w ຖ෼ຖඵେྔʹૹΒΕͯ͘Δ໊ࢗΛετΞͰ͖Δ͚ͩͷ8SJUFͷύϑΥʔϚϯε͕ඞཁ

    w ͜ΕΛຬͨ͢άϥϑσʔλϕʔε͸ଘࡏ͠ͳ͔ͬͨ طଘͷάϥϑσʔλϕʔεͷ໰୊఺

  13. ©2018 Wantedly, Inc. εέʔϧ͠ͳ͍ߏ଄ • αʔϏεͷ੒௕ʹεέʔϧͰԠ͡ΒΕΔඞཁ͕͋Δ w /FPK͸σʔλϕʔεଆͰͷσʔλͷγϟʔσΟϯάʹରԠ͍ͯ͠ͳ͍ w 0SJFOU%#͸$MBTTʢςʔϒϧʣຖʹγϟʔσΟϯάΛ͢Δ

    w ͋Δςʔϒϧ͚ͩॻ͖ࠐΈ͕ܹ͍͠ɺΈ͍ͨͳέʔεʹରԠͰ͖ͳ͍ طଘͷάϥϑσʔλϕʔεͷ໰୊఺

  14. ©2018 Wantedly, Inc. %HSBQIͷ঺հ

  15. ©2018 Wantedly, Inc. (P੡ͷ෼ࢄάϥϑσʔλϕʔε w (SBQI2-ϥΠΫͳΫΤϦγϯλοΫε w H31$ͱ)551ͷαϙʔτ w ϋΠύϑΥʔϚϯε

    %HSBQIͱ͸

  16. ©2018 Wantedly, Inc. ॻ͖ࠐΈͷύϑΥʔϚϯε͕༏Ε͍ͯΔ w ެࣜͷϕϯνϚʔΫ w ಡΈॻ͖ڞʹ/FPKͷ໿ഒ w 1FPQMFͷ̎೥ޙΛ૝ఆͨ͠σʔλྔ


    εϧʔϓοτͰϕϯνϚʔΫΛ࣮ࢪ w εέʔϧͳ͠Ͱ଱͑ΒΕͨ %HSBQIͱ͸ https://blog.dgraph.io/post/benchmark-

  17. ©2018 Wantedly, Inc. %HSBQIͷσʔλϞσϧ w %HSBQIͰ͸άϥϑϞσϧͱͯ͠3%'5SJQMFΛ࠾༻ w 8FCϦιʔεʹϝλ৘ใΛ෇༩͢ΔͨΊͷ࢓૊Έ w Ϧιʔεͷؔ܎Λ4VCKFDU

    1SFEJDBUF 0CKFDUͰදݱ w 4VCKFDUࢧ఺ͱͳΔ/PEF w 1SFEJDBUFؔ܎Λද͢&EHF w 0CKFDUऴ఺ͱͳΔ/PEF %HSBQIͱ͸ <a> <is> <person> . <b> <is> <person> . <a> <named> <Alice> . <b> <named> <Bob> . <a> <friend> <b> . RDF ٖࣅίʔυʹ͓͚ΔϑϨϯυϦϨʔγϣϯ

  18. ©2018 Wantedly, Inc. 3%'5SJQMF Dgraphͱ͸ <a> <is> <person> . <b>

    <is> <person> . <a> <named> <Alice> . <b> <named> <Bob> . <a> <friend> <b> . RDF ٖࣅίʔυʹ͓͚ΔϑϨϯυϦϨʔγϣϯ C Bob b a Alice friend named ==

  19. ©2018 Wantedly, Inc. 3%'5SJQMF w QFSTPO͕ͳ͍  w 0CKFDU4VCKFDU͸&OUJUZͱ"UUSJCVUF͕͋Δ w

    &OUJUZ6TFSͳͲͷ֓೦Λද͢ w "UUSJCVUF஋Λද͢ w ຊ࣭తʹ͸ϞσϦϯάʹ͸ؔ܎͕ͳ͍ w OBNFEQSFEJDBUF͕͋ΔͳΒͦΕ͸QFSTPO w ·ΔͰ%VDL5ZQJOH Dgraphͱ͸ C Bob b a Alice friend named

  20. ©2018 Wantedly, Inc. %HSBQIͷσʔλߏ଄

  21. ©2018 Wantedly, Inc. άϥϑ͸̎ͭͷσʔλߏ଄ʹΑͬͯදݱ͞Ε͍ͯΔ w 1PTUJOH w 1PTUJOH-JTU %HSBQIͷσʔλߏ଄

  22. ©2018 Wantedly, Inc. 1PTUJOH w %HSBQIͷσʔλͷ࠷খ୯Ґ w 1SFEJDBUFͱͦΕʹඥͮ͘
 4VCKFDUͱ0CKFDUΛ಺แ͢Δ w

    WBMVF@UZQF͕6*% w 4VCKFDU͸&OUJUZͰ͋Δ w ͦΕҎ֎͸"UUSJCVUF w WBMVFʹ͸࣮σʔλ͕ೖΔ %HSBQIͷσʔλߏ଄ message Posting { fixed64 uid = 1; bytes value = 2; enum ValType { DEFAULT = 0; BINARY = 1; INT = 2; FLOAT = 3; BOOL = 4; DATETIME = 5; GEO = 6; UID = 7; PASSWORD = 8; STRING = 9; } ValType val_type = 3; ... ... enum PostingType { REF=0; VALUE=1; VALUE_LANG=2; } PostingType posting_type = 4; bytes lang_tag = 5; string label = 6; repeated api.Facet facets = 9; uint32 op = 12; uint64 start_ts = 13; uint64 commit_ts = 14; } Protobuf ʹΑΔ Posting ͷఆٛ

  23. ©2018 Wantedly, Inc. 1PTUJOH-JTU w 1SFEJDBUFͱ4VCKFDUͷϖΞʹඥͮ͘1PTUJOHΛͻͱ·ͱΊʹͨ͠΋ͷ w ͻͱ·ͱΊʹ͢Δ͜ͱͰɺ͋ΔϢʔβͷ༑ୡҰཡ
 ΛҰճͷ୳ࡧͰऔಘͰ͖Δ w

    ,74ͷLFZʹ1SFEJDBUFͱ4VCKFDUؚ͕·ΕΔ w ͦͷҝ1PTUJOHʹ͸ࣗ෼͕ԿऀͰ͋Δ͔Λ
 දݱ͢ΔGJFME͕ͳ͍ %HSBQIͷσʔλߏ଄ message PostingList { repeated Posting postings = 1; bytes checksum = 2; uint64 commit = 3; bytes uids = 4; } Protobuf ʹΑΔ Posting ͷఆٛ (Predicate,Subject) -> PostingList KVS ΁ͷετΞํ๏

  24. ©2018 Wantedly, Inc. #BEHFS w %HSBQIͷ։ൃνʔϜ͕࡞੒ͨ͠(PͰॻ͔Εͨ,74 w ϋΠύϑΥʔϚϯε w 3BOEPN3FBE͸'BDFCPPL੡ͷ,74Ͱ͋Δ3PDLT%#ͷഒ

    w 44%΁ͷ࠷దԽ w "$*%5SBOTBDUJPOͷαϙʔτ DdgraphͷσʔλετϨʔδ

  25. ©2018 Wantedly, Inc. Կނ#BEHFSΛ࡞ͬͨͷ͔ w ΋ͱ΋ͱ͸3PDLT%#Λ࢖͍ͬͯͨ w ͔͠͠$HPΛ͔ͭ͏σϝϦοτ͕େ͖͗ͨ͢ w (PͷΤίγεςϜʹ৐Εͳ͍

    w QQSPGSBDFEFUFDUPS w QUISFBE@DSFBUFͰόάΛҾ͖ى͜͠ɺύϑΥʔϚϯε͕௿Լ͢Δ w (Pͷࠒ w ($͕ޮ͔ͳ͍ͷͰ$GSFF Λ๨ΕΔͱϝϞϦϦʔΫΛҾ͖ى͜͢ DdgraphͷσʔλετϨʔδ

  26. ©2018 Wantedly, Inc. Կނ#BEHFSΛ࡞ͬͨͷ͔ w (PͰॻ͔Ε͍ͯΔ#PMU%#͸Ͳ͏ͩͬͨͷ͔ w #PMU%#͸3FBE8SJUFڞʹάϩʔόϧۭؒʹ͋ΔTZOD38.VUFYͷϩοΫΛͱΔ w ฒྻॲཧ͕ग़དྷͳ͍

    w 1PTUJOH-JTUͷมߋʹେ͖ͳӨڹΛ༩͑ͨ DdgraphͷσʔλετϨʔδ

  27. ©2018 Wantedly, Inc. #BEHFSͷ࢓૊Έ

  28. ©2018 Wantedly, Inc. -PH4USVDUVSFE.FSHF5SFF w ίϯηϓτ͸Ͱ͖Δ͚ͩγʔΫΛগͳ͘͢Δ w ॻ͖ࠐΈͷϑϩʔ  ॻ͖ࠐΈ͕͋ͬͨ৔߹͸8"-ͱϝϞϦʢ.FN5BCMFʣʹॻ͖ग़͢

     ॻ͖ࠐΉͱ͖ʹ.FN5BCMFʹσʔλ͕ೖΒͳ͍৔߹͸ϑΝΠϧʹϑϥογϡ  γʔΫͷྔΛݮΒͨ͢ΊʹɺϑΝΠϧ͕ଟ͘ͳ͖ͬͯͨΒϑΝΠϧΛϚʔδͤ͞Δ w $PNQBDUJPO Storage architecture of Badger

  29. ©2018 Wantedly, Inc. -PH4USVDUVSFE.FSHF5SFF Add: (muni, 30) Storage architecture of

    Badger L W Alice 10 Bob 20 Level 1 L W Carol 30 Level 0 (Mem Table)

  30. ©2018 Wantedly, Inc. -PH4USVDUVSFE.FSHF5SFF Add: (muni, 30) Storage architecture of

    Badger L W Alice 10 Bob 20 Level 1 L W Carol 30 muni 30 Level 0 (Mem Table)

  31. ©2018 Wantedly, Inc. -PH4USVDUVSFE.FSHF5SFF Add: (muni, 30) Storage architecture of

    Badger L W Alice 10 Bob 20 Level 1 L W Level 0 (Mem Table) L W Carol 10 muni 20

  32. ©2018 Wantedly, Inc. -PH4USVDUVSFE.FSHF5SFF Set: (Alice, 20) Storage architecture of

    Badger L W Alice 10 Bob 20 Level 1 L W Level 0 (Mem Table) L W Carol 10 muni 20

  33. ©2018 Wantedly, Inc. -PH4USVDUVSFE.FSHF5SFF Set: (Alice, 20) ... will remove

    (Alice, 10) when merge Storage architecture of Badger L W Alice 10 Bob 20 Level 1 L W Alice 20 Level 0 (Mem Table) L W Carol 10 muni 20

  34. ©2018 Wantedly, Inc. -PH4USVDUVSFE.FSHF5SFF Add: (mike, 50) Storage architecture of

    Badger L W Alice 10 Bob 20 Level 1 L W Alice 20 Level 0 (Mem Table) L W Carol 10 muni 20

  35. ©2018 Wantedly, Inc. -PH4USVDUVSFE.FSHF5SFF Add: (mike, 50) Storage architecture of

    Badger L W Alice 10 Bob 20 Level 1 L W Alice 20 mike 50 Level 0 (Mem Table) L W Carol 10 muni 20

  36. ©2018 Wantedly, Inc. -PH4USVDUVSFE.FSHF5SFF Add: (mike, 50) Storage architecture of

    Badger L W Alice 20 mike 50 Level 1 L W Level 0 (Mem Table) L W Carol 10 muni 20 Bob 20 Level 2

  37. ©2018 Wantedly, Inc. -PH4USVDUVSFE.FSHF5SFF w Կ͕خ͍͠ʁ w 8SJUF͕૸Δͷ͸ϚʔδͷλΠϛϯά͚ͩ w 8"-͸ॻ͍͍ͯΔ͚Ͳɺ࣮σʔλʹൺ΂Ε͹௿ίετ

    w ಡΈࠐΉͱ͖͸௿Ϩϕϧ͔Βॱʹߦ͏ͷͰɺஞ࣍σʔλΛߋ৽͢Δඞཁ͕ͳ͍ w ϑΝΠϧͷ਺͚ͩγʔΫ͢Ε͹Α͍ w ϑΝΠϧʹ͸Ͳͷ,FZؚ͕·Ε͍ͯΔͷ͔֬ೝ͢ΔͨΊͷ#MPPN'JMUFS͕͋Δ w #MPPN'JMUFS͕GBMTFΛฦͨ͠৔߹͸ͦͷϑΝΠϧʹ͸,FZ͸ඞͣଘࡏ͠ͳ͍ Storage architecture of Badger

  38. ©2018 Wantedly, Inc. "$*%5SBOTBDUJPO w ࣮૷͸.VMUJ7FSTJPO$PODVSSFODZ$POUSPM w 3FBEͨ͠ͱ͖ɺߦʹUJNFTUBNQΛ෇༩͢Δ w ผͷτϥϯβΫγϣϯͰಉ͡ߦʹະίϛοτͷมߋ͕૸ͬͨ

    w มߋ͸ϝϞϦʹରͯ͠ߦͳΘΕɺଞͷτϥϯβΫγϣϯ͔Β͸ݟ͑ͳ͍ w ίϛοτͯ͠ߦͷ࣮σʔλͷUJNFTUBNQΛߋ৽ w τϥϯβΫγϣϯ಺ͰҰճ3FBEͨ͠ߦΛ΋͏Ұ౓ࢀর w طʹUJNFTUBNQ͕෇༩͞Ε͍ͯΔͷͰલͷ஋Λฦ͢ ACID Transaction of Badger

  39. ©2018 Wantedly, Inc. %HSBQIͷεέʔϦϯά

  40. ©2018 Wantedly, Inc. %HSBQIΛߏ੒͢Δίϯϙʔωϯτ w [FSP w Ϋϥελͷ؅ཧϊʔυ w (SPVQʢޙड़ʣͷ؅ཧ

    w όοΫΞοϓ΍γϟοτμ΢ϯͱ໋͍ͬͨྩΛ(SPVQʹ఻ൖͤ͞Δ w TFSWFS w σʔλετΞɺΫΤϦͷ࣮ߦϊʔυ w ̍ͭͷ(SPVQʹඞͣଐ͢Δ %HSBQIͷεέʔϦϯά

  41. ©2018 Wantedly, Inc. (SPVQ w ୆Ҏ্ͷTFSWFSʹΑͬͯߏ੒͞ΕΔ w ಉ͡(SPVQʹଐ͢Δશͯͷϊʔυ͸ಉ͡σʔλΛ࣋ͭ w 3BGUͷ2VPSVN

    w γϟʔσΟϯάͷ୯Ґ w (SPVQʹׂΓ౰ͯΒΕΔσʔλ͸1SFEJDBUFຖʹܾ·Δ w ͭ·ΓGSJFOEʹؔ͢Δσʔλ͸શͯಉ͡(SPVQͰ؅ཧ͞ΕΔ w Ͳͷ1SFEJDBUFΛड͚͔࣋ͭ͸[FSPʹΑܾͬͯఆ͞ΕΔ %HSBQIͷεέʔϦϯά

  42. ©2018 Wantedly, Inc. w ͭͳ͕ΓΛอଘ͢ΔσʔλετΞͱͯ͠
 άϥϑσʔλϕʔεΛ࠾༻ w %HSBQI͸طଘͷάϥϑσʔλϕʔε͕ ๊͍͑ͯͨ໰୊Λղܾ͍ͯ͠Δ w

    ॻ͖ࠐΈͷύϑΥʔϚϯε w ਫฏεέʔϦϯά ·ͱΊ