Dgraph - A high performance graph database written in pure Go

Go Conference 2018 Spring

Transcript

1. ©2018 Wantedly, Inc.
   Dgraph - A high performance
   graph database written in pure Go
   Go Conference 2018 Spring - @munisystem
   

   View Slide

2. ©2018 Wantedly, Inc.
   Yuichi Saito
   @munisystem
   • Wantedly, Inc.
   • Infrastructure Engineer
   • Microservices
   • Productivity
   • My Favorite Golang Products
   • Istio / Istio
   • wercker / stern
   • ory / dockertest
   ࣗݾ঺հ
   

   View Slide

3. ©2018 Wantedly, Inc.
   άϥϑͱ͸
   άϥϑͰදݱͰ͖Δ΋ͷ
   ୔ࢁͷ8FC"QQMJDBUJPO͕άϥϑΛ಺แ͍ͯ͠Δ
   w ίʔυϕʔεͱґଘύοέʔδ
   w υΩϡϝϯτͱΧςΰϦλά
   w ঎඼ͱ஫จཤྺ
   w ໊ࢗަ׵
   

   View Slide

4. ©2018 Wantedly, Inc.
   ԿނάϥϑσʔλϕʔεΛ࢖͏ͷ͔
   طଘͷάϥϑσʔλϕʔεͷ໰୊఺
   %HSBQIͷ঺հ
   ·ͱΊ
   ΞδΣϯμ
   

   View Slide

5. ©2018 Wantedly, Inc.
   άϥϑΛ3%#.4Ͱѻ͏ͷ͸೉͍͠
   w 3%#.4͸άϥϑΛϦϨʔγϣϯͱͯ͠දݱ͢Δ
   w άϥϑΛḷΔ౓ʹ+0*/Λ࢖͏ඞཁ͕͋Δ
   w +0*/͸ίετ͕͔͔Δ
   w ਓɺ঎඼ɺ໊ࢗͳͲ༷ʑͳσʔλؒͷάϥϑΛදݱ͢Δʹ͸ɺҎԼͷͲͪΒ͔͔͠ͳ͍
   ਓਓɺਓ঎඼ͳͲͦΕͧΕʹରͯ͠ϦϨʔγϣϯΛఆٛ
   ϙϦϞʔϑΟοΫͳϦϨʔγϣϯΛఆٛ
   ԿނάϥϑσʔλϕʔεΛ࢖͏ͷ͔
   

   View Slide

6. ©2018 Wantedly, Inc.
   άϥϑσʔλϕʔε͸ݸʑͷͭͳ͕ΓϑΥʔΧεͨ͠σʔλετΞ
   w શͯͷσʔλ͸/PEFͱ&EHFͰදݱ͞ΕΔ
   w /PEF
   w σʔλͷΤϯςΟςΟΛࣔ͢
   w FHϢʔβɺ঎඼ɺ໊ࢗ
   w &EHF
   w σʔλؒͷؔ܎ͱํ޲Λࣔ͢
   w FH༑ͩͪɺߪೖɺ໊ࢗަ׵
   ԿނάϥϑσʔλϕʔεΛ࢖͏ͷ͔
   C
   

   View Slide

7. ©2018 Wantedly, Inc.
   Index Free Adjacency
   w /PEF͸࣍ͷ/PEF΁ΞΫηε͢ΔͨΊͷΞυϨεΛ࣋ͭ
   w ୳ࡧ͸ࣗ਎ͷ࣋ͭ&EHFͷ਺͚ͩͰΑ͍
   w -JOLFE-JTUͳͲͰ࣮૷͞ΕΔ
   w 0
   Ͱ࣍ͷ/PEFΛࢀরͰ͖Δ
   w ྫ"͔Β#Λࢀর͢Δ
   ԿނάϥϑσʔλϕʔεΛ࢖͏ͷ͔
   A
   B
   C
   

   View Slide

8. ©2018 Wantedly, Inc.
   Index Free Adjacency
   w /PEF͸࣍ͷ/PEF΁ΞΫηε͢ΔͨΊͷΞυϨεΛ࣋ͭ
   w ୳ࡧ͸ࣗ਎ͷ࣋ͭ&EHFͷ਺͚ͩͰΑ͍
   w -JOLFE-JTUͳͲͰ࣮૷͞ΕΔ
   w 0
   Ͱ࣍ͷ/PEFΛࢀরͰ͖Δ
   w ྫ"͔Β#Λࢀর͢Δ
   ԿނάϥϑσʔλϕʔεΛ࢖͏ͷ͔
   A
   B
   C
   

   View Slide

9. ©2018 Wantedly, Inc.
   Index Free Adjacency
   w /PEF͸࣍ͷ/PEF΁ΞΫηε͢ΔͨΊͷΞυϨεΛ࣋ͭ
   w ୳ࡧ͸ࣗ਎ͷ࣋ͭ&EHFͷ਺͚ͩͰΑ͍
   w -JOLFE-JTUͳͲͰ࣮૷͞ΕΔ
   w 0
   Ͱ࣍ͷ/PEFΛࢀরͰ͖Δ
   w ྫ"͔Β#Λࢀর͢Δ
   ԿނάϥϑσʔλϕʔεΛ࢖͏ͷ͔
   A
   B
   C
   

   View Slide

10. ©2018 Wantedly, Inc.
    8BOUFEMZ1FPQMF͕Ϣʔβʹఏڙ͍ͯ͠ΔՁ஋
    w ʮ໊ࢗΛऔΓɺਓͱͭͳ͕Δʯ
    w ໊ࢗͱਓͷ͍͋ͩͷάϥϑΛετΞ͍ͨ͠
    ԿނάϥϑσʔλϕʔεΛ࢖͏ͷ͔
    

    View Slide

11. ©2018 Wantedly, Inc.
    طଘͷάϥϑσʔλϕʔεͷ໰୊఺
    

    View Slide

12. ©2018 Wantedly, Inc.
    ॻ͖ࠐΈͷੑೳ͕௿͔ͬͨ
    w 3FBE͸ຆͲͷάϥϑσʔλϕʔεͰߴ͍ੑೳΛग़͍ͯ͠Δ
    w ໊ࢗΛࡱΔɺͱ͍͏ߦҝ͸8BOUFEMZ1FPQMFͷίΞͳମݧ
    w ຖ෼ຖඵେྔʹૹΒΕͯ͘Δ໊ࢗΛετΞͰ͖Δ͚ͩͷ8SJUFͷύϑΥʔϚϯε͕ඞཁ
    w ͜ΕΛຬͨ͢άϥϑσʔλϕʔε͸ଘࡏ͠ͳ͔ͬͨ
    طଘͷάϥϑσʔλϕʔεͷ໰୊఺
    

    View Slide

13. ©2018 Wantedly, Inc.
    εέʔϧ͠ͳ͍ߏ଄
    • αʔϏεͷ੒௕ʹεέʔϧͰԠ͡ΒΕΔඞཁ͕͋Δ
    w /FPK͸σʔλϕʔεଆͰͷσʔλͷγϟʔσΟϯάʹରԠ͍ͯ͠ͳ͍
    w 0SJFOU%#͸$MBTTʢςʔϒϧʣຖʹγϟʔσΟϯάΛ͢Δ
    w ͋Δςʔϒϧ͚ͩॻ͖ࠐΈ͕ܹ͍͠ɺΈ͍ͨͳέʔεʹରԠͰ͖ͳ͍
    طଘͷάϥϑσʔλϕʔεͷ໰୊఺
    

    View Slide

14. ©2018 Wantedly, Inc.
    %HSBQIͷ঺հ
    

    View Slide

15. ©2018 Wantedly, Inc.
    (P੡ͷ෼ࢄάϥϑσʔλϕʔε
    w (SBQI2-ϥΠΫͳΫΤϦγϯλοΫε
    w H31$ͱ)551ͷαϙʔτ
    w ϋΠύϑΥʔϚϯε
    %HSBQIͱ͸
    

    View Slide

16. ©2018 Wantedly, Inc.
    ॻ͖ࠐΈͷύϑΥʔϚϯε͕༏Ε͍ͯΔ
    w ެࣜͷϕϯνϚʔΫ
    w ಡΈॻ͖ڞʹ/FPKͷ໿ഒ
    w 1FPQMFͷ̎೥ޙΛ૝ఆͨ͠σʔλྔ

    εϧʔϓοτͰϕϯνϚʔΫΛ࣮ࢪ
    w εέʔϧͳ͠Ͱ଱͑ΒΕͨ
    %HSBQIͱ͸
    https://blog.dgraph.io/post/benchmark-
    

    View Slide

17. ©2018 Wantedly, Inc.
    %HSBQIͷσʔλϞσϧ
    w %HSBQIͰ͸άϥϑϞσϧͱͯ͠3%'5SJQMFΛ࠾༻
    w 8FCϦιʔεʹϝλ৘ใΛ෇༩͢ΔͨΊͷ࢓૊Έ
    w Ϧιʔεͷؔ܎Λ4VCKFDU 1SFEJDBUF 0CKFDUͰදݱ
    w 4VCKFDUࢧ఺ͱͳΔ/PEF
    w 1SFEJDBUFؔ܎Λද͢&EHF
    w 0CKFDUऴ఺ͱͳΔ/PEF
    %HSBQIͱ͸
    .
    .
    .
    .
    .
    RDF ٖࣅίʔυʹ͓͚ΔϑϨϯυϦϨʔγϣϯ
    

    View Slide

18. ©2018 Wantedly, Inc.
    3%'5SJQMF
    Dgraphͱ͸
    .
    .
    .
    .
    .
    RDF ٖࣅίʔυʹ͓͚ΔϑϨϯυϦϨʔγϣϯ
    C
    Bob b
    a Alice
    friend named
    ==
    

    View Slide

19. ©2018 Wantedly, Inc.
    3%'5SJQMF
    w QFSTPO͕ͳ͍
    w 0CKFDU4VCKFDU͸&OUJUZͱ"UUSJCVUF͕͋Δ
    w &OUJUZ6TFSͳͲͷ֓೦Λද͢
    w "UUSJCVUF஋Λද͢
    w ຊ࣭తʹ͸ϞσϦϯάʹ͸ؔ܎͕ͳ͍
    w OBNFEQSFEJDBUF͕͋ΔͳΒͦΕ͸QFSTPO
    w ·ΔͰ%VDL5ZQJOH
    Dgraphͱ͸
    C
    Bob b
    a Alice
    friend named
    

    View Slide

20. ©2018 Wantedly, Inc.
    %HSBQIͷσʔλߏ଄
    

    View Slide

21. ©2018 Wantedly, Inc.
    άϥϑ͸̎ͭͷσʔλߏ଄ʹΑͬͯදݱ͞Ε͍ͯΔ
    w 1PTUJOH
    w 1PTUJOH-JTU
    %HSBQIͷσʔλߏ଄
    

    View Slide

22. ©2018 Wantedly, Inc.
    1PTUJOH
    w %HSBQIͷσʔλͷ࠷খ୯Ґ
    w 1SFEJDBUFͱͦΕʹඥͮ͘

    4VCKFDUͱ0CKFDUΛ಺แ͢Δ
    w WBMVF@UZQF͕6*%
    w 4VCKFDU͸&OUJUZͰ͋Δ
    w ͦΕҎ֎͸"UUSJCVUF
    w WBMVFʹ͸࣮σʔλ͕ೖΔ
    %HSBQIͷσʔλߏ଄
    message Posting {
    fixed64 uid = 1;
    bytes value = 2;
    enum ValType {
    DEFAULT = 0;
    BINARY = 1;
    INT = 2;
    FLOAT = 3;
    BOOL = 4;
    DATETIME = 5;
    GEO = 6;
    UID = 7;
    PASSWORD = 8;
    STRING = 9;
    }
    ValType val_type = 3;
    ...
    ...
    enum PostingType {
    REF=0;
    VALUE=1;
    VALUE_LANG=2;
    }
    PostingType posting_type = 4;
    bytes lang_tag = 5;
    string label = 6;
    repeated api.Facet facets = 9;
    uint32 op = 12;
    uint64 start_ts = 13;
    uint64 commit_ts = 14;
    }
    Protobuf ʹΑΔ Posting ͷఆٛ
    

    View Slide

23. ©2018 Wantedly, Inc.
    1PTUJOH-JTU
    w 1SFEJDBUFͱ4VCKFDUͷϖΞʹඥͮ͘1PTUJOHΛͻͱ·ͱΊʹͨ͠΋ͷ
    w ͻͱ·ͱΊʹ͢Δ͜ͱͰɺ͋ΔϢʔβͷ༑ୡҰཡ

    ΛҰճͷ୳ࡧͰऔಘͰ͖Δ
    w ,74ͷLFZʹ1SFEJDBUFͱ4VCKFDUؚ͕·ΕΔ
    w ͦͷҝ1PTUJOHʹ͸ࣗ෼͕ԿऀͰ͋Δ͔Λ

    දݱ͢ΔGJFME͕ͳ͍
    %HSBQIͷσʔλߏ଄
    message PostingList {
    repeated Posting postings = 1;
    bytes checksum = 2;
    uint64 commit = 3;
    bytes uids = 4;
    }
    Protobuf ʹΑΔ Posting ͷఆٛ
    (Predicate,Subject) -> PostingList
    KVS ΁ͷετΞํ๏
    

    View Slide

24. ©2018 Wantedly, Inc.
    #BEHFS
    w %HSBQIͷ։ൃνʔϜ͕࡞੒ͨ͠(PͰॻ͔Εͨ,74
    w ϋΠύϑΥʔϚϯε
    w 3BOEPN3FBE͸'BDFCPPL੡ͷ,74Ͱ͋Δ3PDLT%#ͷഒ
    w 44%΁ͷ࠷దԽ
    w "$*%5SBOTBDUJPOͷαϙʔτ
    DdgraphͷσʔλετϨʔδ
    

    View Slide

25. ©2018 Wantedly, Inc.
    Կނ#BEHFSΛ࡞ͬͨͷ͔
    w ΋ͱ΋ͱ͸3PDLT%#Λ࢖͍ͬͯͨ
    w ͔͠͠$HPΛ͔ͭ͏σϝϦοτ͕େ͖͗ͨ͢
    w (PͷΤίγεςϜʹ৐Εͳ͍
    w QQSPGSBDFEFUFDUPS
    w QUISFBE@DSFBUFͰόάΛҾ͖ى͜͠ɺύϑΥʔϚϯε͕௿Լ͢Δ
    w (Pͷࠒ
    w ($͕ޮ͔ͳ͍ͷͰ$GSFF
    Λ๨ΕΔͱϝϞϦϦʔΫΛҾ͖ى͜͢
    DdgraphͷσʔλετϨʔδ
    

    View Slide

26. ©2018 Wantedly, Inc.
    Կނ#BEHFSΛ࡞ͬͨͷ͔
    w (PͰॻ͔Ε͍ͯΔ#PMU%#͸Ͳ͏ͩͬͨͷ͔
    w #PMU%#͸3FBE8SJUFڞʹάϩʔόϧۭؒʹ͋ΔTZOD38.VUFYͷϩοΫΛͱΔ
    w ฒྻॲཧ͕ग़དྷͳ͍
    w 1PTUJOH-JTUͷมߋʹେ͖ͳӨڹΛ༩͑ͨ
    DdgraphͷσʔλετϨʔδ
    

    View Slide

27. ©2018 Wantedly, Inc.
    #BEHFSͷ࢓૊Έ
    

    View Slide

28. ©2018 Wantedly, Inc.
    -PH4USVDUVSFE.FSHF5SFF
    w ίϯηϓτ͸Ͱ͖Δ͚ͩγʔΫΛগͳ͘͢Δ
    w ॻ͖ࠐΈͷϑϩʔ
    ॻ͖ࠐΈ͕͋ͬͨ৔߹͸8"-ͱϝϞϦʢ.FN5BCMFʣʹॻ͖ग़͢
    ॻ͖ࠐΉͱ͖ʹ.FN5BCMFʹσʔλ͕ೖΒͳ͍৔߹͸ϑΝΠϧʹϑϥογϡ
    γʔΫͷྔΛݮΒͨ͢ΊʹɺϑΝΠϧ͕ଟ͘ͳ͖ͬͯͨΒϑΝΠϧΛϚʔδͤ͞Δ
    w $PNQBDUJPO
    Storage architecture of Badger
    

    View Slide

29. ©2018 Wantedly, Inc.
    -PH4USVDUVSFE.FSHF5SFF
    Add: (muni, 30)
    Storage architecture of Badger
    L W
    Alice 10
    Bob 20
    Level 1
    L W
    Carol 30
    Level 0 (Mem Table)
    

    View Slide

30. ©2018 Wantedly, Inc.
    -PH4USVDUVSFE.FSHF5SFF
    Add: (muni, 30)
    Storage architecture of Badger
    L W
    Alice 10
    Bob 20
    Level 1
    L W
    Carol 30
    muni 30
    Level 0 (Mem Table)
    

    View Slide

31. ©2018 Wantedly, Inc.
    -PH4USVDUVSFE.FSHF5SFF
    Add: (muni, 30)
    Storage architecture of Badger
    L W
    Alice 10
    Bob 20
    Level 1
    L W
    Level 0 (Mem Table)
    L W
    Carol 10
    muni 20
    

    View Slide

32. ©2018 Wantedly, Inc.
    -PH4USVDUVSFE.FSHF5SFF
    Set: (Alice, 20)
    Storage architecture of Badger
    L W
    Alice 10
    Bob 20
    Level 1
    L W
    Level 0 (Mem Table)
    L W
    Carol 10
    muni 20
    

    View Slide

33. ©2018 Wantedly, Inc.
    -PH4USVDUVSFE.FSHF5SFF
    Set: (Alice, 20) ... will remove (Alice, 10) when merge
    Storage architecture of Badger
    L W
    Alice 10
    Bob 20
    Level 1
    L W
    Alice 20
    Level 0 (Mem Table)
    L W
    Carol 10
    muni 20
    

    View Slide

34. ©2018 Wantedly, Inc.
    -PH4USVDUVSFE.FSHF5SFF
    Add: (mike, 50)
    Storage architecture of Badger
    L W
    Alice 10
    Bob 20
    Level 1
    L W
    Alice 20
    Level 0 (Mem Table)
    L W
    Carol 10
    muni 20
    

    View Slide

35. ©2018 Wantedly, Inc.
    -PH4USVDUVSFE.FSHF5SFF
    Add: (mike, 50)
    Storage architecture of Badger
    L W
    Alice 10
    Bob 20
    Level 1
    L W
    Alice 20
    mike 50
    Level 0 (Mem Table)
    L W
    Carol 10
    muni 20
    

    View Slide

36. ©2018 Wantedly, Inc.
    -PH4USVDUVSFE.FSHF5SFF
    Add: (mike, 50)
    Storage architecture of Badger
    L W
    Alice 20
    mike 50
    Level 1
    L W
    Level 0 (Mem Table)
    L W
    Carol 10
    muni 20
    Bob 20
    Level 2
    

    View Slide

37. ©2018 Wantedly, Inc.
    -PH4USVDUVSFE.FSHF5SFF
    w Կ͕خ͍͠ʁ
    w 8SJUF͕૸Δͷ͸ϚʔδͷλΠϛϯά͚ͩ
    w 8"-͸ॻ͍͍ͯΔ͚Ͳɺ࣮σʔλʹൺ΂Ε͹௿ίετ
    w ಡΈࠐΉͱ͖͸௿Ϩϕϧ͔Βॱʹߦ͏ͷͰɺஞ࣍σʔλΛߋ৽͢Δඞཁ͕ͳ͍
    w ϑΝΠϧͷ਺͚ͩγʔΫ͢Ε͹Α͍
    w ϑΝΠϧʹ͸Ͳͷ,FZؚ͕·Ε͍ͯΔͷ͔֬ೝ͢ΔͨΊͷ#MPPN'JMUFS͕͋Δ
    w #MPPN'JMUFS͕GBMTFΛฦͨ͠৔߹͸ͦͷϑΝΠϧʹ͸,FZ͸ඞͣଘࡏ͠ͳ͍
    Storage architecture of Badger
    

    View Slide

38. ©2018 Wantedly, Inc.
    "$*%5SBOTBDUJPO
    w ࣮૷͸.VMUJ7FSTJPO$PODVSSFODZ$POUSPM
    w 3FBEͨ͠ͱ͖ɺߦʹUJNFTUBNQΛ෇༩͢Δ
    w ผͷτϥϯβΫγϣϯͰಉ͡ߦʹະίϛοτͷมߋ͕૸ͬͨ
    w มߋ͸ϝϞϦʹରͯ͠ߦͳΘΕɺଞͷτϥϯβΫγϣϯ͔Β͸ݟ͑ͳ͍
    w ίϛοτͯ͠ߦͷ࣮σʔλͷUJNFTUBNQΛߋ৽
    w τϥϯβΫγϣϯ಺ͰҰճ3FBEͨ͠ߦΛ΋͏Ұ౓ࢀর
    w طʹUJNFTUBNQ͕෇༩͞Ε͍ͯΔͷͰલͷ஋Λฦ͢
    ACID Transaction of Badger
    

    View Slide

39. ©2018 Wantedly, Inc.
    %HSBQIͷεέʔϦϯά
    

    View Slide

40. ©2018 Wantedly, Inc.
    %HSBQIΛߏ੒͢Δίϯϙʔωϯτ
    w [FSP
    w Ϋϥελͷ؅ཧϊʔυ
    w (SPVQʢޙड़ʣͷ؅ཧ
    w όοΫΞοϓ΍γϟοτμ΢ϯͱ໋͍ͬͨྩΛ(SPVQʹ఻ൖͤ͞Δ
    w TFSWFS
    w σʔλετΞɺΫΤϦͷ࣮ߦϊʔυ
    w ̍ͭͷ(SPVQʹඞͣଐ͢Δ
    %HSBQIͷεέʔϦϯά
    

    View Slide

41. ©2018 Wantedly, Inc.
    (SPVQ
    w ୆Ҏ্ͷTFSWFSʹΑͬͯߏ੒͞ΕΔ
    w ಉ͡(SPVQʹଐ͢Δશͯͷϊʔυ͸ಉ͡σʔλΛ࣋ͭ
    w 3BGUͷ2VPSVN
    w γϟʔσΟϯάͷ୯Ґ
    w (SPVQʹׂΓ౰ͯΒΕΔσʔλ͸1SFEJDBUFຖʹܾ·Δ
    w ͭ·ΓGSJFOEʹؔ͢Δσʔλ͸શͯಉ͡(SPVQͰ؅ཧ͞ΕΔ
    w Ͳͷ1SFEJDBUFΛड͚͔࣋ͭ͸[FSPʹΑܾͬͯఆ͞ΕΔ
    %HSBQIͷεέʔϦϯά
    

    View Slide

42. ©2018 Wantedly, Inc.
    w ͭͳ͕ΓΛอଘ͢ΔσʔλετΞͱͯ͠

    άϥϑσʔλϕʔεΛ࠾༻
    w %HSBQI͸طଘͷάϥϑσʔλϕʔε͕
    ๊͍͑ͯͨ໰୊Λղܾ͍ͯ͠Δ
    w ॻ͖ࠐΈͷύϑΥʔϚϯε
    w ਫฏεέʔϦϯά
    ·ͱΊ
    

    View Slide