Slide 1
Slide 1 text
©2018 Wantedly, Inc.
Dgraph - A high performance
graph database written in pure Go
Go Conference 2018 Spring - @munisystem
Slide 2
Slide 2 text
©2018 Wantedly, Inc.
Yuichi Saito
@munisystem
• Wantedly, Inc.
• Infrastructure Engineer
• Microservices
• Productivity
• My Favorite Golang Products
• Istio / Istio
• wercker / stern
• ory / dockertest
ࣗݾհ
Slide 3
Slide 3 text
©2018 Wantedly, Inc.
άϥϑͱ
άϥϑͰදݱͰ͖Δͷ
ࢁͷ8FC"QQMJDBUJPO͕άϥϑΛแ͍ͯ͠Δ
w ίʔυϕʔεͱґଘύοέʔδ
w υΩϡϝϯτͱΧςΰϦλά
w ͱจཤྺ
w ໊ަ
Slide 4
Slide 4 text
©2018 Wantedly, Inc.
ԿނάϥϑσʔλϕʔεΛ͏ͷ͔
طଘͷάϥϑσʔλϕʔεͷ
%HSBQIͷհ
·ͱΊ
ΞδΣϯμ
Slide 5
Slide 5 text
©2018 Wantedly, Inc.
άϥϑΛ3%#.4Ͱѻ͏ͷ͍͠
w 3%#.4άϥϑΛϦϨʔγϣϯͱͯ͠දݱ͢Δ
w άϥϑΛḷΔʹ+0*/Λ͏ඞཁ͕͋Δ
w +0*/ίετ͕͔͔Δ
w ਓɺɺ໊ͳͲ༷ʑͳσʔλؒͷάϥϑΛදݱ͢ΔʹɺҎԼͷͲͪΒ͔͔͠ͳ͍
ਓਓɺਓͳͲͦΕͧΕʹରͯ͠ϦϨʔγϣϯΛఆٛ
ϙϦϞʔϑΟοΫͳϦϨʔγϣϯΛఆٛ
ԿނάϥϑσʔλϕʔεΛ͏ͷ͔
Slide 6
Slide 6 text
©2018 Wantedly, Inc.
άϥϑσʔλϕʔεݸʑͷͭͳ͕ΓϑΥʔΧεͨ͠σʔλετΞ
w શͯͷσʔλ/PEFͱ&EHFͰදݱ͞ΕΔ
w /PEF
w σʔλͷΤϯςΟςΟΛࣔ͢
w FHϢʔβɺɺ໊
w &EHF
w σʔλؒͷؔͱํΛࣔ͢
w FH༑ͩͪɺߪೖɺ໊ަ
ԿނάϥϑσʔλϕʔεΛ͏ͷ͔
C
Slide 7
Slide 7 text
©2018 Wantedly, Inc.
Index Free Adjacency
w /PEF࣍ͷ/PEFΞΫηε͢ΔͨΊͷΞυϨεΛ࣋ͭ
w ୳ࡧࣗͷ࣋ͭ&EHFͷ͚ͩͰΑ͍
w -JOLFE-JTUͳͲͰ࣮͞ΕΔ
w 0
Ͱ࣍ͷ/PEFΛࢀরͰ͖Δ
w ྫ"͔Β#Λࢀর͢Δ
ԿނάϥϑσʔλϕʔεΛ͏ͷ͔
A
B
C
Slide 8
Slide 8 text
©2018 Wantedly, Inc.
Index Free Adjacency
w /PEF࣍ͷ/PEFΞΫηε͢ΔͨΊͷΞυϨεΛ࣋ͭ
w ୳ࡧࣗͷ࣋ͭ&EHFͷ͚ͩͰΑ͍
w -JOLFE-JTUͳͲͰ࣮͞ΕΔ
w 0
Ͱ࣍ͷ/PEFΛࢀরͰ͖Δ
w ྫ"͔Β#Λࢀর͢Δ
ԿނάϥϑσʔλϕʔεΛ͏ͷ͔
A
B
C
Slide 9
Slide 9 text
©2018 Wantedly, Inc.
Index Free Adjacency
w /PEF࣍ͷ/PEFΞΫηε͢ΔͨΊͷΞυϨεΛ࣋ͭ
w ୳ࡧࣗͷ࣋ͭ&EHFͷ͚ͩͰΑ͍
w -JOLFE-JTUͳͲͰ࣮͞ΕΔ
w 0
Ͱ࣍ͷ/PEFΛࢀরͰ͖Δ
w ྫ"͔Β#Λࢀর͢Δ
ԿނάϥϑσʔλϕʔεΛ͏ͷ͔
A
B
C
Slide 10
Slide 10 text
©2018 Wantedly, Inc.
8BOUFEMZ1FPQMF͕Ϣʔβʹఏڙ͍ͯ͠ΔՁ
w ʮ໊ΛऔΓɺਓͱͭͳ͕Δʯ
w ໊ͱਓͷ͍͋ͩͷάϥϑΛετΞ͍ͨ͠
ԿނάϥϑσʔλϕʔεΛ͏ͷ͔
Slide 11
Slide 11 text
©2018 Wantedly, Inc.
طଘͷάϥϑσʔλϕʔεͷ
Slide 12
Slide 12 text
©2018 Wantedly, Inc.
ॻ͖ࠐΈͷੑೳ͕͔ͬͨ
w 3FBEຆͲͷάϥϑσʔλϕʔεͰߴ͍ੑೳΛग़͍ͯ͠Δ
w ໊ΛࡱΔɺͱ͍͏ߦҝ8BOUFEMZ1FPQMFͷίΞͳମݧ
w ຖຖඵେྔʹૹΒΕͯ͘Δ໊ΛετΞͰ͖Δ͚ͩͷ8SJUFͷύϑΥʔϚϯε͕ඞཁ
w ͜ΕΛຬͨ͢άϥϑσʔλϕʔεଘࡏ͠ͳ͔ͬͨ
طଘͷάϥϑσʔλϕʔεͷ
Slide 13
Slide 13 text
©2018 Wantedly, Inc.
εέʔϧ͠ͳ͍ߏ
• αʔϏεͷʹεέʔϧͰԠ͡ΒΕΔඞཁ͕͋Δ
w /FPKσʔλϕʔεଆͰͷσʔλͷγϟʔσΟϯάʹରԠ͍ͯ͠ͳ͍
w 0SJFOU%#$MBTTʢςʔϒϧʣຖʹγϟʔσΟϯάΛ͢Δ
w ͋Δςʔϒϧ͚ͩॻ͖ࠐΈ͕ܹ͍͠ɺΈ͍ͨͳέʔεʹରԠͰ͖ͳ͍
طଘͷάϥϑσʔλϕʔεͷ
Slide 14
Slide 14 text
©2018 Wantedly, Inc.
%HSBQIͷհ
Slide 15
Slide 15 text
©2018 Wantedly, Inc.
(Pͷࢄάϥϑσʔλϕʔε
w (SBQI2-ϥΠΫͳΫΤϦγϯλοΫε
w H31$ͱ)551ͷαϙʔτ
w ϋΠύϑΥʔϚϯε
%HSBQIͱ
Slide 16
Slide 16 text
©2018 Wantedly, Inc.
ॻ͖ࠐΈͷύϑΥʔϚϯε͕༏Ε͍ͯΔ
w ެࣜͷϕϯνϚʔΫ
w ಡΈॻ͖ڞʹ/FPKͷഒ
w 1FPQMFͷ̎ޙΛఆͨ͠σʔλྔ
εϧʔϓοτͰϕϯνϚʔΫΛ࣮ࢪ
w εέʔϧͳ͠Ͱ͑ΒΕͨ
%HSBQIͱ
https://blog.dgraph.io/post/benchmark-
Slide 17
Slide 17 text
©2018 Wantedly, Inc.
%HSBQIͷσʔλϞσϧ
w %HSBQIͰάϥϑϞσϧͱͯ͠3%'5SJQMFΛ࠾༻
w 8FCϦιʔεʹϝλใΛ༩͢ΔͨΊͷΈ
w ϦιʔεͷؔΛ4VCKFDU
1SFEJDBUF
0CKFDUͰදݱ
w 4VCKFDUࢧͱͳΔ/PEF
w 1SFEJDBUFؔΛද͢&EHF
w 0CKFDUऴͱͳΔ/PEF
%HSBQIͱ
.
.
.
.
.
RDF ٖࣅίʔυʹ͓͚ΔϑϨϯυϦϨʔγϣϯ
Slide 18
Slide 18 text
©2018 Wantedly, Inc.
3%'5SJQMF
Dgraphͱ
.
.
.
.
.
RDF ٖࣅίʔυʹ͓͚ΔϑϨϯυϦϨʔγϣϯ
C
Bob b
a Alice
friend named
==
Slide 19
Slide 19 text
©2018 Wantedly, Inc.
3%'5SJQMF
w QFSTPO͕ͳ͍
w 0CKFDU4VCKFDU&OUJUZͱ"UUSJCVUF͕͋Δ
w &OUJUZ6TFSͳͲͷ֓೦Λද͢
w "UUSJCVUFΛද͢
w ຊ࣭తʹϞσϦϯάʹ͕ؔͳ͍
w OBNFEQSFEJDBUF͕͋ΔͳΒͦΕQFSTPO
w ·ΔͰ%VDL5ZQJOH
Dgraphͱ
C
Bob b
a Alice
friend named
Slide 20
Slide 20 text
©2018 Wantedly, Inc.
%HSBQIͷσʔλߏ
Slide 21
Slide 21 text
©2018 Wantedly, Inc.
άϥϑ̎ͭͷσʔλߏʹΑͬͯදݱ͞Ε͍ͯΔ
w 1PTUJOH
w 1PTUJOH-JTU
%HSBQIͷσʔλߏ
Slide 22
Slide 22 text
©2018 Wantedly, Inc.
1PTUJOH
w %HSBQIͷσʔλͷ࠷খ୯Ґ
w 1SFEJDBUFͱͦΕʹඥͮ͘
4VCKFDUͱ0CKFDUΛแ͢Δ
w WBMVF@UZQF͕6*%
w 4VCKFDU&OUJUZͰ͋Δ
w ͦΕҎ֎"UUSJCVUF
w WBMVFʹ࣮σʔλ͕ೖΔ
%HSBQIͷσʔλߏ
message Posting {
fixed64 uid = 1;
bytes value = 2;
enum ValType {
DEFAULT = 0;
BINARY = 1;
INT = 2;
FLOAT = 3;
BOOL = 4;
DATETIME = 5;
GEO = 6;
UID = 7;
PASSWORD = 8;
STRING = 9;
}
ValType val_type = 3;
...
...
enum PostingType {
REF=0;
VALUE=1;
VALUE_LANG=2;
}
PostingType posting_type = 4;
bytes lang_tag = 5;
string label = 6;
repeated api.Facet facets = 9;
uint32 op = 12;
uint64 start_ts = 13;
uint64 commit_ts = 14;
}
Protobuf ʹΑΔ Posting ͷఆٛ
Slide 23
Slide 23 text
©2018 Wantedly, Inc.
1PTUJOH-JTU
w 1SFEJDBUFͱ4VCKFDUͷϖΞʹඥͮ͘1PTUJOHΛͻͱ·ͱΊʹͨ͠ͷ
w ͻͱ·ͱΊʹ͢Δ͜ͱͰɺ͋ΔϢʔβͷ༑ୡҰཡ
ΛҰճͷ୳ࡧͰऔಘͰ͖Δ
w ,74ͷLFZʹ1SFEJDBUFͱ4VCKFDUؚ͕·ΕΔ
w ͦͷҝ1PTUJOHʹ͕ࣗԿऀͰ͋Δ͔Λ
දݱ͢ΔGJFME͕ͳ͍
%HSBQIͷσʔλߏ
message PostingList {
repeated Posting postings = 1;
bytes checksum = 2;
uint64 commit = 3;
bytes uids = 4;
}
Protobuf ʹΑΔ Posting ͷఆٛ
(Predicate,Subject) -> PostingList
KVS ͷετΞํ๏
Slide 24
Slide 24 text
©2018 Wantedly, Inc.
#BEHFS
w %HSBQIͷ։ൃνʔϜ͕࡞ͨ͠(PͰॻ͔Εͨ,74
w ϋΠύϑΥʔϚϯε
w 3BOEPN3FBE'BDFCPPLͷ,74Ͱ͋Δ3PDLT%#ͷഒ
w 44%ͷ࠷దԽ
w "$*%5SBOTBDUJPOͷαϙʔτ
DdgraphͷσʔλετϨʔδ
Slide 25
Slide 25 text
©2018 Wantedly, Inc.
Կނ#BEHFSΛ࡞ͬͨͷ͔
w ͱͱ3PDLT%#Λ͍ͬͯͨ
w ͔͠͠$HPΛ͔ͭ͏σϝϦοτ͕େ͖͗ͨ͢
w (PͷΤίγεςϜʹΕͳ͍
w QQSPGSBDFEFUFDUPS
w QUISFBE@DSFBUFͰόάΛҾ͖ى͜͠ɺύϑΥʔϚϯε͕Լ͢Δ
w (Pͷࠒ
w ($͕ޮ͔ͳ͍ͷͰ$GSFF
ΛΕΔͱϝϞϦϦʔΫΛҾ͖ى͜͢
DdgraphͷσʔλετϨʔδ
Slide 26
Slide 26 text
©2018 Wantedly, Inc.
Կނ#BEHFSΛ࡞ͬͨͷ͔
w (PͰॻ͔Ε͍ͯΔ#PMU%#Ͳ͏ͩͬͨͷ͔
w #PMU%#3FBE8SJUFڞʹάϩʔόϧۭؒʹ͋ΔTZOD38.VUFYͷϩοΫΛͱΔ
w ฒྻॲཧ͕ग़དྷͳ͍
w 1PTUJOH-JTUͷมߋʹେ͖ͳӨڹΛ༩͑ͨ
DdgraphͷσʔλετϨʔδ
Slide 27
Slide 27 text
©2018 Wantedly, Inc.
#BEHFSͷΈ
Slide 28
Slide 28 text
©2018 Wantedly, Inc.
-PH4USVDUVSFE.FSHF5SFF
w ίϯηϓτͰ͖Δ͚ͩγʔΫΛগͳ͘͢Δ
w ॻ͖ࠐΈͷϑϩʔ
ॻ͖ࠐΈ͕͋ͬͨ߹8"-ͱϝϞϦʢ.FN5BCMFʣʹॻ͖ग़͢
ॻ͖ࠐΉͱ͖ʹ.FN5BCMFʹσʔλ͕ೖΒͳ͍߹ϑΝΠϧʹϑϥογϡ
γʔΫͷྔΛݮΒͨ͢ΊʹɺϑΝΠϧ͕ଟ͘ͳ͖ͬͯͨΒϑΝΠϧΛϚʔδͤ͞Δ
w $PNQBDUJPO
Storage architecture of Badger
Slide 29
Slide 29 text
©2018 Wantedly, Inc.
-PH4USVDUVSFE.FSHF5SFF
Add: (muni, 30)
Storage architecture of Badger
L W
Alice 10
Bob 20
Level 1
L W
Carol 30
Level 0 (Mem Table)
Slide 30
Slide 30 text
©2018 Wantedly, Inc.
-PH4USVDUVSFE.FSHF5SFF
Add: (muni, 30)
Storage architecture of Badger
L W
Alice 10
Bob 20
Level 1
L W
Carol 30
muni 30
Level 0 (Mem Table)
Slide 31
Slide 31 text
©2018 Wantedly, Inc.
-PH4USVDUVSFE.FSHF5SFF
Add: (muni, 30)
Storage architecture of Badger
L W
Alice 10
Bob 20
Level 1
L W
Level 0 (Mem Table)
L W
Carol 10
muni 20
Slide 32
Slide 32 text
©2018 Wantedly, Inc.
-PH4USVDUVSFE.FSHF5SFF
Set: (Alice, 20)
Storage architecture of Badger
L W
Alice 10
Bob 20
Level 1
L W
Level 0 (Mem Table)
L W
Carol 10
muni 20
Slide 33
Slide 33 text
©2018 Wantedly, Inc.
-PH4USVDUVSFE.FSHF5SFF
Set: (Alice, 20) ... will remove (Alice, 10) when merge
Storage architecture of Badger
L W
Alice 10
Bob 20
Level 1
L W
Alice 20
Level 0 (Mem Table)
L W
Carol 10
muni 20
Slide 34
Slide 34 text
©2018 Wantedly, Inc.
-PH4USVDUVSFE.FSHF5SFF
Add: (mike, 50)
Storage architecture of Badger
L W
Alice 10
Bob 20
Level 1
L W
Alice 20
Level 0 (Mem Table)
L W
Carol 10
muni 20
Slide 35
Slide 35 text
©2018 Wantedly, Inc.
-PH4USVDUVSFE.FSHF5SFF
Add: (mike, 50)
Storage architecture of Badger
L W
Alice 10
Bob 20
Level 1
L W
Alice 20
mike 50
Level 0 (Mem Table)
L W
Carol 10
muni 20
Slide 36
Slide 36 text
©2018 Wantedly, Inc.
-PH4USVDUVSFE.FSHF5SFF
Add: (mike, 50)
Storage architecture of Badger
L W
Alice 20
mike 50
Level 1
L W
Level 0 (Mem Table)
L W
Carol 10
muni 20
Bob 20
Level 2
Slide 37
Slide 37 text
©2018 Wantedly, Inc.
-PH4USVDUVSFE.FSHF5SFF
w Կ͕خ͍͠ʁ
w 8SJUF͕ΔͷϚʔδͷλΠϛϯά͚ͩ
w 8"-ॻ͍͍ͯΔ͚Ͳɺ࣮σʔλʹൺΕίετ
w ಡΈࠐΉͱ͖Ϩϕϧ͔Βॱʹߦ͏ͷͰɺஞ࣍σʔλΛߋ৽͢Δඞཁ͕ͳ͍
w ϑΝΠϧͷ͚ͩγʔΫ͢ΕΑ͍
w ϑΝΠϧʹͲͷ,FZؚ͕·Ε͍ͯΔͷ͔֬ೝ͢ΔͨΊͷ#MPPN'JMUFS͕͋Δ
w #MPPN'JMUFS͕GBMTFΛฦͨ͠߹ͦͷϑΝΠϧʹ,FZඞͣଘࡏ͠ͳ͍
Storage architecture of Badger
Slide 38
Slide 38 text
©2018 Wantedly, Inc.
"$*%5SBOTBDUJPO
w ࣮.VMUJ7FSTJPO$PODVSSFODZ$POUSPM
w 3FBEͨ͠ͱ͖ɺߦʹUJNFTUBNQΛ༩͢Δ
w ผͷτϥϯβΫγϣϯͰಉ͡ߦʹະίϛοτͷมߋ͕ͬͨ
w มߋϝϞϦʹରͯ͠ߦͳΘΕɺଞͷτϥϯβΫγϣϯ͔Βݟ͑ͳ͍
w ίϛοτͯ͠ߦͷ࣮σʔλͷUJNFTUBNQΛߋ৽
w τϥϯβΫγϣϯͰҰճ3FBEͨ͠ߦΛ͏Ұࢀর
w طʹUJNFTUBNQ͕༩͞Ε͍ͯΔͷͰલͷΛฦ͢
ACID Transaction of Badger
Slide 39
Slide 39 text
©2018 Wantedly, Inc.
%HSBQIͷεέʔϦϯά
Slide 40
Slide 40 text
©2018 Wantedly, Inc.
%HSBQIΛߏ͢Δίϯϙʔωϯτ
w [FSP
w Ϋϥελͷཧϊʔυ
w (SPVQʢޙड़ʣͷཧ
w όοΫΞοϓγϟοτμϯͱ໋͍ͬͨྩΛ(SPVQʹൖͤ͞Δ
w TFSWFS
w σʔλετΞɺΫΤϦͷ࣮ߦϊʔυ
w ̍ͭͷ(SPVQʹඞͣଐ͢Δ
%HSBQIͷεέʔϦϯά
Slide 41
Slide 41 text
©2018 Wantedly, Inc.
(SPVQ
w Ҏ্ͷTFSWFSʹΑͬͯߏ͞ΕΔ
w ಉ͡(SPVQʹଐ͢Δશͯͷϊʔυಉ͡σʔλΛ࣋ͭ
w 3BGUͷ2VPSVN
w γϟʔσΟϯάͷ୯Ґ
w (SPVQʹׂΓͯΒΕΔσʔλ1SFEJDBUFຖʹܾ·Δ
w ͭ·ΓGSJFOEʹؔ͢Δσʔλશͯಉ͡(SPVQͰཧ͞ΕΔ
w Ͳͷ1SFEJDBUFΛड͚͔࣋ͭ[FSPʹΑܾͬͯఆ͞ΕΔ
%HSBQIͷεέʔϦϯά
Slide 42
Slide 42 text
©2018 Wantedly, Inc.
w ͭͳ͕ΓΛอଘ͢ΔσʔλετΞͱͯ͠
άϥϑσʔλϕʔεΛ࠾༻
w %HSBQIطଘͷάϥϑσʔλϕʔε͕
๊͍͑ͯͨΛղܾ͍ͯ͠Δ
w ॻ͖ࠐΈͷύϑΥʔϚϯε
w ਫฏεέʔϦϯά
·ͱΊ