Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up for free
LevelDB on S3 As A KVS
shunsukeaihara
June 02, 2015
Programming
1
2.3k
LevelDB on S3 As A KVS
shunsukeaihara
June 02, 2015
Tweet
Share
More Decks by shunsukeaihara
See All by shunsukeaihara
BONXを支える技術:発話区間検出(VAD)の話/Akerun & BONX Tech Talk
shunsukeaihara
4
6.6k
Goのnet.TCPConnの話/shibuya.go01
shunsukeaihara
2
610
Norikra in Gunosy Network Ads@Norikra meetup #2
shunsukeaihara
1
5.5k
色恒常性仮説に基づく色補正ライブラリcolorcorrect / 2015-01-31-kantocv27
shunsukeaihara
3
1.8k
Sparkによる分散処理 / 2015-01-16 PyData.Tokyo#3
shunsukeaihara
11
3k
ゼロから始めた Gunosyアドサーバ開発運用記 / 2014-12-16-dots
shunsukeaihara
6
1k
Gunosy.Go#5 index/io/log
shunsukeaihara
0
87
Gunosy.go#2 package/compress
shunsukeaihara
0
69
Other Decks in Programming
See All in Programming
Pythonで鉄道指向プログラミング
usabarashi
0
140
Git Rebase
bkuhlmann
7
1.1k
Introduction to Property-Based Testing @ COSCUP 2022
cybai
1
150
RustのWebフレームワーク周りの概観
hayao
0
190
Atomic Design とテストの○○な話
takfjp
2
830
料理の注文メニューの3D化への挑戦
hideg
0
300
Pluggable Storage in PostgreSQL
sira
1
200
Google IO 2022 社内LT会 / What's new in Android development tools
shingo_kobayashi
1
430
Now in Android Overview
aosa4054
1
420
How to Test Your Compose UI (Droidcon Berlin 2022)
stewemetal
1
130
테라폼으로 ECR 관리하기 (How to Manage ECR with Terraform)
posquit0
0
530
モデルの定義に基づくバリデーションを実現するためのpydantic入門
daikikatsuragawa
0
120
Featured
See All Featured
Navigating Team Friction
lara
175
11k
The Invisible Side of Design
smashingmag
290
48k
ピンチをチャンスに:未来をつくるプロダクトロードマップ #pmconf2020
aki_i
25
16k
Rebuilding a faster, lazier Slack
samanthasiow
62
7.3k
Robots, Beer and Maslow
schacon
152
7.2k
Building Better People: How to give real-time feedback that sticks.
wjessup
344
17k
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
danielanewman
212
20k
How GitHub Uses GitHub to Build GitHub
holman
465
280k
A Tale of Four Properties
chriscoyier
149
21k
5 minutes of I Can Smell Your CMS
philhawksworth
196
18k
Embracing the Ebb and Flow
colly
73
3.4k
Designing with Data
zakiwarfel
91
4k
Transcript
LevelDB'on'S3'As'A'KVS
Who$am$I • Shunsuke)Aihara(@shunsukeaihara) • ΞυͬͯΔ • ܭࢉݴޠֶ • ը૾ॲཧɾԻॲཧɾݴޠॲཧͷϥΠ ϒϥϦ͍Ζ͍Ζॻ͍ͯΔ
• h1ps:/ /bitbucket.org/aihara
ΞυαʔόΛϦϓϨΠεͨ͠ • Python(tornado)͔Βgo.+.kamiʹॻ͖͑ • όουϊϋΛ՚ྷʹશճආ
50ms%or%DieΛ્Ήͷ 1. Redis 2. Redis 3. Redis 4. GIL 5.
etc
ՄೳͳݶΓڞ༗DBΛ৮Βͳ͍ ΫϥελͷϝϯςͭΒ͍ RedisγϯάϧεϨου
ΞυḉͰRDB৮ΔౕԿΛͬͯμϝ
AWSͰҰ൪҆ఆ͍ͯ҆͠ࢄDB?
S3
S3ͷ͍υίϩ • S3Λͦͷ··KVSͱͯ͠͏ • ผͷαʔϏεͰ࣮ࢪ&/&݁ߏεέʔϧ • shard&keyͱ͔ࢦఆͰ͖Δ͚ͲΞυαʔόͰϨΠςϯγͷ ͕ • S3্ʹϑΝΠϧDBΛஔ
• ϑΝΠϧDBΛόϥϚΩ
LevelDBΛS3ʹஔ • Norikra(+(Redshi/(+(PythonΫϥελͰ౷ܭใɺλʔήςΟ ϯάใΛ͍ΖΜͳϥΠϑαΠΫϧͰߋ৽ • MessagepackܗࣜͰɺLevelDBʹ์ΓࠐΉ • h=ps:/ /github.com/shunsukeaihara/goDleveldbDobjectstorage/ •
֎෦ެ։൛
goଆ࣮ • golevelΛར༻ • ىಈ࣌ʹLevelDBΛμϯϩʔυ • λΠϚʔͰఆظతʹ࠶μϯϩʔυ • ҰͭͷLevelDBϑΝΠϧͷΞΫηεҰͭͷGorou-neͰ •
ϑΝΠϧͷચ͍ସ͑ͳΜ͔ಛʹؾʹ͠ͳ͍
CommandύλʔϯͬΆ͍࣮ • get(unpack)/set/expireͱ͔ • ΑΓChannelΛ༻͍ͨՄಡੑΛ • MessagepackΛUnpackͨ͠σʔλΦϯϝϞϦͰΩϟογϡ • RedisͬΆ͍ExpireΛ࣮ •
Expire:merͱChannelͰ࣮
gcTick := time.NewTicker(10 * time.Second) defer gcTick.Stop() updateTick := time.NewTicker(time.Duration(ldb.dbConf.Options.UpdateInterval)
* time.Second) defer updateTick.Stop() for { select { case cmd := <-ldb.get: cmd.result <- ldb.execGet(cmd) case <-ldb.reset: ldb.cache = map[string]*cachedItem{} case <-updateTick.C: ldb.mu.RLock() if !ldb.downloading { go ldb.download(ctx, true) } ldb.mu.RUnlock() case msg := <-ldb.switchDB: oldDB := ldb.db oldPath := ldb.dbpath ldb.db = msg.db ldb.dbpath = msg.dbpath oldDB.Close() os.RemoveAll(oldPath) case <-ldb.expire: ldb.simpleExpire(1) case <-gcTick.C: ldb.simpleExpire(1) case msg := <-ldb.exit: ldb.db.Close() os.RemoveAll(ldb.dbpath) ldb.cache = map[string]*cachedItem{} msg <- struct{}{} break } }
IFͱ͔ • ςΩτʹΦϨΦϨDSL͔ΒδΣωͬͯΔ func GetCandidate(ctx context.Context, key string) ([]*data.Candidate, bool)
{ ldb, ok := levelDBFromContext(ctx, candidateDBKey) if !ok { panic("no candidate db in context") } res, ok, _ := ldb.getCandidate(key) return res, ok } func (ldb *LevelDB) getCandidate(key string) ([]*data.Candidate, bool, bool) { cmd := NewDBGetCmd(key, data.UnmarshalCandidates, int64(ldb.dbConf.Options.CacheExpire)) ldb.get <- cmd r := <-cmd.result if !r.ok { return nil, false, false } return r.val.([]*data.Candidate), r.ok, r.hit }
ͦΕͰRedisɾMemcache͕ඞཁͳॴ…… • Redisࣗମਨࢄ/ਫฏࢄ • RedisΫϥελʹରͯ͠LevelDBΞΫηεͱಉ༷ɺػೳຖʹ 1Goru2neΛׂΓͯ • MessagepackΛUnpackͨ͠σʔλΩϟογϡ • ͬͪ͜ʹRedisͬΆ͍expireઓུΛ࣮
݁Ռ • ςΩτʹॻ͍ͯेૣ͍ • PythonͰॻ͍͍ͯͨαʔόͷ8ഒͷεϧʔϓοτ • 10ms-or-Die͙Β͍ͷͰಈ࡞ • ܕ͕༗Δͷେخ͍͠ͱ͍͏ؾ࣋ͪ •
ίʔυҾ͖ܧ͗࣌ͷܕͷ͋Γ͕ͨ͞