Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Redis Hacks
Search
David Cramer
May 03, 2014
Technology
3
240
Redis Hacks
Python Nordeste 2014 - Lightning Talk
David Cramer
May 03, 2014
Tweet
Share
More Decks by David Cramer
See All by David Cramer
Mastering Duct Tape (PyCon Balkan 2018)
zeeg
2
890
Open Source as a Business (PyCon SG 2014)
zeeg
0
380
Angular.js Workshop (PyCon SG 2014)
zeeg
0
250
Architecting a Culture of Quality
zeeg
2
320
Release Faster
zeeg
12
1.4k
Open Source as a Business (EuroPython 2013)
zeeg
18
17k
Building to Scale (PyCon TW 2013)
zeeg
18
1.3k
Building to Scale
zeeg
28
24k
Lessons in Testing - DjangoCon 2012
zeeg
8
1.4k
Other Decks in Technology
See All in Technology
品質保証に注目したAIプロダクト開発
sansantech
PRO
1
140
Wantedlyの開発組織における生成AIの浸透プロジェクトについて
kotominaga
2
190
組織を巻き込む大規模プラットフォーム移行戦略 〜50+サービスのマルチリージョン・マルチプロダクト化で学んだステークホルダー協働の実践〜 / Platform migration strategy engaging all stakeholders
toshi0607
2
450
LINEヤフーのQA組織について
lycorptech_jp
PRO
0
390
測りにくい成果を測る — BtoB SaaSにおける効果検証への挑戦 / Shirokane Kougyou vol 20
sansan_randd
3
480
AIエージェントで90秒の広告動画を制作!台本・音声・映像・編集をつなぐAWS最新アーキテクチャの実践
nasuvitz
3
520
slog.Handlerのよくある実装ミス
sakiengineer
5
600
そのJavaScript、V8が泣いてます。V8の気持ちを理解して書くパフォーマンス最適化
riyaamemiya
15
5.9k
サーバなしで対戦ゲームが作れる!? 純正フレームワークで実現するリアルタイム通信
kuromelon257
0
600
メルカリIBISの紹介
0gm
1
1.2k
AIフレンドリーなコードベースを目指して/登壇資料(高橋 悟生)
hacobu
PRO
3
580
エンジニアがデザインまで担うための AI駆動UIデザイン/フロントエンド開発実践
kitami
4
770
Featured
See All Featured
Six Lessons from altMBA
skipperchong
28
4k
ピンチをチャンスに:未来をつくるプロダクトロードマップ #pmconf2020
aki_iinuma
127
53k
10 Git Anti Patterns You Should be Aware of
lemiorhan
PRO
657
61k
GraphQLとの向き合い方2022年版
quramy
49
14k
Speed Design
sergeychernyshev
32
1.1k
Keith and Marios Guide to Fast Websites
keithpitt
411
22k
The World Runs on Bad Software
bkeepers
PRO
71
11k
Learning to Love Humans: Emotional Interface Design
aarron
274
40k
The Straight Up "How To Draw Better" Workshop
denniskardys
237
140k
How To Stay Up To Date on Web Technology
chriscoyier
791
250k
The Art of Programming - Codeland 2020
erikaheidi
56
13k
Typedesign – Prime Four
hannesfritz
42
2.8k
Transcript
David Cramer twitter.com/zeeg Redis Hacks (or “How Sentry Scales”)
Buffering Writes
r = Redis() ! def incr(type, id): key = 'pending:{}'.format(type)
! r.zincrby(key, id, 1)
r = Redis() ! def flush(type): key = 'pending:{}'.format(type) result
= r.zrange(key, 0, -1, withscores=True) ! for id, count in result: prms = {'type': type, 'count': count, 'id': id} ! sql(""" update %(type)s set count = count + % (count)d where id = %(id)s """, prms)
Rate Limiting
r = Redis() ! def process_hit(project_id): epoch = time() /
60 key = ‘{}:{}’.format(project_id, epoch) ! pipe = r.pipeline() pipe.incr(key) pipe.expire(key, 60) result = pipe.execute() ! # return current value return int(result[0])
def request(project_id): result = process_hit(project_id) if result > 20: return
Response(status=429) return Response(status=200)
Time Series Data
def count_hits_today(project_id): start = time() end = now + DAY_SECONDS
! pipe = r.pipeline() for epoch in xrange(now, end, 10): key = ‘{}:{}’.format( project_id, epoch) pipe.get(key) results = pipe.execute() ! # remove non-zero results results = filter(bool, results) # coerce remainder to ints results = map(int, results) # return sum of buckets return sum(results)
Good-enough Locks
from contextlib import contextmanager ! r = Redis() ! @contextmanager
def lock(key, nowait=True): while not r.setnx(key, '1'): if nowait: raise Locked('try again soon!') sleep(0.01) ! # limit lock time to 10 seconds r.expire(key, 10) ! # do something crazy yield ! # explicitly unlock r.delete(key)
def do_something_crazy(): with lock('crazy'): print 'Hello World!'
Basic Sharding via Nydus
from nydus.db import create_cluster ! redis = create_cluster({ 'backend': 'nydus.db.backends.redis.Redis',
'hosts': { 0: {'db': 0}, 1: {'db': 1}, }, 'router': 'nydus.db.routers.keyvalue.PartitionRouter', })
def count_hits_today(project_id): start = time() end = now + DAY_SECONDS
! keys = [] for epoch in xrange(now, end, 10): key = '{}:{}'.format(project_id, epoch) keys.append(key) ! with redis.map() as conn: results = map(conn.get, keys) ! # remove non-zero results results = filter(bool, results) # coerce remainder to ints results = map(int, results) # return sum of buckets return sum(results)