Redis for Database Fanatics

Slide 1

Slide 1 text

(for database fanatics) Sunny Gleason ConFoo February 27, 2014

Slide 2

Slide 2 text

whoami •distributed systems engineer  sunnycloud - boston, ma •previous work @ Amazon, Ning •github: sunnygleason  twitter: @sunnygleason  speakerdeck: sunnygleason •don’t be a stranger!

Slide 3

Slide 3 text

what’s this all about? • Databases are awesome • They store and protect data • We need them to be faster • ... or to use different data   access patterns • So let’s sacriﬁce some durability for performance and nice APIs

Slide 4

Slide 4 text

agenda • What is redis? • … redis and Database Models • … redis API Overview • Sample Features implemented using redis

Slide 5

Slide 5 text

redis == fast data structures • Map / Dictionary: associate keys & values • ArrayList / Deque: lists supporting operations on both ends + random access • Heap / Priority Queue: sorted maps,   key/value + score

Slide 6

Slide 6 text

redis extras • “Transactions” : multi-operations with watch values • Lua Scripting : essentially stored procedures • Publish / Subscribe : channel subscriptions with real-time notiﬁcations

Slide 7

Slide 7 text

database recap source: http://www.cnblogs.com/fangwenyu/archive/2012/07/08/2581417.html

Slide 8

Slide 8 text

database recap • Buffer Pool • Pages live durably on-disk • Indexes sit in memory • Minor disk seeks for record retrieval • Major disk seeks for table scans • Limited by IOPS: i/o operations per second

Slide 9

Slide 9 text

in-memory model • All data stored in RAM • If swaps occur, very bad news • Can be spooled to disk periodically • Disk not recommended for normal operations - leads to false sense of security

Slide 10

Slide 10 text

why use redis? • Backend/Caching: page fragments, object cache, web sessions, ephemeral data from external or internal services • Site Features: Counters, Membership, Job/Message Queues, Leaderboards

Slide 11

Slide 11 text

redis KV API • GET [key] • SET [key] [value] • SETEX [key] [exp] [value] • DEL [key] • SCAN [cursor] • KEYS [pattern]

Slide 12

Slide 12 text

redis KV API • GET [key] : gets the value of a key • SET [key] [value] : sets the value of a key • SETEX [key] [exp] [value] : sets an expiring key/value • DEL [key] : removes a key/value pair • SCAN [cursor] : iterates over keys • KEYS [pattern] : list keys that match a pattern

Slide 13

Slide 13 text

redis string API • MGET [k1] [k2] … • MSET [k1] [v1] [k2] [v2] … • INCR [k1] / DECR [k1] • INCRBY [k1] [n] / DECRBY [k1] [n]

Slide 14

Slide 14 text

redis string API • MGET [k1] [k2] … : retrieve multiple values • MSET [k1] [v1] [k2] [v2] … : set multiple KV pairs • INCR [k1] / DECR [k1] : increment / decrement value • INCRBY [k1] [n] / DECRBY [k1] [n] : incr/decr by N

Slide 15

Slide 15 text

redis hash API • HGET / HSET / HDEL • HMGET / HMSET / HGETALL • HKEYS / HSCAN

Slide 16

Slide 16 text

redis hash API • HGET / HSET / HDEL : get / set / delete hash entry • HMGET / HMSET / HGETALL : multiget, multiset, get all entries of hash • HKEYS / HSCAN : list / traverse hash keys

Slide 17

Slide 17 text

redis list API • LPUSH / LPOP / LLEN / LREM / LRANGE • LSET • RPOPLPUSH

Slide 18

Slide 18 text

redis set API • SADD / SCARD / SISMEMBER • SDIFF / SINTER / SUNION  + "STORE" variants

Slide 19

Slide 19 text

redis sorted set API • ZADD / ZCARD / ZCOUNT / ZRANGE • ZINCRBY • ZRANK • ZREV* - reverse variants

Slide 20

Slide 20 text

redis pubsub API • PUBLISH / SUBSCRIBE / UNSUBSCRIBE • PSUBSCRIBE / PUNSUBCRIBE

Slide 21

Slide 21 text

redis transactions • MULTI / EXEC • WATCH

Slide 22

Slide 22 text

redis scripting • SCRIPT LOAD • EVAL / EVALSHA

Slide 23

Slide 23 text

building stuff • Counters • Queues • Membership • Leader Boards

Slide 24

Slide 24 text

database counters (ok) • CREATE TABLE counters (  int id PRIMARY KEY,  object_id int,  count int) • SELECT * FROM leaderboard  WHERE object_id = ? • UPDATE object_id  SET count = ?  WHERE object_id = ?

Slide 25

Slide 25 text

database counters (BAD!) • SELECT count(*)   FROM collection_table   WHERE object_id = ?

Slide 26

Slide 26 text

redis counters • Use KV or Hash API • Key: page id • Value: count • Use INCR / INCRBY • Alternatively HINCR / HINCRBY for Hash API

Slide 27

Slide 27 text

database queue • CREATE TABLE queue (  int id PRIMARY KEY,  queue_id int,  object_id int) • SELECT * FROM queue  WHERE queue_id = ?  ORDER BY id  LIMIT 1 • INSERT INTO queue VALUES (?, ?, ?); • DELETE FROM queue  WHERE id = ?

Slide 28

Slide 28 text

redis queue • Use the List API • Key: queue name • Value: list of entries • LPUSH [queue] [item]  LPOP [queue] (for LIFO)  RPOP [queue] (for FIFO) • BONUS! RPUSHLPOP : atomic transfer from one queue to another

Slide 29

Slide 29 text

database membership • CREATE TABLE members (  int id PRIMARY KEY,  group_id int,  user_id int,  UNIQUE KEY(group_id, user_id)) • SELECT * FROM members  WHERE group_id = ?  AND user_id = ? • INSERT INTO members VALUES (?, ?, ?);

Slide 30

Slide 30 text

redis membership • Use the Set API • Key: group name • Value: set of members • SADD [group] [user]  SISMEMBER [group] [user] • BONUS! SUNION / SINTER / SDIFF

Slide 31

Slide 31 text

database leaderboard • CREATE TABLE leaderboard (  int id PRIMARY KEY,  board_id int,  board_member_id int,  score int) • SELECT * FROM leaderboard  WHERE board_id = ?  ORDER BY score  LIMIT 100 • UPDATE leaderboard  SET score = ?  WHERE board_id = ?  AND board_member_id = ?

Slide 32

Slide 32 text

redis leaderboard • Use the Sorted Set API • Key: leaderboard name • Value: priority queue • ZADD [board] [user] [score]  ZINCRBY [board] [user] [n]  ZRANGE [board] [m] [n] • ZREV* for reverse sort order

Slide 33

Slide 33 text

Publish / Subscribe • Real-time communications without polling • Machines talking to each other: status/performance • Humans talking to each other:  chat/notiﬁcations

Slide 34

Slide 34 text

pubsub applications

Slide 35

Slide 35 text

pubsub applications • Chat / Presence / Notiﬁcations • Location / Asset Tracking (taxi/mobile) • Real-time Data Streams (stocks) • System Monitoring / Status / Alerts • Device Sharing & Synchronization (mobile)

Slide 36

Slide 36 text

redis pubsub • Single node, high performance • Very easy to get started • Adaptors for socket.io and atmosphere • Missing items: durability, scalability, presence, history, massive broadcast

Slide 37

Slide 37 text

scaling pubsub into the cloud • Global presence with over 14 data centers worldwide, < 250ms latency • Provide presence, history, storage,   playback, iOS notiﬁcations • Massive broadcast, millions of messages / sec • No additional back-end services - just HTML5 + JS • APIs in 20+ languages (Ruby, JS, Python, PHP, Java, C#, Erlang, Lua, …)

Slide 38

Slide 38 text

database conclusion • Databases aren't going anywhere • Most practical solution for primary storage of data • Know their limitations (performance & operational), so you can make tradeoffs accordingly • In addition, there are plenty of NoSQL solutions (please check out my talk tomorrow on "Supersize your Key- Value Store”!)

Slide 39

Slide 39 text

redis conclusion • Try redis for high-performance  application needs • Use it as a throwaway store, and  you’ll be happiest • Watch your applications become simpler,  and faster!

Slide 40

Slide 40 text

questions? thank you!