Redis for Database Fanatics

Presented at ConFoo.ca February 27, 2014

Most database lovers have heard of memcached for application-level data caching, but many haven't had the opportunity to play with Redis, a powerful in-memory data structure store.

In this presentation, we explore Redis data structure operations through a number of patterns: key-value storage, leaderboards, membership, counters and message queues. To keep things fair, we'll provide relational database versions for comparison as well.

Transcript

1. (for database fanatics) Sunny Gleason ConFoo February 27, 2014
2. whoami •distributed systems engineer
 sunnycloud - boston, ma •previous work @ Amazon, Ning •github: sunnygleason
 twitter: @sunnygleason
 speakerdeck: sunnygleason •don’t be a stranger!
3. what’s this all about? • Databases are awesome • They store and protect data • We need them to be faster • ... or to use different data 
 access patterns • So let’s sacrifice some durability for performance and nice APIs
4. agenda • What is redis? • … redis and Database Models • … redis API Overview • Sample Features implemented using redis
5. redis == fast data structures • Map / Dictionary: associate keys & values • ArrayList / Deque: lists supporting operations on both ends + random access • Heap / Priority Queue: sorted maps, 
 key/value + score
6. redis extras • “Transactions” : multi-operations with watch values • Lua Scripting : essentially stored procedures • Publish / Subscribe : channel subscriptions with real-time notifications
7. database recap source: http://www.cnblogs.com/fangwenyu/archive/2012/07/08/2581417.html
8. database recap • Buffer Pool • Pages live durably on-disk • Indexes sit in memory • Minor disk seeks for record retrieval • Major disk seeks for table scans • Limited by IOPS: i/o operations per second
9. in-memory model • All data stored in RAM • If swaps occur, very bad news • Can be spooled to disk periodically • Disk not recommended for normal operations - leads to false sense of security
10. why use redis? • Backend/Caching: page fragments, object cache, web sessions, ephemeral data from external or internal services • Site Features: Counters, Membership, Job/Message Queues, Leaderboards
11. redis KV API • GET [key] • SET [key] [value] • SETEX [key] [exp] [value] • DEL [key] • SCAN [cursor] • KEYS [pattern]
12. redis KV API • GET [key] : gets the value of a key • SET [key] [value] : sets the value of a key • SETEX [key] [exp] [value] : sets an expiring key/value • DEL [key] : removes a key/value pair • SCAN [cursor] : iterates over keys • KEYS [pattern] : list keys that match a pattern
13. redis string API • MGET [k1] [k2] … • MSET [k1] [v1] [k2] [v2] … • INCR [k1] / DECR [k1] • INCRBY [k1] [n] / DECRBY [k1] [n]
14. redis string API • MGET [k1] [k2] … : retrieve multiple values • MSET [k1] [v1] [k2] [v2] … : set multiple KV pairs • INCR [k1] / DECR [k1] : increment / decrement value • INCRBY [k1] [n] / DECRBY [k1] [n] : incr/decr by N
15. redis hash API • HGET / HSET / HDEL • HMGET / HMSET / HGETALL • HKEYS / HSCAN
16. redis hash API • HGET / HSET / HDEL : get / set / delete hash entry • HMGET / HMSET / HGETALL : multiget, multiset, get all entries of hash • HKEYS / HSCAN : list / traverse hash keys
17. redis list API • LPUSH / LPOP / LLEN / LREM / LRANGE • LSET • RPOPLPUSH
18. redis set API • SADD / SCARD / SISMEMBER • SDIFF / SINTER / SUNION
 + "STORE" variants
19. redis sorted set API • ZADD / ZCARD / ZCOUNT / ZRANGE • ZINCRBY • ZRANK • ZREV* - reverse variants
20. redis pubsub API • PUBLISH / SUBSCRIBE / UNSUBSCRIBE • PSUBSCRIBE / PUNSUBCRIBE
21. redis transactions • MULTI / EXEC • WATCH
22. redis scripting • SCRIPT LOAD • EVAL / EVALSHA
23. building stuff • Counters • Queues • Membership • Leader Boards
24. database counters (ok) • CREATE TABLE counters (
 int id PRIMARY KEY,
 object_id int,
 count int) • SELECT * FROM leaderboard
 WHERE object_id = ? • UPDATE object_id
 SET count = ?
 WHERE object_id = ?
25. database counters (BAD!) • SELECT count(*) 
 FROM collection_table 
 WHERE object_id = ?
26. redis counters • Use KV or Hash API • Key: page id • Value: count • Use INCR / INCRBY • Alternatively HINCR / HINCRBY for Hash API
27. database queue • CREATE TABLE queue (
 int id PRIMARY KEY,
 queue_id int,
 object_id int) • SELECT * FROM queue
 WHERE queue_id = ?
 ORDER BY id
 LIMIT 1 • INSERT INTO queue VALUES (?, ?, ?); • DELETE FROM queue
 WHERE id = ?
28. redis queue • Use the List API • Key: queue name • Value: list of entries • LPUSH [queue] [item]
 LPOP [queue] (for LIFO)
 RPOP [queue] (for FIFO) • BONUS! RPUSHLPOP : atomic transfer from one queue to another
29. database membership • CREATE TABLE members (
 int id PRIMARY KEY,
 group_id int,
 user_id int,
 UNIQUE KEY(group_id, user_id)) • SELECT * FROM members
 WHERE group_id = ?
 AND user_id = ? • INSERT INTO members VALUES (?, ?, ?);
30. redis membership • Use the Set API • Key: group name • Value: set of members • SADD [group] [user]
 SISMEMBER [group] [user] • BONUS! SUNION / SINTER / SDIFF
31. database leaderboard • CREATE TABLE leaderboard (
 int id PRIMARY KEY,
 board_id int,
 board_member_id int,
 score int) • SELECT * FROM leaderboard
 WHERE board_id = ?
 ORDER BY score
 LIMIT 100 • UPDATE leaderboard
 SET score = ?
 WHERE board_id = ?
 AND board_member_id = ?
32. redis leaderboard • Use the Sorted Set API • Key: leaderboard name • Value: priority queue • ZADD [board] [user] [score]
 ZINCRBY [board] [user] [n]
 ZRANGE [board] [m] [n] • ZREV* for reverse sort order
33. Publish / Subscribe • Real-time communications without polling • Machines talking to each other: status/performance • Humans talking to each other:
 chat/notifications
34. pubsub applications
35. pubsub applications • Chat / Presence / Notifications • Location / Asset Tracking (taxi/mobile) • Real-time Data Streams (stocks) • System Monitoring / Status / Alerts • Device Sharing & Synchronization (mobile)
36. redis pubsub • Single node, high performance • Very easy to get started • Adaptors for socket.io and atmosphere • Missing items: durability, scalability, presence, history, massive broadcast
37. scaling pubsub into the cloud • Global presence with over 14 data centers worldwide, < 250ms latency • Provide presence, history, storage, 
 playback, iOS notifications • Massive broadcast, millions of messages / sec • No additional back-end services - just HTML5 + JS • APIs in 20+ languages (Ruby, JS, Python, PHP, Java, C#, Erlang, Lua, …)
38. database conclusion • Databases aren't going anywhere • Most practical solution for primary storage of data • Know their limitations (performance & operational), so you can make tradeoffs accordingly • In addition, there are plenty of NoSQL solutions (please check out my talk tomorrow on "Supersize your Key- Value Store”!)
39. redis conclusion • Try redis for high-performance
 application needs • Use it as a throwaway store, and
 you’ll be happiest • Watch your applications become simpler,
 and faster!
40. questions? thank you!