Redis (for database fanatics) REVISITED!

Most database lovers have heard of memcached for application-level data caching, but many haven't had the opportunity to play with Redis, a powerful in-memory data structure store. In this presentation, we explore Redis data structure operations through a number of patterns: key-value storage, leaderboards, membership, counters and message queues. To keep things fair, we'll provide database versions for comparison as well.

Transcript

1. (for database fanatics) Sunny Gleason ConFoo February 27, 2014 2019/03/14 (Happy Day!) REVISITED!
2. whoami • Sunny Gleason
 @sunnygleason, sunnygleason 
 SunnyCloud, Boston MA • Application Development (Web & Mobile)
 Distributed Systems (Cloud)
 Security & Compliance • Prior Work • Independent developer/army of one…
3. what’s this all about? • Databases are awesome • They store and protect data • We need them to be faster • ... or to use different data 
 access patterns • So let’s sacrifice some durability for performance and nice APIs
4. agenda • What is redis? • … redis and Database Models • … redis API Overview • Sample Features implemented using redis
5. redis == fast data structures • Map / Dictionary: associate keys & values • ArrayList / Deque: lists supporting operations on both ends + random access • Heap / Priority Queue: sorted maps, 
 key/value + score
6. redis extras • “Transactions” : multi-operations with watch values • Lua Scripting : a.k.a. stored procedures • Publish / Subscribe : channel subscriptions with real-time notifications • Streams : data structures for log-like data
7. database recap source: http://www.cnblogs.com/fangwenyu/archive/2012/07/08/2581417.html
8. database recap • Buffer Pool • Pages live durably on-disk • Indexes sit in memory • Minor disk seeks for record retrieval • Major disk seeks for table scans • Limited by IOPS: i/o operations per second
9. in-memory model • All data stored in RAM • If swaps occur, very bad news • Can be spooled to disk periodically • Disk not recommended for normal operations - leads to false sense of security
10. why use redis? • Backend/Caching: page fragments, object cache, web sessions, ephemeral data from external or internal services • Site Features: Counters, Membership, Job/Message Queues, Leaderboards
11. why use redis? Requests/sec 0.00 450000.00 900000.00 1350000.00 1800000.00 PING_INLINE PING_BULK SET GET INCR LPUSH RPUSH LPOP RPOP TCP LOCALHOST UNIX DOMAIN SOCKET $ redis-benchmark -q -n 2000000 -c 1000 -P 40 $ redis-benchmark -q -n 2000000 -c 1000 -P 40 -s /var/run/redis/redis-server.sock
12. why not use redis? • Durability : the recovery/failover processes are just ok, not fantastic • Multiprocessing : each redis process is (approximately) single-threaded • Transactions and semantics of other operations (sets, etc.) get tricky under sharding models
13. why not use redis? https://ungleich.ch/en-us/cms/blog/2018/12/24/1a-who-does-not-need-be-afraid-oom-killer/
14. redis KV API • GET [key] • SET [key] [value] • SETEX [key] [exp] [value] • DEL [key] • SCAN [cursor] • KEYS [pattern] https://redis.io/commands#generic
15. redis KV API • GET [key] : gets the value of a key • SET [key] [value] : sets the value of a key • SETEX [key] [exp] [value] : sets an expiring key/value • DEL [key] : removes a key/value pair • SCAN [cursor] : iterates over keys • KEYS [pattern] : list keys that match a pattern https://redis.io/commands#generic
16. redis KV API - Pro Tip • SET / EXPIRE, or SETEX are useful for giving entries finite lifetimes • Super useful for ensuring redis store doesn’t grow without bound • Configure your redis memory limit and eviction policy accordingly https://redis.io/commands#generic
17. redis string API • MGET [k1] [k2] … • MSET [k1] [v1] [k2] [v2] … • INCR [k1] / DECR [k1] • INCRBY [k1] [n] / DECRBY [k1] [n] https://redis.io/commands#string
18. redis string API • MGET [k1] [k2] … : retrieve multiple values (!!!) • MSET [k1] [v1] [k2] [v2] … : set many KV pairs (!!!) • INCR [k1] / DECR [k1] : increment / decrement value (!!!) • INCRBY [k1] [n] / DECRBY [k1] [n] : inc/dec by N (!!!) https://redis.io/commands#string
19. redis string API - Pro Tip • Use MGET/MSET to reduce operation count! • Lodash has the zipObject operation to turn separate key and value arrays into a unified object https://redis.io/commands#string
20. redis hash API • HGET / HSET / HDEL • HMGET / HMSET / HGETALL • HKEYS / HSCAN https://redis.io/commands#hash
21. redis hash API • HGET / HSET / HDEL : get / set / delete hash entry • HMGET / HMSET / HGETALL : multiget, multiset, get all entries of hash • HKEYS / HSCAN : list / traverse hash keys https://redis.io/commands#hash
22. redis hash API - Pro Tip https://redis.io/commands#hash • In Node.js, HGETALL returns a JSON-like object • Use hash API plus HGETALL for objects that have independently- updated parts…
23. redis list API • LPUSH / LPOP / LLEN / LREM / LRANGE • LSET • RPOPLPUSH https://redis.io/commands#list
24. redis list API - Pro Tip https://redis.io/commands#list • LLEN & LRANGE are great for paginated collections • The parameters “0” and “99” are the offset and limit to retrieve • LLEN gives the total list length
25. redis list API - Pro Tip 2 • Queue “Drain”: Use a redis transaction to get the “left range” and “left trim” it as part of the same operation • Other clients will see a consistent view of the list before and after
26. redis set API • SADD / SCARD / SISMEMBER • SDIFF / SINTER / SUNION
 + "STORE" variants https://redis.io/commands#set
27. redis set API - Pro Tip https://redis.io/commands#set • Use SDIFF to compare progress between two workers • Similarly, SINTER gives common set
28. redis sorted set API • ZADD / ZCARD / ZCOUNT / ZRANGE • ZINCRBY • ZRANK • ZREV* - reverse variants https://redis.io/commands#sorted_set
29. redis pubsub API • PUBLISH / SUBSCRIBE / UNSUBSCRIBE • PSUBSCRIBE / PUNSUBCRIBE https://redis.io/commands#pubsub
30. redis pubsub API - Pro Tip https://redis.io/commands#pubsub • redis PubSub is great for quick publish/ suscribe prototyping with WebSockets • Be careful as message volume and number of subscribers increase
31. redis transactions • MULTI / EXEC (!!!) • WATCH https://redis.io/commands#transactions
32. redis scripting • SCRIPT LOAD • EVAL / EVALSHA
33. building stuff • Counters • Queues • Membership • Leader Boards
34. database counters (ok) • CREATE TABLE counters (
 int id PRIMARY KEY,
 object_id int,
 count int) • SELECT * FROM leaderboard
 WHERE object_id = ? • UPDATE object_id
 SET count = ?
 WHERE object_id = ?
35. database counters (BAD!) • SELECT count(*) 
 FROM collection_table 
 WHERE object_id = ?
36. redis counters • Use KV or Hash API • Key: page id • Value: count • Use INCR / INCRBY • Alternatively HINCR / HINCRBY for Hash API
37. database queue • CREATE TABLE queue (
 int id PRIMARY KEY,
 queue_id int,
 object_id int) • SELECT * FROM queue
 WHERE queue_id = ?
 ORDER BY id
 LIMIT 1 • INSERT INTO queue VALUES (?, ?, ?); • DELETE FROM queue
 WHERE id = ?
38. redis queue • Use the List API • Key: queue name • Value: list of entries • LPUSH [queue] [item]
 LPOP [queue] (for LIFO)
 RPOP [queue] (for FIFO) • BONUS! RPUSHLPOP : atomic transfer from one queue to another
39. database membership • CREATE TABLE members (
 int id PRIMARY KEY,
 group_id int,
 user_id int,
 UNIQUE KEY(group_id, user_id)) • SELECT * FROM members
 WHERE group_id = ?
 AND user_id = ? • INSERT INTO members VALUES (?, ?, ?);
40. redis membership • Use the Set API • Key: group name • Value: set of members • SADD [group] [user]
 SISMEMBER [group] [user] • BONUS! SUNION / SINTER / SDIFF
41. database leaderboard • CREATE TABLE leaderboard (
 int id PRIMARY KEY,
 board_id int,
 board_member_id int,
 score int) • SELECT * FROM leaderboard
 WHERE board_id = ?
 ORDER BY score
 LIMIT 100 • UPDATE leaderboard
 SET score = ?
 WHERE board_id = ?
 AND board_member_id = ?
42. redis leaderboard • Use the Sorted Set API • Key: leaderboard name • Value: priority queue • ZADD [board] [user] [score]
 ZINCRBY [board] [user] [n]
 ZRANGE [board] [m] [n] • ZREV* for reverse sort order
43. Publish / Subscribe • Real-time communications without polling • Machines talking to each other: status/performance • Humans talking to each other:
 chat/notifications
44. pubsub applications
45. pubsub applications • Chat / Presence / Notifications • Location / Asset Tracking (taxi/mobile) • Real-time Data Streams (stocks) • System Monitoring / Status / Alerts • Device Sharing & Synchronization (mobile)
46. redis pubsub • Single node, high performance • Very easy to get started • Adaptors for socket.io and atmosphere • Missing items: durability, scalability, presence, history, massive broadcast
47. redis client libraries • Node.js : node_redis • Java : jedis, lettuce • Scala : scala-redis • C#/.NET : StackExchange.Redis, ServiceStack.Redis • PHP : phpredis • Ruby : redis-rb • Go : redigo • PHP : phpredis • Rust : redis-rs • And many more… ! https://redis.io/clients
48. database conclusion • We love Databases, hopefully they aren't going anywhere • Most practical solution for primary storage of data • Know their limitations (performance & operational), so you can make tradeoffs accordingly • In addition, be aware of relevant NoSQL solutions: ElasticSearch & others…
49. redis conclusion • Try redis for high-performance
 application needs • Use it as a throwaway store, and
 you’ll be happiest • Watch your applications become simpler,
 and faster!
50. questions? thank you!