Redis (for database fanatics) REVISITED!

Redis (for database fanatics) REVISITED!

Most database lovers have heard of memcached for application-level data caching, but many haven't had the opportunity to play with Redis, a powerful in-memory data structure store. In this presentation, we explore Redis data structure operations through a number of patterns: key-value storage, leaderboards, membership, counters and message queues. To keep things fair, we'll provide database versions for comparison as well.

Ac1a1d449af259bfb7191105db5212a2?s=128

Sunny Gleason

March 14, 2019
Tweet

Transcript

  1. (for database fanatics) Sunny Gleason ConFoo February 27, 2014 2019/03/14

    (Happy Day!) REVISITED!
  2. whoami • Sunny Gleason
 @sunnygleason, sunnygleason 
 SunnyCloud, Boston MA

    • Application Development (Web & Mobile)
 Distributed Systems (Cloud)
 Security & Compliance • Prior Work • Independent developer/army of one…
  3. what’s this all about? • Databases are awesome • They

    store and protect data • We need them to be faster • ... or to use different data 
 access patterns • So let’s sacrifice some durability for performance and nice APIs
  4. agenda • What is redis? • … redis and Database

    Models • … redis API Overview • Sample Features implemented using redis
  5. redis == fast data structures • Map / Dictionary: associate

    keys & values • ArrayList / Deque: lists supporting operations on both ends + random access • Heap / Priority Queue: sorted maps, 
 key/value + score
  6. redis extras • “Transactions” : multi-operations with watch values •

    Lua Scripting : a.k.a. stored procedures • Publish / Subscribe : channel subscriptions with real-time notifications • Streams : data structures for log-like data
  7. database recap source: http://www.cnblogs.com/fangwenyu/archive/2012/07/08/2581417.html

  8. database recap • Buffer Pool • Pages live durably on-disk

    • Indexes sit in memory • Minor disk seeks for record retrieval • Major disk seeks for table scans • Limited by IOPS: i/o operations per second
  9. in-memory model • All data stored in RAM • If

    swaps occur, very bad news • Can be spooled to disk periodically • Disk not recommended for normal operations - leads to false sense of security
  10. why use redis? • Backend/Caching: page fragments, object cache, web

    sessions, ephemeral data from external or internal services • Site Features: Counters, Membership, Job/Message Queues, Leaderboards
  11. why use redis? Requests/sec 0.00 450000.00 900000.00 1350000.00 1800000.00 PING_INLINE

    PING_BULK SET GET INCR LPUSH RPUSH LPOP RPOP TCP LOCALHOST UNIX DOMAIN SOCKET $ redis-benchmark -q -n 2000000 -c 1000 -P 40 $ redis-benchmark -q -n 2000000 -c 1000 -P 40 -s /var/run/redis/redis-server.sock
  12. why not use redis? • Durability : the recovery/failover processes

    are just ok, not fantastic • Multiprocessing : each redis process is (approximately) single-threaded • Transactions and semantics of other operations (sets, etc.) get tricky under sharding models
  13. why not use redis? https://ungleich.ch/en-us/cms/blog/2018/12/24/1a-who-does-not-need-be-afraid-oom-killer/

  14. redis KV API • GET [key] • SET [key] [value]

    • SETEX [key] [exp] [value] • DEL [key] • SCAN [cursor] • KEYS [pattern] https://redis.io/commands#generic
  15. redis KV API • GET [key] : gets the value

    of a key • SET [key] [value] : sets the value of a key • SETEX [key] [exp] [value] : sets an expiring key/value • DEL [key] : removes a key/value pair • SCAN [cursor] : iterates over keys • KEYS [pattern] : list keys that match a pattern https://redis.io/commands#generic
  16. redis KV API - Pro Tip • SET / EXPIRE,

    or SETEX are useful for giving entries finite lifetimes • Super useful for ensuring redis store doesn’t grow without bound • Configure your redis memory limit and eviction policy accordingly https://redis.io/commands#generic
  17. redis string API • MGET [k1] [k2] … • MSET

    [k1] [v1] [k2] [v2] … • INCR [k1] / DECR [k1] • INCRBY [k1] [n] / DECRBY [k1] [n] https://redis.io/commands#string
  18. redis string API • MGET [k1] [k2] … : retrieve

    multiple values (!!!) • MSET [k1] [v1] [k2] [v2] … : set many KV pairs (!!!) • INCR [k1] / DECR [k1] : increment / decrement value (!!!) • INCRBY [k1] [n] / DECRBY [k1] [n] : inc/dec by N (!!!) https://redis.io/commands#string
  19. redis string API - Pro Tip • Use MGET/MSET to

    reduce operation count! • Lodash has the zipObject operation to turn separate key and value arrays into a unified object https://redis.io/commands#string
  20. redis hash API • HGET / HSET / HDEL •

    HMGET / HMSET / HGETALL • HKEYS / HSCAN https://redis.io/commands#hash
  21. redis hash API • HGET / HSET / HDEL :

    get / set / delete hash entry • HMGET / HMSET / HGETALL : multiget, multiset, get all entries of hash • HKEYS / HSCAN : list / traverse hash keys https://redis.io/commands#hash
  22. redis hash API - Pro Tip https://redis.io/commands#hash • In Node.js,

    HGETALL returns a JSON-like object • Use hash API plus HGETALL for objects that have independently- updated parts…
  23. redis list API • LPUSH / LPOP / LLEN /

    LREM / LRANGE • LSET • RPOPLPUSH https://redis.io/commands#list
  24. redis list API - Pro Tip https://redis.io/commands#list • LLEN &

    LRANGE are great for paginated collections • The parameters “0” and “99” are the offset and limit to retrieve • LLEN gives the total list length
  25. redis list API - Pro Tip 2 • Queue “Drain”:

    Use a redis transaction to get the “left range” and “left trim” it as part of the same operation • Other clients will see a consistent view of the list before and after
  26. redis set API • SADD / SCARD / SISMEMBER •

    SDIFF / SINTER / SUNION
 + "STORE" variants https://redis.io/commands#set
  27. redis set API - Pro Tip https://redis.io/commands#set • Use SDIFF

    to compare progress between two workers • Similarly, SINTER gives common set
  28. redis sorted set API • ZADD / ZCARD / ZCOUNT

    / ZRANGE • ZINCRBY • ZRANK • ZREV* - reverse variants https://redis.io/commands#sorted_set
  29. redis pubsub API • PUBLISH / SUBSCRIBE / UNSUBSCRIBE •

    PSUBSCRIBE / PUNSUBCRIBE https://redis.io/commands#pubsub
  30. redis pubsub API - Pro Tip https://redis.io/commands#pubsub • redis PubSub

    is great for quick publish/ suscribe prototyping with WebSockets • Be careful as message volume and number of subscribers increase
  31. redis transactions • MULTI / EXEC (!!!) • WATCH https://redis.io/commands#transactions

  32. redis scripting • SCRIPT LOAD • EVAL / EVALSHA

  33. building stuff • Counters • Queues • Membership • Leader

    Boards
  34. database counters (ok) • CREATE TABLE counters (
 int id

    PRIMARY KEY,
 object_id int,
 count int) • SELECT * FROM leaderboard
 WHERE object_id = ? • UPDATE object_id
 SET count = ?
 WHERE object_id = ?
  35. database counters (BAD!) • SELECT count(*) 
 FROM collection_table 


    WHERE object_id = ?
  36. redis counters • Use KV or Hash API • Key:

    page id • Value: count • Use INCR / INCRBY • Alternatively HINCR / HINCRBY for Hash API
  37. database queue • CREATE TABLE queue (
 int id PRIMARY

    KEY,
 queue_id int,
 object_id int) • SELECT * FROM queue
 WHERE queue_id = ?
 ORDER BY id
 LIMIT 1 • INSERT INTO queue VALUES (?, ?, ?); • DELETE FROM queue
 WHERE id = ?
  38. redis queue • Use the List API • Key: queue

    name • Value: list of entries • LPUSH [queue] [item]
 LPOP [queue] (for LIFO)
 RPOP [queue] (for FIFO) • BONUS! RPUSHLPOP : atomic transfer from one queue to another
  39. database membership • CREATE TABLE members (
 int id PRIMARY

    KEY,
 group_id int,
 user_id int,
 UNIQUE KEY(group_id, user_id)) • SELECT * FROM members
 WHERE group_id = ?
 AND user_id = ? • INSERT INTO members VALUES (?, ?, ?);
  40. redis membership • Use the Set API • Key: group

    name • Value: set of members • SADD [group] [user]
 SISMEMBER [group] [user] • BONUS! SUNION / SINTER / SDIFF
  41. database leaderboard • CREATE TABLE leaderboard (
 int id PRIMARY

    KEY,
 board_id int,
 board_member_id int,
 score int) • SELECT * FROM leaderboard
 WHERE board_id = ?
 ORDER BY score
 LIMIT 100 • UPDATE leaderboard
 SET score = ?
 WHERE board_id = ?
 AND board_member_id = ?
  42. redis leaderboard • Use the Sorted Set API • Key:

    leaderboard name • Value: priority queue • ZADD [board] [user] [score]
 ZINCRBY [board] [user] [n]
 ZRANGE [board] [m] [n] • ZREV* for reverse sort order
  43. Publish / Subscribe • Real-time communications without polling • Machines

    talking to each other: status/performance • Humans talking to each other:
 chat/notifications
  44. pubsub applications

  45. pubsub applications • Chat / Presence / Notifications • Location

    / Asset Tracking (taxi/mobile) • Real-time Data Streams (stocks) • System Monitoring / Status / Alerts • Device Sharing & Synchronization (mobile)
  46. redis pubsub • Single node, high performance • Very easy

    to get started • Adaptors for socket.io and atmosphere • Missing items: durability, scalability, presence, history, massive broadcast
  47. redis client libraries • Node.js : node_redis • Java :

    jedis, lettuce • Scala : scala-redis • C#/.NET : StackExchange.Redis, ServiceStack.Redis • PHP : phpredis • Ruby : redis-rb • Go : redigo • PHP : phpredis • Rust : redis-rs • And many more… ! https://redis.io/clients
  48. database conclusion • We love Databases, hopefully they aren't going

    anywhere • Most practical solution for primary storage of data • Know their limitations (performance & operational), so you can make tradeoffs accordingly • In addition, be aware of relevant NoSQL solutions: ElasticSearch & others…
  49. redis conclusion • Try redis for high-performance
 application needs •

    Use it as a throwaway store, and
 you’ll be happiest • Watch your applications become simpler,
 and faster!
  50. questions? thank you!