whoami
• Sunny Gleason
@sunnygleason, sunnygleason
SunnyCloud, Boston MA
• Application Development (Web & Mobile)
Distributed Systems (Cloud)
Security & Compliance
• Prior Work
• Independent developer/army of one…
Slide 3
Slide 3 text
what’s this all about?
• Databases are awesome
• They store and protect data
• We need them to be faster
• ... or to use different data
access patterns
• So let’s sacrifice some durability for
performance and nice APIs
Slide 4
Slide 4 text
agenda
• What is
redis?
• …
redis and Database Models
• …
redis API Overview
• Sample Features implemented using
redis
Slide 5
Slide 5 text
redis == fast data structures
• Map / Dictionary: associate keys & values
• ArrayList / Deque: lists supporting
operations on both ends + random access
• Heap / Priority Queue: sorted maps,
key/value + score
Slide 6
Slide 6 text
redis extras
• “Transactions” : multi-operations with
watch values
• Lua Scripting : a.k.a. stored procedures
• Publish / Subscribe : channel
subscriptions with real-time notifications
• Streams : data structures for log-like data
database recap
• Buffer Pool
• Pages live durably on-disk
• Indexes sit in memory
• Minor disk seeks for record retrieval
• Major disk seeks for table scans
• Limited by IOPS: i/o operations per second
Slide 9
Slide 9 text
in-memory model
• All data stored in RAM
• If swaps occur, very bad news
• Can be spooled to disk periodically
• Disk not recommended for normal
operations - leads to false sense of
security
Slide 10
Slide 10 text
why use redis?
• Backend/Caching: page fragments,
object cache, web sessions, ephemeral
data from external or internal services
• Site Features: Counters, Membership,
Job/Message Queues, Leaderboards
why not use redis?
• Durability : the recovery/failover
processes are just ok, not fantastic
• Multiprocessing : each redis process is
(approximately) single-threaded
• Transactions and semantics of other
operations (sets, etc.) get tricky under
sharding models
Slide 13
Slide 13 text
why not use redis?
https://ungleich.ch/en-us/cms/blog/2018/12/24/1a-who-does-not-need-be-afraid-oom-killer/
Slide 14
Slide 14 text
redis KV API
• GET [key]
• SET [key] [value]
• SETEX [key] [exp] [value]
• DEL [key]
• SCAN [cursor]
• KEYS [pattern]
https://redis.io/commands#generic
Slide 15
Slide 15 text
redis KV API
• GET [key] : gets the value of a key
• SET [key] [value] : sets the value of a key
• SETEX [key] [exp] [value] : sets an expiring key/value
• DEL [key] : removes a key/value pair
• SCAN [cursor] : iterates over keys
• KEYS [pattern] : list keys that match a pattern
https://redis.io/commands#generic
Slide 16
Slide 16 text
redis KV API - Pro Tip
• SET / EXPIRE, or SETEX
are useful for giving
entries finite lifetimes
• Super useful for ensuring
redis store doesn’t grow
without bound
• Configure your redis
memory limit and
eviction policy
accordingly
https://redis.io/commands#generic
redis string API
• MGET [k1] [k2] … : retrieve multiple values (!!!)
• MSET [k1] [v1] [k2] [v2] … : set many KV pairs (!!!)
• INCR [k1] / DECR [k1] : increment / decrement value (!!!)
• INCRBY [k1] [n] / DECRBY [k1] [n] : inc/dec by N (!!!)
https://redis.io/commands#string
Slide 19
Slide 19 text
redis string API - Pro Tip
• Use MGET/MSET to
reduce operation
count!
• Lodash has the
zipObject operation to
turn separate key and
value arrays into a
unified object
https://redis.io/commands#string
redis hash API
• HGET / HSET / HDEL : get / set / delete
hash entry
• HMGET / HMSET / HGETALL : multiget,
multiset, get all entries of hash
• HKEYS / HSCAN : list / traverse hash
keys
https://redis.io/commands#hash
Slide 22
Slide 22 text
redis hash API - Pro Tip
https://redis.io/commands#hash
• In Node.js, HGETALL
returns a JSON-like
object
• Use hash API plus
HGETALL for objects
that have
independently-
updated parts…
Slide 23
Slide 23 text
redis list API
• LPUSH / LPOP / LLEN /
LREM / LRANGE
• LSET
• RPOPLPUSH
https://redis.io/commands#list
Slide 24
Slide 24 text
redis list API - Pro Tip
https://redis.io/commands#list
• LLEN & LRANGE are
great for paginated
collections
• The parameters “0”
and “99” are the
offset and limit to
retrieve
• LLEN gives the total
list length
Slide 25
Slide 25 text
redis list API - Pro Tip 2
• Queue “Drain”: Use a
redis transaction to
get the “left range”
and “left trim” it as
part of the same
operation
• Other clients will see
a consistent view of
the list before and
after
Slide 26
Slide 26 text
redis set API
• SADD / SCARD / SISMEMBER
• SDIFF / SINTER / SUNION
+ "STORE" variants
https://redis.io/commands#set
Slide 27
Slide 27 text
redis set API - Pro Tip
https://redis.io/commands#set
• Use SDIFF to
compare progress
between two
workers
• Similarly, SINTER
gives common set
redis pubsub API - Pro Tip
https://redis.io/commands#pubsub
• redis PubSub is great
for quick publish/
suscribe prototyping
with WebSockets
• Be careful as message
volume and number
of subscribers
increase
Slide 31
Slide 31 text
redis transactions
• MULTI / EXEC (!!!)
• WATCH
https://redis.io/commands#transactions
database counters (ok)
• CREATE TABLE counters (
int id PRIMARY KEY,
object_id int,
count int)
• SELECT * FROM leaderboard
WHERE object_id = ?
• UPDATE object_id
SET count = ?
WHERE object_id = ?
Slide 35
Slide 35 text
database counters (BAD!)
• SELECT count(*)
FROM collection_table
WHERE object_id = ?
Slide 36
Slide 36 text
redis counters
• Use KV or Hash API
• Key: page id
• Value: count
• Use INCR / INCRBY
• Alternatively HINCR / HINCRBY for Hash API
Slide 37
Slide 37 text
database queue
• CREATE TABLE queue (
int id PRIMARY KEY,
queue_id int,
object_id int)
• SELECT * FROM queue
WHERE queue_id = ?
ORDER BY id
LIMIT 1
• INSERT INTO queue VALUES (?, ?, ?);
• DELETE FROM queue
WHERE id = ?
Slide 38
Slide 38 text
redis queue
• Use the List API
• Key: queue name
• Value: list of entries
• LPUSH [queue] [item]
LPOP [queue] (for LIFO)
RPOP [queue] (for FIFO)
• BONUS! RPUSHLPOP : atomic transfer from one
queue to another
Slide 39
Slide 39 text
database membership
• CREATE TABLE members (
int id PRIMARY KEY,
group_id int,
user_id int,
UNIQUE KEY(group_id, user_id))
• SELECT * FROM members
WHERE group_id = ?
AND user_id = ?
• INSERT INTO members VALUES (?, ?, ?);
Slide 40
Slide 40 text
redis membership
• Use the Set API
• Key: group name
• Value: set of members
• SADD [group] [user]
SISMEMBER [group] [user]
• BONUS! SUNION / SINTER / SDIFF
Slide 41
Slide 41 text
database leaderboard
• CREATE TABLE leaderboard (
int id PRIMARY KEY,
board_id int,
board_member_id int,
score int)
• SELECT * FROM leaderboard
WHERE board_id = ?
ORDER BY score
LIMIT 100
• UPDATE leaderboard
SET score = ?
WHERE board_id = ?
AND board_member_id = ?
Slide 42
Slide 42 text
redis leaderboard
• Use the Sorted Set API
• Key: leaderboard name
• Value: priority queue
• ZADD [board] [user] [score]
ZINCRBY [board] [user] [n]
ZRANGE [board] [m] [n]
• ZREV* for reverse sort order
Slide 43
Slide 43 text
Publish / Subscribe
• Real-time communications without
polling
• Machines talking to each other:
status/performance
• Humans talking to each other:
chat/notifications
redis pubsub
• Single node, high performance
• Very easy to get started
• Adaptors for socket.io and atmosphere
• Missing items: durability, scalability,
presence, history, massive broadcast
database conclusion
• We love Databases, hopefully they aren't going anywhere
• Most practical solution for primary storage of data
• Know their limitations (performance & operational), so
you can make tradeoffs accordingly
• In addition, be aware of relevant NoSQL solutions:
ElasticSearch & others…
Slide 49
Slide 49 text
redis conclusion
• Try redis for high-performance
application needs
• Use it as a throwaway store, and
you’ll be happiest
• Watch your applications become simpler,
and faster!