Redis — memcached on steroids

Presentation held in the seminar on "NotOnlySQL databases."

Transcript

1. memcached on steroids
2. “blazingly fast” – Kirk Haines, Engineyard
3. Back in the day… Salvatore Sanfilippo
4. Back in the day… Salvatore Sanfilippo
5. Back in the day… LLOOGG Google Analytics + Realtime
6. Back in the day… LLOOGG Google Analytics + Realtime
7. Back in the day… LLOOGG Google Analytics + Realtime
8. Back in the day… LLOOGG Google Analytics + Realtime “ Redis is working very well… We never experienced any stability problem – Salvatore Sanfilippo
9. Back in the day… community growth @simonw Simon Willison @antirez just posted a redis feature request http://groups.google.com/group/redis-db/.../cd7e7c65dba53e27 - a way of randomly fetching an item from a set without also deleting it 20 Oct 09 via web
10. Back in the day… community growth @simonw Simon Willison @antirez just posted a redis feature request http://groups.google.com/group/redis-db/.../cd7e7c65dba53e27 - a way of randomly fetching an item from a set without also deleting it 20 Oct 09 via web
11. Back in the day… VMWare acquisition
12. A little server of awesome http://www.flickr.com/photos/ajc1/4663140532 man, that’s totally stolen from Simon Willison
13. Redis Manifesto 1. A DSL for Abstract Data Types 2. Memory storage is #1 3. Fundamental data structures for a fundamental API 4. Code is like a poem 5. We’re against complexity 6. Two levels of API 7. We optimize for joy
14. a key/value store? A DSL For Abstract Data Types redis> SET key hello OK redis> GET key “hello”
15. a key/value store? A DSL For Abstract Data Types redis> SET key hello OK redis> GET key “hello” request
16. a key/value store? A DSL For Abstract Data Types redis> SET key hello OK redis> GET key “hello” request awesome, a REPL! redis-cli
17. on par with A DSL For Abstract Data Types redis> APPEND key world (integer) 10 redis> SETNX key bye (integer) 1 redis> GET key “helloworld” redis> DEL key (integer) 1 memcache
18. on par with memcache: atomic counters A DSL For Abstract Data Types redis> SET participants 10 OK redis> INCR participants (integer) 11 “It gives Memcached a serious run for its money.” – Mathias Meyer, Peritor GmbH
19. on par with memcache: cache invalidation A DSL For Abstract Data Types redis> EXPIRE key 60 OK redis> ...
20. on par with memcache: cache invalidation A DSL For Abstract Data Types redis> EXPIRE key 60 OK redis> TTL key (integer) 53 redis>
21. on par with memcache: cache invalidation A DSL For Abstract Data Types redis> EXPIRE key 60 OK redis> TTL key (integer) 53 redis> GET key “helloworld” redis>
22. on par with memcache: cache invalidation A DSL For Abstract Data Types redis> EXPIRE key 60 OK redis> TTL key (integer) 53 redis> GET key “helloworld” redis> ...
23. on par with memcache: cache invalidation A DSL For Abstract Data Types redis> EXPIRE key 60 OK redis> TTL key (integer) 53 redis> GET key “helloworld” redis> GET key (nil)
24. “ Memory is the new disk. –– Jim Gray, Turing Award laureate
25. Memory storage is #1 Redis key space http://spotfireblog.tibco.com/wp-content/uploads/in-memory-analytics.jpg
26. Memory storage is #1 snapshotting mode http://www.flickr.com/photos/bionicteaching/3212235059/
27. Memory storage is #1 snapshotting mode http://www.flickr.com/photos/bionicteaching/3212235059/ every x seconds every y operations on SAVE / BGSAVE
28. Memory storage is #1 snapshotting mode http://www.flickr.com/photos/bionicteaching/3212235059/ every x seconds every y operations on SAVE / BGSAVE fork(); // copy on write! dump(); move();
29. Memory storage is #1 Append-Only File – Write-Ahead Logging http://www.flickr.com/photos/generationbass/4827013488/
30. Memory storage is #1 Append-Only File – Write-Ahead Logging http://www.flickr.com/photos/generationbass/4827013488/ fsync() on every command
31. Memory storage is #1 Append-Only File – Write-Ahead Logging http://www.flickr.com/photos/generationbass/4827013488/ fsync() on every command fsync() every second
32. Memory storage is #1 Append-Only File – Write-Ahead Logging http://www.flickr.com/photos/generationbass/4827013488/ fsync() on every command fsync() every second fsync() per OS
33. Memory storage is #1 Append-Only File – Write-Ahead Logging http://www.flickr.com/photos/generationbass/4827013488/ fsync() on every command fsync() every second fsync() per OS BGREWRITEAOF trims AOF
34. Memory storage is #1 meh, persistence http://www.flickr.com/photos/generationbass/4827013488/ memory overhead
35. Memory storage is #1 memory contention ?
36. Memory storage is #1 memory contention ?
37. Memory storage is #1 memory contention ? Virtual Memory to the rescue
38. Memory storage is #1 memory contention ? Virtual Memory to the rescue because Salvatore is much brighter than all of Microsoft Research
39. Memory storage is #1 memory contention ? Virtual Memory to the rescue because Salvatore is much brighter than all of Microsoft Research …and Redis types do not map to OS pages
40. Memory storage is #1 memory contention ? Virtual Memory to the rescue because Salvatore is much brighter than all of Microsoft Research …and Redis types do not map to OS pages all keys need to stay in memory values must be swapped in their entirety swapped by age and size
41. Fundamental data structures for a fundamental API
42. lists, we can redis> LPUSH databases mysql (integer) 1 redis> LPUSH databases mongodb (integer) 2 Fundamental data structures for a fundamental API
43. lists, we can redis> LPUSH databases mysql (integer) 1 redis> LPUSH databases mongodb (integer) 2 redis> LRANGE databases 0 -1 1) “mongodb” 2) “mysql” Fundamental data structures for a fundamental API
44. lists, we can redis> LPUSH databases mysql (integer) 1 redis> LPUSH databases mongodb (integer) 2 redis> LRANGE databases 0 -1 1) “mongodb” 2) “mysql” Fundamental data structures for a fundamental API Bulk reply
45. lists, we can redis> LPUSH databases mysql (integer) 1 redis> LPUSH databases mongodb (integer) 2 redis> LRANGE databases 0 -1 1) “mongodb” 2) “mysql” Fundamental data structures for a fundamental API Bulk reply implemented as Linked Lists
46. lists, we can redis> LPUSH databases mysql (integer) 1 redis> LPUSH databases mongodb (integer) 2 redis> LRANGE databases 0 -1 1) “mongodb” 2) “mysql” Fundamental data structures for a fundamental API Bulk reply implemented as Linked Lists PUSH POP O(1)
47. a data structure server hash tables sets sorted sets Fundamental data structures for a fundamental API lists
48. a data structure server hash tables sets sorted sets Fundamental data structures for a fundamental API lists
49. a data structure server hash tables sets sorted sets Fundamental data structures for a fundamental API lists
50. RENAME key newkey RENAMENX key newkey RPOP key RPOPLPUSH source destination RPUSH key value RPUSHX key value SADD key member SCARD key SDIFF key [key ...] SDIFFSTORE destination key [key ...] SELECT index SET key value SETBIT key offset value SETEX key seconds value SETNX key value SETRANGE key offset value SINTER key [key ...] SINTERSTORE destination key [key ...] SISMEMBER key member SMEMBERS key SMOVE source destination member SORT key [BY pattern] [LIMIT offset count] [GET pattern [GET pattern ...]] [ASC|DESC] [ALPHA] SPOP key SRANDMEMBER key SREM key member STRLEN key SUNION key [key ...] SUNIONSTORE destination key [key ...] TTL key TYPE key ZADD key score member ZCARD key ZCOUNT key min max ZINCRBY key increment member ZINTERSTORE destination numkeys key [key ...] ZRANGE key start stop [WITHSCORES] ZRANGEBYSCORE key min max [WITHSCORES] [LIMIT offset count] ZRANK key member ZREM key member ZREMRANGEBYRANK key start stop ZREMRANGEBYSCORE key min max ZREVRANGE key start stop [WITHSCORES] ZREVRANGEBYSCORE key max min [WITHSCORES] [LIMIT offset count] ZREVRANK key member ZSCORE key member ZUNIONSTORE destination numkeys key [key ...] APPEND key value BLPOP key [key ...] BRPOP key [key ...] timeout BRPOPLPUSH source destination timeout DECR key DECRBY key decrement DEL key [key ...] EXISTS key EXPIRE key seconds EXPIREAT key timestamp FLUSHALL R FLUSHDB GET key GETBIT key offset GETRANGE key start end GETSET key value HDEL key field HEXISTS key field HGET key field HGETALL key HINCRBY key field increment HKEYS key HLEN key HMGET key field [field ...] HMSET key field value [field value ...] HSET key field value HSETNX key field value HVALS key INCR key INCRBY key increment KEYS pattern LINDEX key index LINSERT key BEFORE|AFTER pivot value LLEN key LPOP key LPUSH key value LPUSHX key value LRANGE key start stop LREM key count value LSET key index value LTRIM key start stop MGET key [key ...] MOVE key db MSET key value [key value ...] MSETNX key value [key value ...] PERSIST key RANDOMKEY a data processing server Fundamental data structures for a fundamental API
51. RENAME key newkey RENAMENX key newkey RPOP key RPOPLPUSH source destination RPUSH key value RPUSHX key value SADD key member SCARD key SDIFF key [key ...] SDIFFSTORE destination key [key ...] SELECT index SET key value SETBIT key offset value SETEX key seconds value SETNX key value SETRANGE key offset value SINTER key [key ...] SINTERSTORE destination key [key ...] SISMEMBER key member SMEMBERS key SMOVE source destination member SORT key [BY pattern] [LIMIT offset count] [GET pattern [GET pattern ...]] [ASC|DESC] [ALPHA] SPOP key SRANDMEMBER key SREM key member STRLEN key SUNION key [key ...] SUNIONSTORE destination key [key ...] TTL key TYPE key ZADD key score member ZCARD key ZCOUNT key min max ZINCRBY key increment member ZINTERSTORE destination numkeys key [key ...] ZRANGE key start stop [WITHSCORES] ZRANGEBYSCORE key min max [WITHSCORES] [LIMIT offset count] ZRANK key member ZREM key member ZREMRANGEBYRANK key start stop ZREMRANGEBYSCORE key min max ZREVRANGE key start stop [WITHSCORES] ZREVRANGEBYSCORE key max min [WITHSCORES] [LIMIT offset count] ZREVRANK key member ZSCORE key member ZUNIONSTORE destination numkeys key [key ...] APPEND key value BLPOP key [key ...] BRPOP key [key ...] timeout BRPOPLPUSH source destination timeout DECR key DECRBY key decrement DEL key [key ...] EXISTS key EXPIRE key seconds EXPIREAT key timestamp FLUSHALL R FLUSHDB GET key GETBIT key offset GETRANGE key start end GETSET key value HDEL key field HEXISTS key field HGET key field HGETALL key HINCRBY key field increment HKEYS key HLEN key HMGET key field [field ...] HMSET key field value [field value ...] HSET key field value HSETNX key field value HVALS key INCR key INCRBY key increment KEYS pattern LINDEX key index LINSERT key BEFORE|AFTER pivot value LLEN key LPOP key LPUSH key value LPUSHX key value LRANGE key start stop LREM key count value LSET key index value LTRIM key start stop MGET key [key ...] MOVE key db MSET key value [key value ...] MSETNX key value [key value ...] PERSIST key RANDOMKEY a data processing server Fundamental data structures for a fundamental API every operation is atomic
52. RENAME key newkey RENAMENX key newkey RPOP key RPOPLPUSH source destination RPUSH key value RPUSHX key value SADD key member SCARD key SDIFF key [key ...] SDIFFSTORE destination key [key ...] SELECT index SET key value SETBIT key offset value SETEX key seconds value SETNX key value SETRANGE key offset value SINTER key [key ...] SINTERSTORE destination key [key ...] SISMEMBER key member SMEMBERS key SMOVE source destination member SORT key [BY pattern] [LIMIT offset count] [GET pattern [GET pattern ...]] [ASC|DESC] [ALPHA] SPOP key SRANDMEMBER key SREM key member STRLEN key SUNION key [key ...] SUNIONSTORE destination key [key ...] TTL key TYPE key ZADD key score member ZCARD key ZCOUNT key min max ZINCRBY key increment member ZINTERSTORE destination numkeys key [key ...] ZRANGE key start stop [WITHSCORES] ZRANGEBYSCORE key min max [WITHSCORES] [LIMIT offset count] ZRANK key member ZREM key member ZREMRANGEBYRANK key start stop ZREMRANGEBYSCORE key min max ZREVRANGE key start stop [WITHSCORES] ZREVRANGEBYSCORE key max min [WITHSCORES] [LIMIT offset count] ZREVRANK key member ZSCORE key member ZUNIONSTORE destination numkeys key [key ...] APPEND key value BLPOP key [key ...] BRPOP key [key ...] timeout BRPOPLPUSH source destination timeout DECR key DECRBY key decrement DEL key [key ...] EXISTS key EXPIRE key seconds EXPIREAT key timestamp FLUSHALL R FLUSHDB GET key GETBIT key offset GETRANGE key start end GETSET key value HDEL key field HEXISTS key field HGET key field HGETALL key HINCRBY key field increment HKEYS key HLEN key HMGET key field [field ...] HMSET key field value [field value ...] HSET key field value HSETNX key field value HVALS key INCR key INCRBY key increment KEYS pattern LINDEX key index LINSERT key BEFORE|AFTER pivot value LLEN key LPOP key LPUSH key value LPUSHX key value LRANGE key start stop LREM key count value LSET key index value LTRIM key start stop MGET key [key ...] MOVE key db MSET key value [key value ...] MSETNX key value [key value ...] PERSIST key RANDOMKEY a data processing server Fundamental data structures for a fundamental API every operation is atomic Mind blown.
53. CAP & ACID promises consistency partition tolerance availability durability atomicity isolation consistency Fundamental data structures for a fundamental API “kicks ACID out the door” –Mathias Meyer, Chief Cloud Officer
54. CAP & ACID promises consistency partition tolerance availability durability atomicity isolation consistency Fundamental data structures for a fundamental API “kicks ACID out the door” –Mathias Meyer, Chief Cloud Officer “that’s like saying why aren’t filesystems ACID”
55. Code is like a poem C 23.000 lines of code
56. We’re against complexity
57. Two levels of APIs transactions redis> MULTI OK redis> GET key QUEUED redis> DEL key QUEUED redis> EXEC 1) “helloworld” 2) (integer) 1
58. Two levels of APIs transactions redis> MULTI OK redis> GET key QUEUED redis> DEL key QUEUED redis> EXEC 1) “helloworld” 2) (integer) 1 EXEC DISCARD
59. Two levels of APIs transactions redis> MULTI OK redis> GET key QUEUED redis> DEL key QUEUED redis> EXEC 1) “helloworld” 2) (integer) 1 atomic
60. Two levels of APIs transactions redis> MULTI OK redis> GET key QUEUED redis> DEL key QUEUED redis> EXEC 1) “helloworld” 2) (integer) 1 atomic there’s always a but: server crash = partial execution
61. Two levels of APIs transactions redis> MULTI OK redis> GET key QUEUED redis> DEL key QUEUED redis> EXEC 1) “helloworld” 2) (integer) 1 atomic there’s always a but: server crash = partial execution there’s always a but but: detected on startup => exit
62. Two levels of APIs transactions redis> MULTI OK redis> GET key QUEUED redis> DEL key QUEUED redis> EXEC 1) “helloworld” 2) (integer) 1 atomic there’s always a but: server crash = partial execution there’s always a but but: detected on startup => exit there’s always a but but but: redis-check-aof repairs it
63. Two levels of APIs optimistic locking redis> WATCH key OK redis> MULTI OK
64. Two levels of APIs optimistic locking redis> WATCH key OK redis> MULTI OK SET key ciao <redis OK
65. Two levels of APIs optimistic locking redis> WATCH key OK redis> MULTI OK SET key ciao <redis OK redis> SET key bye QUEUED redis> EXEC (nil)
66. Two levels of APIs optimistic locking redis> WATCH key OK redis> MULTI OK SET key ciao <redis OK redis> SET key bye QUEUED redis> EXEC (nil) GET key <redis “ciao”
67. We optimize for joy Pub/Sub redis> SUBSCRIBE chat Reading messages… (press Ctrl-C to quit) 1) “subscribe” 2) “chat” 3) (integer) 1
68. We optimize for joy Pub/Sub redis> SUBSCRIBE chat Reading messages… (press Ctrl-C to quit) 1) “subscribe” 2) “chat” 3) (integer) 1 type
69. We optimize for joy Pub/Sub redis> SUBSCRIBE chat Reading messages… (press Ctrl-C to quit) 1) “subscribe” 2) “chat” 3) (integer) 1 channel
70. We optimize for joy Pub/Sub redis> SUBSCRIBE chat Reading messages… (press Ctrl-C to quit) 1) “subscribe” 2) “chat” 3) (integer) 1 message
71. We optimize for joy Pub/Sub redis> SUBSCRIBE chat Reading messages… (press Ctrl-C to quit) 1) “subscribe” 2) “chat” 3) (integer) 1 PUBLISH chat “asl?” <redis (integer) 1
72. We optimize for joy Pub/Sub redis> SUBSCRIBE chat Reading messages… (press Ctrl-C to quit) 1) “subscribe” 2) “chat” 3) (integer) 1 PUBLISH chat “asl?” <redis (integer) 1 1) “message” 2) “chat” 3) “asl?” ^C
73. We optimize for joy Pub/Sub redis> SUBSCRIBE chat Reading messages… (press Ctrl-C to quit) 1) “subscribe” 2) “chat” 3) (integer) 1 PUBLISH chat “asl?” <redis (integer) 1 1) “message” 2) “chat” 3) “asl?” ^C in other news: PSUBSCRIBE UNSUBSCRIBE
74. “ Redis is more than just a key-value store, it’s a lifestyle. –– Mathias Meyer
75. Benchmarks redis-benchmark $ redis-benchmark ====== SET ====== 100000 requests completed in 0.88 seconds 50 parallel clients 3 bytes payload ====== GET ====== 100000 requests completed in 1.23 seconds Linux 2.6, Xeon X3320 2.5GHz, loopback
76. Benchmarks Colin Howe 3.7x writes 2.1x reads speed improvement for over MySQL
77. Replication master-slave scenario http://www.flickr.com/photos/joiearai/4393860397/
78. Replication master-slave scenario http://www.flickr.com/photos/joiearai/4393860397/
79. Replication master-slave scenario http://www.flickr.com/photos/joiearai/4393860397/
80. Replication master-slave scenario http://www.flickr.com/photos/joiearai/4393860397/ SLAVEOF host port
81. Replication master-slave scenario http://www.flickr.com/photos/joiearai/4393860397/ SLAVEOF host port save + re-issuing
82. Who’s using it Github Resque message queue service Configuration Management Routing Information
83. Who’s using it Digg Digg Streaming API personalized news data real time view click counts
84. Who’s using it wooga Monster World Happy Hospital inventory system user system
85. Who’s using it Peter Noordhuis # Author: Peter Noordhuis # Description: Simple demo to showcase Redis PubSub with EventMachine <170 lines of Ruby code> <220 lines of HTML+JS>
86. Who’s using it WildlifeNearYou Vanity API rate limiting Crowdsourcing taxonomies A/B testing The Guardian Crowdsourced analysis of MP expenses BNP membership list analysis Hurl HTTP request testing
87. Who’s using it WildlifeNearYou Vanity API rate limiting Crowdsourcing taxonomies A/B testing The Guardian Crowdsourced analysis of MP expenses BNP membership list analysis Hurl HTTP request testing general blogosphere hype storm http://redis.io/topics/using-redis lists about 30 others
88. “ I think the… problem is… believing that there can be “one true datastore”. Different technologies excel at different things. –– Weixi Yen
89. Check out… Simon Willison’s tutorial http://simonwillison.net/static/2010/redis-tutorial
90. Check out… http://blog.mjrusso.com/2010/10/17/redis-from-the-ground-up Redis, from the Ground Up http://try.redis-db.com/ Try Redis coming August 2011, Salvatore & Pieter, published by O’Reilly Redis: The Definitive Guide Data modeling, caching, and messaging also check my Redis bookmarks: http://del.icio.us/lehmannro/redis http://rediscookbook.org/ The Redis Cookbook
91. man, that was a hell of a ride. Thanks. Questions? Now go use Redis.io!