Redis Steady Go (Nov 2012)

Slide 1

Slide 1 text

Peter Cooper presents.. A rapid tour of Redis, the super-fast data structure server! Friday, 23 November 12

Slide 2

Slide 2 text

Friday, 23 November 12

Slide 3

Slide 3 text

Salvatore San!lippo a.k.a. antirez Friday, 23 November 12

Slide 4

Slide 4 text

WHAT IS REDIS? Friday, 23 November 12

Slide 5

Slide 5 text

remote dictionary server Friday, 23 November 12

Slide 6

Slide 6 text

remote hash server Friday, 23 November 12

Slide 7

Slide 7 text

data structure server! Friday, 23 November 12

Slide 8

Slide 8 text

data structure server! isn’t that a “database”? Friday, 23 November 12

Slide 9

Slide 9 text

in a way, but.. no “tables” no SQL no enforced relationships lots of working with “primitives” isn’t that a “database”? Friday, 23 November 12

Slide 10

Slide 10 text

think less.. and more.. name age user 1 jim 21 user 2 amy 44 user:1:name = Jim user:1:age = 21 user:2:name = Amy user:2:age = 44 just keys and values.. a la SQLite, MySQL, etc.. Friday, 23 November 12

Slide 11

Slide 11 text

In Ruby: a = “Hello world” b = “This is a test” storing some strings Friday, 23 November 12

Slide 12

Slide 12 text

In Redis: SET A “Hello world” SET B “This is a test” storing some strings Friday, 23 November 12

Slide 13

Slide 13 text

And to retrieve: GET A GET B storing some strings Friday, 23 November 12

Slide 14

Slide 14 text

1 0 1 0 1 0 1 0 like an assembly language? mov rax, 0x0a68732f push rax xor rax, rax mov rsi, rsp ... Friday, 23 November 12

Slide 15

Slide 15 text

1 0 1 0 1 0 1 0 SURE! :-) but it’s hard to categorize Friday, 23 November 12

Slide 16

Slide 16 text

DSL for abstract data types and that’s from the Redis manifesto! google “redis manifesto” Friday, 23 November 12

Slide 17

Slide 17 text

open source (BSD) !rst public release in 2009 VMware hired Salvatore in 2010 (and Pieter Noordhuis!) Friday, 23 November 12

Slide 18

Slide 18 text

add more commands add persistence add more data types and more.. familiar with memcached? Friday, 23 November 12

Slide 19

Slide 19 text

LIVE DEMO TIME! Friday, 23 November 12

Slide 20

Slide 20 text

REDIS HOSTING Friday, 23 November 12

Slide 21

Slide 21 text

redistogo.com garantiadata.com openredis.com redis4you.com redisgreen.net Friday, 23 November 12

Slide 22

Slide 22 text

USES FOR REDIS Friday, 23 November 12

Slide 23

Slide 23 text

The three biggies.. Friday, 23 November 12

Slide 24

Slide 24 text

The three biggies.. Database Friday, 23 November 12

Slide 25

Slide 25 text

The three biggies.. Database Messaging Friday, 23 November 12

Slide 26

Slide 26 text

The three biggies.. Database Messaging Cache Friday, 23 November 12

Slide 27

Slide 27 text

Real time logging LLOOGGs! Just keep pushing stu" onto a list.. LLOOGG was/is a service run by Salvatore that Redis was born out of. Friday, 23 November 12

Slide 28

Slide 28 text

INCR pageviews:127.0.0.1:2012-11-23 INCR pageviews:127.0.0.1:2012-11-23-[HH] SADD users:logged_in:2012-11-23 [user_id] HINCRBY users:gina total_logins Friday, 23 November 12

Slide 29

Slide 29 text

Rate limiting INCR hits:33.12.32.22 EXPIRE hits:33.12.32.22 3600 INCR hits:33.12.32.22 It’s a bit naive but.. you get the idea. Friday, 23 November 12

Slide 30

Slide 30 text

Scoreboarding Sony did/does this Sorted sets are ideal! Add entries to a sorted set and query for results later. SuperDude1 20 pts xx_idiot_xx 19 pts mrkewl 11 pts freddieM 2 pts Friday, 23 November 12

Slide 31

Slide 31 text

Inter Process Communication (IPC) Talking between programs Use lists, a pre-agreed set of keys OR.. PUB/SUB! Friday, 23 November 12

Slide 32

Slide 32 text

Job queues resque does this! Push jobs onto a queue and forget about them (sort of..!) Friday, 23 November 12

Slide 33

Slide 33 text

Session storage redis-store library can help with this for your Ruby webapps SET session:[session_key] “{ stuff: 123, blah: 234 }” Friday, 23 November 12

Slide 34

Slide 34 text

Cache Think like old school memcached usage Some folks use Redis and memcached, keeping memcached for cache only and Redis for the more complex stu" Friday, 23 November 12

Slide 35

Slide 35 text

YouP***.com Yep, really. Redis is their primary datastore. Alexa ranking 104 Friday, 23 November 12

Slide 36

Slide 36 text

GroupOn, Stack Over#ow, Craigslist Tumblr, Microsoft There’s a lot more than this but just to make the point.. Redis is not some crazy far out technology. & 200+ Redis servers at Pinterest..! Friday, 23 November 12

Slide 37

Slide 37 text

Friday, 23 November 12

Slide 38

Slide 38 text

coder.io Friday, 23 November 12

Slide 39

Slide 39 text

CHARACTERISTICS Friday, 23 November 12

Slide 40

Slide 40 text

Single threaded and event driven Think Node.js or a simple EventMachine loop (technically in Redis 2.4+ some threads are used for background IO tasks but not servicing requests) Friday, 23 November 12

Slide 41

Slide 41 text

Individual operations are atomic Nothing can interrupt or clash with a command you issue. Remember: Redis is single threaded processed just one at a time.. Friday, 23 November 12

Slide 42

Slide 42 text

Data can be “expired” Want a key/value to die in 60 seconds? You got it. INCR hits:33.12.32.22 EXPIRE hits:33.12.32.22 3600 EXPIREAT hits:33.12.32.22 1928374758 Or cache invalidation! Friday, 23 November 12

Slide 43

Slide 43 text

Everything is stored in memory It has persistence but.. your entire dataset is in memory too. If it can’t !t, stu"* will be pruned. * - special technical term Friday, 23 November 12

Slide 44

Slide 44 text

Everything is stored in memory THIS MAKES REDIS SUPER FAST! Friday, 23 November 12

Slide 45

Slide 45 text

But it does have persistence! This makes Redis actually useful ;-) RDB “snapshots” and AOF logs Forks in order to do dumps and “copy on write” saves a lot of pain Friday, 23 November 12

Slide 46

Slide 46 text

Did I mention SUPER FAST? Think 50k+ GETs and SETs per second. Friday, 23 November 12

Slide 47

Slide 47 text

Lots of Redis clients C, C#, C++, Clojure, Lisp, Ruby, Python, Go, Erlang Scala, Objective C, Lua, Node, PHP, Smalltalk, Tcl Some languages even have ORMs supporting Redis. Like Ohm in Ruby. DataMapper has an adapter too.And Python’s redis_wrap. Friday, 23 November 12

Slide 48

Slide 48 text

Scriptable (with Lua) A handy way to keep more complex operations atomic Server can also store scripts and recall them using SHA1 hash Friday, 23 November 12

Slide 49

Slide 49 text

EXEC if (redis.call("type",KEYS[1]) ~= "string" or redis.call("get",KEYS[1]) ~= ARGV[1]) then redis.call("set",KEYS[1],ARGV[2]) end , 1, “some key” Friday, 23 November 12

Slide 50

Slide 50 text

EXECSHA 1ad692e76fa983ee712cabb, 1, “some key” Friday, 23 November 12

Slide 51

Slide 51 text

Can be used from PostgreSQL with a foreign data wrapper CREATE EXTENSION redis_fdw; CREATE SERVER redis_server FOREIGN DATA WRAPPER redis_fdw OPTIONS (address '127.0.0.1', port '6379'); CREATE FOREIGN TABLE redis_db0 (key text, value text) SERVER redis_server OPTIONS (database '0'); ... SELECT id, email, value as visits FROM users, redis_db0 WHERE ('user_' || cast(id as text)) = cast(redis_db0.key as text) AND cast(value as int) > 40; Friday, 23 November 12

Slide 52

Slide 52 text

DATA TYPES Friday, 23 November 12

Slide 53

Slide 53 text

5 Data Types Strings Lists Sets Sorted Sets Hashes Friday, 23 November 12

Slide 54

Slide 54 text

STRINGS As simple as it gets.. Friday, 23 November 12

Slide 55

Slide 55 text

SET mystring “hello world” ./redis-cli Redis command line client app GET mystring returns “hello world” ./redis-cli Key Value Friday, 23 November 12

Slide 56

Slide 56 text

GETSET MGET SETNX SETEX MSET MSETNX INCR INCRBY DECR DECRBY APPEND SUBSTR Works on strings that appear to be integers. Magic! Friday, 23 November 12

Slide 57

Slide 57 text

GET hits:33.12.32.22 #=> (nil) INCR hits:33.12.32.22 #=> 1 It even does the initializing.. Friday, 23 November 12

Slide 58

Slide 58 text

GENERIC OPERATIONS Yes, a short break with those data types.. Friday, 23 November 12

Slide 59

Slide 59 text

TYPE EXISTS RENAME DEL EXPIRE EXPIREAT TTL PERSIST Friday, 23 November 12

Slide 60

Slide 60 text

LISTS Lurvely linked lists with super-fast PUSH and POP Friday, 23 November 12

Slide 61

Slide 61 text

f e d c b a RPUSH LPOP LPUSH RPOP RPUSH my_q f e.g. Friday, 23 November 12

Slide 62

Slide 62 text

f e d c b a } LRANGE 2 3 LLEN == 6 LINDEX 5 X LREM 1 b Friday, 23 November 12

Slide 63

Slide 63 text

f e d c b a RPUSH LPOP RPUSH my_q a RPUSH my_q b LPOP my_q == “a” LPOP my_q == “b” LPOP my_q == (nil) Or BLPOP to block (wait) until something can be popped Queues Not a native data type. Just a list! Friday, 23 November 12

Slide 64

Slide 64 text

def enqueue(name, data) SADD queues [name] RPUSH queue:[name] [data.to_json] def dequeue(name) LPOP queue:[name] def length(name) LLEN queue:[name] def clear(name) DEL queue:[name] Friday, 23 November 12

Slide 65

Slide 65 text

def enqueue(name, data) SADD queues [name] RPUSH queue:[name] [data.to_json] def dequeue(name) BLPOP queue:[name] 60 def length(name) LLEN queue:[name] def clear(name) DEL queue:[name] Friday, 23 November 12

Slide 66

Slide 66 text

def enqueue(name, data) SADD queues [name] RPUSH queue:[name] [data.to_json] def dequeue(names) BLPOP *names (e.g. BLPOP queue:h queue:m queue:l) def length(name) LLEN queue:[name] def clear(name) DEL queue:[name] Friday, 23 November 12

Slide 67

Slide 67 text

LTRIM myqueue 0 99 Want to keep a list just 100 items long? Friday, 23 November 12

Slide 68

Slide 68 text

SETS Think a list but with only unique elements. Friday, 23 November 12

Slide 69

Slide 69 text

SADD uniques 127.0.0.1 SADD uniques 127.0.0.2 SADD uniques 127.0.0.3 SCARD uniques # => 3 Friday, 23 November 12

Slide 70

Slide 70 text

SADD contains:aba abacus SADD contains:aba cabal SADD contains:aba baba # .. etc .. SISMEMBER contains:aba abacus #=> 1 (or true) Friday, 23 November 12

Slide 71

Slide 71 text

contains:aba contains:ase abacus cabal baba hello teabag base cabaret database vase vaseline baseline uncase unbased phase database tease SADD contains:ase suitcase SREM contains:aba hello SMOVE contains:aba contains:ase base Friday, 23 November 12

Slide 72

Slide 72 text

contains:aba contains:ase abacus cabal baba teabag cabaret database vase vaseline baseline unbased phase database suitcase SCARD contains:aba == 6 SISMEMBER contains:aba chips == 0 (meaning false) SRANDMEMBER contains:aba == “teabag” SMEMBERS contains:ase == vase, vaseline, baseline, unbased, phase, database, suitcase Friday, 23 November 12

Slide 73

Slide 73 text

contains:aba contains:ase abacus cabal baba teabag cabaret vase vaseline baseline unbased phase suitcase SINTER contains:aba contains:ase == database database This is only a simple example. SINTER can take any number of arguments! SUNION is another command that will join sets together. SDIFF will give you the difference. Friday, 23 November 12

Slide 74

Slide 74 text

contains:aba contains:ase abacus cabal baba teabag cabaret vase vaseline baseline unbased phase suitcase SINTERSTORE resultset contains:aba contains:ase database SUNIONSTORE does the same for set unions. database resultset Friday, 23 November 12

Slide 75

Slide 75 text

$id = INCR next_user_id SET user:$id peterc SADD starts:p $id SADD starts:pe $id SADD starts:pet $id ... etc. Imagine an autocomplete system.. Friday, 23 November 12

Slide 76

Slide 76 text

$text = “pet” $ids = SMEMBERS starts:$text MGET *user:$ids And to query.. Not actual syntax, implying.. MGET user:1 user:2 user:3, etc. Friday, 23 November 12

Slide 77

Slide 77 text

Things like inverted indexes, graphs, spell checkers, and similar data oriented tasks are also simple using Redis’ primitives. Friday, 23 November 12

Slide 78

Slide 78 text

SORTED SETS a.k.a. “scored” sets Like the sets we saw before, but with a “score” for each member too. Friday, 23 November 12

Slide 79

Slide 79 text

ZADD scoreboard 10 “Johnny” ZADD scoreboard 8 “Fred” ZADD scoreboard 20 “Amy” ZREVRANGE scoreboard 0 1 A very short example.. .. gets us the top 2 scorers! Friday, 23 November 12

Slide 80

Slide 80 text

HASHES Like Ruby’s hashes but with string-only values. Friday, 23 November 12

Slide 81

Slide 81 text

Yep, you can only use strings in a hash. No super duper weird recursive data structures for you! Friday, 23 November 12

Slide 82

Slide 82 text

HSET user:1 name “John” HGET user:1 name Yet another very short example.. Other commands include HGETALL, HKEYS, HVALUES, HINCRBY, etc.. name John age 20 gender male user:1 Friday, 23 November 12

Slide 83

Slide 83 text

OTHER STUFF! Friday, 23 November 12

Slide 84

Slide 84 text

SERVER COMMANDS Friday, 23 November 12

Slide 85

Slide 85 text

SAVE BGSAVE INFO MONITOR SHUTDOWN FLUSHDB QUIT and more.. Friday, 23 November 12

Slide 86

Slide 86 text

TRANSACTIONS Friday, 23 November 12

Slide 87

Slide 87 text

MULTI INCR a INCR b EXEC This is when “stu!” actually happens. Friday, 23 November 12

Slide 88

Slide 88 text

WATCH somekey GET somekey MULTI SET someotherkey $x EXEC You can also ensure a transaction fails if a dependency is broken. Put result into $x This fails if somekey changed. (check and set) Friday, 23 November 12

Slide 89

Slide 89 text

REPLICATION Friday, 23 November 12

Slide 90

Slide 90 text

Very simple master/slave replication. Server 1 Server 2 12.34.56.78:6378 12.34.56.78:6379 Friday, 23 November 12

Slide 91

Slide 91 text

Simple master/slave replication. Server 1 Server 2 12.34.56.78:6378 12.34.56.78:6379 SLAVEOF 12.34.56.78 6378 Friday, 23 November 12

Slide 92

Slide 92 text

Clients can hammer server 1 while server 2 does persistence, perhaps? Server 1 Server 2 SAVE disk Friday, 23 November 12

Slide 93

Slide 93 text

Want to do a live replacement of Redis without a!ecting clients? REPLICATE AND SWITCH! Friday, 23 November 12

Slide 94

Slide 94 text

FUTURE Friday, 23 November 12

Slide 95

Slide 95 text

Redis Sentinel Distributed monitoring Noti"cations Auto failover Client con"guration server Friday, 23 November 12

Slide 96

Slide 96 text

Redis Cluster Distributed and fault tolerant Redis Essentially.. auto-sharding (to 4096 slots) Only a subset of Redis (no complex multi-key ops) Friday, 23 November 12

Slide 97

Slide 97 text

Redis on Windows Slightly touchy point Both the community & Microsoft have done some work MS has 64 bit version coming soon Friday, 23 November 12

Slide 98

Slide 98 text

IN CLOSING Friday, 23 November 12

Slide 99

Slide 99 text

search for.. “try redis” “redis documentation” “the little redis book” “redis [your language]” Friday, 23 November 12

Slide 100

Slide 100 text

or visit.. redis.io #redis on irc.freenode.net @redisfeed Friday, 23 November 12

Slide 101

Slide 101 text

redis.io Friday, 23 November 12

Slide 102

Slide 102 text

coming mid 2013.. out in 2011 very short & simple beta out, print in 2013 Friday, 23 November 12

Slide 103

Slide 103 text

bye! [email protected] @peterc Friday, 23 November 12