Slide 1

Slide 1 text

(advanced) Redis Advanced is between brackets because we don’t claim to be advanced users, we just have gained a lot of experience

Slide 2

Slide 2 text

Jan De Poorter @defv

Slide 3

Slide 3 text

Redis?

Slide 4

Slide 4 text

NoSQL

Slide 5

Slide 5 text

:-D

Slide 6

Slide 6 text

No, but srsly

Slide 7

Slide 7 text

Redis is an in-memory key-value data-structure server

Slide 8

Slide 8 text

in memory with “best effort” persistence

Slide 9

Slide 9 text

Data structure server Strings SET, SETNX, GET, INCR, DECR, APPEND, ... Hashes HSET, HGET, HGETALL, HDEL, HLEN, ... Lists LPUSH, LPOP, LINDEX, LLEN, LRANGE, ... Sets & Sorted sets SADD, SPOP, SMEMBERS, SDIFF, ZADD, ...

Slide 10

Slide 10 text

some libraries Ruby redis-rb (gem) hiredis-rb (gem) Node redis (npm) hiredis (npm)

Slide 11

Slide 11 text

Q-Music Our use case was the API for the new Q-Music website and the iPhone and Android applications.

Slide 12

Slide 12 text

High volume

Slide 13

Slide 13 text

Pub-Sub

Slide 14

Slide 14 text

BG jobs with Sidekiq

Slide 15

Slide 15 text

Some do’s

Slide 16

Slide 16 text

Serialize values

Slide 17

Slide 17 text

2 class QApi::Redis 3 class << self 4 def redis 5 @redis ||= Redis.new(config) 6 end ... 181 def set(key, data) 182 redis.set key, to_json(data) 183 end 184 185 def get(key) 186 from_json redis.get(key) 187 end 188 216 private 217 def from_json(result) 218 case result 219 when Array 220 result.map { |r| from_json(r) } 221 when Hash 222 result 223 when nil 224 nil 225 else 226 MultiJson.decode(result) 227 end 228 rescue MultiJson::DecodeError 229 result 230 end 231 232 def to_json(data) 233 MultiJson.encode(data) 234 end 262 end 263 end

Slide 18

Slide 18 text

Save references

Slide 19

Slide 19 text

# Write LPUSH plays {“id”: 42, ”title”:”99 Bottles of Rum”} LPUSH plays {“id”: 49, ”title”:”The Drunken Sailor”} LPUSH plays {“id”: 42, ”title”:”99 Bottles of Rum”} # Read LRANGE plays 0 -1 don’t do this

Slide 20

Slide 20 text

it’s fine with 100 items

Slide 21

Slide 21 text

but becomes huge with 90 000 items

Slide 22

Slide 22 text

or 1 000 000 Justin Bieber mentions ;-)

Slide 23

Slide 23 text

# Write SET tracks:42 {“id”: 42, ”title”:”99 Bottles of Rum”} SET tracks:49 {“id”: 42, ”title”:” The Drunken Sailor”} LPUSH plays tracks:42 LPUSH plays tracks:49 LPUSH plays tracks:42 # Read LRANGE plays 0 -1 MGET tracks:42 tracks:49 do this instead

Slide 24

Slide 24 text

or in Ruby 82 def list_from_references(list, options) 83 references = redis.lrange list, 84 options[:start], options[:stop] 85 if references.any? 86 from_json redis.mget(*references) 87 else 88 [] 89 end 90 end

Slide 25

Slide 25 text

Some don’t’s

Slide 26

Slide 26 text

RDB vs AOF

Slide 27

Slide 27 text

RDB does fork()

Slide 28

Slide 28 text

doubling memory usage

Slide 29

Slide 29 text

Our dataset was 4GB

Slide 30

Slide 30 text

We had ~8GB of memory

Slide 31

Slide 31 text

No BGSAVE’s for 4 days

Slide 32

Slide 32 text

:-(

Slide 33

Slide 33 text

appendonly yes

Slide 34

Slide 34 text

Slow(er) start-up

Slide 35

Slide 35 text

Can be optimized

Slide 36

Slide 36 text

Monitoring

Slide 37

Slide 37 text

data:~$ redis-cli redis > INFO redis_version:2.4.16 ... connected_clients:152 ... used_memory:2744963744 used_memory_human:2.56G used_memory_peak:6150240632 used_memory_peak_human:5.73G ... changes_since_last_save:9576 last_save_time:1360661315 redis >

Slide 38

Slide 38 text

PUBLISH & SUBSCRIBE overload

Slide 39

Slide 39 text

WebSocket clients

Slide 40

Slide 40 text

1 client == 1 connection == 1 subscription

Slide 41

Slide 41 text

Works perfect in development

Slide 42

Slide 42 text

Works perfect in staging

Slide 43

Slide 43 text

but in production...

Slide 44

Slide 44 text

1 new event

Slide 45

Slide 45 text

sent to 3500 connected clients

Slide 46

Slide 46 text

Takes about 0.3 seconds

Slide 47

Slide 47 text

Up to 4 messages per second

Slide 48

Slide 48 text

:-(

Slide 49

Slide 49 text

Move logic to websocket server

Slide 50

Slide 50 text

:-(

Slide 51

Slide 51 text

1 type of msg == 1 subscription

Slide 52

Slide 52 text

:-)

Slide 53

Slide 53 text

MULTI / EXEC

Slide 54

Slide 54 text

> MULTI OK > RENAME bl:1 bld:1 QUEUED > RENAME bl:2 bld:2 QUEUED > EXEC 1) OK 2) OK

Slide 55

Slide 55 text

Works great on 1000’s of commands

Slide 56

Slide 56 text

Not so much on 1 million commands...

Slide 57

Slide 57 text

Don’t exceed ~10k commands in 1 transaction

Slide 58

Slide 58 text

master/slave

Slide 59

Slide 59 text

no library support

Slide 60

Slide 60 text

DIY

Slide 61

Slide 61 text

We didn’t

Slide 62

Slide 62 text

Limited connections

Slide 63

Slide 63 text

Redis < 2.6

Slide 64

Slide 64 text

/* Max number of fd supported */ #define AE_SETSIZE (1024*10)

Slide 65

Slide 65 text

No more then ~10k connections allowed

Slide 66

Slide 66 text

Redis >= 2.6

Slide 67

Slide 67 text

The sky is the limit

Slide 68

Slide 68 text

(actually, the linux max FD is the limit)

Slide 69

Slide 69 text

Questions?