Learn Redis the hard way ... in production

Slide 1

Slide 1 text

Learn Redis the hard way ... in production Sunday, 20. August 2017 @ in Sankt Augustin Andy Grunwald / @andygrunwald

Slide 2

Slide 2 text

Andy Grunwald

Slide 3

Slide 3 text

Andy Grunwald

Slide 4

Slide 4 text

Andy Grunwald Site Reliability- and Platform Engineering @andygrunwald andygrunwald

Slide 5

Slide 5 text

No content

Slide 6

Slide 6 text

No content

Slide 7

Slide 7 text

6 Redis is an [...], in-memory data structure store, used as a database, cache and message broker. Redis website redis.io

Slide 8

Slide 8 text

7 Redis supports data structures such as strings, hashes, lists, sets, sorted sets with range queries, bitmaps, hyperloglogs and geospatial indexes with radius queries. Redis website redis.io

Slide 9

Slide 9 text

7 Redis supports data structures such as strings, hashes, lists, sets, sorted sets with range queries, bitmaps, hyperloglogs and geospatial indexes with radius queries. Redis website redis.io

Slide 10

Slide 10 text

8 Redis supports data structures such as strings, hashes, lists, sets, sorted sets with range queries, bitmaps, hyperloglogs and geospatial indexes with radius queries. Redis website redis.io "php", "conference" => "FrOSCon"];

Slide 11

Slide 11 text

9 Redis supports data structures such as strings, hashes, lists, sets, sorted sets with range queries, bitmaps, hyperloglogs and geospatial indexes with radius queries. Redis website redis.io

Slide 12

Slide 12 text

10 Redis supports data structures such as strings, hashes, lists, sets, sorted sets with range queries, bitmaps, hyperloglogs and geospatial indexes with radius queries. Redis website redis.io true, "conference" => true];

Slide 13

Slide 13 text

Slide 14

Slide 14 text

Slide 15

Slide 15 text

The story ...

Slide 16

Slide 16 text

No content

Slide 17

Slide 17 text

No content

Slide 18

Slide 18 text

No content

Slide 19

Slide 19 text

September, 2010

Slide 20

Slide 20 text

No content

Slide 21

Slide 21 text

February, 2013

Slide 22

Slide 22 text

phpredis (PHP-Ext) predis (PHP)

Slide 23

Slide 23 text

February, 2014

Slide 24

Slide 24 text

20 https://www.trivago.com/

Slide 25

Slide 25 text

www7.trivago.com 20140202102259941 | WARN | ... Redis\ConnectException: Unable to connect: read error on connection ... #0 /.../vendor/.../Redis/RedisPool.php(106): ...\Redis\RedisPool- >connect(Object(Redis), Object(...\Redis\RedisServerConfiguration)) #1 /.../vendor/.../Redis/RedisClient.php(130): ...\Redis\RedisPool->get('default', true) #2 /.../vendor/.../Redis/RedisClient.php(94): ...\Redis\RedisClient->setMode(false) ... #17 /.../app/bootstrap.php.cache(551): Symfony\Bundle\FrameworkBundle\HttpKernel- >handle(Object(Symfony\Component\HttpFoundation\Request), 1, true) #18 /.../web/app.php(15): Symfony\Component\HttpKernel\Kernel- >handle(Object(Symfony\Component\HttpFoundation\Request)) #19 {main} | 12.34.56.78 | www.trivago.de | /? aDateRange%5Barr%5D=2014-05-20&aDateRange%5Bdep%5D=2014-05-21&iRoomType=1&iPathId=44 742... | Mozilla/5.0 (WindowsNT 6.1; Trident/7.0; rv:11.0) like Gecko

Slide 26

Slide 26 text

No content

Slide 27

Slide 27 text

No content

Slide 28

Slide 28 text

Oct 30, 2011

Slide 29

Slide 29 text

Oct 30, 2011 Open

Slide 30

Slide 30 text

Oct 30, 2011 Open 132 comments

Slide 31

Slide 31 text

24 PHP • Raising PHP connection and command timeouts from 500ms to 2.5 seconds • Disabling the PHP setting default_socket_timeout Operating System • Disabling SYN cookies on the host systems • Checking the number of file descriptors on Redis and Webservers • Raising the mbuffer of the host systems • Control and adjust the TCP backlog sizes

Slide 32

Slide 32 text

No content

Slide 33

Slide 33 text

Slide 34

Slide 34 text

26 Closing Redis connections at the end of a web request February 2014

Slide 35

Slide 35 text

26 Closing Redis connections at the end of a web request February 2014 A/B-Testing of connection libraries March 2014

Slide 36

Slide 36 text

26 Closing Redis connections at the end of a web request February 2014 A/B-Testing of connection libraries March 2014 Upgrade Redis May 2014

Slide 37

Slide 37 text

No content

Slide 38

Slide 38 text

May, 2014

Slide 39

Slide 39 text

No content

Slide 40

Slide 40 text

30 Anything that can go wrong, will go wrong. Murphy

Slide 41

Slide 41 text

1571] 21 May 18:06:26.550 * Background saving started by pid 20267 [20267] 21 May 18:08:25.195 * DB saved on disk [1571 | signal handler] (1400688506) --- WATCHDOG TIMER EXPIRED --- [1571] 21 May 18:08:26.331 # === REDIS BUG REPORT START: Cut & paste starting from here === [1571] 21 May 18:08:26.331 # Redis 2.8.9 crashed by signal: 11 [1571] 21 May 18:08:26.331 # Failed assertion: (:0) [1571] 21 May 18:08:26.331 # --- STACK TRACE _._ _.-``__ ''-._ _.-`` `. `_. ''-._ Redis 2.8.9 (00000000/0) 64 bit .-`` .-```. ```\/ _.,_ ''-._ ( ' , .-` | `, ) Running in stand alone mode |`-._`-...-` __...-.``-._|'` _.-'| Port: 6380 | `-._ `._ / _.-' | PID: 20398 `-._ `-._ `-./ _.-' _.-' |`-._`-._ `-.__.-' _.-'_.-'| | `-._`-._ _.-'_.-' | http://redis.io `-._ `-._`-.__.-'_.-' _.-'

Slide 42

Slide 42 text

Tip #1  Redis is single threaded Every command will block for a certain time.

Slide 43

Slide 43 text

... [20398] 22 May 09:20:55.351 * 10000 changes in 60 seconds. Saving... [20398] 22 May 09:20:55.759 * Background saving started by pid 41941 [41941] 22 May 09:22:48.197 * DB saved on disk [20398] 22 May 09:22:49.321 * Background saving terminated with success [20398] 22 May 09:25:23.299 * 10000 changes in 60 seconds. Saving... [20398] 22 May 09:25:23.644 * Background saving started by pid 42027 [20398] 22 May 09:26:50.646 # Accepting client connection: accept: Software caused connection abort [20398] 22 May 09:26:50.900 # Accepting client connection: accept: Software caused connection abort ...

Slide 44

Slide 44 text

Slide 45

Slide 45 text

Slide 46

Slide 46 text

Tip #2  Review your persistence requirements Sane defaults, but does those apply to you? BGSAVE or AOF?

Slide 47

Slide 47 text

No content

Slide 48

Slide 48 text

Pipelining

Slide 49

Slide 49 text

Tip #3  Shared data contexts might block each other. Different use cases and commands. SELECT == Antipattern

Slide 50

Slide 50 text

June, 2014

Slide 51

Slide 51 text

Bachmann-Landau Notation

Slide 52

Slide 52 text

Big O Notation

Slide 53

Slide 53 text

No content

Slide 54

Slide 54 text

No content

Slide 55

Slide 55 text

O(n) can kill you

Slide 56

Slide 56 text

Tip #4  Check the time complexity of your commands High Big O estimations can slow down the performance

Slide 57

Slide 57 text

August, 2014

Slide 58

Slide 58 text

No content

Slide 59

Slide 59 text

No content

Slide 60

Slide 60 text

Tip #5  Consider a proxy in between Reduce connection overhead and block expensive / dangerous commands

Slide 61

Slide 61 text

September, 2014 up to now

Slide 62

Slide 62 text

If {# of data} > available RAM

Slide 63

Slide 63 text

No content

Slide 64

Slide 64 text

51 Eviction policies • noeviction: return errors when the memory limit was reached ...  • allkeys-lru: [...] less recently used (LRU) keys first • volatile-lru: [...] less recently used (LRU) keys first (with TTL)  • allkeys-random: evict random keys ...  • volatile-random: evict random keys (with TTL)  • volatile-ttl: [...] evict keys with an expire set, with a shorter time to live (TTL) first.

Slide 65

Slide 65 text

Tip #6  Review your runtime requirements Plan your maxmemory limit and set the right maxmemory-policy

Slide 66

Slide 66 text

If {# of data} > available RAM

Slide 67

Slide 67 text

No content

Slide 68

Slide 68 text

Tip #7  Plan your capacity and use it Shard your data over multiple machines to store more and be resistent to failure

Slide 69

Slide 69 text

No content

Slide 70

Slide 70 text

57 Redis is a very sharp knife.   It is perfect for some problems, and a horrible fit for others. btilly HackerNews, 31. March 2017

Slide 71

Slide 71 text

58 One problem with Redis is that it looks superficially simple, but actually, like any tool used in critical contexts, there are two possibilities:   1) Know how it works very well and do great things with it.   2) Don't understand it properly and find yourself in big troubles soon or later. antirez HackerNews, 31. March 2017

Slide 72

Slide 72 text

Learn Redis the hard way ... in production Sunday, 20. August 2017 @ in Sankt Augustin Andy Grunwald / @andygrunwald