Redis, The DB Faster Than You!

Applicazioni real time, logging, scalibilità, sharding, le infinite possibilità attuali di Redis con uno sguardo alla futura versione 3.

Transcript

1. Sys App Architect by Luca Cipriani 0 Redis The DB Faster Than You! Monday, September 30, 13
2. Redis, the DB Faster Than You! Sys App Architect by Luca Cipriani 1 Over the sessions * pub/sub (chat) * hipache * AWS ElastiCache for Redis Master/slave config * The Twilio issue Partitioning * Client side partitioning * Proxy partitioning (twemproxy) * Query routing The Future * Partial resync e sentinel * Redis Cluster * Redis logger and queue broker (Logstash and App Logging) Monday, September 30, 13
3. Pub/Sub 2 Pattern matching with PSUBSCRIBE news.* Superfast chat Use a callback on a PUBLISH event (notification systems, alert solutions) Warning: Data not stored persistently (use lists with BLPOP) Canali e chat Monday, September 30, 13
4. Hipache Node + Redis = Websocket + http Proxy 3 The configuration is a redis list (I personally prefer a set) Active checks on the client (I prefer to use active check BY the client with Redis TTL) Hot plug/unplug backends Fast, really fast Websockets support (thanks to node-http-proxy) Monday, September 30, 13
5. AWS ElastiCache Redis on AWS 4 Monday, September 30, 13
6. AWS ElastiCache Redis on AWS 5 Import via RDS snapshot file on S3 Version 2.6.13 (latest stable) Auto minor version upgrades Multiple nodes Zero configuration, zero maintenance, just pay. Monday, September 30, 13
7. Master/Slave Maste/Slave senza farsi male 6 Async replica, Redis 2.8 will support slave acknowledgement You can have slaves of slave (but just one master) Replication is not blocking (it uses another thread) Useful for Scalability or Redundancy Use your slave to save/backup your data Use read only slaves Use auth Redis 2.8 feature: Allow writes only with N attached replicas (relaxed CAP Theorem) • min-slaves-to-write <number of slaves> • min-slaves-max-lag <number of seconds> Monday, September 30, 13
8. The Twilio Incident The Antirez Blog Post 7 Twilio * Messaging/Chat company Two main issues: * Replication of many slaves * Misconfigured servers Redis 2.6 needs a full resync between master and slave after a connection issue. Redis 2.8 has partial resync It uses additional memory every attached slave (I know you were using EC2, slow disks, dude!) They used config-set without rewriting the config file Redis 2.8 has CONFIG REWRITE command Monday, September 30, 13
9. Partitioning Data Scaling 8 Partitioning: * larger DB (sum of memory) * multiple cores (redis is mostly single threaded) * Larger networking throughput (more TCP connections) Range Partitioning (the simple and wrong way) users from ID 0 to ID 10000 will go into instance R0 users form ID 10001 to ID 20000 will go into instance R1 ... Problems: * a table mapping is needed (where is what) * user:1, user:2 (id needed in the key) Monday, September 30, 13
10. Partitioning Divide et Impera 9 Hash Partitioning * No need for id in the key i.e. hash(foobar) = 43591045039 * Use the modulo operator to choose one of 4 backends, instance = 43591045039 % 4 Monday, September 30, 13
11. Partitioning Complex partitioning 10 Proxy assisted partitioning * clients send requests to a proxy that is able to speak the Redis protocol, no development effort Twemproxy (Redis or Memcached) from Twitter: * pipe, * sharding, * backlog * multiple hash algorithms https://github.com/twitter/twemproxy Query routing: * Redis Cluster sends your query to a random instance, the instance will forward your query to the right node. Monday, September 30, 13
12. What’s Bad With Partitions Warnings 11 Be aware of: * Multiple key insertion not directly supported (sets in different keys) * Transactions involving multiple keys do not work * The partitioning granuliary is the key, do not use big sets for it, they will be not splitted. * Complex backup restore Summary * Redis for cache: scaling up and down using consistent hashing. * Redis to store data: the map between keys and nodes fixed is needed, balance of keys is done by Redis Cluster Monday, September 30, 13
13. Presharding The way to start 12 * Redis memory footprint is small (1MB) * Start multiple redis on the same server * The number of data to move will be small * Use docker.io, it is cool! (tested with 300 containers in the same VM) Monday, September 30, 13
14. Redis Cluster The Future 13 How it works * Mix between query routing and client side partitioning. * Live check (PING PONG) * Reconfiguration of nodes (ask redirection) * Slave election * Pub/sub for node insertion notifications Monday, September 30, 13
15. Case History Cloud9 IDE uses Redis 14 Cloud9 IDE uses/used it for: * User’s data * Dynamic proxy * Queue * Logstash (Redis + ElasticSearch + Kibana) = logs with steroids * Sessions (2k users always on) Monday, September 30, 13
16. Too Fast For You DDOS Yourself 15 Monday, September 30, 13
17. Too Fast For You Ops.. DevOps. 16 Monday, September 30, 13
18. Sys App Architect Cloud Consulting Company 17 Luca Cipriani @sysapp sysapparchitect.com Sys App Architect Monday, September 30, 13