Redis - Lightning Talk @ RefME

Transcript

2. what? • “Redis is an open source, BSD licensed, advanced key-value cache and store. It is often referred to as a data structure server” • Not quite a NOSQL database • Not quite a key value store
3. what? • Single threaded and event driven • Everything in memory • Individual operations are atomic • Fast (~10k EC2 t1-micro, ~30k m1-medium) • Clients in many languages
4. GETTING STARTED brew install redis # start the server redis-server # run commands redis-cli
5. BASICS SET mykey 1000 # -> 1 (success) GET mykey # -> 1000 EXPIRE mykey 5 # 6 seconds later ... GET mykey # -> nil
6. BASICS GET counter # -> nil INCR counter # -> 1 SET counter_2 "100" DECRBY counter_2 10 # -> 90
7. LISTS LPUSH mylist 100 LPUSH mylist 200 LRANGE mylist 0 100 # -> 1) 100 # -> 2) 200 RPUSH mylist -100 LLEN mylist # -> 3
8. LISTS AS QUEUES RPUSH myqueue 1000 RPUSH myqueue 1001 LRANGE myqueue 0 -1 # -> 998,999,1000,1001 LPOP myqueue # -> 998 LTRIM myqueue 0 99
9. SETS SADD neds_kids 'Arya' SMEMBERS neds_kids # -> Jon,Robb,Arya,Sansa,Brann,Rickon SCARD neds_kids # -> 6 SISMEMBER nightswatch 'Brann' # -> 0 SINTER nightswatch neds_kids # -> Jon
10. SORTED SETS ZADD scoreboard 8 "Jesse" ZADD scoreboard 7 "Walter" ZADD scoreboard 10 "Badger" ZREVRANGE scoreboard 0 1 # -> Badger, Jesse
11. HASHES HSET user:1 name "Luke" HSET user:1 role "Developer" HGET user:1 name # -> "Luke" # also HVALUES, HKEYS, HINCR, HGETALL
12. USES? • Counters (view counters, sign in counters) • Logs • Queues / Background Jobs (Resque, Sidekiq) • Cache • Inter-process & Inter-machine communication • Pub/Sub
13. BATCH PROCESSING • 5-10m line file on AWS s3, split file into 1000 line chunks • Parallelize work across N processes on M workers • Use Redis keys to track progress on UI of overall job e.g job:1010:status, job:1010:parts_remaining • Heavy API usage, use redis-based queue to cycle through API keys cross machine. Also mutex locks
14. BAYES CLASSIFIER require 'bayes_classifier' c = BayesClassifier.new c.train('Liked', 'Fantastic product, would recommend') c.train('Liked', 'Very sturdy bed, easy to build.') c.train('Disliked', 'Terrible design, already broken') c.train('Disliked', 'I am disappointed') c.classify('Terrible and rubbish') # -> Disliked
15. BAYES CLASSIFIER c.train('Disliked', 'Terrible design, already broken') # under the hood @words_count['Disliked'] += 3 @category_counts['Disliked']['terrible'] += 1 @category_counts['Disliked']['design'] += 1 @category_counts['Disliked']['broken'] += 1 c.classify("Terrible and rubbish") # .... maths!(logarithms)
16. BAYES CLASSIFIER c.train('Disliked', 'Terrible design, already broken') # under the hood redis.incrby('category_counts:disliked', 3) redis.hincrby('category_words:disliked', 'terrible', 1) redis.hincrby('category_words:disliked', 'design', 1) redis.hincrby('category_words:disliked', 'broken', 1) c.classify("Terrible and rubbish") # .... same maths, but more redis
17. BATCH PROCESSING • 10 million keywords trained in 2 hours using AWS Elasticache and Resque • Classification is O(1) due to data structures, same speed regardless of size of trained data set
18. CAVEATS • Persistence is not easy, snapshots or Append Only File and playback • No evictions by default • No partitions by default • New Redis Cluster may solve the above (April 1st 2015)
19. STEP 1:
20. STEP 2: ???
21. STEP 3: PROFIT