Blazing Data with Redis (TrueNorthPHP)

There are many fast data stores, and then there is Redis. Learn about this excellent NoSQL solution that is a powerful in-memory key-value store. Learn how to solve traditionally difficult problems with Redis, and how you can benefit from 100,000 reads/writes a second on commodity hardware. We’ll discuss how and when to use the different datatypes and commands to fit your needs. We’ll discuss the different PHP libraries with their pros and cons. We’ll then show some live examples on how to use it for a chatroom, and how Redis manages a billion data points for our dating matching system. Finally, we’ll discuss some of the upcoming features in the near future, such as clustering and scripting.

Transcript

1. Blazing Data with Redis By: Justin Carmony (and LEGOs!)
2. About Me • Director of Development
 for DeseretNews.com • Utah PHP Usergroup
 President • I Make (and Break) 
 Web Stuff • Focus on Scalable, 
 Real-time Websites
 & APIs (I <3 Redis)
3. About Presentation • We’ll ask for questions several times during presentation & the end. • I will post links, slides, resource, etc. • Goal: Educate & Inspire you on how, when, and
 why to use Redis!
4. Lets Start by Measuring the Audience
5. Let’s Talk about a Common Problem
6. You’re a Ninja Awesome Developer
7. Boss: “We Want Real-Time Data!” We Want to Track Online USers!
8. You’re a Ninja Awesome Developer We Want To Know Who 
 Is Viewing Our Site!
9. “I’ll Just Add a Table
 to MySQL....”
18. Real-Time Data
19. Real-Time Data • High Volume Writes • High Volume Reads • Low Latency / Delay • Grows Exponentially compared to Rest of your Data
20. Examples of Real-Time Data
21. Real-Time Collaboration
22. API Limiting
23. Real-Time Logging
24. Traditional Data Stores • Examples: Databases (i.e. MySQL, Postgre) • Writing is Resource Intensive • Stored on Disk (aka Slower) • Slower as Volume Grows • Challenge for Replication (in other words...)
25. Meltdowns with High Volume Reads & Writes
26. Many NoSQL Solutions
27. Redis - Built for Speed • Key-Value Storage • In-Memory • Persistence • Multiple Data Types • Transactions • Pub/Sub w/ Blocking
 (and it’s Fast...)
28. Extremely Fast...
29. Ludicrous Speed Fast
30. Ludicrous Speed Fast 100,000+ Reads & Writes Per Second
31. It’s gone Plaid!
32. API Limiting 40,000 Foot View of Redis
33. Redis is Awesome For: • Data with a high volume of Reads & Writes • Simple, Less-Relational Data Sets • Data that doesn’t have to be 100% durable • Data that can shard well.
34. How To Use Redis
35. Step One:
    Install & Run • Linux & Mac: Compile from Source • No Dependencies • make && make install • Windows • Download Binaries • Compile? Best of Luck
36. Running Redis • Run: redis-server • Can use configuration file • Run as a “service”:
 http://redis.io/topics/ quickstart • redis-cli to try out commands

37. Step Two:
    Understand Keys
38. Redis Keys • Keys MUST be UNIQUE • Keys are Binary Safe Strings • Super Long Keys (i.e. 1024 bytes) are costly to look up • Cryptic Short Keys (i.e. u:1000:pwd) have little performance improvement over a descriptive key (i.e. user:1000:password)
39. Step Three:
    Thinking Key-Value
40. This is not an RMDBS • It’s All About the Keys,
 Not the Values • There is no “querying” * • There are no “indexes” • There are no “schemas” * - not 100% true since redis 2.8.0, will cover why later
41. Document Your Data!
42. Step Four: Get to Know
    your Data Types Like Knowing Your Lego Pieces!
43. Redis Strings • Store Simple Strings • Binary Safe • Great for JSON • Advanced Commands
44. Data-Type “Matrix” Sorted Unsorted Comparable Sorted
 Sets Sets Stand Alone Lists Hashes
45. Quick Variable Guide • Hashes - Small in Size, Very Efficient • Lists - Awesome Queues, Size Doesn’t Affect Performance • Sets - Great for Intersecting with others • Sorted Sets - Use to Keep “Indexes”, Sorts using Scores
46. Step Five:
    Learn the Commands
47. Learning the Commands • Generic Commands for
 All Keys • Commands for Each
 Data Type • Each Command has 
 Big O Notation for 
 Performance • Simple, Yet Powerful
49. Interactive Examples
50. Putting It All Together
51. Using Redis With PHP • Predis • PHP Library • Easy to Use • Very Fast to Update New Features • phpredis • PHP Extension • Faster, but requires
 compiling module We’ll Be Using
 Predis
52. Example: Simple Caching
53. Simple Cache • Data Types: • Strings • Commands: • SETEX <key> <seconds to expire> <value> • GET <key> • EXPIREAT <key> <timestamp>
54. Connecting <?php
 
 // Include the Predis Autoloader
 require 'predis/lib/Predis/Autoloader.php';
 
 // Register the Autoloader
 Predis\Autoloader::register();
 
 // Create a Client with defaults
 $redis = new Predis\Client();
 
 // Create with Connection String
 $redis = new Predis\Client('tcp://10.0.0.1:6379');
 " /** Our Examples Will Assume This Is Already Done **/
55. Simple Cache $key = 'cache.user:justin';
 
 $data_str = $redis->get($key);
 
 if($data_str)
 {
 " $data = unserialize($data_str);
 }
 else
 {
 " // Really Expensive Method of Getting This Data
 " $data = MyDatabase::GetExpensiveData();
 " $redis->setex($key, 60, serialize($data));
 }
 
 /* Do something with the $data */
56. Example: Users Online
57. Example: Online Users • Data Types: • Sets • Commands: • SADD <key> <value> • SUNION <key1> <key2> <key3> <key....> • EXPIRE <key> <timestamp>
58. Marking Users Online /* Store Current User */
 // Current User ID
 $user_id = 1234;
 " // Get Current Time
 $now = time();
 $minute = date("i",$now);
 " // Generate the Key
 $key = "online:".$minute;
 " // Adding user to online users
 $redis->sadd($key, $user_id);
 $redis->expire($key, 60 * 10); // Expire in 10 minutes
59. Getting Online Users /* Getting Onling Users */
 $keys = array();
 // Get Current Time
 $now = time();
 $minute = date("i",$now);
 
 $count = 0;
 $minutes_ago = 5;
 while($count < $minutes_ago)
 {
 " $keys[] = "online:".$minute;
 " 
 " $count++;
 " $min--;
 " if($min < 0)
 " {
 " " $min = 59;
 " }
 }
 // create command: SUNION online:10 online:9 online:8 online:7 online:6
 $scmd = $redis->createCommand("sunion",$keys);
 $online_ids = $redis->executeCommand($scmd);

60. Example: Friends Online • Data Types: • Sets • Additional Commands: • SUNIONSTORE <dest> <key1> <key2> <key....> • SINTER <key1> <key2> <key...>
61. My Friends Online /* My Friends Online */
 $keys = array('online_users');
 $user_id = '1234';
 // Get Current Time
 $minute = date("i",time());
 
 $count = 0;
 $minutes_ago = 5;
 while($count < $minutes_ago)
 {
 " $keys[] = "online:".$minute;
 " $count++;
 " $min--;
 " if($min < 0) { $min = 59; }
 }
 
 // SUNIONSTORE online_users online:10 online:9 online:8 online:7 online:6
 $scmd = $redis->createCommand("sunionstore",$keys);
 $redis->executeCommand($scmd);
 // SINTER online_users user_id:1234.friends_ids
 $online_friend_ids = $redis->sinter('online_users'
 " , 'user:'.$user_id.'.friend_ids');
62. Under The Hood
63. Few Things About Redis • Single Threaded • Can “Shard” For More Capacity / Performance • All Commands are Atomic • Transactions for Multiple Atomic Commands • Pipelining for High Performance
64. Persistence • Snapshots on a Configurable Schedule • Will “fork” the process and write the DataSet to Disk • Append-Only File • Will Let You Survive a “Reboot”
65. Lua Scripting • Introduced: Redis 2.6.0 • Allows for “querying”, more like Map/ Reduce • Great for common, reusable logic
66. Pattern Scanning • In Redis 2.8.0 support was added for cursor-based scanning keys, hashes, sets, sorted sets, and lists.
67. Redis Cluster • redis-cluster is currently being developed & a release candidate can be downloaded to try out. • Allows for master-master cluster • Right now only recommended for advanced users at the moment.
68. How About a Live Demo?
69. One Redis Server
70. 25 Servers * 10 Clients = 
 250 Workers
71. Vagrant
    Digital Ocean
    Salt Stack
    Redis & PHP
72. Demo Time!
73. Questions?
74. Your Homework Download & Play Around with Redis! It’s Awesome!
75. More Info? http://www.justincarmony.com/redis
76. Flickr Credits & Images: http://gim.ie/jsLJ
77. Thanks! Rate My Talk (PLEASE): https://joind.in/6486 " Twitter: @JustinCarmony " IRC: carmony #uphpu #phpc #salt " Website: http://www.justincarmony.com/blog " Email: justin@justincarmony.com
81. Misc. Unused Slides
82. Example: Chat Room • Additional Commands: • ZREVRANGEBYSCORE <key> <max> <min>
83. Example: Chat Room • Data Types: Hashes, Sorted-Sets, Pub/Sub • Commands: • INCR <key> • HSET <key> <field> <val> • HMGET <key> <field1> <field2> <field...> • ZADD <key> <score> <member>
84. 25 Client Servers
85. Getting Users Online