Blazing Data with Redis (TrueNorthPHP)

Blazing Data with Redis (TrueNorthPHP)

There are many fast data stores, and then there is Redis. Learn about this excellent NoSQL solution that is a powerful in-memory key-value store. Learn how to solve traditionally difficult problems with Redis, and how you can benefit from 100,000 reads/writes a second on commodity hardware. We’ll discuss how and when to use the different datatypes and commands to fit your needs. We’ll discuss the different PHP libraries with their pros and cons. We’ll then show some live examples on how to use it for a chatroom, and how Redis manages a billion data points for our dating matching system. Finally, we’ll discuss some of the upcoming features in the near future, such as clustering and scripting.

42e57550044496027f9a3a4303f13362?s=128

Justin Carmony

December 30, 2013
Tweet

Transcript

  1. Blazing Data with Redis By: Justin Carmony (and LEGOs!)

  2. About Me • Director of Development
 for DeseretNews.com • Utah

    PHP Usergroup
 President • I Make (and Break) 
 Web Stuff • Focus on Scalable, 
 Real-time Websites
 & APIs (I <3 Redis)
  3. About Presentation • We’ll ask for questions several times during

    presentation & the end. • I will post links, slides, resource, etc. • Goal: Educate & Inspire you on how, when, and
 why to use Redis!
  4. Lets Start by Measuring the Audience

  5. Let’s Talk about a Common Problem

  6. You’re a Ninja Awesome Developer

  7. Boss: “We Want Real-Time Data!” We Want to Track Online

    USers!
  8. You’re a Ninja Awesome Developer We Want To Know Who

    
 Is Viewing Our Site!
  9. “I’ll Just Add a Table
 to MySQL....”

  10. None
  11. None
  12. None
  13. None
  14. None
  15. None
  16. None
  17. None
  18. Real-Time Data

  19. Real-Time Data • High Volume Writes • High Volume Reads

    • Low Latency / Delay • Grows Exponentially compared to Rest of your Data
  20. Examples of Real-Time Data

  21. Real-Time Collaboration

  22. API Limiting

  23. Real-Time Logging

  24. Traditional Data Stores • Examples: Databases (i.e. MySQL, Postgre) •

    Writing is Resource Intensive • Stored on Disk (aka Slower) • Slower as Volume Grows • Challenge for Replication (in other words...)
  25. Meltdowns with High Volume Reads & Writes

  26. Many NoSQL Solutions

  27. Redis - Built for Speed • Key-Value Storage • In-Memory

    • Persistence • Multiple Data Types • Transactions • Pub/Sub w/ Blocking
 (and it’s Fast...)
  28. Extremely Fast...

  29. Ludicrous Speed Fast

  30. Ludicrous Speed Fast 100,000+ Reads & Writes Per Second

  31. It’s gone Plaid!

  32. API Limiting 40,000 Foot View of Redis

  33. Redis is Awesome For: • Data with a high volume

    of Reads & Writes • Simple, Less-Relational Data Sets • Data that doesn’t have to be 100% durable • Data that can shard well.
  34. How To Use Redis

  35. Step One:  Install & Run • Linux & Mac:

    Compile from Source • No Dependencies • make && make install • Windows • Download Binaries • Compile? Best of Luck
  36. Running Redis • Run: redis-server • Can use configuration file

    • Run as a “service”:
 http://redis.io/topics/ quickstart • redis-cli to try out commands

  37. Step Two:  Understand Keys

  38. Redis Keys • Keys MUST be UNIQUE • Keys are

    Binary Safe Strings • Super Long Keys (i.e. 1024 bytes) are costly to look up • Cryptic Short Keys (i.e. u:1000:pwd) have little performance improvement over a descriptive key (i.e. user:1000:password)
  39. Step Three:  Thinking Key-Value

  40. This is not an RMDBS • It’s All About the

    Keys,
 Not the Values • There is no “querying” * • There are no “indexes” • There are no “schemas” * - not 100% true since redis 2.8.0, will cover why later
  41. Document Your Data!

  42. Step Four: Get to Know  your Data Types Like

    Knowing Your Lego Pieces!
  43. Redis Strings • Store Simple Strings • Binary Safe •

    Great for JSON • Advanced Commands
  44. Data-Type “Matrix” Sorted Unsorted Comparable Sorted
 Sets Sets Stand Alone

    Lists Hashes
  45. Quick Variable Guide • Hashes - Small in Size, Very

    Efficient • Lists - Awesome Queues, Size Doesn’t Affect Performance • Sets - Great for Intersecting with others • Sorted Sets - Use to Keep “Indexes”, Sorts using Scores
  46. Step Five: Learn the Commands

  47. Learning the Commands • Generic Commands for
 All Keys •

    Commands for Each
 Data Type • Each Command has 
 Big O Notation for 
 Performance • Simple, Yet Powerful
  48. None
  49. Interactive Examples

  50. Putting It All Together

  51. Using Redis With PHP • Predis • PHP Library •

    Easy to Use • Very Fast to Update New Features • phpredis • PHP Extension • Faster, but requires
 compiling module We’ll Be Using
 Predis
  52. Example: Simple Caching

  53. Simple Cache • Data Types: • Strings • Commands: •

    SETEX <key> <seconds to expire> <value> • GET <key> • EXPIREAT <key> <timestamp>
  54. Connecting <?php
 
 // Include the Predis Autoloader
 require 'predis/lib/Predis/Autoloader.php';


    
 // Register the Autoloader
 Predis\Autoloader::register();
 
 // Create a Client with defaults
 $redis = new Predis\Client();
 
 // Create with Connection String
 $redis = new Predis\Client('tcp://10.0.0.1:6379');
 " /** Our Examples Will Assume This Is Already Done **/
  55. Simple Cache $key = 'cache.user:justin';
 
 $data_str = $redis->get($key);
 


    if($data_str)
 {
 " $data = unserialize($data_str);
 }
 else
 {
 " // Really Expensive Method of Getting This Data
 " $data = MyDatabase::GetExpensiveData();
 " $redis->setex($key, 60, serialize($data));
 }
 
 /* Do something with the $data */
  56. Example: Users Online

  57. Example: Online Users • Data Types: • Sets • Commands:

    • SADD <key> <value> • SUNION <key1> <key2> <key3> <key....> • EXPIRE <key> <timestamp>
  58. Marking Users Online /* Store Current User */
 // Current

    User ID
 $user_id = 1234;
 " // Get Current Time
 $now = time();
 $minute = date("i",$now);
 " // Generate the Key
 $key = "online:".$minute;
 " // Adding user to online users
 $redis->sadd($key, $user_id);
 $redis->expire($key, 60 * 10); // Expire in 10 minutes
  59. Getting Online Users /* Getting Onling Users */
 $keys =

    array();
 // Get Current Time
 $now = time();
 $minute = date("i",$now);
 
 $count = 0;
 $minutes_ago = 5;
 while($count < $minutes_ago)
 {
 " $keys[] = "online:".$minute;
 " 
 " $count++;
 " $min--;
 " if($min < 0)
 " {
 " " $min = 59;
 " }
 }
 // create command: SUNION online:10 online:9 online:8 online:7 online:6
 $scmd = $redis->createCommand("sunion",$keys);
 $online_ids = $redis->executeCommand($scmd);

  60. Example: Friends Online • Data Types: • Sets • Additional

    Commands: • SUNIONSTORE <dest> <key1> <key2> <key....> • SINTER <key1> <key2> <key...>
  61. My Friends Online /* My Friends Online */
 $keys =

    array('online_users');
 $user_id = '1234';
 // Get Current Time
 $minute = date("i",time());
 
 $count = 0;
 $minutes_ago = 5;
 while($count < $minutes_ago)
 {
 " $keys[] = "online:".$minute;
 " $count++;
 " $min--;
 " if($min < 0) { $min = 59; }
 }
 
 // SUNIONSTORE online_users online:10 online:9 online:8 online:7 online:6
 $scmd = $redis->createCommand("sunionstore",$keys);
 $redis->executeCommand($scmd);
 // SINTER online_users user_id:1234.friends_ids
 $online_friend_ids = $redis->sinter('online_users'
 " , 'user:'.$user_id.'.friend_ids');
  62. Under The Hood

  63. Few Things About Redis • Single Threaded • Can “Shard”

    For More Capacity / Performance • All Commands are Atomic • Transactions for Multiple Atomic Commands • Pipelining for High Performance
  64. Persistence • Snapshots on a Configurable Schedule • Will “fork”

    the process and write the DataSet to Disk • Append-Only File • Will Let You Survive a “Reboot”
  65. Lua Scripting • Introduced: Redis 2.6.0 • Allows for “querying”,

    more like Map/ Reduce • Great for common, reusable logic
  66. Pattern Scanning • In Redis 2.8.0 support was added for

    cursor-based scanning keys, hashes, sets, sorted sets, and lists.
  67. Redis Cluster • redis-cluster is currently being developed & a

    release candidate can be downloaded to try out. • Allows for master-master cluster • Right now only recommended for advanced users at the moment.
  68. How About a Live Demo?

  69. One Redis Server

  70. 25 Servers * 10 Clients = 
 250 Workers

  71. Vagrant Digital Ocean Salt Stack Redis & PHP

  72. Demo Time!

  73. Questions?

  74. Your Homework Download & Play Around with Redis! It’s Awesome!

  75. More Info? http://www.justincarmony.com/redis

  76. Flickr Credits & Images: http://gim.ie/jsLJ

  77. Thanks! Rate My Talk (PLEASE): https://joind.in/6486 " Twitter: @JustinCarmony "

    IRC: carmony #uphpu #phpc #salt " Website: http://www.justincarmony.com/blog " Email: justin@justincarmony.com
  78. None
  79. None
  80. None
  81. Misc. Unused Slides

  82. Example: Chat Room • Additional Commands: • ZREVRANGEBYSCORE <key> <max>

    <min>
  83. Example: Chat Room • Data Types: Hashes, Sorted-Sets, Pub/Sub •

    Commands: • INCR <key> • HSET <key> <field> <val> • HMGET <key> <field1> <field2> <field...> • ZADD <key> <score> <member>
  84. 25 Client Servers

  85. Getting Users Online