Blazing Data with Redis (TrueNorthPHP)

There are many fast data stores, and then there is Redis. Learn about this excellent NoSQL solution that is a powerful in-memory key-value store. Learn how to solve traditionally difficult problems with Redis, and how you can benefit from 100,000 reads/writes a second on commodity hardware. We’ll discuss how and when to use the different datatypes and commands to fit your needs. We’ll discuss the different PHP libraries with their pros and cons. We’ll then show some live examples on how to use it for a chatroom, and how Redis manages a billion data points for our dating matching system. Finally, we’ll discuss some of the upcoming features in the near future, such as clustering and scripting.

Transcript

1. Blazing Data with Redis
   By: Justin Carmony
   (and LEGOs!)
   

   View Slide

2. About Me
   • Director of Development

   for DeseretNews.com
   
   • Utah PHP Usergroup

   President
   
   • I Make (and Break) 

   Web Stuff
   
   • Focus on Scalable, 

   Real-time Websites

   & APIs
   (I <3 Redis)
   

   View Slide

3. About Presentation
   • We’ll ask for questions
   several times during
   presentation & the end.
   
   • I will post links, slides,
   resource, etc.
   
   • Goal: Educate & Inspire
   you on how, when, and

   why to use Redis!
   

   View Slide

4. Lets Start by
   Measuring the
   Audience
   

   View Slide

5. Let’s Talk about a
   Common Problem
   

   View Slide

6. You’re a Ninja
   Awesome
   Developer
   

   View Slide

7. Boss: “We Want
   Real-Time Data!”
   We Want to
   Track Online
   USers!
   

   View Slide

8. You’re a Ninja
   Awesome
   Developer
   We Want To Know Who 

   Is Viewing Our Site!
   

   View Slide

9. “I’ll Just Add a Table

   to MySQL....”
   

   View Slide

10. View Slide

11. View Slide

12. View Slide

13. View Slide

14. View Slide

15. View Slide

16. View Slide

17. View Slide

18. Real-Time Data
    

    View Slide

19. Real-Time Data
    • High Volume Writes
    
    • High Volume Reads
    
    • Low Latency / Delay
    
    • Grows Exponentially
    compared to Rest of
    your Data
    

    View Slide

20. Examples of Real-Time Data
    

    View Slide

21. Real-Time Collaboration
    

    View Slide

22. API Limiting
    

    View Slide

23. Real-Time Logging
    

    View Slide

24. Traditional Data Stores
    • Examples: Databases (i.e.
    MySQL, Postgre)
    
    • Writing is Resource
    Intensive
    
    • Stored on Disk (aka
    Slower)
    
    • Slower as Volume Grows
    
    • Challenge for Replication
    (in other words...)
    

    View Slide

25. Meltdowns with High Volume
    Reads & Writes
    

    View Slide

26. Many NoSQL Solutions
    

    View Slide

27. Redis - Built for Speed
    • Key-Value Storage
    
    • In-Memory
    
    • Persistence
    
    • Multiple Data Types
    
    • Transactions
    
    • Pub/Sub w/ Blocking

    (and it’s Fast...)
    

    View Slide

28. Extremely Fast...
    

    View Slide

29. Ludicrous Speed Fast
    

    View Slide

30. Ludicrous Speed Fast
    100,000+
    
    Reads & Writes
    
    Per Second
    

    View Slide

31. It’s gone Plaid!
    

    View Slide

32. API Limiting
    40,000 Foot View of Redis
    

    View Slide

33. Redis is Awesome For:
    • Data with a high volume of
    Reads & Writes
    
    • Simple, Less-Relational
    Data Sets
    
    • Data that doesn’t have to
    be 100% durable
    
    • Data that can shard well.
    

    View Slide

34. How To
    Use Redis
    

    View Slide

35. Step One:
    Install & Run
    • Linux & Mac: Compile
    from Source
    
    • No Dependencies
    
    • make && make install
    
    • Windows
    
    • Download Binaries
    
    • Compile? Best of Luck
    

    View Slide

36. Running Redis
    • Run: redis-server
    
    • Can use configuration
    file
    
    • Run as a “service”:

    http://redis.io/topics/
    quickstart
    
    • redis-cli to try out
    commands

    

    View Slide

37. Step Two:
    Understand Keys
    

    View Slide

38. Redis Keys
    • Keys MUST be UNIQUE
    
    • Keys are Binary Safe Strings
    
    • Super Long Keys (i.e. 1024 bytes) are costly to
    look up
    
    • Cryptic Short Keys (i.e. u:1000:pwd) have
    little performance improvement over a
    descriptive key (i.e. user:1000:password)
    

    View Slide

39. Step Three:
    Thinking Key-Value
    

    View Slide

40. This is not an RMDBS
    • It’s All About the Keys,

    Not the Values
    
    • There is no “querying” *
    
    • There are no “indexes”
    
    • There are no “schemas”
    * - not 100% true since redis 2.8.0, will cover why later
    

    View Slide

41. Document
    Your Data!
    

    View Slide

42. Step Four: Get to Know
    your Data Types
    Like Knowing Your
    Lego Pieces!
    

    View Slide

43. Redis Strings
    • Store Simple
    Strings
    
    • Binary Safe
    
    • Great for JSON
    
    • Advanced
    Commands
    

    View Slide

44. Data-Type “Matrix”
    Sorted Unsorted
    Comparable
    Sorted

    Sets
    Sets
    Stand Alone Lists Hashes
    

    View Slide

45. Quick Variable Guide
    • Hashes - Small in Size,
    Very Efficient
    
    • Lists - Awesome
    Queues, Size Doesn’t
    Affect Performance
    
    • Sets - Great for
    Intersecting with others
    
    • Sorted Sets - Use to
    Keep “Indexes”, Sorts
    using Scores
    

    View Slide

46. Step Five:
    Learn the Commands
    

    View Slide

47. Learning the Commands
    • Generic Commands for

    All Keys
    
    • Commands for Each

    Data Type
    
    • Each Command has 

    Big O Notation for 

    Performance
    
    • Simple, Yet Powerful
    

    View Slide

48. View Slide

49. Interactive
    Examples
    

    View Slide

50. Putting It
    
    All Together
    

    View Slide

51. Using Redis With PHP
    • Predis
    
    • PHP Library
    
    • Easy to Use
    
    • Very Fast to Update
    New Features
    
    • phpredis
    
    • PHP Extension
    
    • Faster, but requires

    compiling module We’ll Be Using

    Predis
    

    View Slide

52. Example: Simple Caching
    

    View Slide

53. Simple Cache
    • Data Types:
    
    • Strings
    
    • Commands:
    
    • SETEX
    
    • GET
    
    • EXPIREAT
    

    View Slide

54. Connecting
    

    // Include the Predis Autoloader

    require 'predis/lib/Predis/Autoloader.php';

    

    // Register the Autoloader

    Predis\Autoloader::register();

    

    // Create a Client with defaults

    $redis = new Predis\Client();

    

    // Create with Connection String

    $redis = new Predis\Client('tcp://10.0.0.1:6379');

    "
    /** Our Examples Will Assume This Is Already Done **/
    

    View Slide

55. Simple Cache
    $key = 'cache.user:justin';

    

    $data_str = $redis->get($key);

    

    if($data_str)

    {

    " $data = unserialize($data_str);

    }

    else

    {

    " // Really Expensive Method of Getting This Data

    " $data = MyDatabase::GetExpensiveData();

    " $redis->setex($key, 60, serialize($data));

    }

    

    /* Do something with the $data */
    

    View Slide

56. Example: Users Online
    

    View Slide

57. Example: Online Users
    • Data Types:
    
    • Sets
    
    • Commands:
    
    • SADD
    
    • SUNION
    
    • EXPIRE
    

    View Slide

58. Marking Users Online
    /* Store Current User */

    // Current User ID

    $user_id = 1234;

    "
    // Get Current Time

    $now = time();

    $minute = date("i",$now);

    "
    // Generate the Key

    $key = "online:".$minute;

    "
    // Adding user to online users

    $redis->sadd($key, $user_id);

    $redis->expire($key, 60 * 10); // Expire in 10 minutes
    

    View Slide

59. Getting Online Users
    /* Getting Onling Users */

    $keys = array();

    // Get Current Time

    $now = time();

    $minute = date("i",$now);

    

    $count = 0;

    $minutes_ago = 5;

    while($count < $minutes_ago)

    {

    " $keys[] = "online:".$minute;

    " 

    " $count++;

    " $min--;

    " if($min < 0)

    " {

    " " $min = 59;

    " }

    }

    // create command: SUNION online:10 online:9 online:8 online:7 online:6

    $scmd = $redis->createCommand("sunion",$keys);

    $online_ids = $redis->executeCommand($scmd);

    

    View Slide

60. Example: Friends Online
    • Data Types:
    
    • Sets
    
    • Additional Commands:
    
    • SUNIONSTORE
    
    • SINTER
    

    View Slide

61. My Friends Online
    /* My Friends Online */

    $keys = array('online_users');

    $user_id = '1234';

    // Get Current Time

    $minute = date("i",time());

    

    $count = 0;

    $minutes_ago = 5;

    while($count < $minutes_ago)

    {

    " $keys[] = "online:".$minute;

    " $count++;

    " $min--;

    " if($min < 0) { $min = 59; }

    }

    

    // SUNIONSTORE online_users online:10 online:9 online:8 online:7 online:6

    $scmd = $redis->createCommand("sunionstore",$keys);

    $redis->executeCommand($scmd);

    // SINTER online_users user_id:1234.friends_ids

    $online_friend_ids = $redis->sinter('online_users'

    " , 'user:'.$user_id.'.friend_ids');
    

    View Slide

62. Under The Hood
    

    View Slide

63. Few Things About Redis
    • Single Threaded
    
    • Can “Shard” For More Capacity /
    Performance
    
    • All Commands are Atomic
    
    • Transactions for Multiple Atomic
    Commands
    
    • Pipelining for High Performance
    

    View Slide

64. Persistence
    • Snapshots on a Configurable Schedule
    
    • Will “fork” the process and write the
    DataSet to Disk
    
    • Append-Only File
    
    • Will Let You Survive a “Reboot”
    

    View Slide

65. Lua Scripting
    • Introduced: Redis 2.6.0
    
    • Allows for “querying”, more like Map/
    Reduce
    
    • Great for common, reusable logic
    

    View Slide

66. Pattern Scanning
    • In Redis 2.8.0 support was added for
    cursor-based scanning keys, hashes, sets,
    sorted sets, and lists.
    

    View Slide

67. Redis Cluster
    • redis-cluster is currently being developed &
    a release candidate can be downloaded to
    try out.
    
    • Allows for master-master cluster
    
    • Right now only recommended for advanced
    users at the moment.
    

    View Slide

68. How About a Live
    Demo?
    

    View Slide

69. One Redis Server
    

    View Slide

70. 25 Servers * 10 Clients = 

    250 Workers
    

    View Slide

71. Vagrant
    Digital Ocean
    Salt Stack
    Redis & PHP
    

    View Slide

72. Demo Time!
    

    View Slide

73. Questions?
    

    View Slide

74. Your Homework
    Download & Play Around with Redis!
    
    It’s Awesome!
    

    View Slide

75. More Info?
    http://www.justincarmony.com/redis
    

    View Slide

76. Flickr Credits & Images:
    http://gim.ie/jsLJ
    

    View Slide

77. Thanks!
    Rate My Talk (PLEASE): https://joind.in/6486
    
    "
    Twitter: @JustinCarmony
    
    "
    IRC: carmony
    
    #uphpu #phpc #salt
    
    "
    Website:
    
    http://www.justincarmony.com/blog
    
    "
    Email:
    
    justin@justincarmony.com
    

    View Slide

78. View Slide

79. View Slide

80. View Slide

81. Misc. Unused Slides
    

    View Slide

82. Example: Chat Room
    • Additional Commands:
    
    • ZREVRANGEBYSCORE
    
    

    View Slide

83. Example: Chat Room
    • Data Types: Hashes, Sorted-Sets, Pub/Sub
    
    • Commands:
    
    • INCR
    
    • HSET <field>
    
    • HMGET <field1> <field2> <field...>
    
    • ZADD
    

    View Slide

84. 25 Client Servers
    

    View Slide

85. Getting Users Online
    

    View Slide