Scaling PHP Applications With Redis (Nomad PHP July 2015)

Transcript

1. Scaling PHP Applications With Redis Josh Butts Nomad PHP, July 2015 https://joind.in/14896
2. About Me • VP of Engineering at Offers.com in Austin, Texas • Long-time Austin PHP organizer • Competitive Skee-baller • Follow me online: • github.com/jimbojsb • @jimbojsb
3. Agenda • Redis Overview • Types of keys • PHP Strategies
4. What is Redis? • Redis is an “Advanced” key-value store database • Backed by VMWare • Redis works like memcached, but better: • Atomic commands & transactions • Server-side data structures • Comparable speed
5. Redis in an in-memory database • You need as much RAM as you have data • Disk persistence is exactly that • Customize disk writes interval to suit your pain threshold • Something interesting happens when you run out of memory
6. Redis is Single-Threaded • Redis uses an event-loop style architecture, like node, nginx • It can peg one CPU core but that’s it • Redis Cluster can solve this with automatic sharding • Below Redis 3.0, it’s easier to get a bigger server than fix your code (In my opinion)
7. How to get Redis • http://redis.io • Doesn’t run (well) on Windows • Has virtually no compile dependencies • apt-get, homebrew, yum, etc • Amazon ElastiCache • No reason not to run the latest and greatest • but at get at least 2.6
8. How to explore • There aren’t any good GUI tools out there • redis-cli is your friend
9. A bit about organization • Redis can have multiple databases • use one database per application • Within a database, namespace your keys • Ex: classname:uid:datafield • Keep them concise but useful. Keys take memory too!
10. Types of Keys
11. String Keys • Simple key-value • Memcache equivalent • Bitwise ops in newer versions • Common Commands • SET • GET • INCR http://redis.io/commands#string
12. Hash Keys • Key + multiple fields / values • Think 1-dimensional associative array • mykey => [field1 => value1, field2 => value2] http://redis.io/commands#hash • Common commands • HSET • HGET • HGETALL • HDEL • HVALS • HKEYS
13. Hash Keys
14. Set Keys • key + unordered list of strings • myset => [item2, item5, item1] • Common Commands • SADD • SMEMBERS • SISMEMBER • SREM http://redis.io/commands#set
15. Set Keys
16. List Keys • Like sets, except insertion order matters • Build queues or stacks • Optional blocking • Common commands • RPUSH, LPUSH • RPOP, LPOP, BRPOP, BLPOP • LLEN http://redis.io/commands#list
17. List Keys
18. Sorted Set Keys • Like sets, but sorted by a user-provided score value • Extremely fast access by score or range of scores, because it’s sorted in storage • Common commands • ZADD • ZRANGE • ZREVRANGE
19. Sorted Set Keys
20. Other commands that work on all keys • DEL - delete a key, regardless of type • KEYS - search for keys (usually with a wildcard) • EXPIRE / PERSIST - change expiration values for keys
21. Connecting from PHP
22. Connecting from PHP • Several libraries out there • Just use Predis (https://github.com/nrk/predis) • Requires PHP 5.3
23. Predis • All the examples here assume you’re using Predis • Connect to a localhost redis: $p = new Predis\Client(); • Redis commands implemented as magic methods on Predis\Client • $p->set($key, $val); • $val = $p->get($key);
24. Examples
25. Attribute Display Logic items id INT(11) name VARCHAR(32) items_attributes id INT(11) item_id INT(11) attr_name VARCHAR(32) attr_value VARCHAR(32) 10k rows 100k rows • An offer, from Dell, has: • Free Shipping, Financing Available, Expires Soon, etc
26. Attribute Display Logic • Display “Free Shipping” graphic on the item if it has a free shipping attribute row
27. Attribute Display - Traditional class Item { public function hasAttribute($name) { $sql = "SELECT 1 FROM items_attributes WHERE item_id = $this->id AND attr_name='$name' LIMIT 1"; $result = $this->pdo->execute($sql); return $result != false; } }
28. Denormalize data from MySQL to Redis • Smart, selective caching • Define a consistent way to name a relational object in Redis • I prefer [object class]:[object id]:[attribute] • ex: product:13445:num_comments • This prevents data collisions, and makes it easy to work with data on the command line
29. Attribute Display - Redis class Item { public function hasAttribute($name) { return $this->redis->sismember(“item:$this->id:attributes”, $name); } public function addAttribute($name, $value) { //traditional mysql stuff here still $this->redis->sadd('item:$this->id:attributes', $name); } public function deleteAttribute($name) { //traditional mysql stuff here $this->redis->srem(‘item:$this->id:attributes’, $name); } }
30. Advantages • The more items you have, the less MySQL will scale this solution for you on it’s own • Frequently updating your items kills the MySQL query cache • Checking existence of a set member is O(1) time • On a mid-range laptop, I can check roughly 10,000 attributes per second
31. MVC Routing • Example: • Offers.com has over 10000 stores • Store pages live at /[store]/
32. Old Store Routing class Store_Route implements Zend_Controller_Router_Route_Interface { public function route($url) { $sql = “SELECT 1 FROM stores WHERE url=’$url’”; $store = $this->pdo->execute($sql); // make sure $store is properly validated and then... return array(“module” => “core”, “controller” => “store”, “action” => “index”); } }
33. New Routing class Redis_Route implements Zend_Controller_Router_Route_Interface { public function route($url) { $p = $this->predis; if ($p->exists($url)) { list($module, $controller, $action) = $this->redis->hvals($url); return array(“module” => $module, “controller” => $controller, “action” => $action); } return false; } }
34. Filling in the Redis keys class Store { public function create(array $data) { // ... traditional SQL stuff to put store in the database ... // $route = array(“module” => “core”, “controller” => “store”, “action” => “index”); $this->predis->hmset($data[“url”], $route); } }
35. Advantages • I can now create offers.com/[anything]/ and route it to the right place, in O(1) time • I’m only adding a few lines of code to my existing models • One custom Redis route can handle any dynamic url
36. Event Throttling • A common task is to only let a user do something N number of times • This becomes more complex when you add a time window to that constraint • We can use key expirations in Redis to help with this
37. A Simple Event Throttling Class class Event { public static function log($event, $uid, $duration = null) { $r = self::$redis; $r->incr("event:$event:$uid"); if ($duration) { if ($r->ttl("event:$event:$uid") < 0) { $r->expires("event:$event:$uid", $duration); } } } public static function isAllowed($event, $uid, $threshold = null) { $r = self::$redis; if ($r->exists("event:$event:$uid")) { if ($threshold) { $currentValue = $r->get("even:$event:$uid"); if ($currentValue <= $threshold) { return true; } } return false; } return true; } }
38. Using the Event Class class MyController { public function clickAction() { $userId = $_SESSION["uid"]; if (Event::isAllowed("click", $userId, 5)) { // magic happens here Event::log('click', $userId, 3600); } else { // send an error message } } }
39. Most Popular [insert widget here] • Everyone likes to vote on things • Facebook like? • You probably want to store this 2 ways • in MySQL for reporting, long term aggregation, etc • something faster so your website will actually work
40. Using Zsets for “Most Popular” <?php class MyWidget { public static function getMostPopular($limit = null) { $rangeStart = 0; $rangeEnd = ($limit ? $limit - 1 : -1); return $redis->zrevrange("widgets:popular", $rangeStart, $rangeEnd, true); } public function logVote() { $pdo->query(" UDPATE widgets SET votes=votes+1 WHERE id='$this->id' "); $redis->zincrby("widgets:popular", 1, $this->name); } }
41. Questions?
42. Job Queues • Redis lists make great job queues • Offload your intensive workloads to some other process • Blocking I/O allows you to easily build long-running daemons • Be aware of scale tradeoffs vs. data availability
43. Job Queues class Queue { protected $name; protected $predis; public function push($job) { $this->predis->lpush($this->name, json_encode($job)); } public function pop($block = false) { $job = null; if ($block) { $data = $this->predis->brpop($this->name, 0); $job = $data[1]; } else { $job = $this->predis->rpop($this->name); } if ($job) { return json_decode($job); } } }
44. Queuing Jobs $q = new Queue(‘test_queue'); if ($form->isValid($_POST)) { $q->push($form->getValues()); $message = “Thanks for your submission”; } else { $message = “Error - something went wrong”; } echo “<h1>$message</h1>”;
45. Processing Jobs - Crontab style function processJob(Array $job) { //...something really cool here // throw an exception on error } // process all pending jobs $q = new Queue(‘email_confirmations’); while ($job = $q->pop()) { try { processJob($job); } catch (Exception $e) { echo “error processing job”; $q = new Queue(‘errors’); $q->push($job); } }
46. Processing Jobs - Worker style function processJob(Array $job) { //...something really cool here // throw an exception on error } // keep processing jobs as they become available $q = new Queue(‘test_queue’); while ($job = $q->pop(true)) { try { processJob($job); } catch (Exception $e) { echo “error processing job”; $q = new Queue(‘errors’); $q->push($job); } }