Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Scaling PHP Applications With Redis (Nomad PHP July 2015)

Scaling PHP Applications With Redis (Nomad PHP July 2015)

44a352b02a91a9e841da7533bc5d9b8e?s=128

Josh Butts

July 24, 2015
Tweet

Transcript

  1. Scaling PHP Applications With Redis Josh Butts Nomad PHP, July

    2015 https://joind.in/14896
  2. About Me • VP of Engineering at Offers.com in Austin,

    Texas • Long-time Austin PHP organizer • Competitive Skee-baller • Follow me online: • github.com/jimbojsb • @jimbojsb
  3. Agenda • Redis Overview • Types of keys • PHP

    Strategies
  4. What is Redis? • Redis is an “Advanced” key-value store

    database • Backed by VMWare • Redis works like memcached, but better: • Atomic commands & transactions • Server-side data structures • Comparable speed
  5. Redis in an in-memory database • You need as much

    RAM as you have data • Disk persistence is exactly that • Customize disk writes interval to suit your pain threshold • Something interesting happens when you run out of memory
  6. Redis is Single-Threaded • Redis uses an event-loop style architecture,

    like node, nginx • It can peg one CPU core but that’s it • Redis Cluster can solve this with automatic sharding • Below Redis 3.0, it’s easier to get a bigger server than fix your code (In my opinion)
  7. How to get Redis • http://redis.io • Doesn’t run (well)

    on Windows • Has virtually no compile dependencies • apt-get, homebrew, yum, etc • Amazon ElastiCache • No reason not to run the latest and greatest • but at get at least 2.6
  8. How to explore • There aren’t any good GUI tools

    out there • redis-cli is your friend
  9. A bit about organization • Redis can have multiple databases

    • use one database per application • Within a database, namespace your keys • Ex: classname:uid:datafield • Keep them concise but useful. Keys take memory too!
  10. Types of Keys

  11. String Keys • Simple key-value • Memcache equivalent • Bitwise

    ops in newer versions • Common Commands • SET • GET • INCR http://redis.io/commands#string
  12. Hash Keys • Key + multiple fields / values •

    Think 1-dimensional associative array • mykey => [field1 => value1, field2 => value2] http://redis.io/commands#hash • Common commands • HSET • HGET • HGETALL • HDEL • HVALS • HKEYS
  13. Hash Keys

  14. Set Keys • key + unordered list of strings •

    myset => [item2, item5, item1] • Common Commands • SADD • SMEMBERS • SISMEMBER • SREM http://redis.io/commands#set
  15. Set Keys

  16. List Keys • Like sets, except insertion order matters •

    Build queues or stacks • Optional blocking • Common commands • RPUSH, LPUSH • RPOP, LPOP, BRPOP, BLPOP • LLEN http://redis.io/commands#list
  17. List Keys

  18. Sorted Set Keys • Like sets, but sorted by a

    user-provided score value • Extremely fast access by score or range of scores, because it’s sorted in storage • Common commands • ZADD • ZRANGE • ZREVRANGE
  19. Sorted Set Keys

  20. Other commands that work on all keys • DEL -

    delete a key, regardless of type • KEYS - search for keys (usually with a wildcard) • EXPIRE / PERSIST - change expiration values for keys
  21. Connecting from PHP

  22. Connecting from PHP • Several libraries out there • Just

    use Predis (https://github.com/nrk/predis) • Requires PHP 5.3
  23. Predis • All the examples here assume you’re using Predis

    • Connect to a localhost redis: $p = new Predis\Client(); • Redis commands implemented as magic methods on Predis\Client • $p->set($key, $val); • $val = $p->get($key);
  24. Examples

  25. Attribute Display Logic items id INT(11) name VARCHAR(32) items_attributes id

    INT(11) item_id INT(11) attr_name VARCHAR(32) attr_value VARCHAR(32) 10k rows 100k rows • An offer, from Dell, has: • Free Shipping, Financing Available, Expires Soon, etc
  26. Attribute Display Logic • Display “Free Shipping” graphic on the

    item if it has a free shipping attribute row
  27. Attribute Display - Traditional class Item { public function hasAttribute($name)

    { $sql = "SELECT 1 FROM items_attributes WHERE item_id = $this->id AND attr_name='$name' LIMIT 1"; $result = $this->pdo->execute($sql); return $result != false; } }
  28. Denormalize data from MySQL to Redis • Smart, selective caching

    • Define a consistent way to name a relational object in Redis • I prefer [object class]:[object id]:[attribute] • ex: product:13445:num_comments • This prevents data collisions, and makes it easy to work with data on the command line
  29. Attribute Display - Redis class Item { public function hasAttribute($name)

    { return $this->redis->sismember(“item:$this->id:attributes”, $name); } public function addAttribute($name, $value) { //traditional mysql stuff here still $this->redis->sadd('item:$this->id:attributes', $name); } public function deleteAttribute($name) { //traditional mysql stuff here $this->redis->srem(‘item:$this->id:attributes’, $name); } }
  30. Advantages • The more items you have, the less MySQL

    will scale this solution for you on it’s own • Frequently updating your items kills the MySQL query cache • Checking existence of a set member is O(1) time • On a mid-range laptop, I can check roughly 10,000 attributes per second
  31. MVC Routing • Example: • Offers.com has over 10000 stores

    • Store pages live at /[store]/
  32. Old Store Routing class Store_Route implements Zend_Controller_Router_Route_Interface { public function

    route($url) { $sql = “SELECT 1 FROM stores WHERE url=’$url’”; $store = $this->pdo->execute($sql); // make sure $store is properly validated and then... return array(“module” => “core”, “controller” => “store”, “action” => “index”); } }
  33. New Routing class Redis_Route implements Zend_Controller_Router_Route_Interface { public function route($url)

    { $p = $this->predis; if ($p->exists($url)) { list($module, $controller, $action) = $this->redis->hvals($url); return array(“module” => $module, “controller” => $controller, “action” => $action); } return false; } }
  34. Filling in the Redis keys class Store { public function

    create(array $data) { // ... traditional SQL stuff to put store in the database ... // $route = array(“module” => “core”, “controller” => “store”, “action” => “index”); $this->predis->hmset($data[“url”], $route); } }
  35. Advantages • I can now create offers.com/[anything]/ and route it

    to the right place, in O(1) time • I’m only adding a few lines of code to my existing models • One custom Redis route can handle any dynamic url
  36. Event Throttling • A common task is to only let

    a user do something N number of times • This becomes more complex when you add a time window to that constraint • We can use key expirations in Redis to help with this
  37. A Simple Event Throttling Class class Event { public static

    function log($event, $uid, $duration = null) { $r = self::$redis; $r->incr("event:$event:$uid"); if ($duration) { if ($r->ttl("event:$event:$uid") < 0) { $r->expires("event:$event:$uid", $duration); } } } public static function isAllowed($event, $uid, $threshold = null) { $r = self::$redis; if ($r->exists("event:$event:$uid")) { if ($threshold) { $currentValue = $r->get("even:$event:$uid"); if ($currentValue <= $threshold) { return true; } } return false; } return true; } }
  38. Using the Event Class class MyController { public function clickAction()

    { $userId = $_SESSION["uid"]; if (Event::isAllowed("click", $userId, 5)) { // magic happens here Event::log('click', $userId, 3600); } else { // send an error message } } }
  39. Most Popular [insert widget here] • Everyone likes to vote

    on things • Facebook like? • You probably want to store this 2 ways • in MySQL for reporting, long term aggregation, etc • something faster so your website will actually work
  40. Using Zsets for “Most Popular” <?php class MyWidget { public

    static function getMostPopular($limit = null) { $rangeStart = 0; $rangeEnd = ($limit ? $limit - 1 : -1); return $redis->zrevrange("widgets:popular", $rangeStart, $rangeEnd, true); } public function logVote() { $pdo->query(" UDPATE widgets SET votes=votes+1 WHERE id='$this->id' "); $redis->zincrby("widgets:popular", 1, $this->name); } }
  41. Questions?

  42. Job Queues • Redis lists make great job queues •

    Offload your intensive workloads to some other process • Blocking I/O allows you to easily build long-running daemons • Be aware of scale tradeoffs vs. data availability
  43. Job Queues class Queue { protected $name; protected $predis; public

    function push($job) { $this->predis->lpush($this->name, json_encode($job)); } public function pop($block = false) { $job = null; if ($block) { $data = $this->predis->brpop($this->name, 0); $job = $data[1]; } else { $job = $this->predis->rpop($this->name); } if ($job) { return json_decode($job); } } }
  44. Queuing Jobs $q = new Queue(‘test_queue'); if ($form->isValid($_POST)) { $q->push($form->getValues());

    $message = “Thanks for your submission”; } else { $message = “Error - something went wrong”; } echo “<h1>$message</h1>”;
  45. Processing Jobs - Crontab style function processJob(Array $job) { //...something

    really cool here // throw an exception on error } // process all pending jobs $q = new Queue(‘email_confirmations’); while ($job = $q->pop()) { try { processJob($job); } catch (Exception $e) { echo “error processing job”; $q = new Queue(‘errors’); $q->push($job); } }
  46. Processing Jobs - Worker style function processJob(Array $job) { //...something

    really cool here // throw an exception on error } // keep processing jobs as they become available $q = new Queue(‘test_queue’); while ($job = $q->pop(true)) { try { processJob($job); } catch (Exception $e) { echo “error processing job”; $q = new Queue(‘errors’); $q->push($job); } }