Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Scaling PHP Applications With Redis (Nomad PHP July 2015)

Scaling PHP Applications With Redis (Nomad PHP July 2015)

Josh Butts

July 24, 2015
Tweet

More Decks by Josh Butts

Other Decks in Technology

Transcript

  1. About Me • VP of Engineering at Offers.com in Austin,

    Texas • Long-time Austin PHP organizer • Competitive Skee-baller • Follow me online: • github.com/jimbojsb • @jimbojsb
  2. What is Redis? • Redis is an “Advanced” key-value store

    database • Backed by VMWare • Redis works like memcached, but better: • Atomic commands & transactions • Server-side data structures • Comparable speed
  3. Redis in an in-memory database • You need as much

    RAM as you have data • Disk persistence is exactly that • Customize disk writes interval to suit your pain threshold • Something interesting happens when you run out of memory
  4. Redis is Single-Threaded • Redis uses an event-loop style architecture,

    like node, nginx • It can peg one CPU core but that’s it • Redis Cluster can solve this with automatic sharding • Below Redis 3.0, it’s easier to get a bigger server than fix your code (In my opinion)
  5. How to get Redis • http://redis.io • Doesn’t run (well)

    on Windows • Has virtually no compile dependencies • apt-get, homebrew, yum, etc • Amazon ElastiCache • No reason not to run the latest and greatest • but at get at least 2.6
  6. How to explore • There aren’t any good GUI tools

    out there • redis-cli is your friend
  7. A bit about organization • Redis can have multiple databases

    • use one database per application • Within a database, namespace your keys • Ex: classname:uid:datafield • Keep them concise but useful. Keys take memory too!
  8. String Keys • Simple key-value • Memcache equivalent • Bitwise

    ops in newer versions • Common Commands • SET • GET • INCR http://redis.io/commands#string
  9. Hash Keys • Key + multiple fields / values •

    Think 1-dimensional associative array • mykey => [field1 => value1, field2 => value2] http://redis.io/commands#hash • Common commands • HSET • HGET • HGETALL • HDEL • HVALS • HKEYS
  10. Set Keys • key + unordered list of strings •

    myset => [item2, item5, item1] • Common Commands • SADD • SMEMBERS • SISMEMBER • SREM http://redis.io/commands#set
  11. List Keys • Like sets, except insertion order matters •

    Build queues or stacks • Optional blocking • Common commands • RPUSH, LPUSH • RPOP, LPOP, BRPOP, BLPOP • LLEN http://redis.io/commands#list
  12. Sorted Set Keys • Like sets, but sorted by a

    user-provided score value • Extremely fast access by score or range of scores, because it’s sorted in storage • Common commands • ZADD • ZRANGE • ZREVRANGE
  13. Other commands that work on all keys • DEL -

    delete a key, regardless of type • KEYS - search for keys (usually with a wildcard) • EXPIRE / PERSIST - change expiration values for keys
  14. Connecting from PHP • Several libraries out there • Just

    use Predis (https://github.com/nrk/predis) • Requires PHP 5.3
  15. Predis • All the examples here assume you’re using Predis

    • Connect to a localhost redis: $p = new Predis\Client(); • Redis commands implemented as magic methods on Predis\Client • $p->set($key, $val); • $val = $p->get($key);
  16. Attribute Display Logic items id INT(11) name VARCHAR(32) items_attributes id

    INT(11) item_id INT(11) attr_name VARCHAR(32) attr_value VARCHAR(32) 10k rows 100k rows • An offer, from Dell, has: • Free Shipping, Financing Available, Expires Soon, etc
  17. Attribute Display Logic • Display “Free Shipping” graphic on the

    item if it has a free shipping attribute row
  18. Attribute Display - Traditional class Item { public function hasAttribute($name)

    { $sql = "SELECT 1 FROM items_attributes WHERE item_id = $this->id AND attr_name='$name' LIMIT 1"; $result = $this->pdo->execute($sql); return $result != false; } }
  19. Denormalize data from MySQL to Redis • Smart, selective caching

    • Define a consistent way to name a relational object in Redis • I prefer [object class]:[object id]:[attribute] • ex: product:13445:num_comments • This prevents data collisions, and makes it easy to work with data on the command line
  20. Attribute Display - Redis class Item { public function hasAttribute($name)

    { return $this->redis->sismember(“item:$this->id:attributes”, $name); } public function addAttribute($name, $value) { //traditional mysql stuff here still $this->redis->sadd('item:$this->id:attributes', $name); } public function deleteAttribute($name) { //traditional mysql stuff here $this->redis->srem(‘item:$this->id:attributes’, $name); } }
  21. Advantages • The more items you have, the less MySQL

    will scale this solution for you on it’s own • Frequently updating your items kills the MySQL query cache • Checking existence of a set member is O(1) time • On a mid-range laptop, I can check roughly 10,000 attributes per second
  22. Old Store Routing class Store_Route implements Zend_Controller_Router_Route_Interface { public function

    route($url) { $sql = “SELECT 1 FROM stores WHERE url=’$url’”; $store = $this->pdo->execute($sql); // make sure $store is properly validated and then... return array(“module” => “core”, “controller” => “store”, “action” => “index”); } }
  23. New Routing class Redis_Route implements Zend_Controller_Router_Route_Interface { public function route($url)

    { $p = $this->predis; if ($p->exists($url)) { list($module, $controller, $action) = $this->redis->hvals($url); return array(“module” => $module, “controller” => $controller, “action” => $action); } return false; } }
  24. Filling in the Redis keys class Store { public function

    create(array $data) { // ... traditional SQL stuff to put store in the database ... // $route = array(“module” => “core”, “controller” => “store”, “action” => “index”); $this->predis->hmset($data[“url”], $route); } }
  25. Advantages • I can now create offers.com/[anything]/ and route it

    to the right place, in O(1) time • I’m only adding a few lines of code to my existing models • One custom Redis route can handle any dynamic url
  26. Event Throttling • A common task is to only let

    a user do something N number of times • This becomes more complex when you add a time window to that constraint • We can use key expirations in Redis to help with this
  27. A Simple Event Throttling Class class Event { public static

    function log($event, $uid, $duration = null) { $r = self::$redis; $r->incr("event:$event:$uid"); if ($duration) { if ($r->ttl("event:$event:$uid") < 0) { $r->expires("event:$event:$uid", $duration); } } } public static function isAllowed($event, $uid, $threshold = null) { $r = self::$redis; if ($r->exists("event:$event:$uid")) { if ($threshold) { $currentValue = $r->get("even:$event:$uid"); if ($currentValue <= $threshold) { return true; } } return false; } return true; } }
  28. Using the Event Class class MyController { public function clickAction()

    { $userId = $_SESSION["uid"]; if (Event::isAllowed("click", $userId, 5)) { // magic happens here Event::log('click', $userId, 3600); } else { // send an error message } } }
  29. Most Popular [insert widget here] • Everyone likes to vote

    on things • Facebook like? • You probably want to store this 2 ways • in MySQL for reporting, long term aggregation, etc • something faster so your website will actually work
  30. Using Zsets for “Most Popular” <?php class MyWidget { public

    static function getMostPopular($limit = null) { $rangeStart = 0; $rangeEnd = ($limit ? $limit - 1 : -1); return $redis->zrevrange("widgets:popular", $rangeStart, $rangeEnd, true); } public function logVote() { $pdo->query(" UDPATE widgets SET votes=votes+1 WHERE id='$this->id' "); $redis->zincrby("widgets:popular", 1, $this->name); } }
  31. Job Queues • Redis lists make great job queues •

    Offload your intensive workloads to some other process • Blocking I/O allows you to easily build long-running daemons • Be aware of scale tradeoffs vs. data availability
  32. Job Queues class Queue { protected $name; protected $predis; public

    function push($job) { $this->predis->lpush($this->name, json_encode($job)); } public function pop($block = false) { $job = null; if ($block) { $data = $this->predis->brpop($this->name, 0); $job = $data[1]; } else { $job = $this->predis->rpop($this->name); } if ($job) { return json_decode($job); } } }
  33. Queuing Jobs $q = new Queue(‘test_queue'); if ($form->isValid($_POST)) { $q->push($form->getValues());

    $message = “Thanks for your submission”; } else { $message = “Error - something went wrong”; } echo “<h1>$message</h1>”;
  34. Processing Jobs - Crontab style function processJob(Array $job) { //...something

    really cool here // throw an exception on error } // process all pending jobs $q = new Queue(‘email_confirmations’); while ($job = $q->pop()) { try { processJob($job); } catch (Exception $e) { echo “error processing job”; $q = new Queue(‘errors’); $q->push($job); } }
  35. Processing Jobs - Worker style function processJob(Array $job) { //...something

    really cool here // throw an exception on error } // keep processing jobs as they become available $q = new Queue(‘test_queue’); while ($job = $q->pop(true)) { try { processJob($job); } catch (Exception $e) { echo “error processing job”; $q = new Queue(‘errors’); $q->push($job); } }