Scaling PHP to 40 Million Uniques

Scaling PHP to 40 Million Uniques

This is a talk I gave at Northeast PHP 2013 in Boston, MA.

Links: jkle.in/nephp

72f8175ac13e367bb500dd4da1f1aa32?s=128

Jonathan Klein

August 17, 2013
Tweet

Transcript

  1. 1.

    Scaling PHP to 40 Million Uniques Northeast PHP 2013 Jonathan

    Klein @jonathanklein Tuesday, September 24, 13
  2. 2.

    Scaling PHP to 40 Million Uniques Northeast PHP 2013 Jonathan

    Klein @jonathanklein Tuesday, September 24, 13
  3. 3.

    Scaling PHP to 60 Million Uniques Northeast PHP 2013 Jonathan

    Klein @jonathanklein Tuesday, September 24, 13
  4. 6.

    Who Am I? • Performance Engineer at Etsy • Boston

    Web Perf Meetup Organizer Tuesday, September 24, 13
  5. 7.

    Who Am I? • Performance Engineer at Etsy • Boston

    Web Perf Meetup Organizer • Previously at Wayfair Tuesday, September 24, 13
  6. 8.

    Who Am I? • Performance Engineer at Etsy • Boston

    Web Perf Meetup Organizer • Previously at Wayfair • Led team that converted to PHP Tuesday, September 24, 13
  7. 13.

    Some Etsy Stats • 1.5 billion page views/month • Almost

    $1B in sales last year Tuesday, September 24, 13
  8. 14.

    Some Etsy Stats • 1.5 billion page views/month • Almost

    $1B in sales last year • Over 1M lines of PHP Tuesday, September 24, 13
  9. 15.

    Some Etsy Stats • 1.5 billion page views/month • Almost

    $1B in sales last year • Over 1M lines of PHP • 60M+ unique visitors/month Tuesday, September 24, 13
  10. 26.

    A few others... • Search (Solr) • Gearman • Redis

    • Postgres (legacy) Tuesday, September 24, 13
  11. 31.
  12. 36.

    Apache • Apache 2.2 • Prefork MPM • mod_php5 StartServers

    30 MinSpareServers 30 MaxSpareServers 60 ServerLimit 60 MaxClients 60 MaxRequestsPerChild 0 Tuesday, September 24, 13
  13. 40.

    PHP • PHP 5.4 • Zend OPCache (3GB Memory Segment)

    • memory_limit: 128M Tuesday, September 24, 13
  14. 41.

    PHP • PHP 5.4 • Zend OPCache (3GB Memory Segment)

    • memory_limit: 128M • max_execution_time: 30 Tuesday, September 24, 13
  15. 44.

    Optimizing PHP • See last year’s talk • Use an

    Opcode Cache http://www.slideshare.net/jnklein/northeast-php-high-performance-php Tuesday, September 24, 13
  16. 45.

    Optimizing PHP • See last year’s talk • Use an

    Opcode Cache • xhprof http://www.slideshare.net/jnklein/northeast-php-high-performance-php Tuesday, September 24, 13
  17. 46.

    Optimizing PHP • See last year’s talk • Use an

    Opcode Cache • xhprof • StatsD/Graphite http://www.slideshare.net/jnklein/northeast-php-high-performance-php Tuesday, September 24, 13
  18. 47.

    Optimizing PHP • See last year’s talk • Use an

    Opcode Cache • xhprof • StatsD/Graphite • Find hotspots http://www.slideshare.net/jnklein/northeast-php-high-performance-php Tuesday, September 24, 13
  19. 48.

    Optimizing PHP • See last year’s talk • Use an

    Opcode Cache • xhprof • StatsD/Graphite • Find hotspots • Upgrade it http://www.slideshare.net/jnklein/northeast-php-high-performance-php Tuesday, September 24, 13
  20. 50.

    Translations class language_de { static $translations = array( "<hash>" =>

    array("content" => 'Accessoires', "file" => "TAXONOMY"), "<hash>" => array("content" => 'Keramik', "file" => "CATEGORY"), "<hash>" => array("content" => 'Dekorieren', "file" => "CATEGORY"), "<hash>" => array("content" => 'Getaggt %s', "file" => "Foo.php"), "<hash>" => array("content" => 'Badezusatz', "file" => "CATEGORY"), "<hash>" => array("content" => 'Datum:{% $date %}', "file" => "bar.tpl") ... snip ... ); } Tuesday, September 24, 13
  21. 56.

    Deploying PHP • mod_realdoc - Apache module • incpath -

    PHP Extension http://codeascraft.com/2013/07/01/atomic-deploys-at-etsy/ Tuesday, September 24, 13
  22. 57.

    Deploying PHP • mod_realdoc - Apache module • incpath -

    PHP Extension • A/B symlink swap http://codeascraft.com/2013/07/01/atomic-deploys-at-etsy/ Tuesday, September 24, 13
  23. 58.

    Deploying PHP • mod_realdoc - Apache module • incpath -

    PHP Extension • A/B symlink swap • Avoid recompilation to opcodes http://codeascraft.com/2013/07/01/atomic-deploys-at-etsy/ Tuesday, September 24, 13
  24. 60.

    PHP 5.4 vs. 5.3 - CPU PHP 5.3 PHP 5.4

    Tuesday, September 24, 13
  25. 61.

    PHP 5.4 vs. 5.3 - Memory PHP 5.3 PHP 5.4

    Tuesday, September 24, 13
  26. 73.

    Memcached • Roughly same hardware as webs • More memory

    (48GB) • Shard keys across servers Tuesday, September 24, 13
  27. 74.

    Memcached • Roughly same hardware as webs • More memory

    (48GB) • Shard keys across servers • mctop - https://github.com/etsy/mctop Tuesday, September 24, 13
  28. 80.

    MySQL • MySQL 5.5 (http://jkle.in/nephp) • ~20 master/master pairs •

    innodb • thread_cache=800 Tuesday, September 24, 13
  29. 81.

    MySQL • MySQL 5.5 (http://jkle.in/nephp) • ~20 master/master pairs •

    innodb • thread_cache=800 • max_connections=2500 Tuesday, September 24, 13
  30. 82.

    MySQL • MySQL 5.5 (http://jkle.in/nephp) • ~20 master/master pairs •

    innodb • thread_cache=800 • max_connections=2500 • [Almost] no joins Tuesday, September 24, 13
  31. 91.

    Offloading Requests • Multiple CDNs • Img hit rate >

    90% • CSS/JS hit rate ~100% Tuesday, September 24, 13
  32. 92.

    Offloading Requests • Multiple CDNs • Img hit rate >

    90% • CSS/JS hit rate ~100% • Edge serves ~7000 reqs/sec Tuesday, September 24, 13
  33. 98.

    // Owner: Username $server_config['foo']['bar'] = array( 'users' => array('jsmith', 'jdoe'),

    ); // Owner: Username $server_config['baz'] = [ 'enabled' => [ 'variant1' => 0, 'variant2' => 0, 'variant3' => 50 ], ]; // Owner: Username $server_config['qux'] = array( 'enabled' => 100, 'data' => array( 'somedata' ) ); Tuesday, September 24, 13
  34. 116.
  35. 117.

    Options • Scale Horizontally (more shards) • Scale Vertically (SSD,

    bigger drives) • Change architecture Tuesday, September 24, 13
  36. 123.
  37. 124.

    Architecture Changes • Ongoing conversation • Separate logical/physical shards •

    Complex • Server failure • Migrating data Tuesday, September 24, 13
  38. 125.

    Architecture Changes • Ongoing conversation • Separate logical/physical shards •

    Complex • Server failure • Migrating data • Consistency Tuesday, September 24, 13