Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Scaling PHP to 40 Million Uniques

72f8175ac13e367bb500dd4da1f1aa32?s=47 Jonathan Klein
August 17, 2013
Technology
3
340

Scaling PHP to 40 Million Uniques

This is a talk I gave at Northeast PHP 2013 in Boston, MA.

Links: jkle.in/nephp

72f8175ac13e367bb500dd4da1f1aa32?s=128

Jonathan Klein

August 17, 2013
Tweet

More Decks by Jonathan Klein

See All by Jonathan Klein
72f8175ac13e367bb500dd4da1f1aa32?s=48 jklein
15
240
72f8175ac13e367bb500dd4da1f1aa32?s=48 jklein
1
440
72f8175ac13e367bb500dd4da1f1aa32?s=48 jklein
2
600
72f8175ac13e367bb500dd4da1f1aa32?s=48 jklein
4
1.3k
72f8175ac13e367bb500dd4da1f1aa32?s=48 jklein
1
310
72f8175ac13e367bb500dd4da1f1aa32?s=48 jklein
1
200
72f8175ac13e367bb500dd4da1f1aa32?s=48 jklein
1
460
72f8175ac13e367bb500dd4da1f1aa32?s=48 jklein
0
120
72f8175ac13e367bb500dd4da1f1aa32?s=48 jklein
1
830

Other Decks in Technology

See All in Technology
9df0566fad9c9ca3bf669917848878fe?s=48 i35_267
0
280
928b1395afedebb5ee48d44ab917c5f1?s=48 ryusa
2
380
D9e65f4b0af059ae9ba243c8c2265e4f?s=48 pohjus
1
2.1k
E7c0cdb6e91562ab5995348cd2b2110e?s=48 gkzz
0
360
1397c76a3ecccf382a6da698c503b849?s=48 siroemk
0
280
346406bb33c5e2389c5df9c74b0b98da?s=48 yoshiki0705
0
560
1bdd9c71c3e2bb8fe1fac700c6ba3b34?s=48 yamasaki696
0
400
6ed12627fec46a135f1bce5d56f3568e?s=48 nwiizo
1
370
19fc8f6113c5c3d86e6176362ff29479?s=48 kanaugust
PRO
0
190
74948d1118cd84528f7861a3069dd8d9?s=48 qryuu
7
5k
7cd783377515bdf8207062840b7b2f4e?s=48 soracom
0
140
86e4f6731e81df0591bd9cb66ddbd347?s=48 sugimomoto
1
420

Featured

See All Featured
B83d5b590577969ee79a5ad845411a7d?s=48 ammeep
656
54k
3d4519fd34b76fb265fc0237f3792bd4?s=48 danielanewman
1
480
Ae14cc4491ac334f9cd23f9f93b4305e?s=48 orderedlist
PRO
329
35k
9fa239b3154a0169d99ab7bf1422dd12?s=48 marcelosomers
220
15k
96270e4c3e5e9806cf7245475c00b275?s=48 addyosmani
1347
190k
A415db3ac9e9668acbc1fb36eead84ac?s=48 mthomps
39
2.3k
1519587b1a0febb658c36c94f6272b0d?s=48 roundedbygravity
241
21k
C4de88ea47538e4594750e3861a341c8?s=48 brettharned
93
3k
8081b26e05bb4354f7d65ffc34cbbd67?s=48 chriscoyier
499
130k
016edace78ffe826f2348d1e0c504afd?s=48 maggiecrowley
8
430
C0390e8b11079f8761c0cd3274ed61cb?s=48 productmarketing
5
660
B3d6434763caa0ef5dc4b792662c49f7?s=48 smashingmag
283
47k

Transcript

  1. Scaling PHP to 40 Million Uniques Northeast PHP 2013 Jonathan

    Klein @jonathanklein Tuesday, September 24, 13

  2. Scaling PHP to 40 Million Uniques Northeast PHP 2013 Jonathan

    Klein @jonathanklein Tuesday, September 24, 13

  3. Scaling PHP to 60 Million Uniques Northeast PHP 2013 Jonathan

    Klein @jonathanklein Tuesday, September 24, 13

  4. Who Am I? Tuesday, September 24, 13

  5. Who Am I? • Performance Engineer at Etsy Tuesday, September

    24, 13

  6. Who Am I? • Performance Engineer at Etsy • Boston

    Web Perf Meetup Organizer Tuesday, September 24, 13

  7. Who Am I? • Performance Engineer at Etsy • Boston

    Web Perf Meetup Organizer • Previously at Wayfair Tuesday, September 24, 13

  8. Who Am I? • Performance Engineer at Etsy • Boston

    Web Perf Meetup Organizer • Previously at Wayfair • Led team that converted to PHP Tuesday, September 24, 13

  9. Slides, Links: jkle.in/nephp Tuesday, September 24, 13

  10. codeascraft.com Tuesday, September 24, 13

  11. Some Etsy Stats Tuesday, September 24, 13

  12. Some Etsy Stats • 1.5 billion page views/month Tuesday, September

    24, 13

  13. Some Etsy Stats • 1.5 billion page views/month • Almost

    $1B in sales last year Tuesday, September 24, 13

  14. Some Etsy Stats • 1.5 billion page views/month • Almost

    $1B in sales last year • Over 1M lines of PHP Tuesday, September 24, 13

  15. Some Etsy Stats • 1.5 billion page views/month • Almost

    $1B in sales last year • Over 1M lines of PHP • 60M+ unique visitors/month Tuesday, September 24, 13

  16. 2012 Traffic Tuesday, September 24, 13

  17. August 2012 Traffic Tuesday, September 24, 13

  18. August 2012 Traffic 50% Higher Tuesday, September 24, 13

  19. Holidays: ~90M Uniques Tuesday, September 24, 13

  20. Tuesday, September 24, 13

  21. Tuesday, September 24, 13

  22. A few others... Tuesday, September 24, 13

  23. A few others... • Search (Solr) Tuesday, September 24, 13

  24. A few others... • Search (Solr) • Gearman Tuesday, September

    24, 13

  25. A few others... • Search (Solr) • Gearman • Redis

    Tuesday, September 24, 13

  26. A few others... • Search (Solr) • Gearman • Redis

    • Postgres (legacy) Tuesday, September 24, 13

  27. Tuesday, September 24, 13

  28. Hardware (Supermicro) Tuesday, September 24, 13

  29. Hardware (Supermicro) 2x 8-core Intel E5-2960 CPUs Tuesday, September 24,

    13

  30. Hardware (Supermicro) 2x 8-core Intel E5-2960 CPUs 24GB of RAM

    Tuesday, September 24, 13

  31. Hardware (Supermicro) 2x 8-core Intel E5-2960 CPUs 24GB of RAM

    160GB SSD Tuesday, September 24, 13

  32. Apache Tuesday, September 24, 13

  33. Apache • Apache 2.2 Tuesday, September 24, 13

  34. Apache • Apache 2.2 • Prefork MPM Tuesday, September 24,

    13

  35. Apache • Apache 2.2 • Prefork MPM • mod_php5 Tuesday,

    September 24, 13

  36. Apache • Apache 2.2 • Prefork MPM • mod_php5 StartServers

    30 MinSpareServers 30 MaxSpareServers 60 ServerLimit 60 MaxClients 60 MaxRequestsPerChild 0 Tuesday, September 24, 13

  37. PHP Tuesday, September 24, 13

  38. PHP • PHP 5.4 Tuesday, September 24, 13

  39. PHP • PHP 5.4 • Zend OPCache (3GB Memory Segment)

    Tuesday, September 24, 13

  40. PHP • PHP 5.4 • Zend OPCache (3GB Memory Segment)

    • memory_limit: 128M Tuesday, September 24, 13

  41. PHP • PHP 5.4 • Zend OPCache (3GB Memory Segment)

    • memory_limit: 128M • max_execution_time: 30 Tuesday, September 24, 13

  42. Optimizing PHP http://www.slideshare.net/jnklein/northeast-php-high-performance-php Tuesday, September 24, 13

  43. Optimizing PHP • See last year’s talk http://www.slideshare.net/jnklein/northeast-php-high-performance-php Tuesday, September

    24, 13

  44. Optimizing PHP • See last year’s talk • Use an

    Opcode Cache http://www.slideshare.net/jnklein/northeast-php-high-performance-php Tuesday, September 24, 13

  45. Optimizing PHP • See last year’s talk • Use an

    Opcode Cache • xhprof http://www.slideshare.net/jnklein/northeast-php-high-performance-php Tuesday, September 24, 13

  46. Optimizing PHP • See last year’s talk • Use an

    Opcode Cache • xhprof • StatsD/Graphite http://www.slideshare.net/jnklein/northeast-php-high-performance-php Tuesday, September 24, 13

  47. Optimizing PHP • See last year’s talk • Use an

    Opcode Cache • xhprof • StatsD/Graphite • Find hotspots http://www.slideshare.net/jnklein/northeast-php-high-performance-php Tuesday, September 24, 13

  48. Optimizing PHP • See last year’s talk • Use an

    Opcode Cache • xhprof • StatsD/Graphite • Find hotspots • Upgrade it http://www.slideshare.net/jnklein/northeast-php-high-performance-php Tuesday, September 24, 13

  49. Static Arrays Tuesday, September 24, 13

  50. Translations class language_de { static $translations = array( "<hash>" =>

    array("content" => 'Accessoires', "file" => "TAXONOMY"), "<hash>" => array("content" => 'Keramik', "file" => "CATEGORY"), "<hash>" => array("content" => 'Dekorieren', "file" => "CATEGORY"), "<hash>" => array("content" => 'Getaggt %s', "file" => "Foo.php"), "<hash>" => array("content" => 'Badezusatz', "file" => "CATEGORY"), "<hash>" => array("content" => 'Datum:{% $date %}', "file" => "bar.tpl") ... snip ... ); } Tuesday, September 24, 13

  51. Tuesday, September 24, 13

  52. Tuesday, September 24, 13

  53. Atomic Deploys Tuesday, September 24, 13

  54. Deploying PHP http://codeascraft.com/2013/07/01/atomic-deploys-at-etsy/ Tuesday, September 24, 13

  55. Deploying PHP • mod_realdoc - Apache module http://codeascraft.com/2013/07/01/atomic-deploys-at-etsy/ Tuesday, September

    24, 13

  56. Deploying PHP • mod_realdoc - Apache module • incpath -

    PHP Extension http://codeascraft.com/2013/07/01/atomic-deploys-at-etsy/ Tuesday, September 24, 13

  57. Deploying PHP • mod_realdoc - Apache module • incpath -

    PHP Extension • A/B symlink swap http://codeascraft.com/2013/07/01/atomic-deploys-at-etsy/ Tuesday, September 24, 13

  58. Deploying PHP • mod_realdoc - Apache module • incpath -

    PHP Extension • A/B symlink swap • Avoid recompilation to opcodes http://codeascraft.com/2013/07/01/atomic-deploys-at-etsy/ Tuesday, September 24, 13

  59. Upgrading PHP Tuesday, September 24, 13

  60. PHP 5.4 vs. 5.3 - CPU PHP 5.3 PHP 5.4

    Tuesday, September 24, 13

  61. PHP 5.4 vs. 5.3 - Memory PHP 5.3 PHP 5.4

    Tuesday, September 24, 13

  62. Tuesday, September 24, 13

  63. Understand Framework Overhead Tuesday, September 24, 13

  64. http://systemsarchitect.net/performance-benchmark-of-popular-php-frameworks/ Tuesday, September 24, 13

  65. Tuesday, September 24, 13

  66. Tuesday, September 24, 13

  67. Tuesday, September 24, 13

  68. Frameworks are Harder to Scale Tuesday, September 24, 13

  69. Tuesday, September 24, 13

  70. Memcached Tuesday, September 24, 13

  71. Memcached • Roughly same hardware as webs Tuesday, September 24,

    13

  72. Memcached • Roughly same hardware as webs • More memory

    (48GB) Tuesday, September 24, 13

  73. Memcached • Roughly same hardware as webs • More memory

    (48GB) • Shard keys across servers Tuesday, September 24, 13

  74. Memcached • Roughly same hardware as webs • More memory

    (48GB) • Shard keys across servers • mctop - https://github.com/etsy/mctop Tuesday, September 24, 13

  75. Tuesday, September 24, 13

  76. MySQL Tuesday, September 24, 13

  77. MySQL • MySQL 5.5 (http://jkle.in/nephp) Tuesday, September 24, 13

  78. MySQL • MySQL 5.5 (http://jkle.in/nephp) • ~20 master/master pairs Tuesday,

    September 24, 13

  79. MySQL • MySQL 5.5 (http://jkle.in/nephp) • ~20 master/master pairs •

    innodb Tuesday, September 24, 13

  80. MySQL • MySQL 5.5 (http://jkle.in/nephp) • ~20 master/master pairs •

    innodb • thread_cache=800 Tuesday, September 24, 13

  81. MySQL • MySQL 5.5 (http://jkle.in/nephp) • ~20 master/master pairs •

    innodb • thread_cache=800 • max_connections=2500 Tuesday, September 24, 13

  82. MySQL • MySQL 5.5 (http://jkle.in/nephp) • ~20 master/master pairs •

    innodb • thread_cache=800 • max_connections=2500 • [Almost] no joins Tuesday, September 24, 13

  83. Theoretically we can serve ~9000 req/sec Tuesday, September 24, 13

  84. Tuesday, September 24, 13

  85. Static Content Tuesday, September 24, 13

  86. Tuesday, September 24, 13

  87. ~500 Million Images Tuesday, September 24, 13

  88. Offloading Requests Tuesday, September 24, 13

  89. Offloading Requests • Multiple CDNs Tuesday, September 24, 13

  90. Offloading Requests • Multiple CDNs • Img hit rate >

    90% Tuesday, September 24, 13

  91. Offloading Requests • Multiple CDNs • Img hit rate >

    90% • CSS/JS hit rate ~100% Tuesday, September 24, 13

  92. Offloading Requests • Multiple CDNs • Img hit rate >

    90% • CSS/JS hit rate ~100% • Edge serves ~7000 reqs/sec Tuesday, September 24, 13

  93. What about misses? Tuesday, September 24, 13

  94. Tuesday, September 24, 13

  95. Change Management Tuesday, September 24, 13

  96. Feature Flags Tuesday, September 24, 13

  97. Tuesday, September 24, 13

  98. // Owner: Username $server_config['foo']['bar'] = array( 'users' => array('jsmith', 'jdoe'),

    ); // Owner: Username $server_config['baz'] = [ 'enabled' => [ 'variant1' => 0, 'variant2' => 0, 'variant3' => 50 ], ]; // Owner: Username $server_config['qux'] = array( 'enabled' => 100, 'data' => array( 'somedata' ) ); Tuesday, September 24, 13

  99. if (Feature::isEnabled('foo.bar')) { // Code } Tuesday, September 24, 13

  100. Ramp-up: 1%, 5%, 25%, 100% Tuesday, September 24, 13

  101. Deployinator Tuesday, September 24, 13

  102. Tuesday, September 24, 13

  103. Schemanator Tuesday, September 24, 13

  104. Tuesday, September 24, 13

  105. Recent Improvements Tuesday, September 24, 13

  106. Tuesday, September 24, 13

  107. Sandy Bridge Tuesday, September 24, 13

  108. Baseline Performance (median) Tuesday, September 24, 13

  109. Small code change Baseline Performance (median) Tuesday, September 24, 13

  110. Small code change PHP 5.4 Baseline Performance (median) Tuesday, September

    24, 13

  111. Small code change PHP 5.4 Disabled Hyperthreading Baseline Performance (median)

    Tuesday, September 24, 13

  112. Challenges Tuesday, September 24, 13

  113. Tuesday, September 24, 13

  114. Options Tuesday, September 24, 13

  115. Options • Scale Horizontally (more shards) Tuesday, September 24, 13

  116. Options • Scale Horizontally (more shards) • Scale Vertically (SSD,

    bigger drives) Tuesday, September 24, 13

  117. Options • Scale Horizontally (more shards) • Scale Vertically (SSD,

    bigger drives) • Change architecture Tuesday, September 24, 13

  118. Tuesday, September 24, 13

  119. Architecture Changes Tuesday, September 24, 13

  120. Architecture Changes • Ongoing conversation Tuesday, September 24, 13

  121. Architecture Changes • Ongoing conversation • Separate logical/physical shards Tuesday,

    September 24, 13

  122. Architecture Changes • Ongoing conversation • Separate logical/physical shards •

    Complex Tuesday, September 24, 13

  123. Architecture Changes • Ongoing conversation • Separate logical/physical shards •

    Complex • Server failure Tuesday, September 24, 13

  124. Architecture Changes • Ongoing conversation • Separate logical/physical shards •

    Complex • Server failure • Migrating data Tuesday, September 24, 13

  125. Architecture Changes • Ongoing conversation • Separate logical/physical shards •

    Complex • Server failure • Migrating data • Consistency Tuesday, September 24, 13

  126. Monitoring Tuesday, September 24, 13

  127. Graphite Tuesday, September 24, 13

  128. Stacked Search Timers Tuesday, September 24, 13

  129. Ganglia Tuesday, September 24, 13

  130. Tuesday, September 24, 13

  131. Nagios Tuesday, September 24, 13

  132. Tuesday, September 24, 13

  133. Tuesday, September 24, 13

  134. Takeaways Tuesday, September 24, 13

  135. K.I.S.S. Tuesday, September 24, 13

  136. Use Proven Technologies Tuesday, September 24, 13

  137. Tuesday, September 24, 13

  138. Understand Your Stack Tuesday, September 24, 13

  139. Measure Everything Tuesday, September 24, 13

  140. Work at Etsy ;-) Tuesday, September 24, 13

  141. Connect http://www.meetup.com/Web-Performance-Boston/ www.etsy.com/careers jonathan@etsy.com @jonathanklein Tuesday, September 24, 13