Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Scaling PHP to 40 Million Uniques
Search
Jonathan Klein
August 17, 2013
Technology
3
1.2k
Scaling PHP to 40 Million Uniques
This is a talk I gave at Northeast PHP 2013 in Boston, MA.
Links: jkle.in/nephp
Jonathan Klein
August 17, 2013
Tweet
Share
More Decks by Jonathan Klein
See All by Jonathan Klein
Cognitive Biases in Engineering Organizations - Craft Conf
jklein
15
410
Demystifying SPDY and HTTP/2
jklein
1
620
Demystifying SPDY and HTTP/2
jklein
2
1.1k
Cognitive Biases in Engineering Organizations
jklein
4
2k
Composer: From Beginner to Expert
jklein
1
960
Upgrading the Web: Boston Web Performance Meetup
jklein
1
290
Upgrading the Web: Driving Support For New Standards
jklein
1
800
Northeastern Lunch and Learn
jklein
0
250
Profiling PHP With XHProf
jklein
1
1k
Other Decks in Technology
See All in Technology
Four Keysから始める信頼性の改善 - SRE NEXT 2025
ozakikota
0
210
AWS CDKの仕組み / how-aws-cdk-works
gotok365
10
890
マーケットプレイス版Oracle WebCenter Content For OCI
oracle4engineer
PRO
3
990
american airlines®️ USA Contact Numbers: Complete 2025 Support Guide
supportflight
1
120
Delta airlines®️ USA Contact Numbers: Complete 2025 Support Guide
airtravelguide
0
350
How Do I Contact HP Printer Support? [Full 2025 Guide for U.S. Businesses]
harrry1211
0
130
Enhancing SaaS Product Reliability and Release Velocity through Optimized Testing Approach
ropqa
1
250
大量配信システムにおけるSLOの実践:「見えない」信頼性をSLOで可視化
plaidtech
PRO
0
290
VS CodeとGitHub Copilotで爆速開発!アップデートの波に乗るおさらい会 / Rapid Development with VS Code and GitHub Copilot: Catch the Latest Wave
yamachu
2
340
[ JAWS-UG千葉支部 x 彩の国埼玉支部 ]ムダ遣い卒業!FinOpsで始めるAWSコスト最適化の第一歩
sh_fk2
2
150
AIエージェントが書くのなら直接CloudFormationを書かせればいいじゃないですか何故AWS CDKを使う必要があるのさ
watany
18
7k
60以上のプロダクトを持つ組織における開発者体験向上への取り組み - チームAPIとBackstageで構築する組織の可視化基盤 - / sre next 2025 Efforts to Improve Developer Experience in an Organization with Over 60 Products
vtryo
3
980
Featured
See All Featured
Why You Should Never Use an ORM
jnunemaker
PRO
58
9.4k
実際に使うSQLの書き方 徹底解説 / pgcon21j-tutorial
soudai
PRO
181
54k
ピンチをチャンスに:未来をつくるプロダクトロードマップ #pmconf2020
aki_iinuma
126
53k
Being A Developer After 40
akosma
90
590k
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
eileencodes
35
2.4k
Site-Speed That Sticks
csswizardry
10
700
YesSQL, Process and Tooling at Scale
rocio
173
14k
Keith and Marios Guide to Fast Websites
keithpitt
411
22k
Making the Leap to Tech Lead
cromwellryan
134
9.4k
Helping Users Find Their Own Way: Creating Modern Search Experiences
danielanewman
29
2.7k
Visualizing Your Data: Incorporating Mongo into Loggly Infrastructure
mongodb
47
9.6k
How GitHub (no longer) Works
holman
314
140k
Transcript
Scaling PHP to 40 Million Uniques Northeast PHP 2013 Jonathan
Klein @jonathanklein Tuesday, September 24, 13
Scaling PHP to 40 Million Uniques Northeast PHP 2013 Jonathan
Klein @jonathanklein Tuesday, September 24, 13
Scaling PHP to 60 Million Uniques Northeast PHP 2013 Jonathan
Klein @jonathanklein Tuesday, September 24, 13
Who Am I? Tuesday, September 24, 13
Who Am I? • Performance Engineer at Etsy Tuesday, September
24, 13
Who Am I? • Performance Engineer at Etsy • Boston
Web Perf Meetup Organizer Tuesday, September 24, 13
Who Am I? • Performance Engineer at Etsy • Boston
Web Perf Meetup Organizer • Previously at Wayfair Tuesday, September 24, 13
Who Am I? • Performance Engineer at Etsy • Boston
Web Perf Meetup Organizer • Previously at Wayfair • Led team that converted to PHP Tuesday, September 24, 13
Slides, Links: jkle.in/nephp Tuesday, September 24, 13
codeascraft.com Tuesday, September 24, 13
Some Etsy Stats Tuesday, September 24, 13
Some Etsy Stats • 1.5 billion page views/month Tuesday, September
24, 13
Some Etsy Stats • 1.5 billion page views/month • Almost
$1B in sales last year Tuesday, September 24, 13
Some Etsy Stats • 1.5 billion page views/month • Almost
$1B in sales last year • Over 1M lines of PHP Tuesday, September 24, 13
Some Etsy Stats • 1.5 billion page views/month • Almost
$1B in sales last year • Over 1M lines of PHP • 60M+ unique visitors/month Tuesday, September 24, 13
2012 Traffic Tuesday, September 24, 13
August 2012 Traffic Tuesday, September 24, 13
August 2012 Traffic 50% Higher Tuesday, September 24, 13
Holidays: ~90M Uniques Tuesday, September 24, 13
Tuesday, September 24, 13
Tuesday, September 24, 13
A few others... Tuesday, September 24, 13
A few others... • Search (Solr) Tuesday, September 24, 13
A few others... • Search (Solr) • Gearman Tuesday, September
24, 13
A few others... • Search (Solr) • Gearman • Redis
Tuesday, September 24, 13
A few others... • Search (Solr) • Gearman • Redis
• Postgres (legacy) Tuesday, September 24, 13
Tuesday, September 24, 13
Hardware (Supermicro) Tuesday, September 24, 13
Hardware (Supermicro) 2x 8-core Intel E5-2960 CPUs Tuesday, September 24,
13
Hardware (Supermicro) 2x 8-core Intel E5-2960 CPUs 24GB of RAM
Tuesday, September 24, 13
Hardware (Supermicro) 2x 8-core Intel E5-2960 CPUs 24GB of RAM
160GB SSD Tuesday, September 24, 13
Apache Tuesday, September 24, 13
Apache • Apache 2.2 Tuesday, September 24, 13
Apache • Apache 2.2 • Prefork MPM Tuesday, September 24,
13
Apache • Apache 2.2 • Prefork MPM • mod_php5 Tuesday,
September 24, 13
Apache • Apache 2.2 • Prefork MPM • mod_php5 StartServers
30 MinSpareServers 30 MaxSpareServers 60 ServerLimit 60 MaxClients 60 MaxRequestsPerChild 0 Tuesday, September 24, 13
PHP Tuesday, September 24, 13
PHP • PHP 5.4 Tuesday, September 24, 13
PHP • PHP 5.4 • Zend OPCache (3GB Memory Segment)
Tuesday, September 24, 13
PHP • PHP 5.4 • Zend OPCache (3GB Memory Segment)
• memory_limit: 128M Tuesday, September 24, 13
PHP • PHP 5.4 • Zend OPCache (3GB Memory Segment)
• memory_limit: 128M • max_execution_time: 30 Tuesday, September 24, 13
Optimizing PHP http://www.slideshare.net/jnklein/northeast-php-high-performance-php Tuesday, September 24, 13
Optimizing PHP • See last year’s talk http://www.slideshare.net/jnklein/northeast-php-high-performance-php Tuesday, September
24, 13
Optimizing PHP • See last year’s talk • Use an
Opcode Cache http://www.slideshare.net/jnklein/northeast-php-high-performance-php Tuesday, September 24, 13
Optimizing PHP • See last year’s talk • Use an
Opcode Cache • xhprof http://www.slideshare.net/jnklein/northeast-php-high-performance-php Tuesday, September 24, 13
Optimizing PHP • See last year’s talk • Use an
Opcode Cache • xhprof • StatsD/Graphite http://www.slideshare.net/jnklein/northeast-php-high-performance-php Tuesday, September 24, 13
Optimizing PHP • See last year’s talk • Use an
Opcode Cache • xhprof • StatsD/Graphite • Find hotspots http://www.slideshare.net/jnklein/northeast-php-high-performance-php Tuesday, September 24, 13
Optimizing PHP • See last year’s talk • Use an
Opcode Cache • xhprof • StatsD/Graphite • Find hotspots • Upgrade it http://www.slideshare.net/jnklein/northeast-php-high-performance-php Tuesday, September 24, 13
Static Arrays Tuesday, September 24, 13
Translations class language_de { static $translations = array( "<hash>" =>
array("content" => 'Accessoires', "file" => "TAXONOMY"), "<hash>" => array("content" => 'Keramik', "file" => "CATEGORY"), "<hash>" => array("content" => 'Dekorieren', "file" => "CATEGORY"), "<hash>" => array("content" => 'Getaggt %s', "file" => "Foo.php"), "<hash>" => array("content" => 'Badezusatz', "file" => "CATEGORY"), "<hash>" => array("content" => 'Datum:{% $date %}', "file" => "bar.tpl") ... snip ... ); } Tuesday, September 24, 13
Tuesday, September 24, 13
Tuesday, September 24, 13
Atomic Deploys Tuesday, September 24, 13
Deploying PHP http://codeascraft.com/2013/07/01/atomic-deploys-at-etsy/ Tuesday, September 24, 13
Deploying PHP • mod_realdoc - Apache module http://codeascraft.com/2013/07/01/atomic-deploys-at-etsy/ Tuesday, September
24, 13
Deploying PHP • mod_realdoc - Apache module • incpath -
PHP Extension http://codeascraft.com/2013/07/01/atomic-deploys-at-etsy/ Tuesday, September 24, 13
Deploying PHP • mod_realdoc - Apache module • incpath -
PHP Extension • A/B symlink swap http://codeascraft.com/2013/07/01/atomic-deploys-at-etsy/ Tuesday, September 24, 13
Deploying PHP • mod_realdoc - Apache module • incpath -
PHP Extension • A/B symlink swap • Avoid recompilation to opcodes http://codeascraft.com/2013/07/01/atomic-deploys-at-etsy/ Tuesday, September 24, 13
Upgrading PHP Tuesday, September 24, 13
PHP 5.4 vs. 5.3 - CPU PHP 5.3 PHP 5.4
Tuesday, September 24, 13
PHP 5.4 vs. 5.3 - Memory PHP 5.3 PHP 5.4
Tuesday, September 24, 13
Tuesday, September 24, 13
Understand Framework Overhead Tuesday, September 24, 13
http://systemsarchitect.net/performance-benchmark-of-popular-php-frameworks/ Tuesday, September 24, 13
Tuesday, September 24, 13
Tuesday, September 24, 13
Tuesday, September 24, 13
Frameworks are Harder to Scale Tuesday, September 24, 13
Tuesday, September 24, 13
Memcached Tuesday, September 24, 13
Memcached • Roughly same hardware as webs Tuesday, September 24,
13
Memcached • Roughly same hardware as webs • More memory
(48GB) Tuesday, September 24, 13
Memcached • Roughly same hardware as webs • More memory
(48GB) • Shard keys across servers Tuesday, September 24, 13
Memcached • Roughly same hardware as webs • More memory
(48GB) • Shard keys across servers • mctop - https://github.com/etsy/mctop Tuesday, September 24, 13
Tuesday, September 24, 13
MySQL Tuesday, September 24, 13
MySQL • MySQL 5.5 (http://jkle.in/nephp) Tuesday, September 24, 13
MySQL • MySQL 5.5 (http://jkle.in/nephp) • ~20 master/master pairs Tuesday,
September 24, 13
MySQL • MySQL 5.5 (http://jkle.in/nephp) • ~20 master/master pairs •
innodb Tuesday, September 24, 13
MySQL • MySQL 5.5 (http://jkle.in/nephp) • ~20 master/master pairs •
innodb • thread_cache=800 Tuesday, September 24, 13
MySQL • MySQL 5.5 (http://jkle.in/nephp) • ~20 master/master pairs •
innodb • thread_cache=800 • max_connections=2500 Tuesday, September 24, 13
MySQL • MySQL 5.5 (http://jkle.in/nephp) • ~20 master/master pairs •
innodb • thread_cache=800 • max_connections=2500 • [Almost] no joins Tuesday, September 24, 13
Theoretically we can serve ~9000 req/sec Tuesday, September 24, 13
Tuesday, September 24, 13
Static Content Tuesday, September 24, 13
Tuesday, September 24, 13
~500 Million Images Tuesday, September 24, 13
Offloading Requests Tuesday, September 24, 13
Offloading Requests • Multiple CDNs Tuesday, September 24, 13
Offloading Requests • Multiple CDNs • Img hit rate >
90% Tuesday, September 24, 13
Offloading Requests • Multiple CDNs • Img hit rate >
90% • CSS/JS hit rate ~100% Tuesday, September 24, 13
Offloading Requests • Multiple CDNs • Img hit rate >
90% • CSS/JS hit rate ~100% • Edge serves ~7000 reqs/sec Tuesday, September 24, 13
What about misses? Tuesday, September 24, 13
Tuesday, September 24, 13
Change Management Tuesday, September 24, 13
Feature Flags Tuesday, September 24, 13
Tuesday, September 24, 13
// Owner: Username $server_config['foo']['bar'] = array( 'users' => array('jsmith', 'jdoe'),
); // Owner: Username $server_config['baz'] = [ 'enabled' => [ 'variant1' => 0, 'variant2' => 0, 'variant3' => 50 ], ]; // Owner: Username $server_config['qux'] = array( 'enabled' => 100, 'data' => array( 'somedata' ) ); Tuesday, September 24, 13
if (Feature::isEnabled('foo.bar')) { // Code } Tuesday, September 24, 13
Ramp-up: 1%, 5%, 25%, 100% Tuesday, September 24, 13
Deployinator Tuesday, September 24, 13
Tuesday, September 24, 13
Schemanator Tuesday, September 24, 13
Tuesday, September 24, 13
Recent Improvements Tuesday, September 24, 13
Tuesday, September 24, 13
Sandy Bridge Tuesday, September 24, 13
Baseline Performance (median) Tuesday, September 24, 13
Small code change Baseline Performance (median) Tuesday, September 24, 13
Small code change PHP 5.4 Baseline Performance (median) Tuesday, September
24, 13
Small code change PHP 5.4 Disabled Hyperthreading Baseline Performance (median)
Tuesday, September 24, 13
Challenges Tuesday, September 24, 13
Tuesday, September 24, 13
Options Tuesday, September 24, 13
Options • Scale Horizontally (more shards) Tuesday, September 24, 13
Options • Scale Horizontally (more shards) • Scale Vertically (SSD,
bigger drives) Tuesday, September 24, 13
Options • Scale Horizontally (more shards) • Scale Vertically (SSD,
bigger drives) • Change architecture Tuesday, September 24, 13
Tuesday, September 24, 13
Architecture Changes Tuesday, September 24, 13
Architecture Changes • Ongoing conversation Tuesday, September 24, 13
Architecture Changes • Ongoing conversation • Separate logical/physical shards Tuesday,
September 24, 13
Architecture Changes • Ongoing conversation • Separate logical/physical shards •
Complex Tuesday, September 24, 13
Architecture Changes • Ongoing conversation • Separate logical/physical shards •
Complex • Server failure Tuesday, September 24, 13
Architecture Changes • Ongoing conversation • Separate logical/physical shards •
Complex • Server failure • Migrating data Tuesday, September 24, 13
Architecture Changes • Ongoing conversation • Separate logical/physical shards •
Complex • Server failure • Migrating data • Consistency Tuesday, September 24, 13
Monitoring Tuesday, September 24, 13
Graphite Tuesday, September 24, 13
Stacked Search Timers Tuesday, September 24, 13
Ganglia Tuesday, September 24, 13
Tuesday, September 24, 13
Nagios Tuesday, September 24, 13
Tuesday, September 24, 13
Tuesday, September 24, 13
Takeaways Tuesday, September 24, 13
K.I.S.S. Tuesday, September 24, 13
Use Proven Technologies Tuesday, September 24, 13
Tuesday, September 24, 13
Understand Your Stack Tuesday, September 24, 13
Measure Everything Tuesday, September 24, 13
Work at Etsy ;-) Tuesday, September 24, 13
Connect http://www.meetup.com/Web-Performance-Boston/ www.etsy.com/careers
[email protected]
@jonathanklein Tuesday, September 24, 13