Speaker Deck
Sign in
Sign up
for free
Scaling PHP to 40 Million Uniques
Jonathan Klein
August 17, 2013
Technology
3
310
Scaling PHP to 40 Million Uniques
This is a talk I gave at Northeast PHP 2013 in Boston, MA.
Links: jkle.in/nephp
Jonathan Klein
August 17, 2013
Tweet
Share
Want more?
jklein
12
220
jklein
1
420
jklein
2
560
reverentgeek
2
160
rmw
1
83
skipperchong
0
74
carmenhchung
5
150
keathley
4
160
productmarketing
2
220
clairelew
2
440
thoeni
1
120
jcasabona
4
200
Transcript
1
.
Scaling PHP to 40 Million Uniques Northeast PHP 2013 Jonathan
Klein @jonathanklein Tuesday, September 24, 13
2
.
Scaling PHP to 40 Million Uniques Northeast PHP 2013 Jonathan
Klein @jonathanklein Tuesday, September 24, 13
3
.
Scaling PHP to 60 Million Uniques Northeast PHP 2013 Jonathan
Klein @jonathanklein Tuesday, September 24, 13
4
.
Who Am I? Tuesday, September 24, 13
5
.
Who Am I? • Performance Engineer at Etsy Tuesday, September
24, 13
6
.
Who Am I? • Performance Engineer at Etsy • Boston
Web Perf Meetup Organizer Tuesday, September 24, 13
7
.
Who Am I? • Performance Engineer at Etsy • Boston
Web Perf Meetup Organizer • Previously at Wayfair Tuesday, September 24, 13
8
.
Who Am I? • Performance Engineer at Etsy • Boston
Web Perf Meetup Organizer • Previously at Wayfair • Led team that converted to PHP Tuesday, September 24, 13
9
.
Slides, Links: jkle.in/nephp Tuesday, September 24, 13
10
.
codeascraft.com Tuesday, September 24, 13
11
.
Some Etsy Stats Tuesday, September 24, 13
12
.
Some Etsy Stats • 1.5 billion page views/month Tuesday, September
24, 13
13
.
Some Etsy Stats • 1.5 billion page views/month • Almost
$1B in sales last year Tuesday, September 24, 13
14
.
Some Etsy Stats • 1.5 billion page views/month • Almost
$1B in sales last year • Over 1M lines of PHP Tuesday, September 24, 13
15
.
Some Etsy Stats • 1.5 billion page views/month • Almost
$1B in sales last year • Over 1M lines of PHP • 60M+ unique visitors/month Tuesday, September 24, 13
16
.
2012 Traffic Tuesday, September 24, 13
17
.
August 2012 Traffic Tuesday, September 24, 13
18
.
August 2012 Traffic 50% Higher Tuesday, September 24, 13
19
.
Holidays: ~90M Uniques Tuesday, September 24, 13
20
.
Tuesday, September 24, 13
21
.
Tuesday, September 24, 13
22
.
A few others... Tuesday, September 24, 13
23
.
A few others... • Search (Solr) Tuesday, September 24, 13
24
.
A few others... • Search (Solr) • Gearman Tuesday, September
24, 13
25
.
A few others... • Search (Solr) • Gearman • Redis
Tuesday, September 24, 13
26
.
A few others... • Search (Solr) • Gearman • Redis
• Postgres (legacy) Tuesday, September 24, 13
27
.
Tuesday, September 24, 13
28
.
Hardware (Supermicro) Tuesday, September 24, 13
29
.
Hardware (Supermicro) 2x 8-core Intel E5-2960 CPUs Tuesday, September 24,
13
30
.
Hardware (Supermicro) 2x 8-core Intel E5-2960 CPUs 24GB of RAM
Tuesday, September 24, 13
31
.
Hardware (Supermicro) 2x 8-core Intel E5-2960 CPUs 24GB of RAM
160GB SSD Tuesday, September 24, 13
32
.
Apache Tuesday, September 24, 13
33
.
Apache • Apache 2.2 Tuesday, September 24, 13
34
.
Apache • Apache 2.2 • Prefork MPM Tuesday, September 24,
13
35
.
Apache • Apache 2.2 • Prefork MPM • mod_php5 Tuesday,
September 24, 13
36
.
Apache • Apache 2.2 • Prefork MPM • mod_php5 StartServers
30 MinSpareServers 30 MaxSpareServers 60 ServerLimit 60 MaxClients 60 MaxRequestsPerChild 0 Tuesday, September 24, 13
37
.
PHP Tuesday, September 24, 13
38
.
PHP • PHP 5.4 Tuesday, September 24, 13
39
.
PHP • PHP 5.4 • Zend OPCache (3GB Memory Segment)
Tuesday, September 24, 13
40
.
PHP • PHP 5.4 • Zend OPCache (3GB Memory Segment)
• memory_limit: 128M Tuesday, September 24, 13
41
.
PHP • PHP 5.4 • Zend OPCache (3GB Memory Segment)
• memory_limit: 128M • max_execution_time: 30 Tuesday, September 24, 13
42
.
Optimizing PHP http://www.slideshare.net/jnklein/northeast-php-high-performance-php Tuesday, September 24, 13
43
.
Optimizing PHP • See last year’s talk http://www.slideshare.net/jnklein/northeast-php-high-performance-php Tuesday, September
24, 13
44
.
Optimizing PHP • See last year’s talk • Use an
Opcode Cache http://www.slideshare.net/jnklein/northeast-php-high-performance-php Tuesday, September 24, 13
45
.
Optimizing PHP • See last year’s talk • Use an
Opcode Cache • xhprof http://www.slideshare.net/jnklein/northeast-php-high-performance-php Tuesday, September 24, 13
46
.
Optimizing PHP • See last year’s talk • Use an
Opcode Cache • xhprof • StatsD/Graphite http://www.slideshare.net/jnklein/northeast-php-high-performance-php Tuesday, September 24, 13
47
.
Optimizing PHP • See last year’s talk • Use an
Opcode Cache • xhprof • StatsD/Graphite • Find hotspots http://www.slideshare.net/jnklein/northeast-php-high-performance-php Tuesday, September 24, 13
48
.
Optimizing PHP • See last year’s talk • Use an
Opcode Cache • xhprof • StatsD/Graphite • Find hotspots • Upgrade it http://www.slideshare.net/jnklein/northeast-php-high-performance-php Tuesday, September 24, 13
49
.
Static Arrays Tuesday, September 24, 13
50
.
Translations class language_de { static $translations = array( "<hash>" =>
array("content" => 'Accessoires', "file" => "TAXONOMY"), "<hash>" => array("content" => 'Keramik', "file" => "CATEGORY"), "<hash>" => array("content" => 'Dekorieren', "file" => "CATEGORY"), "<hash>" => array("content" => 'Getaggt %s', "file" => "Foo.php"), "<hash>" => array("content" => 'Badezusatz', "file" => "CATEGORY"), "<hash>" => array("content" => 'Datum:{% $date %}', "file" => "bar.tpl") ... snip ... ); } Tuesday, September 24, 13
51
.
Tuesday, September 24, 13
52
.
Tuesday, September 24, 13
53
.
Atomic Deploys Tuesday, September 24, 13
54
.
Deploying PHP http://codeascraft.com/2013/07/01/atomic-deploys-at-etsy/ Tuesday, September 24, 13
55
.
Deploying PHP • mod_realdoc - Apache module http://codeascraft.com/2013/07/01/atomic-deploys-at-etsy/ Tuesday, September
24, 13
56
.
Deploying PHP • mod_realdoc - Apache module • incpath -
PHP Extension http://codeascraft.com/2013/07/01/atomic-deploys-at-etsy/ Tuesday, September 24, 13
57
.
Deploying PHP • mod_realdoc - Apache module • incpath -
PHP Extension • A/B symlink swap http://codeascraft.com/2013/07/01/atomic-deploys-at-etsy/ Tuesday, September 24, 13
58
.
Deploying PHP • mod_realdoc - Apache module • incpath -
PHP Extension • A/B symlink swap • Avoid recompilation to opcodes http://codeascraft.com/2013/07/01/atomic-deploys-at-etsy/ Tuesday, September 24, 13
59
.
Upgrading PHP Tuesday, September 24, 13
60
.
PHP 5.4 vs. 5.3 - CPU PHP 5.3 PHP 5.4
Tuesday, September 24, 13
61
.
PHP 5.4 vs. 5.3 - Memory PHP 5.3 PHP 5.4
Tuesday, September 24, 13
62
.
Tuesday, September 24, 13
63
.
Understand Framework Overhead Tuesday, September 24, 13
64
.
http://systemsarchitect.net/performance-benchmark-of-popular-php-frameworks/ Tuesday, September 24, 13
65
.
Tuesday, September 24, 13
66
.
Tuesday, September 24, 13
67
.
Tuesday, September 24, 13
68
.
Frameworks are Harder to Scale Tuesday, September 24, 13
69
.
Tuesday, September 24, 13
70
.
Memcached Tuesday, September 24, 13
71
.
Memcached • Roughly same hardware as webs Tuesday, September 24,
13
72
.
Memcached • Roughly same hardware as webs • More memory
(48GB) Tuesday, September 24, 13
73
.
Memcached • Roughly same hardware as webs • More memory
(48GB) • Shard keys across servers Tuesday, September 24, 13
74
.
Memcached • Roughly same hardware as webs • More memory
(48GB) • Shard keys across servers • mctop - https://github.com/etsy/mctop Tuesday, September 24, 13
75
.
Tuesday, September 24, 13
76
.
MySQL Tuesday, September 24, 13
77
.
MySQL • MySQL 5.5 (http://jkle.in/nephp) Tuesday, September 24, 13
78
.
MySQL • MySQL 5.5 (http://jkle.in/nephp) • ~20 master/master pairs Tuesday,
September 24, 13
79
.
MySQL • MySQL 5.5 (http://jkle.in/nephp) • ~20 master/master pairs •
innodb Tuesday, September 24, 13
80
.
MySQL • MySQL 5.5 (http://jkle.in/nephp) • ~20 master/master pairs •
innodb • thread_cache=800 Tuesday, September 24, 13
81
.
MySQL • MySQL 5.5 (http://jkle.in/nephp) • ~20 master/master pairs •
innodb • thread_cache=800 • max_connections=2500 Tuesday, September 24, 13
82
.
MySQL • MySQL 5.5 (http://jkle.in/nephp) • ~20 master/master pairs •
innodb • thread_cache=800 • max_connections=2500 • [Almost] no joins Tuesday, September 24, 13
83
.
Theoretically we can serve ~9000 req/sec Tuesday, September 24, 13
84
.
Tuesday, September 24, 13
85
.
Static Content Tuesday, September 24, 13
86
.
Tuesday, September 24, 13
87
.
~500 Million Images Tuesday, September 24, 13
88
.
Offloading Requests Tuesday, September 24, 13
89
.
Offloading Requests • Multiple CDNs Tuesday, September 24, 13
90
.
Offloading Requests • Multiple CDNs • Img hit rate >
90% Tuesday, September 24, 13
91
.
Offloading Requests • Multiple CDNs • Img hit rate >
90% • CSS/JS hit rate ~100% Tuesday, September 24, 13
92
.
Offloading Requests • Multiple CDNs • Img hit rate >
90% • CSS/JS hit rate ~100% • Edge serves ~7000 reqs/sec Tuesday, September 24, 13
93
.
What about misses? Tuesday, September 24, 13
94
.
Tuesday, September 24, 13
95
.
Change Management Tuesday, September 24, 13
96
.
Feature Flags Tuesday, September 24, 13
97
.
Tuesday, September 24, 13
98
.
// Owner: Username $server_config['foo']['bar'] = array( 'users' => array('jsmith', 'jdoe'),
); // Owner: Username $server_config['baz'] = [ 'enabled' => [ 'variant1' => 0, 'variant2' => 0, 'variant3' => 50 ], ]; // Owner: Username $server_config['qux'] = array( 'enabled' => 100, 'data' => array( 'somedata' ) ); Tuesday, September 24, 13
99
.
if (Feature::isEnabled('foo.bar')) { // Code } Tuesday, September 24, 13
100
.
Ramp-up: 1%, 5%, 25%, 100% Tuesday, September 24, 13
101
.
Deployinator Tuesday, September 24, 13
102
.
Tuesday, September 24, 13
103
.
Schemanator Tuesday, September 24, 13
104
.
Tuesday, September 24, 13
105
.
Recent Improvements Tuesday, September 24, 13
106
.
Tuesday, September 24, 13
107
.
Sandy Bridge Tuesday, September 24, 13
108
.
Baseline Performance (median) Tuesday, September 24, 13
109
.
Small code change Baseline Performance (median) Tuesday, September 24, 13
110
.
Small code change PHP 5.4 Baseline Performance (median) Tuesday, September
24, 13
111
.
Small code change PHP 5.4 Disabled Hyperthreading Baseline Performance (median)
Tuesday, September 24, 13
112
.
Challenges Tuesday, September 24, 13
113
.
Tuesday, September 24, 13
114
.
Options Tuesday, September 24, 13
115
.
Options • Scale Horizontally (more shards) Tuesday, September 24, 13
116
.
Options • Scale Horizontally (more shards) • Scale Vertically (SSD,
bigger drives) Tuesday, September 24, 13
117
.
Options • Scale Horizontally (more shards) • Scale Vertically (SSD,
bigger drives) • Change architecture Tuesday, September 24, 13
118
.
Tuesday, September 24, 13
119
.
Architecture Changes Tuesday, September 24, 13
120
.
Architecture Changes • Ongoing conversation Tuesday, September 24, 13
121
.
Architecture Changes • Ongoing conversation • Separate logical/physical shards Tuesday,
September 24, 13
122
.
Architecture Changes • Ongoing conversation • Separate logical/physical shards •
Complex Tuesday, September 24, 13
123
.
Architecture Changes • Ongoing conversation • Separate logical/physical shards •
Complex • Server failure Tuesday, September 24, 13
124
.
Architecture Changes • Ongoing conversation • Separate logical/physical shards •
Complex • Server failure • Migrating data Tuesday, September 24, 13
125
.
Architecture Changes • Ongoing conversation • Separate logical/physical shards •
Complex • Server failure • Migrating data • Consistency Tuesday, September 24, 13
126
.
Monitoring Tuesday, September 24, 13
127
.
Graphite Tuesday, September 24, 13
128
.
Stacked Search Timers Tuesday, September 24, 13
129
.
Ganglia Tuesday, September 24, 13
130
.
Tuesday, September 24, 13
131
.
Nagios Tuesday, September 24, 13
132
.
Tuesday, September 24, 13
133
.
Tuesday, September 24, 13
134
.
Takeaways Tuesday, September 24, 13
135
.
K.I.S.S. Tuesday, September 24, 13
136
.
Use Proven Technologies Tuesday, September 24, 13
137
.
Tuesday, September 24, 13
138
.
Understand Your Stack Tuesday, September 24, 13
139
.
Measure Everything Tuesday, September 24, 13
140
.
Work at Etsy ;-) Tuesday, September 24, 13
141
.
Connect http://www.meetup.com/Web-Performance-Boston/ www.etsy.com/careers jonathan@etsy.com @jonathanklein Tuesday, September 24, 13
Jonathan Klein
jklein
reverentgeek
rmw
skipperchong
carmenhchung
keathley
productmarketing
clairelew
thoeni
jcasabona
![// Owner: Username $server_config['foo']['bar'] = array( 'users' => array('jsmith', 'jdoe'),](https://files.speakerdeck.com/presentations/b4aef6b02d2401312d7e2658b910d553/slide_97.jpg){kind=link}
