@jazzdan
HHVM is Open Source
• Internal diffs developed in the open
• Included in Linux distros
• Over 2000 bugs opened and closed
• Over 1000 pull requests accepted
@jazzdan
HHVM is Compatible* with PHP
• 60% of PHP unit tests fail
• Missing Extensions
• Different error message output
• 20 of the top PHP projects on GitHub do pass
• 97% of unit tests pass among top 50 projects on GitHub
Slide 25
Slide 25 text
@jazzdan
HHVM is Compatible* with PHP
• 99% of Etsy unit tests pass
• (20 suite failures/1,798 test suites)
Fail
20
Pass
1,798
Slide 26
Slide 26 text
@jazzdan
HHVM is Faster
Slide 27
Slide 27 text
@jazzdan
“Between 3x-6x Faster”
Slide 28
Slide 28 text
@jazzdan
(for Facebook)
Slide 29
Slide 29 text
No content
Slide 30
Slide 30 text
Why?
Slide 31
Slide 31 text
No content
Slide 32
Slide 32 text
Model
Controller
API
Business Logic
A
Business Logic
B
Business Logic
A
Slide 33
Slide 33 text
Model
API v3
Business Logic
Slide 34
Slide 34 text
@jazzdan
“Bespoke” Endpoints
• Specific to a view
• Aggregate REST endpoints concurrently
• Return bespoke response
Idea
Code
Release
Idea
Code
A/B Test
Release
“Oh crap…”
<- Wasted effort (maybe)
Slide 91
Slide 91 text
Idea
Code
Release
Idea
Code
A/B Test
Release
Idea
Validate
Prototype
A/B Test
Refinement
A/B Test
Release
Slide 92
Slide 92 text
Idea
Validate
Prototype
A/B Test
Refinement
A/B Test
Release
<- Possibly quite crappy
Slide 93
Slide 93 text
Idea
Validate
Prototype
A/B Test
Refinement
A/B Test
Release
<- Possibly quite crappy
<- Make it less crappy here
Slide 94
Slide 94 text
Idea
Validate
Prototype
A/B Test
Refinement
A/B Test
Release
Can HHVM run etsy.com?
Slide 95
Slide 95 text
Idea
Validate
Prototype
A/B Test
Refinement
A/B Test
Release
Can HHVM run etsy.com?
Is it faster?
Slide 96
Slide 96 text
Scoping
Slide 97
Slide 97 text
API Traffic Web Traffic
Slide 98
Slide 98 text
Run synthetic
benchmarks
Slide 99
Slide 99 text
Response Time as Load Increases
Response Time
0
1000
2000
3000
4000
Requests per Second
10 30 50 70 90 110 130 150 170 190 210 230 250 270
HHVM PHP 5.4
Slide 100
Slide 100 text
Happiness and Performance Correlation
Time
Response Time Happiness
Slide 101
Slide 101 text
Happiness and Performance Correlation
Time
Response Time Happiness
Slide 102
Slide 102 text
Run an experiment
Slide 103
Slide 103 text
Idea
Validate
Prototype
A/B Test
Refinement
A/B Test
Release
Can HHVM run Etsy’s internal API?
Is it faster?
Time to fix the problems we skipped
Slide 104
Slide 104 text
@jazzdan
How do we gain
more confidence?
Slide 105
Slide 105 text
@jazzdan
… and also
validate our hypothesis?
Slide 106
Slide 106 text
Tee Traffic
Slide 107
Slide 107 text
Load Balancer
API
API-HHVM
API-TEST
Slide 108
Slide 108 text
Load Balancer
API
API-HHVM
API-TEST
Slide 109
Slide 109 text
Load Balancer
API
API-HHVM
API-TEST
Slide 110
Slide 110 text
@jazzdan
Infrastructure
experiments are hard
Slide 111
Slide 111 text
@jazzdan
Same hardware
Slide 112
Slide 112 text
@jazzdan
Same traffic profile
Slide 113
Slide 113 text
@jazzdan
Same hacks
Slide 114
Slide 114 text
@jazzdan
Both Machines
• Read Only MySQL Interface
• Read Only memcached Interface
• Read Only Redis interface
• iptables blocking almost all the things
• No log forwarding
Slide 115
Slide 115 text
Happiness and Performance Correlation
Time
Response Time Happiness
Slide 116
Slide 116 text
Happiness and Performance Correlation
Time
Response Time Happiness
Slide 117
Slide 117 text
HHVM CPU 14 hour Zend CPU 14 hour
140 rps peak 140 rps peak
Slide 118
Slide 118 text
@jazzdan
“Between 3x-6x Faster”
Slide 119
Slide 119 text
#hhvm
Repo Authoritative - Extra 20%
• Produce bytecode SQLite database in advance
• Build include map
• Statically resolve file paths
• Do non-type related optimizations at compile time
Slide 120
Slide 120 text
Time
Deploys
About 60 deploys per day
Slide 121
Slide 121 text
@jazzdan
What about writing data?
Slide 122
Slide 122 text
Employee Only Traffic
Slide 123
Slide 123 text
@jazzdan
memcached
Slide 124
Slide 124 text
Memcached operation failed (returned false)
when decrementing KEY
Slide 125
Slide 125 text
Happiness and Performance Correlation
Time
Response Time Happiness
Slide 126
Slide 126 text
Happiness and Performance Correlation
Time
Response Time Happiness
Slide 127
Slide 127 text
No content
Slide 128
Slide 128 text
@jazzdan
All get()s were
returning false
Slide 129
Slide 129 text
Happiness and Performance Correlation
Time
Response Time Happiness
Slide 130
Slide 130 text
Happiness and Performance Correlation
Time
Response Time Happiness
Slide 131
Slide 131 text
No content
Slide 132
Slide 132 text
@jazzdan
Takeaway:
Slide 133
Slide 133 text
@jazzdan
HHVM is rock solid
Slide 134
Slide 134 text
@jazzdan
Extensions
sometimes have bugs
Slide 135
Slide 135 text
Slow Ramp Up
Slide 136
Slide 136 text
[23/janv./2015:22:40:32 +0000]
Slide 137
Slide 137 text
[23/ 1⽉月/2015:23:37:56]
Slide 138
Slide 138 text
request 1
setlocale(“a”) request 2
setlocale(“b”)
strftime()
strftime()
Time
Slide 139
Slide 139 text
@jazzdan
Solution:
newlocale()/uselocale()
Slide 140
Slide 140 text
@jazzdan
Takeaway:
Slide 141
Slide 141 text
@jazzdan
HHVM is threaded
Slide 142
Slide 142 text
Release!
Slide 143
Slide 143 text
No content
Slide 144
Slide 144 text
HHVM
Average Response Time
12 Hour
Zend
Average Response Time
12 Hour
Slide 145
Slide 145 text
HHVM
p95 Response Time
12 Hour
Zend
p95 Response Time
12 Hour
Slide 146
Slide 146 text
HHVM vs PHP 5.5 on Etsy Internal API
Median
p95
p99
Response Time in Milliseconds
0 200 400 600 800
HHVM PHP
Slide 147
Slide 147 text
What else can HHVM do?
Slide 148
Slide 148 text
@jazzdan
Flame Graphs
Slide 149
Slide 149 text
No content
Slide 150
Slide 150 text
@jazzdan
HHVM Debugger
Slide 151
Slide 151 text
No content
Slide 152
Slide 152 text
@jazzdan
pfff
Slide 153
Slide 153 text
@jazzdan
sgrep
Slide 154
Slide 154 text
#hhvm
sgrep: Problem
Find all invocations of foo() where the
second argument is 1, with any number of
arguments after
@jazzdan
If you’re running an old
operating system…
Slide 175
Slide 175 text
@jazzdan
…you’re gonna have
a bad time.
Slide 176
Slide 176 text
@jazzdan
Tee’ing traffic
is a superpower
Slide 177
Slide 177 text
@jazzdan
HHVM is Rock Solid
Slide 178
Slide 178 text
@jazzdan
Extensions
Sometimes Aren't
Slide 179
Slide 179 text
@jazzdan
Threads are Hard
Slide 180
Slide 180 text
@jazzdan
Tooling is Powerful
Slide 181
Slide 181 text
@jazzdan
Lessons Learned
• Do:
• Run a newer Linux distribution
• Ramp up slowly
• Don’t:
• Trust that extensions are 100%
• Assume that processes are like threads