HHVM at Etsy

HHVM at Etsy Harder, Better, Faster, Stronger Dan Miller Core
Platform Engineer Etsy

@jazzdan http://bit.ly/hhvm_etsy

The World’s Handmade Marketplace

Time Deploys About 60 deploys per day

@jazzdan Overview • What is HHVM? • Why were we
interested? • How did we migrate? What problems did we encounter? • What else can it do? • The Future

2009 2015 ‘14 ‘13 ‘12 ‘11 ‘10

2009 2015 ‘14 ‘13 ‘12 ‘11 ‘10 • Linux •
Apache • MySQL • PHP

2009 2015 ‘14 ‘13 ‘12 ‘11 ‘10 • Linux •
Apache • MySQL • PHP • Couldn’t keep up with traffic growth

2009 2015 ‘14 ‘13 ‘12 ‘11 ‘10

2009 2015 ‘14 ‘13 ‘12 ‘11 ‘10 • HipHop (HPHP)
• Compile PHP to C++ • Deploy binary • Separate development environment

@jazzdan HipHop Virtual Machine (HHVM)

@jazzdan HHVM is not a source transformer (That was HPHPc)

Webserver PHP Backend Services FastCGI index.php Logger.php Tpl.php

Webserver HHVM Backend Services FastCGI index.php Logger.php Tpl.php

2009 2015 ‘14 ‘13 ‘12 ‘11 ‘10

@jazzdan HHVM is Open Source • Internal diffs developed in
the open • Included in Linux distros • Over 2000 bugs opened and closed • Over 1000 pull requests accepted

@jazzdan HHVM is Open Source 1. google.com 2. facebook.com 3.
youtube.com 4. yahoo.com 5. baidu.com 6. amazon.com 7. wikipedia.org 8. twitter.com 9. taobao.com 10. qq.com

@jazzdan HHVM is Compatible* with PHP • 60% of PHP
unit tests fail • Missing Extensions • Different error message output • 20 of the top PHP projects on GitHub do pass • 97% of unit tests pass among top 50 projects on GitHub

@jazzdan HHVM is Compatible* with PHP • 99% of Etsy
unit tests pass • (20 suite failures/1,798 test suites) Fail 20 Pass 1,798

@jazzdan HHVM is Faster

@jazzdan “Between 3x-6x Faster”

@jazzdan (for Facebook)

Model Controller API Business Logic A Business Logic B Business
Logic A

Model API v3 Business Logic

@jazzdan “Bespoke” Endpoints • Specific to a view • Aggregate
REST endpoints concurrently • Return bespoke response

@jazzdan Bespoke Endpoints etsy.com/listing/124740565 Bespoke Recent Shop Overview Related Seller
Listing

@jazzdan Overview Related Listing Bespoke Endpoints etsy.com/listing/124740565 Bespoke Recent Shop
Seller

@jazzdan curl_multi_* <?php curl_multi_init(); curl_multi_add(); curl_multi_exec();
while(!$done) { curl_multi_select(); curl_multi_exec(); curl_multi_info_read(); }

@jazzdan Bespoke Endpoints etsy.com/listing/124740565 Bespoke Recent Shop Overview Related Seller
Listing

API Traffic Web Traffic

@jazzdan Lower latency = more RPS/box

@jazzdan More RPS/box = Fewer boxes

@jazzdan Fewer boxes = less $/datacenter

@jazzdan Fewer boxes =

$scp -‐r rlerdorf.vms.etsy.com:/home/rlerdorf/hhvm-‐root .

@jazzdan Missing memcached constants?

@jazzdan /*Missing memcached constants*/

@jazzdan Missing geoip extension?

@jazzdan /*Missing geoip extension?*/

@jazzdan Missing msgpack extension?

@jazzdan Login doesn’t need to work for now

@jazzdan Etc

@jazzdan Ran some benchmarks

@jazzdan It’s faster

@jazzdan Way faster.

Happiness and Performance Correlation Time Response Time Happiness

@jazzdan Time to Compile HHVM

@jazzdan How hard can it be?

@jazzdan September 2nd, 2014: “Started compiling HHVM”

@jazzdan September 23rd, 2014: “strace’d cmake”

@jazzdan ( $git clean -fdx )

@jazzdan September 24th, 2014: “GOT IT TO COMPILE”

@jazzdan September 25th, 2014: “BUILT AN RPM”

libs=$(ldd "${hhvm_binary}" | awk '{print $3}' | grep -‐v '^$'

@jazzdan September 30th, 2014: “Fix the HHVM rpm we made
last week that broke yum on every prod box”

@jazzdan Had to Upgrade • gcc • libmcrypt • gmp
• mpfr • mpc • glog • jemalloc • tbb • libdwarf • libmemcached • libc • cmake • libcurl • more

The Test

Idea Code Release Time

Idea Code Release Idea Code Release A/B Test

Idea Code Release Idea Code A/B Test Release “Oh crap…”
<- Wasted effort (maybe)

Idea Code Release Idea Code A/B Test Release Idea Validate
Prototype A/B Test Reﬁnement A/B Test Release

Idea Validate Prototype A/B Test Reﬁnement A/B Test Release <-
Possibly quite crappy

Idea Validate Prototype A/B Test Reﬁnement A/B Test Release <-
Possibly quite crappy <- Make it less crappy here

Idea Validate Prototype A/B Test Reﬁnement A/B Test Release Can
HHVM run etsy.com?

HHVM run etsy.com? Is it faster?

Scoping

API Traffic Web Traffic

Run synthetic benchmarks

Response Time as Load Increases Response Time 0 1000 2000
3000 4000 Requests per Second 10 30 50 70 90 110 130 150 170 190 210 230 250 270 HHVM PHP 5.4

Run an experiment

HHVM run Etsy’s internal API? Is it faster? Time to fix the problems we skipped

@jazzdan How do we gain more confidence?

@jazzdan … and also validate our hypothesis?

Tee Traffic

Load Balancer API API-HHVM API-TEST

@jazzdan Infrastructure experiments are hard

@jazzdan Same hardware

@jazzdan Same traffic profile

@jazzdan Same hacks

@jazzdan Both Machines • Read Only MySQL Interface • Read
Only memcached Interface • Read Only Redis interface • iptables blocking almost all the things • No log forwarding

HHVM CPU 14 hour Zend CPU 14 hour 140 rps
peak 140 rps peak

@jazzdan “Between 3x-6x Faster”

#hhvm Repo Authoritative - Extra 20% • Produce bytecode SQLite
database in advance • Build include map • Statically resolve file paths • Do non-type related optimizations at compile time

Time Deploys About 60 deploys per day

@jazzdan What about writing data?

Employee Only Traffic

@jazzdan memcached

Memcached operation failed (returned false) when decrementing KEY

@jazzdan All get()s were returning false

@jazzdan Takeaway:

@jazzdan HHVM is rock solid

@jazzdan Extensions sometimes have bugs

Slow Ramp Up

[23/janv./2015:22:40:32 +0000]

[23/ 1⽉月/2015:23:37:56]

request 1 setlocale(“a”) request 2 setlocale(“b”) strftime() strftime() Time

@jazzdan Solution: newlocale()/uselocale()

@jazzdan Takeaway:

@jazzdan HHVM is threaded

Release!

HHVM Average Response Time 12 Hour Zend Average Response Time
12 Hour

HHVM p95 Response Time 12 Hour Zend p95 Response Time
12 Hour

HHVM vs PHP 5.5 on Etsy Internal API Median p95
p99 Response Time in Milliseconds 0 200 400 600 800 HHVM PHP

What else can HHVM do?

@jazzdan Flame Graphs

@jazzdan HHVM Debugger

@jazzdan pfff

@jazzdan sgrep

#hhvm sgrep: Problem Find all invocations of foo() where the
second argument is 1, with any number of arguments after

#hhvm sgrep: Solution $ sgrep -e 'foo(X, 1, ...)' *.php

#hhvm sgrep: Problem Find all && where both operands are
the same

#hhvm sgrep: Solution $ sgrep -e 'X && X' *.php

#hhvm sgrep: Solution $ sgrep -pvar X -e 'X &&
X' *.php

#hhvm sgrep: Problem Find all calls to foo() where the
first argument is 1

#hhvm sgrep: Solution $ sgrep -e 'foo(1, ...)' *.php

#hhvm sgrep: Problem Find all method calls “addPreparable()” with any
number of arguments

#hhvm sgrep: Solution $ sgrep -e 'X->addPreparable(...)'

@jazzdan spatch

#hhvm spatch: Problem Remove the second argument from all invocations
of function foo()

#hhvm spatch: Solution //remove_second_arg_foo.spatch foo(X - ,Y )

#hhvm spatch: Solution $ spatch -f remove_second_arg_foo.spatch *.php

#hhvm spatch: Problem Rename a function with a variable number
of arguments

#hhvm spatch: Solution - foo + bar (...)

@jazzdan perf(1)

@jazzdan perf(1) - 12.09% HPHP::f_sort + PHP::…getAllReplicantNames

Lessons Learned

@jazzdan If you’re running an old operating system…

@jazzdan …you’re gonna have a bad time.

@jazzdan Tee’ing traffic is a superpower

@jazzdan HHVM is Rock Solid

@jazzdan Extensions Sometimes Aren't

@jazzdan Threads are Hard

@jazzdan Tooling is Powerful

@jazzdan Lessons Learned • Do: • Run a newer Linux
distribution • Ramp up slowly • Don’t: • Trust that extensions are 100% • Assume that processes are like threads

The Future

@jazzdan What does the future hold?

@jazzdan No one knows

@jazzdan But now we are better prepared

@jazzdan Questions? ( https://joind.in/13384 )

HHVM at Etsy

HHVM at Etsy

More Decks by Dan Miller

Other Decks in Programming

Featured

Transcript