Upgrade to Pro — share decks privately, control downloads, hide ads and more …

HHVM at Etsy

Dan Miller
February 20, 2015

HHVM at Etsy

In 2014 Etsy’s infrastructure group was handed a challenge: scale Etsy’s API cluster 20x. Many efforts were simultaneously undertaken to meet this challenge, including a migration to HHVM after it showed a promising 5x increase in throughput. While getting our code to run on HHVM was easy, working through the deployment and operationalization proved to be a more difficult challenge.

This was presented at the PHP UK 2015 Conference.

Dan Miller

February 20, 2015
Tweet

More Decks by Dan Miller

Other Decks in Programming

Transcript

  1. @jazzdan Overview • What is HHVM? • Why were we

    interested? • How did we migrate? What problems did we encounter? • What else can it do? • The Future
  2. 2009 2015 ‘14 ‘13 ‘12 ‘11 ‘10 • Linux •

    Apache • MySQL • PHP • Couldn’t keep up with traffic growth
  3. 2009 2015 ‘14 ‘13 ‘12 ‘11 ‘10 • HipHop (HPHP)

    • Compile PHP to C++ • Deploy binary • Separate development environment
  4. @jazzdan HHVM is Open Source • Internal diffs developed in

    the open • Included in Linux distros • Over 2000 bugs opened and closed • Over 1000 pull requests accepted
  5. @jazzdan HHVM is Open Source 1. google.com 2. facebook.com 3.

    youtube.com 4. yahoo.com 5. baidu.com 6. amazon.com 7. wikipedia.org 8. twitter.com 9. taobao.com 10. qq.com
  6. @jazzdan HHVM is Open Source 1. google.com 2. facebook.com 3.

    youtube.com 4. yahoo.com 5. baidu.com 6. amazon.com 7. wikipedia.org 8. twitter.com 9. taobao.com 10. qq.com
  7. @jazzdan HHVM is Compatible* with PHP • 60% of PHP

    unit tests fail • Missing Extensions • Different error message output • 20 of the top PHP projects on GitHub do pass • 97% of unit tests pass among top 50 projects on GitHub
  8. @jazzdan HHVM is Compatible* with PHP • 99% of Etsy

    unit tests pass • (20 suite failures/1,798 test suites) Fail 20 Pass 1,798
  9. @jazzdan “Bespoke” Endpoints • Specific to a view • Aggregate

    REST endpoints concurrently • Return bespoke response
  10. @jazzdan curl_multi_* <?php   curl_multi_init();   curl_multi_add();   curl_multi_exec();  

    while(!$done)  {          curl_multi_select();          curl_multi_exec();          curl_multi_info_read();   }
  11. @jazzdan September 30th, 2014: “Fix the HHVM rpm we made

    last week that broke yum on every prod box”
  12. 6

  13. @jazzdan Had to Upgrade • gcc • libmcrypt • gmp

    • mpfr • mpc • glog • jemalloc • tbb • libdwarf • libmemcached • libc • cmake • libcurl • more
  14. Idea Code Release Idea Code A/B Test Release Idea Validate

    Prototype A/B Test Refinement A/B Test Release
  15. Idea Validate Prototype A/B Test Refinement A/B Test Release <-

    Possibly quite crappy <- Make it less crappy here
  16. Response Time as Load Increases Response Time 0 1000 2000

    3000 4000 Requests per Second 10 30 50 70 90 110 130 150 170 190 210 230 250 270 HHVM PHP 5.4
  17. Idea Validate Prototype A/B Test Refinement A/B Test Release Can

    HHVM run Etsy’s internal API? Is it faster? Time to fix the problems we skipped
  18. @jazzdan Both Machines • Read Only MySQL Interface • Read

    Only memcached Interface • Read Only Redis interface • iptables blocking almost all the things • No log forwarding
  19. #hhvm Repo Authoritative - Extra 20% • Produce bytecode SQLite

    database in advance • Build include map • Statically resolve file paths • Do non-type related optimizations at compile time
  20. HHVM vs PHP 5.5 on Etsy Internal API Median p95

    p99 Response Time in Milliseconds 0 200 400 600 800 HHVM PHP
  21. #hhvm sgrep: Problem Find all invocations of foo() where the

    second argument is 1, with any number of arguments after
  22. @jazzdan Lessons Learned • Do: • Run a newer Linux

    distribution • Ramp up slowly • Don’t: • Trust that extensions are 100% • Assume that processes are like threads