Battle of the Stacks (with presenter notes)

When you first start as a developer all you care about is the code. As you progress and become more concerned with performance you begin to realize that there’s a whole world of possibilities and no clear indication of what works best.

In this presentation we’ll look at different combinations of web server technologies, caching, PHP versions and a lot more; then pit them against each other in a head to head competition to find out which one is the last stack standing.

Transcript

1. Battle of the Stacks by Nic Rosental @nicdev
2. What’s a stack?
3. The components that make up your application. From the web server software to the code.
4. Performance means speed and reliability When you first start coding, you can just run with your default LAMP stack, or shared hosting and never think twice about it. However, as your career progresses you'll quickly find out that those solutions are inadequate for anything but the smallest of projects. Any significant amount of traffic and your site will lag for ages, or even crash.
5. MOAR HARDWARE!!!!!1!!!11 The first instinct is usually to just add more RAM, get a faster CPU or some other server upgrade. This step should be saved until you've exhausted all other possibilities: optimize code (which we won't cover here) and picking the right stack and setting it up appropriately, which we will.
6. Goals for this session • Get a general overview of different stacks • Discuss some of the basics First we'll take a look at some common stacks used for PHP sites and apps, then we'll pit them against each other in a battle to the death to find out which one is the Last Stack Standing. We're going to use a tournament style bracket just for fun. Please DO NOT think for one second that you should base any decisions regarding your stack choice on the results of this “competition." We are not going to exhaust every possible combination of component or learn any of the components in detail.
7. Buyer Beware! I’m only going to cover some basic stuff, and the tests I will show are definitely not the way you should determine which stack is best for you. Again, the goal is to give you just enough to get you informed and ready to go learn more.
8. The Setup All the screenshots, tests, etc. that we’re going to look at today were done on a 1GB server, running Ubuntu 12.04 LTS, hosted at Digital Ocean.
9. LAMP stack The most popular stack out there is undeniably the LAMP stack. This is Linux + Apache + MySQL + PHP This setup can go a long way, if used properly. However, what makes it so popular is also what makes it inadequate in many aspects. The blessing/curse is that it's so easy to set up. Let's say we start a new server, in this case Ubuntu 12.04.
10. With a single command we can install the full LAMP stack
13. What do you get with this type of installation? You get a fully functional LAMP stack with an outdated version of PHP, and a default configuration of Apache 2 running in prefork mode. Of course you also get MySQL, but we will not be looking at databases in this presentation. Ask: How many of you have used this setup or similar before?
14. Prefork vs Worker Prefork MPM uses multiple child processes with one thread each and each process handles one connection at a time. ! Worker MPM uses multiple child processes with many threads each. Each thread handles one connection at a time.
15. #1 Default LAMP Stack The default Apache config is not optimized for high traffic. But because this configuration is so popular, our first contender is going to be the default LAMP stack provided in Ubuntu
16. APC Alternative PHP Cache ! ! In layman's terms, this means that the generated bytecode is stored in memory which helps speed up the overall application.
17. The Alternative PHP Cache (APC) is a free and open opcode cache for PHP. Its goal is to provide a free, open, and robust framework for caching and optimizing PHP intermediate code. ! Source: http://php.net/manual/en/intro.apc.php In layman's terms, this means that the generated bytecode is stored in memory which helps speed up the overall application. No need to interpret and compile on every request.
18. It’s extremely easy to install and works right out of the box. However, it does come with a ton of configuration options which you can review on your own.
19. Included with the installation there's a handy script `apc.php` which you can use to see how APC is performing, and whether it's actually working.
20. Here we see how with very minimal effort and zero dollars we can add to our stack. As you may imagine, this is expected to provide improvements in performance, but let's wait until the next section of the presentation to confirm or deny that.
21. Other alternatives we won’t be looking at in this session There are other so-called accelerators available for PHP. I won't be covering them in this session, so please raise your hand now. I will give you the opportunity to say the name of your favorite accelerator that I neglected to mention, and 15 seconds to tell everyone why it's awesome.
22. eAccelerator ionCube Turck MMCache XCache Nusphere PHPExpress Zend OPcache* et al. http://en.wikipedia.org/wiki/List_of_PHP_accelerators I will cover Zend Opcache later on
23. #2 Default LAMP Stack + APC Our second contender is default LAMP stack + APC To reiterate: Ubuntu 12.04 LTS, Apache 2.2, PHP 5.3.10 + APC
24. The new and shiny. Actually not so new. PHP 5.5 has been around since June 2013, and it's the current stable. How many of you are running 5.5?
25. 5.4? What about 5.4? Although it provides performance improvements over 5.3 in most cases you are either stuck at the default PHP version bundled in your OS of choice, or you upgrade to the latest. Since I picked 12.04 as the OS for this session, PHP 5.4 falls somewhere in between those two categories, so I preferred to skip it altogether.
26. Remember all that APC stuff we talked about a minute ago? Well, PHP 5.5 talks a lot of smack and flat out deprecates APC. That’s because 5.5 comes with Zend OPcache built in !
27. What does it do? Zend OPcache does the same APC does (i.e bytecode caching) but it's the one that's been integrated. APC's latest version (3.1.13) was released on 9/2012 so unless you have a really good reason* to keep it around just go with the bundled one. *If you are using it for user cache, there's APCu.
28. #3 LAMP Stack with PHP 5.5
29. So far we just looked at some low hanging fruit. All you had to do was install some software and done. Let's get our hands dirty with some actual work.
30. For stacks #4, #5 and #6 we're going back to the original default LAMP stack (5.3) and tune-up Apache (just a little bit.) We are not going to look at the gazillion available options in detail. We’ll just cover some basics.
31. Here's the segment of the default configuration we looked at before. I focus on this segment because it's the main place to start looking into changing the way you use Apache. There's a lot more you can work on, but most performance articles (and my experince) are centered around these settings.
32. StartServers, MinSpareServers, MaxSpareServers StartServers, MinSpareServers, MaxSpareServers ! These three directives regulate how many "servers" are started, and how many will stay idle when there's no activity.
33. StartServers In older versions of Apache it made sense to increase this because Apache itself enforced a limit of spawning one child process per second, which could seriously slow down "ramping up." Apache now doubles the number of spawns every second until reaching 32 per second. Once it passes the 4 per second mark it will log an error. I've never had to check on this, but if you see this in your logs, it may be a good idea to increase this number. Otherwise, defaults should be fine.
34. MinSpareServers This is the number of idle servers available at any given time. If the number of servers available falls below this limit, new idle processes will be spun up and sit in waiting. According to the documentation "Tuning of this parameter should only be necessary on very busy sites. Setting this parameter to a large number is almost always a bad idea." There are potential performance improvements, but proceed with care. Setting it too high can be detrimental to performance.
35. MaxSpareServers This is the number of idle servers that will be left running. Setting this too high can mean that you may be using too many resources when not needed. According to the docs: "Tuning of this parameter should only be necessary on very busy sites. Setting this parameter to a large number is almost always a bad idea." By now it should be clear that Apache comes out of the box expecting to be ran on small sites, and uses very conservative values for its default configuration.
36. MaxClients This is where it gets fun. MaxClients is the number of clients that will be serviced at any given time. Anything above this number will be queued up. When setting this up, be mindful that a high setting can exhaust server resources, and too low can keep your clients waiting unnecessarily, making your site appear slow.
37. MaxRequestsPerChild This is the number of requests a child process will handle before dying. The default of 0 means it will never die, which is ok unless you have a good reason to have them die; for example if there's a known issue with your particular OS. Do not set this too low as to avoid having to re-spawn processes unnecessarily.
38. Tune Up Let's tune it up. The Apache documentation recommends focusing your efforts in preserving RAM. Once you exhaust RAM, your server will start swapping memory which will slow it down beyond what's acceptable. There are a ton of settings and tricks available that we won't be discussing. Instead we'll focus on what seems to be the most common strategies for better Apache performance.
39. Quick Math We have a VPS with 1024MB of RAM. We need enough RAM to run the OS, database, etc. So let's suppose we want to limit the web server to using at most, half of the total RAM, 512MB
40. In this case it's about 36MB per process. Since we have a MaxClients set to 150 (default) if we were to become suddenly very popular, we'd need to us 36MB x 150 = 5400MB to service all requests. That's over 5 times the available RAM. Our server would definitely crash. Yes, because of the way Apache spawns new processes and whatnot it’ll never actually reach 5GB of RAM usage.
41. Reliability > Speed Serving pages fast is important, but serving them reliably is fundamental. If we want to keep the server running, we need to limit how far it'll go trying to service requests.
42. 512MB / 36MB = ~14 MaxClients Since we know our average process is 36MB, and we want to keep it down to half of the available RAM, we can do 512MB / 36MB = ~14 MaxClients You shouldn't use this as the setting only after this quick exercise. Consider it a starting point, and test and monitor to determine the best setting for your particular situation. Also remember that this is one of the simplest configuration changes you can make, so don't stop there and continue tweaking and optimizing.
43. #4, #5 and #6 Default LAMP Stack with MaxClients configured For stacks #4, #5 and #6 we are going to change the MaxClients setting. We could change more settings, but I want to focus on the effects this particular setting has on performance.
44. #4
45. #5 has 21 MaxClients but I forgot the screenshot, #6 has 7 MaxClients. So from the basis of 14 MaxClients, we went 7 clients in both directions.
46. PHP-FPM Only two more to go before the Battle of the Stacks. Does anyone use FPM? Next up we have FPM, the FastCGI Process Manager. What does it do? Instead of letting Apache itself process the PHP code (with mod_php), it will pass it along to FPM and serve back the processed response. In this case Apache is set up in Worker mode as opposed to the more common Prefork.
47. Here's the info page. Notice that we left the default PHP 5.3 version installed. Again, I'm trying to show each possible route separately. Most of these options are not mutually exclusive and you should mix and match until you find the one that works best. We won't mess with any additional settings for this presentation, but you should.
48. The memory usage of Apache is minimal, but now we have to look out for FPM’s RAM consumption.
49. #7 LAMP with PHP-FPM Contender #7 is the default LAMP configuration using PHP-FPM isntead of the more popular mod_php.
50. LEMP For the last stack we are going to look at, we ditch Apache altogether and use NGINX instead. This configuration is commonly known as LEMP (Linux, NGINX, MySQL and PHP.) NGINX is another open source web server, but it's been designed with performance in mind.
51. This is a fraud! We’re here for the new hawtness. HHVM is all the hotness right now. I'm sorry to disappoint. There's a lot of exciting stuff going on with HHVM but I have close to zero experience with it. So I won’t be talking about it in this session.
52. So back to LEMP. Like Apache Worker, NGINX will not process the PHP itself, but pass it to PHP-FPM. Still rocking good ole 5.3 with PHP-FPM.
53. Let the games begin! If it isn't clear by now. The results of this "battle" don't mean much. They are just a fun way to see how all these technologies behave with a very simple set of tests and minimal configuration.
54. All the stacks were set up on the exact same server. 1GB RAM, Ubuntu 12.04, hosted on Digital Ocean.
55. The tests consist of installing a default WordPress on each one of the stacks, then running 2 tests on each setup using Loader.io. Then we compare how each setup did (response time and error %.) We will match up each stack in a head to head competition until we are left with the last stack standing.
56. Round 1 tests: Steady: 1 client per second over 60 seconds. Busy: 250 clients over 60 seconds
57. This is what the Loader.io tests look like. All the tests used for this presentation are available online.
59. Round 1! ! #4 LAMP with MaxClients 14! Steady Test http://ldr.io/1jWXpDV Avg time: 265ms Errors: 0 Busy Test http://ldr.io/1myl4rv Avg time: 2218ms Errors: 6 ! #3 LAMP with PHP 5.5 (Zend OPcache)! Steady Test http://ldr.io/1jX5CI7 Avg time: 68ms Errors: 0 Busy Test http://ldr.io/1jX5XdO Avg time: 116ms Errors: 0 What an improvement PHP 5.5 can make! If your code base is 5.5 compatible, there's a lot to be gained by upgrading.
60. Round 1! ! #7 LAMP with PHP-FPM! Steady Test http://ldr.io/1jX7JLZ Avg time: 393ms Errors: 0 Busy Test http://ldr.io/1fpPoTJ Avg time: 7847 Errors: 25 (test aborted for +50% error threshold) ! #6 LAMP with MaxClients 7! Steady Test http://ldr.io/1jWWu6b Avg time: 257ms Errors: 0 Busy Test http://ldr.io/1fpvVTf Avg time: 2031ms Errors: 26 What an upset! Does this mean that mod_php is better than FPM? In most cases, no. But it does show that not every setup fits every situation. Remember that reliability trumps speed.
61. Round 1! ! #1 Default LAMP! Steady Test http://ldr.io/1myp1w4 Avg time: 265ms Errors: 0 Busy Test http://ldr.io/1myoR7Z Avg time: 1979ms Errors: 1 ! #5 LAMP with MaxClients 21! Steady Test http://ldr.io/1mylpdB Avg time: 260ms Errors: 0 Busy Test http://ldr.io/1jWYTxQ Avg time: 3230ms Errors: 15 The default LAMP wins. Even though it performed slightly slower in the Steady Test, it took the Busy Test hands down.
62. Down to the final 4
63. Round 2 Tests Steady: 2 clients per second over 60 seconds. Busy: 250 clients over 30 seconds Double up on the tests
64. Round 2! ! #2 LAMP with APC! Steady Test http://ldr.io/1fpGDZL Avg time: 139ms Errors: 0 Busy Test http://ldr.io/1fpG6Hp Avg time: 1912ms Errors: 0 ! #3 LAMP with PHP 5.5 (Zend OPcache)! Steady Test http://ldr.io/1jX6f4r Avg time: 93ms Errors: 0 Busy Test http://ldr.io/1eaWqrQ Avg time: 341ms Errors: 0 Winner #3, LAMP with PHP 5.5. The improvements in the latest version of PHP are clear.
65. Round 2! ! #6 LAMP with MaxClients 7! Steady Test http://ldr.io/1jWWu6b Avg time: 257ms Errors: 0 Busy Test http://ldr.io/1fpvVTf Avg time: 2031ms Errors: 26 ! #1 Default LAMP! Steady Test http://ldr.io/1fpVT90 Avg time: 464ms Errors: 0 Busy Test http://ldr.io/1jXbHnQ Avg time: 7594ms Errors: 20 (test aborted due to +50% error rate) Here's the proof that once past a certain threshold, the default configuration just doesn't hold up.
67. Final Round ! Test: 500 clients over 30 seconds
68. Final Round! ! #3 LAMP with PHP 5.5 (Zend OPcache) Final Round Test http://ldr.io/1fq0jN2 Avg time: 392ms Errors: 0 ! #6 LAMP with MaxClients 7 Final Round Test http://ldr.io/1fpYhwo Avg time: n/a Errors: n/a Test aborted with zero successful responses Our Last Stack Standing is LAMP with PHP 5.5.
69. Key Takeaways ! There are a lot of options available that don't cost anything. Exhaust all these before upgrading server resources. All the tests were done on a server that costs $10/month. We now know that it can withstand a pretty hefty load with some minimal configuration improvements. Optimize for speed and also for reliability. There's no point in serving pages really fast if the server can’t handle more than a few connections. What works for someone else may not be the best for you. Analyze traffic patterns, test out different setups and then do it again. Experiment, combine and don’t be afraid. There’s a lot to gain.
70. Thanks y’all
71. nic@epiclabs.com ! @nicdev