Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Highly available, fault tolerant, distributed apache for the masses

9ad2a5355d8cfa842e24b7a4322b2535?s=47 Martin Smith
October 17, 2012

Highly available, fault tolerant, distributed apache for the masses


Martin Smith

October 17, 2012


  1. (Attempts at) Highly Available, Fault Tolerant, Distributed Apache for the

    Masses Martin Smith @martinb3 martin@mbs3.org Dan Stoner @thatlinuxbox danstoner@gmail.com
  2. Sysadmins, programmers, software engineers, system architects, customer service representatives, on-call

    staff, marketing, technical support, teachers Who we are
  3. What we're trying to do ("The Goal") - Insulate customers

    from each other - Provide a fault tolerant environment - Create the most performant environment - Provide bulk hosting - Create an architecture that is easy to administer, with fewer special cases
  4. How do we determine success - Websites are available, fast,

    reliable, consistent - Efficient utilization of resources - Keep system load averages below 5 - Protect against massive resource consumption while still allowing burst - Maintainability is also an important factor
  5. Despite what you may have heard, Apache is still popular

    and de-facto Source: http://news.netcraft.com/archives/2012/10/02/october-2012-web- server-survey.html
  6. Busy sites still run Apache Source: http://news.netcraft.com/archives/2012/10/02/october-2012-web- server-survey.html

  7. Starting point: Most basic Apache configuration - One server, multiple

    virtual hosts - Local disk, local logs - mod_php, mod_perl, mod_foo for CGI - suexec for anything that isn't one of the above
  8. Enhancement: Load balancing - Explain different load balancing technologies -

    Explain some of our previous and current decisions for load balancing - Explain the effects on customers, difficulty of clustering different software
  9. None
  10. Graphs - Apache Requests per Second (all nodes total)

  11. Graphs - Apache Requests per Second (single node)

  12. Graphs - System Load Average (single node)

  13. Graphs - System Memory Profile (single node)

  14. CNS Managed Apache Hosting Overview

  15. Enhancement: Document Roots - NFS shares, replication - Giant directory

    vs. home directories - Pitfalls and Problems with this approach - Fun Apache behavior
  16. httpd.conf - Default virtual hosts get important - CGI is

    the hardest problem
  17. Enhancement: CGI - mod_php, mod_perl - Suexec - FastCGI for

    PHP, Perl - other enhancements for speed - Pros and cons with these approaches - What do large scale hosting providers do?
  18. httpd.conf (threading model) # worker MPM # StartServers: initial number

    of server processes to start # MaxClients: maximum number of simultaneous client connections # MinSpareThreads: minimum number of worker threads which are kept spare # MaxSpareThreads: maximum number of worker threads which are kept spare # ThreadsPerChild: constant number of worker threads in each server process # MaxRequestsPerChild: maximum number of requests a server process serves <IfModule worker.c> StartServers 2 ServerLimit 32 MaxClients 2048 MinSpareThreads 32 MaxSpareThreads 128 ThreadsPerChild 64 MaxRequestsPerChild 0 </IfModule>
  19. mod_status (server-status) is cool! Total accesses: 519889 - Total Traffic:

    6.1 GB CPU Usage: u129.93 s40.41 cu0 cs0 - 1.17% CPU load 35.8 requests/sec - 439.7 kB/second - 12.3 kB/request 101 requests currently being processed, 91 idle workers RR___RR_R_RRRCRW____R_W_R_R_____RR_R_RRR_R__RR_RRRR___RRRRR_R_RW RRRRRR__RR_RRWRCR_R______R__RRR_R_R__R___RC_RRR__R__R_____RRR___ RRR_C_R_R_R_R___RRRR_RR_RR__R__R_____WRR_RR_WRRRR__R__RR_R____R_ ................................................................ <snip> Scoreboard Key: "_" Waiting for Connection, " S" Starting up, "R" Reading Request," W" Sending Reply, "K" Keepalive (read), " D" DNS Lookup, "C" Closing connection, " L" Logging, "G" Gracefully finishing, " I" Idle cleanup of worker, "." Open slot with no current process Srv PID Acc M CPU SS Req Conn Child Slot Client VHost Request 0-0 23585 0/218/2420 R 20.03 39 7 0.0 2.48 32.84 ? ? .. reading..
  20. httpd.conf (server-status) LoadModule status_module modules/mod_status.so # ExtendedStatus controls whether Apache

    will generate "full" status # information (ExtendedStatus On) or just basic information (ExtendedStatus # Off) when the "server-status" handler is called. The default is Off. # ExtendedStatus On <Location /server-status> SetHandler server-status Order deny,allow Deny from all Allow from [some list of networks or IP addresses] </Location>
  21. Enhancement: Caching - PHP provides APC (file and opcode cache)

    - File System cache (VFS, backing systems) - Database server cache (maybe) - Applications frequently provide caching (e.g. WP-Cache, WP Super Cache, Joomla System Cache, Drupal Boost, ...) - Others (memcached, redis.... hard in bulk hosting environments) - Bottom line: It depends on the customer ->
  22. APC - Alternative PHP Cache

  23. APC (continued) - APC does not really help in an

    suexec environment - We have seen APC cause issues due to... caching! - Plan on "doing something" when you deploy new PHP code to flush the cache.
  24. Show some data!!! Number of Customers: ~70 Number of apache

    servers: 18 prod (including Gatorlink webmail) Number of vhosts: ~375 Busy site: http://www.ufl.edu www.ufl.edu visits per month: 1.3 million
  25. Server Hardware - All of our Apache servers are virtual

    machines (VMware VSphere 5) - Each virtual machine has modest resources: 2 processor cores 4 GB RAM 18 GB disk (plus NFS-mounted home dirs)
  26. Versions - koolaid or homebrew? Apache 2.2.x, PHP 5.2.x -

    Worth building packages? * security, maintainability, sustainability - Bulk hosting has unique challenges - Panels and 1-clicks and backups
  27. Enhancement: Miscellaneous - Other misc enhancements like Redhat upgrades, configuration

    changes, automation, etc
  28. Questions? Thank you! Please send Dan Stoner all of your

  29. Image sources http://hikaru.tea-nifty.com/robo/cat1176097/index.html http://vi.sualize.us/customer_customer_suggestion_box_picture_3Ukw.html http://www.imdb.com/title/tt0046719/ http://profitablegrowth.com/you-must-understand-we-all-wear-many-hats-here/ http://www.rosehosting.com/blog/how-to-install-lamp-linux-apache-mysql-and-php-on-centos-6-with-phpmyadmin-and- apc-cache/ http://koldfusion.ca/wp/2007/06/trivia-for-hackers-the-movie/