Highly available, fault tolerant, distributed apache for the masses

9ad2a5355d8cfa842e24b7a4322b2535?s=47 Martin Smith
October 17, 2012
18

Highly available, fault tolerant, distributed apache for the masses

9ad2a5355d8cfa842e24b7a4322b2535?s=128

Martin Smith

October 17, 2012
Tweet

Transcript

  1. 1.

    (Attempts at) Highly Available, Fault Tolerant, Distributed Apache for the

    Masses Martin Smith @martinb3 martin@mbs3.org Dan Stoner @thatlinuxbox danstoner@gmail.com
  2. 3.

    What we're trying to do ("The Goal") - Insulate customers

    from each other - Provide a fault tolerant environment - Create the most performant environment - Provide bulk hosting - Create an architecture that is easy to administer, with fewer special cases
  3. 4.

    How do we determine success - Websites are available, fast,

    reliable, consistent - Efficient utilization of resources - Keep system load averages below 5 - Protect against massive resource consumption while still allowing burst - Maintainability is also an important factor
  4. 5.

    Despite what you may have heard, Apache is still popular

    and de-facto Source: http://news.netcraft.com/archives/2012/10/02/october-2012-web- server-survey.html
  5. 7.

    Starting point: Most basic Apache configuration - One server, multiple

    virtual hosts - Local disk, local logs - mod_php, mod_perl, mod_foo for CGI - suexec for anything that isn't one of the above
  6. 8.

    Enhancement: Load balancing - Explain different load balancing technologies -

    Explain some of our previous and current decisions for load balancing - Explain the effects on customers, difficulty of clustering different software
  7. 9.
  8. 15.

    Enhancement: Document Roots - NFS shares, replication - Giant directory

    vs. home directories - Pitfalls and Problems with this approach - Fun Apache behavior
  9. 17.

    Enhancement: CGI - mod_php, mod_perl - Suexec - FastCGI for

    PHP, Perl - other enhancements for speed - Pros and cons with these approaches - What do large scale hosting providers do?
  10. 18.

    httpd.conf (threading model) # worker MPM # StartServers: initial number

    of server processes to start # MaxClients: maximum number of simultaneous client connections # MinSpareThreads: minimum number of worker threads which are kept spare # MaxSpareThreads: maximum number of worker threads which are kept spare # ThreadsPerChild: constant number of worker threads in each server process # MaxRequestsPerChild: maximum number of requests a server process serves <IfModule worker.c> StartServers 2 ServerLimit 32 MaxClients 2048 MinSpareThreads 32 MaxSpareThreads 128 ThreadsPerChild 64 MaxRequestsPerChild 0 </IfModule>
  11. 19.

    mod_status (server-status) is cool! Total accesses: 519889 - Total Traffic:

    6.1 GB CPU Usage: u129.93 s40.41 cu0 cs0 - 1.17% CPU load 35.8 requests/sec - 439.7 kB/second - 12.3 kB/request 101 requests currently being processed, 91 idle workers RR___RR_R_RRRCRW____R_W_R_R_____RR_R_RRR_R__RR_RRRR___RRRRR_R_RW RRRRRR__RR_RRWRCR_R______R__RRR_R_R__R___RC_RRR__R__R_____RRR___ RRR_C_R_R_R_R___RRRR_RR_RR__R__R_____WRR_RR_WRRRR__R__RR_R____R_ ................................................................ <snip> Scoreboard Key: "_" Waiting for Connection, " S" Starting up, "R" Reading Request," W" Sending Reply, "K" Keepalive (read), " D" DNS Lookup, "C" Closing connection, " L" Logging, "G" Gracefully finishing, " I" Idle cleanup of worker, "." Open slot with no current process Srv PID Acc M CPU SS Req Conn Child Slot Client VHost Request 0-0 23585 0/218/2420 R 20.03 39 7 0.0 2.48 32.84 ? ? .. reading..
  12. 20.

    httpd.conf (server-status) LoadModule status_module modules/mod_status.so # ExtendedStatus controls whether Apache

    will generate "full" status # information (ExtendedStatus On) or just basic information (ExtendedStatus # Off) when the "server-status" handler is called. The default is Off. # ExtendedStatus On <Location /server-status> SetHandler server-status Order deny,allow Deny from all Allow from [some list of networks or IP addresses] </Location>
  13. 21.

    Enhancement: Caching - PHP provides APC (file and opcode cache)

    - File System cache (VFS, backing systems) - Database server cache (maybe) - Applications frequently provide caching (e.g. WP-Cache, WP Super Cache, Joomla System Cache, Drupal Boost, ...) - Others (memcached, redis.... hard in bulk hosting environments) - Bottom line: It depends on the customer ->
  14. 23.

    APC (continued) - APC does not really help in an

    suexec environment - We have seen APC cause issues due to... caching! - Plan on "doing something" when you deploy new PHP code to flush the cache.
  15. 24.

    Show some data!!! Number of Customers: ~70 Number of apache

    servers: 18 prod (including Gatorlink webmail) Number of vhosts: ~375 Busy site: http://www.ufl.edu www.ufl.edu visits per month: 1.3 million
  16. 25.

    Server Hardware - All of our Apache servers are virtual

    machines (VMware VSphere 5) - Each virtual machine has modest resources: 2 processor cores 4 GB RAM 18 GB disk (plus NFS-mounted home dirs)
  17. 26.

    Versions - koolaid or homebrew? Apache 2.2.x, PHP 5.2.x -

    Worth building packages? * security, maintainability, sustainability - Bulk hosting has unique challenges - Panels and 1-clicks and backups