Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Site Availability is for Everybody

9c3bc4fea06c0e0bafab417be0bbdb74?s=47 stellacotton
February 12, 2016

Site Availability is for Everybody

9c3bc4fea06c0e0bafab417be0bbdb74?s=128

stellacotton

February 12, 2016
Tweet

More Decks by stellacotton

Other Decks in Programming

Transcript

  1. Stella Cotton | @practice_cactus S I T E AVA I

    L A B I L I T Y I S F O R E V E RY B O D Y —
  2. Stella Cotton | @practice_cactus P O P Q U I

    Z
  3. Stella Cotton | @practice_cactus

  4. Stella Cotton | @practice_cactus €1,600,000,000

  5. Stella Cotton | @practice_cactus

  6. Stella Cotton | @practice_cactus ☺

  7. Stella Cotton | @practice_cactus 1 . G E T S

    TA RT E D 2 . T U R N U P T H E V O L U M E 3 . E X P L O R E T H E R E S U LT S
  8. Stella Cotton | @practice_cactus G E T T I N

    G S TA RT E D
  9. Stella Cotton | @practice_cactus Getting Started P R E PA

    R E
  10. Getting Started: Stella Cotton | @practice_cactus Prepare http://example.com/ -c 1

    -n 1000 ab
  11. Getting Started: Stella Cotton | @practice_cactus Prepare http://example.com/ -c 1

    -n 1000 ab
  12. Getting Started: Stella Cotton | @practice_cactus Don’t DDoS Yourself Prepare

  13. Getting Started: Stella Cotton | @practice_cactus (Yes, technically you are

    only DoS-ing yourself by running Apache Bench on a single machine but you can DDoS with other tools and it sounded funnier. ¯\_(π)_/¯ ) Prepare
  14. Stella Cotton | @practice_cactus Getting Started C O N F

    I G U R E L O A D
  15. Getting Started: Stella Cotton | @practice_cactus Configure Load http://example.com/ -c

    1 -n 1000 ab
  16. Getting Started: Stella Cotton | @practice_cactus “Request” Configure Load

  17. Getting Started: Stella Cotton | @practice_cactus No HTML No JS

    Configure Load
  18. Stella Cotton | @practice_cactus Getting Started A N A LY

    Z E
  19. Getting Started: Stella Cotton | @practice_cactus Analyze Results

  20. Getting Started: Stella Cotton | @practice_cactus Analyze Results

  21. Getting Started: Stella Cotton | @practice_cactus Analyze Results

  22. Getting Started: Stella Cotton | @practice_cactus 3xx/4xx/5xx Analyze Results

  23. Getting Started: Stella Cotton | @practice_cactus Analyze Results http://example.com/ -c

    5 -n 1000 ab -A user:password
  24. Getting Started: Stella Cotton | @practice_cactus Analyze Results

  25. Getting Started: Stella Cotton | @practice_cactus “Failed Requests” Analyze Results

  26. Getting Started: Stella Cotton | @practice_cactus Analyze Results

  27. Stella Cotton | @practice_cactus T U R N I N

    G U P T H E V O L U M E
  28. Stella Cotton | @practice_cactus Turning Up the Volume Y O

    U R A P P + L O A D
  29. Turning Up the Volume: Stella Cotton | @practice_cactus Little’s Law:

    L = λW Your App + Load
  30. Turning Up the Volume: Stella Cotton | @practice_cactus Total Customers

    in Line Your App + Load Arrival
 rate How long it takes to check out = x
  31. Turning Up the Volume: Stella Cotton | @practice_cactus Mean response

    time Your App + Load Mean # in system Mean throughput =
  32. Turning Up the Volume: Stella Cotton | @practice_cactus 1. Proxy

    Server 2. Application Server 3. Database Your App + Load
  33. Turning Up the Volume: Stella Cotton | @practice_cactus Your App

    + Load Proxy Server User Visits Your Site (Yay!) Firewall
  34. Turning Up the Volume: Stella Cotton | @practice_cactus Your App

    + Load Application Server Proxy Server User Visits Your Site (Yay!) Firewall
  35. Turning Up the Volume: Stella Cotton | @practice_cactus Your App

    + Load Proxy Server Application Server (Master Process) Worker (Child Process) Worker (Child Process) Worker (Child Process)
  36. Turning Up the Volume: Stella Cotton | @practice_cactus Your App

    + Load Proxy Server Application Server (Master Process) Database Worker (Child Process) Worker (Child Process) Worker (Child Process)
  37. Turning Up the Volume: Stella Cotton | @practice_cactus Your App

    + Load “504 Gateway Time-out”
  38. Turning Up the Volume: Stella Cotton | @practice_cactus Multiple Queues

    are Less Efficient Your App + Load
  39. Turning Up the Volume: Stella Cotton | @practice_cactus Your App

    + Load PROXY SERVER APP WORKERS
  40. Turning Up the Volume: Stella Cotton | @practice_cactus Your App

    + Load PROXY SERVER APP WORKERS
  41. Turning Up the Volume: Stella Cotton | @practice_cactus Increasing Timeout

    Threshold Your App + Load
  42. Stella Cotton | @practice_cactus Turning Up the Volume Y O

    U R O S + L O A D
  43. Turning Up the Volume: Stella Cotton | @practice_cactus “Too Many

    Open Files” Your OS + Load
  44. Turning Up the Volume: Stella Cotton | @practice_cactus User file

    descriptor limits: ulimit -Hn ulimit -Sn Your OS + Load
  45. Turning Up the Volume: Stella Cotton | @practice_cactus Estimating user

    file limit numbers: • Each file is ~ 1 k of memory • Don’t allocate more than 10% of memory to files • So, approximately 100 file descriptors for every MB of RAM Your OS + Load
  46. Turning Up the Volume: Stella Cotton | @practice_cactus Adjust the

    System Limit: vi /etc/sysctl.conf with fs.file-max = 70000 Your OS + Load
  47. Turning Up the Volume: Stella Cotton | @practice_cactus Adjust the

    User Limit # vi /etc/security/limits.conf with yourusername soft no file 10000 yourusername hard no file 30000 Your OS + Load H T T P : / / W W W. C Y B E R C I T I . B I Z / FA Q / L I N U X - U N I X - N G I N X - T O O - M A N Y- O P E N - F I L E S /
  48. Turning Up the Volume: Stella Cotton | @practice_cactus Reload your

    changes # sysctl -p Your OS + Load
  49. Turning Up the Volume: Stella Cotton | @practice_cactus Adjust the

    Proxy Server e.g. NGINX worker_rlimit_nofile 30000; Your OS + Load
  50. Turning Up the Volume: Stella Cotton | @practice_cactus TCP/IP Port

    Exhaustion Your OS + Load
  51. Turning Up the Volume: Stella Cotton | @practice_cactus Ephemeral Ports

    Your OS + Load
  52. Turning Up the Volume: Stella Cotton | @practice_cactus Decrease TIME

    WAIT Your OS + Load
  53. Turning Up the Volume: Stella Cotton | @practice_cactus Increase Port

    Range Your OS + Load
  54. Turning Up the Volume: Stella Cotton | @practice_cactus Server recommendations

    
 for OS Tuning 
 H T T P : / / U N I C O R N . B O G O M I P S . O R G / T U N I N G . H T M L Your OS + Load
  55. Stella Cotton | @practice_cactus Turning Up the Volume ❄ ❄

  56. Turning Up the Volume: Stella Cotton | @practice_cactus Cache Busting

    ❄❄❄
  57. Turning Up the Volume: Stella Cotton | @practice_cactus Blocking External

    Requests ❄❄❄
  58. Stella Cotton | @practice_cactus E X P L O R

    I N G Y O U R R E S U LT S
  59. Stella Cotton | @practice_cactus Exploring Your Results L I M

    I TAT I O N S O N T H E H O S T M A C H I N E
  60. Exploring Your Results: Stella Cotton | @practice_cactus top Investigating the

    Limits
  61. Exploring Your Results: Stella Cotton | @practice_cactus Investigating the Limits

  62. Exploring Your Results: Stella Cotton | @practice_cactus htop Investigating the

    Limits
  63. Exploring Your Results: Stella Cotton | @practice_cactus Investigating the Limits

  64. Exploring Your Results: Stella Cotton | @practice_cactus Zero Sum Game

    Investigating the Limits
  65. Exploring Your Results: Stella Cotton | @practice_cactus External database Investigating

    the Limits
  66. Exploring Your Results: Stella Cotton | @practice_cactus Don’t DDoS Yourself

    Investigating the Limits
  67. Stella Cotton | @practice_cactus Exploring Your Results T R A

    N S A C T I O N T R A C I N G
  68. Exploring Your Results: Stella Cotton | @practice_cactus Transaction Tracing H

    T T P S : / / D O C S . N E W R E L I C . C O M / D O C S / A P M / T R A N S A C T I O N S / T R A N S A C T I O N - T R A C E S / T R A N S A C T I O N - T R A C E S
  69. Exploring Your Results: Stella Cotton | @practice_cactus rack-mini-profiler Transaction Tracing

  70. Stella Cotton | @practice_cactus Exploring Your Results D ATA B

    A S E W O E S
  71. Exploring Your Results: Stella Cotton | @practice_cactus Slow Query Log

    H T T P : / / D E V. M Y S Q L . C O M / D O C / R E F M A N / 5 . 7 / E N / S L O W - Q U E RY- L O G . H T M L Database Woes
  72. Exploring Your Results: Stella Cotton | @practice_cactus SHOW PROCESSLIST; Database

    Woes
  73. Stella Cotton | @practice_cactus O T H E R L

    O A D T E S T I N G T O O L S
  74. Other Load Testing Tools Stella Cotton | @practice_cactus • Siege

    + Bombard • Bees with Machine Guns • JMeter + Ruby JMeter • ..and many more!
  75. Stella Cotton | @practice_cactus F R A M E W

    O R K F O R C U R I O S I T Y
  76. Stella Cotton | @practice_cactus T H A N K Y

    O U —