A Locust Storm Is A Good Thing!

A Locust Storm Is A Good Thing!

Given at Stir Trek 2016

Have you ever wondered what is involved in doing load testing of web applications? What if you had to support 100,000 simultaneous users on Day 0? Do you have to integrate with 3rd party services? How do you strategically isolate and test those dependencies? What do you do when your tests reveal performance problems? Join us in a talk about web load testing using open source tools, the AWS cloud, dependency isolation, DB profiling, OS tuning, and not being targeted as a botnet.

6b6afbaea3bf1de98975dedc5cd083c1?s=128

stevenjackson

May 06, 2016
Tweet

Transcript

  1. 1.

    A Locust Swarm Can Be a Good Thing! Steve Jackson

    @stevejxsn steve.jackson@leandog.com
  2. 3.

    A Favor? • Something you would change • Something you

    would NOT change • Something you found surprising • Something you found boring
  3. 4.
  4. 5.
  5. 6.
  6. 7.

    A Locust Swarm Can Be a Good Thing! Steve Jackson

    @stevejxsn steve.jackson@leandog.com
  7. 10.

    • How to start? • Why Locust? • How to

    deal with problems in the architecture? • Make an argument for starting load testing early • Tell a compelling story Goals
  8. 11.

    • .realtor starts - Feb 2014 • Load Test start

    - 15 Sep 2014 • Soft Launch - 20 Oct 2014 • Launch - 23 Oct 2014 Timeline
  9. 14.

    Send Emails DNS Mail Forwarding Payment Processor CREA Membership NAR

    Membership Registrar Hosted Website System Architecture
  10. 16.

    NAR Membership System System Send Emails DNS Mail Forwarding Payment

    Processor CREA Membership Registrar Hosted Website Prioritizing Dependencies
  11. 17.

    Fake Member Service
 Sinatra System System Fake DNS
 Sinatra Fake

    Registrar
 EventMachine Fake Payment
 Sinatra Stubbing Dependencies
  12. 19.

    Why Locust? • Could interact with Rails CSRF tokens •

    Could execute end-to-end user interaction with sessions and cookies • Expands to multiple slaves to increase load capacity • Allows for distributed user paths based on percentages
  13. 23.
  14. 24.
  15. 26.

    Methodology • Start Small (1000 simultaneous users) • Gather data

    • Grow infrastructure as we hit bottlenecks
  16. 31.
  17. 32.
  18. 39.

    • Compression = YES, please • Serve static assets directly

    • Reverse proxy (proxy_pass) the rest to unicorn
  19. 40.

    worker_processes 32 worker_rlimit_nofile 65536 use epoll worker_connections 65536 upstream realtor_unicorn

    { server unix:/tmp/realtor.sock fail_timeout=0; } keepalive_requests 0 proxy_read_timeout proxy_write_timeout client_max_body_size Take a look at these nginx.conf sites-available/sitename
  20. 42.

    rails_user hard nofile 65536 /etc/security/limits.conf • ulimits to max #

    Increase size of file handles and inode cache fs.file-max = 100000 /etc/sysctl.conf
  21. 44.
  22. 45.
  23. 46.

    What did I learn? • Don’t wait so long to

    start load testing • The conversations drive new requirements • This stuff is hard to figure out under pressure • Too late to do big changes confidently
  24. 48.
  25. 49.
  26. 50.
  27. 51.
  28. 52.
  29. 53.
  30. 54.
  31. 55.
  32. 56.
  33. 59.
  34. 61.
  35. 63.

    15

  36. 64.
  37. 65.
  38. 66.
  39. 67.

    Send Emails DNS Mail Forwarding Payment Processor CREA Membership NAR

    Membership Registrar Hosted Website System Architecture
  40. 68.

    08

  41. 69.

    07

  42. 70.

    06

  43. 75.
  44. 76.
  45. 77.
  46. 78.
  47. 79.
  48. 81.
  49. 82.
  50. 83.
  51. 84.
  52. 86.
  53. 87.

    “How do I convince others to start earlier?” “How to

    do this cheaper?” “When should we start?”