Going to Eleven

Going to Eleven

100,000 simultaneous users. 38 days. 9 systems. One launch. Ready?

In this retelling of a real-life product "land rush", you'll learn about web load testing of a site that went from zero users to all the users in one day. This talk will cover open source tools, the AWS cloud, dependency isolation, background jobs, and profiling. Hear about some hard lessons learned while negotiating with external partners, OS tuning, and avoiding being targeted as a botnet. Walk away with some idea how to get started load testing and how to deal with problems in the architectures as they come up.

6b6afbaea3bf1de98975dedc5cd083c1?s=128

stevenjackson

March 08, 2017
Tweet

Transcript

  1. 5.

    Goals • How to start? • Why Locust? • How

    to deal with problems in the architecture? • Make an argument for starting load testing early • Tell a compelling story
  2. 6.

    • .realtor starts - Feb 2014 • Load Test start

    - 15 Sep 2014 • Soft Launch - 20 Oct 2014 • Launch - 23 Oct 2014 Timeline
  3. 9.

    System Architecture Send Emails DNS Mail Forwarding Payment Processor CREA

    Membership NAR Membership Registrar Hosted Website
  4. 11.

    NAR Membership Send Emails DNS Mail Forwarding Payment Processor CREA

    Membership Registrar Hosted Website Prioritizing Dependencies
  5. 14.

    Why Locust? • Could interact with Rails CSRF tokens •

    Could execute end-to-end user interaction with sessions and cookies • Expands to multiple slaves to increase load capacity • Allows for distributed user actions based on percentages
  6. 15.
  7. 16.
  8. 17.
  9. 18.
  10. 19.
  11. 21.

    Methodology • Start Small (1000 simultaneous users) • Gather data

    • Grow infrastructure as we hit bottlenecks
  12. 26.
  13. 27.
  14. 34.

    • Compression = YES, please • Serve static assets directly

    • Reverse proxy (proxy_pass) the rest to unicorn
  15. 35.

    worker_processes 32 worker_rlimit_nofile 65536 use epoll worker_connections 65536 upstream realtor_unicorn

    { server unix:/tmp/realtor.sock fail_timeout=0; } keepalive_requests 0 proxy_read_timeout proxy_write_timeout client_max_body_size nginx.conf sites-available/sitename Take a look at these
  16. 37.

    rails_user hard nofile 65536 /etc/security/limits.conf # Increase size of file

    handles and inode cache fs.file-max = 100000 /etc/sysctl.conf ulimits
  17. 39.

    What did I learn? • Don’t wait so long to

    start load testing • The conversations drive new requirements • This stuff is hard to figure out under pressure • Too late to do big changes confidently
  18. 41.
  19. 42.
  20. 43.
  21. 44.
  22. 45.
  23. 46.
  24. 47.
  25. 48.
  26. 49.
  27. 52.
  28. 54.
  29. 56.

    5 1

  30. 57.
  31. 58.
  32. 59.

    System Architecture Send Emails DNS Mail Forwarding Payment Processor CREA

    Membership NAR Membership Registrar Hosted Website
  33. 60.

    8 0

  34. 61.

    7 0

  35. 62.

    6 0

  36. 67.
  37. 68.
  38. 69.
  39. 70.
  40. 71.
  41. 73.
  42. 74.
  43. 75.
  44. 76.
  45. 78.
  46. 79.

    “How do I convince others to start earlier?” “How to

    do this cheaper?” “When should we start?”