High Performance Political Revolutions RailsConf 2017

Presentation video: https://confreaks.tv/videos/railsconf2017-high-performance-political-revolutions

Bernie Sanders has popularized small dollar donations as a mean to fund political campaigns, but having to process millions of small contributions brings technical challenges. The best example is from New Hampshire primary night, when Sanders told a national TV audience to go to berniesanders.com and donate $27. Web traffic peaked at 320,000 requests per minute and 2,600 credit card payments per minute.

ActBlue is a nonprofit tech organization that builds fundraising software for Democratic campaigns and committees, advocacy groups, and nonprofit organizations. Used by the Sanders campaign and 17,000 other organizations since 2004.

This presentation is about what we have learned in ActBlue building a high performance web application. The original version was written in 2004 and migrated to Rails in 2005.

Caching is one of the first steps to address performance, and we will discuss how we are using it. But we will cover other topics as well, like software architecture, which is sometimes overlooked, in spite of being more important.

For each solution, we will explain how it works, the complexities it introduces and how to address them.

We are keeping explanations as simple as possible and we are making few assumptions about the skill level of the audience. We start with basic concepts and build from there, so that even beginners can follow along and take away something useful.

Transcript

2. Political Revolutions • ActBlue is a nonprofit that builds fundraising software for: • Democratic campaigns and committees • advocacy groups • nonprofit organization • Bernie most popular, but 17,500 organizations since 2004 • 3,000 organizations Q1 2017 alone • using Rails since 2005
3. • process donations (payments) • technology + compliance • fundraise tools, stats, A/B tests • 3.8 million ActBlue Express users • $1.6 billion in 31 million contributions ($50 avg) • empower small-dollar donors
7. New Hampshire primary night February 9, 2016
8. Optimize What? • contribution form: • load contribution form • process a contribution
9. Process a Contribution
11. Scaling Challenge • approach • implementation • cost / solution because lots of small-dollar donations Presenting
12. Metrics
14. Metrics # statsd-ruby statsd = Statsd.new(‘chicago-1’) statsd.gauge(‘pending.auths’, pending_auths.count) statsd.timing(‘latency.auths’, Time.now - auth.created_at) • Graphite renders the graphs • collectd has plugins for PostgreSQL, Postfix, … • logs!
15. Metrics Multiple Servers
16. Multiple Servers
17. Multiple Servers # NGINX load balancer configuration http { upstream backend { server 192.168.1.101; server 192.168.1.102; server 192.168.1.103; } server { listen 80; location / { proxy_pass http://backend; }}}
18. Multiple Servers Cost • no file system • handle persistence: Rails sessions, Redis, … • increases required DB connections • LB is single point of failure
19. Metrics Multiple Servers Caching
20. Caching • general term, exists at multiple levels • highest value for the effort • Fastly CDN - huge! • stress tested with DDoS
21. Caching
22. CDN Dashboard
23. Caching Control • HTTP headers • Varnish Configuration Language (VCL) • API Cache-Control: max-age=400 Surrogate-Control: max-age=3600 Vary: Accept-Encoding Surrogate-Key: key1 key2 if (req.url ~ "^/videos") { set req.backend = F_VIDEO_LB; }
24. Caching Cost • expensive • complexity if you want flexibility • intricate SSL termination • contribution form filled by JS
25. Metrics Multiple Servers Caching Separation of Concerns
26. Separation of Concerns • main app in Rails • credit card tokenizer in Node • multiple DB instances • separation useful for compliance as well
27. Separation of Concerns Cost • complex developing and testing
28. Metrics Multiple Servers Caching Separation of Concerns Deferred Tasks
29. Deferred Tasks • CC processing = multiple calls to web services • fault isolation • increased reliability • deferred batch settlements - huge!
30. Queueing System execute authorizations execute settlements send email web server 1 web server 2 web server 3
31. Sidekiq class SettlementProcessor include Sidekiq::Worker def perform(id) Settlement.find(id).settle! end end SettlementProcessor.perform_async(settlement.id) ActiveJob ReceiptMailer.receipt(contribution).deliver_later
32. Deferred Tasks Cost • queuing system is unreliable • coordination • difficult to debug
33. Metrics Multiple Servers Caching Separation of Concerns Deferred Tasks Scalable Architecture
34. Scalable Architecture • mindset, i.e. most important • being fast + being scalable • most difficult to fix if you get it wrong • concurrency instead of central process • avoid premature optimization
35. Scalable Architecture Metrics Multiple Servers Caching Separation of Concerns Deferred Tasks
36. • interested in what we do? • come work with us • talk to me directly • slides on speakerdeck.com • stickers! Braulio Carreño @bcarreno