Builing an HTTP Request Router with NGINX and Lua

Transcript

1. Building an HTTP Request Router with NGINX and Lua Scott Francis - Shopify 23/09/2015
3. Quick Stats 300M U N I Q U E V I S I T S / M O N T H 70ms AV E R AG E R E S P O N S E T I M E 10,000+ M A X C H E C KO U T S P E R M I N U T E 3 DATA C E N T E R S
4. Architecture • Multi-tenant Rails monolith • Sharded MySQL persistent store • Redis and memcached • ElasticSearch • NGINX
5. Architecture
6. Multiple Data Centers • Planned to bring second data center online in early 2015 (Active-Passive). • Downtime costs our merchants money. • How would we perform the DC failover (success-over)? • How could we test the second data center before flipping the switch?
7. Multiple Data Centers
8. 8 DEVELOPING SORTING HAT
9. Sorting Hat Architecture
10. Infrastructure as an Application • Follow Good Software Development Practices • Testing • Continuous Delivery
11. +
12. Middlewares • Like Rack, but for Lua and NGINX – Expose a Lua module that exports two functions: call, and init_worker. • Runs in an NGINX request phase — access_by_lua, rewrite_by_lua, content_by_lua, etc. • Configured via Chef – NGINX configuration file is generated by Chef. • Example Middlewares — request ID generation, Kafka logging, ping, HTTPDB
13. Middleware Configuration { :ping => { :phase => :rewrite }, :request_id => { :phase => :rewrite }, :sorting_hat => { :phase => :rewrite, :dict => { :shop_id_cache => "16M", :pod_id_cache => "16M", :oor_pods => "1M", :pod_to_upstream => "1M", }, }, :monitor => { }, }
14. Ping Middleware
15. 15 RESILIENCY AN D TESTING
16. Integration Testing
17. Circuit Breaker • Don’t access resources when they are down or slow • Have a fallback path — use a cached value, return an error, etc.
18. Toxiproxy • A framework for simulating network conditions in development and test • Platform agnostic — Client libraries for Ruby, Go. .NET, Node, PHP • Toxics manipulate network conditions – latency, down connection, bandwidth, and more
19. A Simple Toxiproxy Test
20. 20 DEPLOYI NG TO PRODUCTION
21. Deployment with ShipIt
22. Monitoring and Alerting
23. Data Centre Move Lead Up • Incremental Rollout — Deployed Sorting Hat into production, though it was effectively doing nothing. • Simple Deployment — One button deploys and rollbacks. • Weekly Practice — Use Sorting Hat to move a test shard back and forth.
24. The Big Day
25. 25 FIN AL THOUGHTS
26. Take Aways • Follow good software development practices when building infrastructure • Testing and resiliency are crucial • Iterate fast and often
27. The Future • POP Servers • MOAR Middlewares — We’re working on more middlewares. • Open source — Contributing our work back to the community.
28. Thanks!
29. References • http://wiki.nginx.org/HttpLuaModule • https://github.com/openresty/test-nginx • https://github.com/Shopify/toxiproxy • https://www.shopify.com/technology/16906928-building-and-testing- resilient-ruby-on-rails-applications • https://github.com/Shopify/shipit-engine