Kill the Fail Whale for your API

Your API exposes your core infrastructure to API requests which are very unpredictable. Usually when serving normal web or mobile app traffic you have reasonable predictability because behind the demand are real users. For an API the consumers are apps, and they can instantly ramp up hundreds or thousands of requests per second. Imagine, just a single rogue app can bring down your entire service for all your consumers! In this session you will learn tools and techniques to proactively stress test your API and related infrastructure components. With better preparation you will succeed in killing the fail whale for your API.

Transcript

1. KILL THE FAIL WHALE FOR YOUR API @MarkCheshire @3scale #APISBERLIN 25 April, 2015
2. @MarkCheshire @3scale #apisberlin Unmatched Flexibility and Scale 3SCALE API MANAGEMENT DESIGNED FOR SPEED AND RELIABILITY 2
3. @MarkCheshire @3scale #apisberlin Unmatched Flexibility and Scale 3 Various – Data-as-a-Service, Business Service, etc Energy Telecom – Include Tier1 Telco Op. Infrastructure Retail / E-Commerce Media Health / Nutrition Transport GVT. / Administration Education Music Finance Charities 600 APIS POWERED BY 3SCALE
4. @MarkCheshire @3scale #apisberlin Unmatched Flexibility and Scale APIS ARE CRITICAL INFRASTRUCTURE 4
5. @MarkCheshire @3scale #apisberlin Unmatched Flexibility and Scale FAIL WHALE WAS A BIG DEAL FOR TWITTER AVOID THE SAME FATE FOR YOUR API 5
6. @MarkCheshire @3scale #apisberlin Unmatched Flexibility and Scale UNDERSTAND PERFORMANCE EXPECTATIONS FOR YOUR API • Average throughput in requests per second • Peak throughput (what is the most traffic that you get over a certain period?) • Throughput distribution – by API endpoint (do you have any endpoint that gets substantially more traffic than any others?) – by users (a few generate most traffic or is it more evenly distributed) 6
7. @MarkCheshire @3scale #apisberlin Unmatched Flexibility and Scale D LADDER TO LOAD TESTING NIRVANA 7 Dumb repetitive load generation Simulated traffic patterns Real log traffic Live traffic
8. @MarkCheshire @3scale #apisberlin Unmatched Flexibility and Scale Tool Ideal use case JMeter functional testing of complex user flows in a web application wrk find what is the maximum load your API can handle Vegeta test how an API behaves at a steady rate of X reqs/s Loader.io SaaS tool. Similar in target to wrk but with zero setup and nice reports Blazemeter SaaS tool, based on JMeter (allows you to upload JMeter test plans and run them from the cloud) LOAD TESTING TOOLS COMPARED
9. @MarkCheshire @3scale #apisberlin Unmatched Flexibility and Scale L FIRST PASS – ESTABLISH THE BASELINE WHAT IS THE MAX LOAD THE API CAN SUPPORT? 9 Loader.io 30 sec test 10K req/sec <40ms response time Only occasional CPU peaks
10. @MarkCheshire @3scale #apisberlin Unmatched Flexibility and Scale WRK TO FIND THE REAL MAXIMUM THROUGHPUT 10 60 sec test 16K req/sec Sustained ~1.4B req/day!
11. @MarkCheshire @3scale #apisberlin Unmatched Flexibility and Scale THE TEST CONFIGURATION 11
12. @MarkCheshire @3scale #apisberlin Unmatched Flexibility and Scale L UH OH! AFTER ADDING THE GATEWAY WE MEET THE FAIL WHALE 12 Error code 500s Timeouts Test terminates
13. @MarkCheshire @3scale #apisberlin Unmatched Flexibility and Scale TYPICAL BOTTLENECK MAXIMIZE THE NUMBER OF AVAILABLE SIMULTANEOUS CONNECTIONS  Increase range of ports allowed for TCP traffic. Default: [1,024 - 5,000] (net.ipv4.ip_local_port_range)  Increase default number of available file descriptors per process (ulimit)  Make connections available for reuse as quickly as possible by reducing the default duration of the TIME_WAIT state (net.ipv4.tcp_fin_timeout) If your API server has concurrency problems, having Nginx (or an API Gateway) in front and enabling keepalives can help reduce the number of connections: http://nginx.com/blog/http-keepalives-and-web-performance/ 13
14. @MarkCheshire @3scale #apisberlin Unmatched Flexibility and Scale L REFINE AND ITERATE UNTIL YOU REACH THE DESIRED PERFORMANCE TARGET 14 10K req/sec <40ms response time
15. @MarkCheshire @3scale #apisberlin Unmatched Flexibility and Scale L ALL THE BENEFITS OF AN API GATEWAY TO PROTECT YOUR API WITH A MINIMAL PERFORMANCE HIT 15
16. @MarkCheshire @3scale #apisberlin Unmatched Flexibility and Scale BLOG POST COMING SOON AT: WWW.3SCALE.NET 16
17. @MarkCheshire @3scale #apisberlin Unmatched Flexibility and Scale PALE ALE IS BETTER THAN THE FAIL WHALE 17
18. @MarkCheshire @3scale #apisberlin Unmatched Flexibility and Scale CREDITS • Fail Whale aka “Lifting a Dreamer” by Yiying Lu http://www.whatisfailwhale.info/ • http://pixshark.com/people-walking-up-stairs-clipart.htm • Fail Whale Pale Ale label by Brian Cook, Manchester, Connecticut, USA http://www.whatisfailwhale.info/ 18 • http://gwan.com/en_apachebench_httperf.html • https://www.usenix.org/legacy/publications/library/proceedings/usits97/full_papers/banga/banga.pdf • http://www.hpl.hp.com/research/linux/httperf/wisp98/httperf.pdf • http://making.fiftythree.com/load-testing-an-unexpected-journey • https://engineering.gosquared.com/optimising-nginx-node-js-and-networking-for-heavy-workloads • https://support.3scale.net/overview#how-it-works • http://www.3scale.net/ RESOURCES