Ryan Kelly: Testing for Graceful Failure with Vaurien and Marteau

= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =
Ryan Kelly:
Testing for Graceful Failure with Vaurien and Marteau
= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =
@ Kiwi PyCon 2013 - Sunday, 08 Sep 2013 - Track 2
http://nz.pycon.org/

**Audience level**

Intermediate

**Description**

This talk shows how the Mozilla Services team test failure scenarios in their web services with two python-based tools: Marteau, a web-based UI for easily running load tests, and Vaurien, a misbehaving TCP proxy that can simulate various backend failures. Used together, these tools help ensure a service will not only scale up to meet demand, but will fail gracefully if it reaches breaking point.

**Abstract**

So you've built an awesome webapp, put it through its paces, and assured yourself that it does what it's supposed to do. Great! Now how does it behave when things start to go wrong?

This talk will demonstrate how the Mozilla Services team tests for failure scenarios in our web services, focusing on two key python-based tools: Marteau, a web-based frontend for easily running load-tests and analyzing the results, and Vaurien, a misbehaving TCP proxy that can simulate a variety of backend failure modes.

Used together, these tools can help ensure that a web service will not only scale up to meet its expected demand, but will fail gracefully when it finally reaches breaking point.

The talk will cover:

* Real-life examples of bugs that only show up when your app is under load; bugs that can turn a brief partial outage into a cascading whole-system failure.
* The basics of writing a load-testing suite for your app.
* How to set up Marteau for easy on-demand load testing.
* How to use Vaurien to simulate various kinds of backend failure, such as an overloaded database, misconfigured DNS, or a suddenly-disappearing job queue.
* Some tips for systematically applying these tools to your own setup.

**YouTube**

http://www.youtube.com/watch?v=WSdyU5s-SMI

Transcript

6. You have a web application You are confident that it works You have a solid deployment and monitoring setup
7. You have a web application You are confident that it works You have a solid deployment and monitoring setup Now...what happens when things start to break?
8. Mozilla Services Firefox Sync Server FirefoxOS Marketplace Push Notification Service More coming... Organic process and tooling This will be a demo, not a tutorial
9. gunicorn MySQL
10. gunicorn MySQL MySQL MySQL
11. gunicorn MySQL MySQL MySQL
12. gunicorn MySQL MySQL MySQL W W W W W
13. gunicorn MySQL MySQL MySQL W W W W W
14. gunicorn MySQL MySQL MySQL W W W W W
15. gunicorn MySQL MySQL MySQL W W W W W
16. gunicorn MySQL MySQL MySQL W W W W W
17. gunicorn MySQL MySQL MySQL W W W W W
20. Deploy it 1. Stress it 2. Break it 3.
21. Deploy it 1. Stress it 2. Break it 3. ??? 4.
22. Deploy it 1. Stress it 2. Break it 3. ??? 4. Profit 5.
27. Our shiny new loadtesting tool Built on existing best-of-breed libraries Designed for distributed real-time operation Highly pluggable
29. runner server
30. runner server agent broker
31. runner server agent broker agent
32. runner server agent broker agent agent
33. runner server agent broker agent agent
34. (opens in new tab...)
35. (opens in new tab)
36. Real-time Graphs and Reports Runners in other languages Integration with server-side monitoring
37. Real-time Graphs and Reports Runners in other languages Integration with server-side monitoring ...your needs here?
40. A misbehaving TCP proxy: Behaviours: delay, hang, error, ... Protocols: raw tcp, http, memcached, mysql, ... Easily extensible for your needs French for "rapscallion"
42. Combine with loadtests to check for cascading failures Integrate into functional tests, for specific error sequences (example opens in new tab...)
44. How does your app interact with the outside world? What happens when the outside world misbehaves? How can you simulate that under controlled conditions? Come hack with us! "Continuous Stress Testing"
46. Tarek Ziade Alexis Metaireau James Bonacci
47. ryan@rfk.id.au http://vaurien.readthedocs.org http://loads.readthedocs.org