Testing at Stream Scale

Testing code is hard. Testing code that's processing 40 billion messages per day is harder. In this talk, I speak a bit about how we do that on MailChimp's Data Systems Team.

Transcript

1. Testing at Stream Scale Matt Farmer | All Things Open 2017
2. About Me
3. About You
4. The MailChimp Data Pipeline
5. The MailChimp Data Pipeline Σ MailChimp MC Data Pipeline Kafka Data Science
6. The MailChimp Data Pipeline - We care about the order of data. - We cannot avoid binary data. (MySQL Binlogs, Thrift) - Our ideal performance is to deliver updates within a minute
7. The time to build a better pipeline
11. The time to build a better pipeline
12. 1. Articulate and communicate
 a testing strategy.
13. Our Testing Guidelines Unit tests should test complex, non obvious behavior and exercise various failure conditions. Should be true unit tests.
14. Our Testing Guidelines Integration tests need to be easier to build, even easier to keep up to date, and stick to testing the "happy path." Test cases should be easy to generate.
15. Our Testing Guidelines Continuous end-to-end testing of accuracy and performance. Reporting and alerting on that testing over time.
16. 1. Articulate and communicate
 a testing strategy.
17. 2. Remove barriers to debugging and testing
18. Prologue Tools
19. 2. Remove barriers to debugging and testing
20. 3. Pursue Greatness in Staging
21. Attributes of a "Great" Streaming Staging Environment - Sourced from the same data as production. - Owned by the team maintaining the streaming application. - Sees a significant % of production data. - Stable and alerts when things break.
22. 3. Pursue Greatness in Staging
23. 4. Continuous End-to-End Testing
24. Kafka Detective
25. Kafka Detective Kafka Detective is a highly configurable utility for reporting on differences between two Kafka topics that should be semantically identical. This enables true, continuous end-to- end testing of a Kafka-based application.
26. Kafka Detective Kafka Detective is a highly configurable utility for reporting on differences between two Kafka topics that should be semantically identical. This enables true, continuous end-to- end testing of a Kafka-based application.
27. Staging Pipeline Production Pipeline Kafka Detective Σ Kafka Detective
28. Kafka Detective Catches Real Issues! A configuration issue that caused our user data to become ordered incorrectly. (e.g. Inserts after Updates for the same DB record.)
29. Kafka Detective Catches Real Issues! An issue constructing our message keys that broke MailChimp Pro.
30. Kafka Detective Features - Pluggable semantics around matching messages - Pluggable reporters (Graphite and Kafka reporter included) - Capable of transforming (e.g. deserializing) mismatches before reporting - Capable of back pressuring when things appear out-of-sync - Capable of handling high volume with low memory
31. Kafka Detective Roadmap - Better support things being unhealthy when Detective starts - Better support for use-cases that repartition data - Better support for use-cases where keys are not globally unique - Look into re-thinking what "success" means
32. 4. Continuous End-to-End Testing
33. TL;DL 1. Articulate and communicate a testing strategy. 2. Remove barriers to debugging and testing. 3. Pursue greatness in staging. 4. Do continuous end-to-end testing.
34. We've open sourced Kafka Detective!
35. detective.frmr.me
36. Thank you! Twitter/GitHub: farmdawgnation farmer@mailchimp.com