Testing at Stream Scale

Testing at Stream Scale

Testing code is hard. Testing code that's processing 40 billion messages per day is harder. In this talk, I speak a bit about how we do that on MailChimp's Data Systems Team.

6aaa707270e134bb72a4c2852f088e22?s=128

Matt Farmer

October 24, 2017
Tweet

Transcript

  1. Testing at Stream Scale Matt Farmer | All Things Open

    2017
  2. About Me

  3. About You

  4. The MailChimp Data Pipeline

  5. The MailChimp Data Pipeline Σ MailChimp MC Data Pipeline Kafka

    Data Science
  6. The MailChimp Data Pipeline - We care about the order

    of data. - We cannot avoid binary data. (MySQL Binlogs, Thrift) - Our ideal performance is to deliver updates within a minute
  7. The time to build a better pipeline

  8. None
  9. None
  10. None
  11. The time to build a better pipeline

  12. 1. Articulate and communicate
 a testing strategy.

  13. Our Testing Guidelines Unit tests should test complex, non obvious

    behavior and exercise various failure conditions. Should be true unit tests.
  14. Our Testing Guidelines Integration tests need to be easier to

    build, even easier to keep up to date, and stick to testing the "happy path." Test cases should be easy to generate.
  15. Our Testing Guidelines Continuous end-to-end testing of accuracy and performance.

    Reporting and alerting on that testing over time.
  16. 1. Articulate and communicate
 a testing strategy.

  17. 2. Remove barriers to debugging and testing

  18. Prologue Tools

  19. 2. Remove barriers to debugging and testing

  20. 3. Pursue Greatness in Staging

  21. Attributes of a "Great" Streaming Staging Environment - Sourced from

    the same data as production. - Owned by the team maintaining the streaming application. - Sees a significant % of production data. - Stable and alerts when things break.
  22. 3. Pursue Greatness in Staging

  23. 4. Continuous End-to-End Testing

  24. Kafka Detective

  25. Kafka Detective Kafka Detective is a highly configurable utility for

    reporting on differences between two Kafka topics that should be semantically identical. This enables true, continuous end-to- end testing of a Kafka-based application.
  26. Kafka Detective Kafka Detective is a highly configurable utility for

    reporting on differences between two Kafka topics that should be semantically identical. This enables true, continuous end-to- end testing of a Kafka-based application.
  27. Staging Pipeline Production Pipeline Kafka Detective Σ Kafka Detective

  28. Kafka Detective Catches Real Issues! A configuration issue that caused

    our user data to become ordered incorrectly. (e.g. Inserts after Updates for the same DB record.)
  29. Kafka Detective Catches Real Issues! An issue constructing our message

    keys that broke MailChimp Pro.
  30. Kafka Detective Features - Pluggable semantics around matching messages -

    Pluggable reporters (Graphite and Kafka reporter included) - Capable of transforming (e.g. deserializing) mismatches before reporting - Capable of back pressuring when things appear out-of-sync - Capable of handling high volume with low memory
  31. Kafka Detective Roadmap - Better support things being unhealthy when

    Detective starts - Better support for use-cases that repartition data - Better support for use-cases where keys are not globally unique - Look into re-thinking what "success" means
  32. 4. Continuous End-to-End Testing

  33. TL;DL 1. Articulate and communicate a testing strategy. 2. Remove

    barriers to debugging and testing. 3. Pursue greatness in staging. 4. Do continuous end-to-end testing.
  34. We've open sourced Kafka Detective!

  35. detective.frmr.me

  36. Thank you! Twitter/GitHub: farmdawgnation farmer@mailchimp.com