Taming the Beast: Automated Testing for Complex Data Pipelines

When you are faced with a system that processes massive datasets with complex data pipelines involving machine learning, how do you test effectively? When your tests results are less “pass” and “fail”, and more “sort of” and “not really”, how do you automate testing?

Trish Khoo draws upon her experience in testing complex data systems to demonstrate proven strategies for testing in this field. Her experience working on ultra-large-scale systems at Google in Mountain View, California shaped her technical approach to testing which she applies in her work as a consultant today.

Transcript

1. TRISH KHOO | TRANSPARO | @HOGFISH TAMING THE BEAST Automated testing for complex data pipelines
2. ABOUT ME Previously: Technical Lead / Manager Google (Geo) California, USA Now: Consultant / Trainer Transparo Brisbane, Australia
3. Netflix ~500 BILLION Events per day ~8 MILLION Events per second during peak hours SEVERAL HUNDRED Event streams
4. COMPLEX DATA PIPELINES
5. Deterministic tests Functional testing E2E testing is simple A SIMPLE SYSTEM Not always deterministic Data driven testing E2E testing hard to scale A COMPLEX DATA PIPELINE
6. TEST DESIGN Data inputs need to be designed for good test coverage. VALIDATION Data outputs need useful analysis. SCALABILITY The entire approach needs to scale. TESTABILITY Integration tests need the system to be built in testable parts.
7. TURN BIG PROBLEMS INTO SMALL PROBLEMS SMALLER TESTS ARE HIGH VALUE
8. ONE TASK ONE TASK ONE TASK TESTABILITY
9. Risk-based Good coverage Regression Exploratory TEST DESIGN
10. VALIDATION PASS What does "good" look like? FAIL What does "bad" look like? INTERESTING A warning - requires manual investigation.
11. SCALABILITY Future-proof your solution PRODUCTION DATA Statistical data sampling AUTOMATE Invest in infrastructure SPEED Do tasks in parallel
12. TEST DESIGN Data inputs need to be designed for good test coverage. VALIDATION Data outputs need useful analysis. SCALABILITY The entire approach needs to scale. TESTABILITY Integration tests need the system to be built in testable parts.
13. TOO MUCH? START SIMPLE. 1. TESTABILITY 2. RISK ANALYSIS 3. REGRESSION TESTS
14. transpa.ro/data-pipelines trish@transpa.ro www.transpa.ro TALK INFO HIRE ME