Puneet Khanduri & Prashant Khanduri

Transcript

1. diffy catching bugs without writing tests Puneet Khanduri(@pzdk) Prashant Khanduri(@khandurip)
2. The Problem “I just refactored a critical part of my service. How do I know I didn’t break anything?” - Every Backend Developer diffy.ai
3. The Other Problem “They just refactored a critical part of my service. How do I know they didn’t break anything?” - Every Site Reliability Engineer diffy.ai
4. The Tipping Point “Does my new microservice behave the same as my monolith?” - Twitter in 2014 - Everyone trying to break a monolith diffy.ai
5. Tier#0 - Unit Tests 5 independent code paths => 5 tests = 100% coverage => 1 test = 20% coverage diffy.ai
6. Tier#1 - Component Tests 5^6 independent code paths => ~15k tests = 100% coverage => 1 test = ~0.06% coverage diffy.ai
7. Tier#2 - Integration Tests A request path has 4 services, 6 methods/service, 5 paths/method => 5^(4x6) independent code paths => ~60 Quadrillion tests = 100% coverage => 1 test = ~0% coverage diffy.ai
8. … emerging pattern ... super exponential cost of coverage diffy.ai
9. Diffy Approach - more coverage for free Free Test Inputs • Sample production traffic • Or whatever traffic source you prefer Free Assertions • Use “known good” versions of your code to generate assertions Note: You still provide the environment (mocked or staged or production) Currently, most customers use a combination of staged and production environments. diffy.ai
10. but … what about noise? • server generated timestamps • random number generators • downstream non-determinism • race conditions diffy.ai
11. Diffy Topology diffy secondary candidate primary raw differences non-deterministic noise filtered differences production cluster staging environment production environment diffy.ai
12. Demo diffy.ai
13. Automation Compare latest in master against last deploy to production • Automatically deploy master as candidate • Automatically deploy last prod tag as primary and secondary Reporting • Diffy emails report with highlighted critical endpoints and fields • Sample requests and responses available for further analysis diffy.ai
14. First version for personal use Generic Thrift version for Twitter HTTP support for Twitter Open Source! Community traction Diffy - the company - diffy.ai … the story so far diffy.ai
15. info@diffy.ai twitter: @diffyproject