Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Puneet Khanduri & Prashant Khanduri

16b6c87229eaf58768d25ed7b2bbbf52?s=47 CodeFest
June 25, 2018

Puneet Khanduri & Prashant Khanduri



June 25, 2018


  1. diffy catching bugs without writing tests Puneet Khanduri(@pzdk) Prashant Khanduri(@khandurip)

  2. The Problem “I just refactored a critical part of my

    service. How do I know I didn’t break anything?” - Every Backend Developer diffy.ai
  3. The Other Problem “They just refactored a critical part of

    my service. How do I know they didn’t break anything?” - Every Site Reliability Engineer diffy.ai
  4. The Tipping Point “Does my new microservice behave the same

    as my monolith?” - Twitter in 2014 - Everyone trying to break a monolith diffy.ai
  5. Tier#0 - Unit Tests 5 independent code paths => 5

    tests = 100% coverage => 1 test = 20% coverage diffy.ai
  6. Tier#1 - Component Tests 5^6 independent code paths => ~15k

    tests = 100% coverage => 1 test = ~0.06% coverage diffy.ai
  7. Tier#2 - Integration Tests A request path has 4 services,

    6 methods/service, 5 paths/method => 5^(4x6) independent code paths => ~60 Quadrillion tests = 100% coverage => 1 test = ~0% coverage diffy.ai
  8. … emerging pattern ... super exponential cost of coverage diffy.ai

  9. Diffy Approach - more coverage for free Free Test Inputs

    • Sample production traffic • Or whatever traffic source you prefer Free Assertions • Use “known good” versions of your code to generate assertions Note: You still provide the environment (mocked or staged or production) Currently, most customers use a combination of staged and production environments. diffy.ai
  10. but … what about noise? • server generated timestamps •

    random number generators • downstream non-determinism • race conditions diffy.ai
  11. Diffy Topology diffy secondary candidate primary raw differences non-deterministic noise

    filtered differences production cluster staging environment production environment diffy.ai
  12. Demo diffy.ai

  13. Automation Compare latest in master against last deploy to production

    • Automatically deploy master as candidate • Automatically deploy last prod tag as primary and secondary Reporting • Diffy emails report with highlighted critical endpoints and fields • Sample requests and responses available for further analysis diffy.ai
  14. First version for personal use Generic Thrift version for Twitter

    HTTP support for Twitter Open Source! Community traction Diffy - the company - diffy.ai … the story so far diffy.ai
  15. info@diffy.ai twitter: @diffyproject