Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Reclaim your time: Automating Canary Analysis

Reclaim your time: Automating Canary Analysis

Automating canary analysis can help you deploy more safely and with more confidence. You can reclaim the time you would have spent verifying deploys or handling incidents. Learn how to use statistical methods to automate the analysis of your canaries.

Bd7faa0afa74f65533784141394adb89?s=128

megankanne

June 05, 2018
Tweet

Other Decks in Programming

Transcript

  1. Automating Canary Analysis @megankanne June 2018 Reclaim your time #autocanary

    • http://bit.ly/autocanary by cjaphotography on Flickr
  2. Photo by Paul Fisher in SOMA

  3. by quintanomedia on Flickr

  4. None
  5. Reclaim your time verification reliability safety confidence #autocanary

  6. “We found out when customers complained” “We caught it before

    it caused any issues” 1 3 “We got alerted about it” 2 Is my build “healthy”? #autocanary
  7. 1913 John Scott Haldane Wikipedia George McCaa, U.S. Bureau of

    Mines
  8. definition A partial deployment of new code by quimby on

    Flickr by midom on Flickr
  9. Portion of Production production canary #autocanary

  10. by gt_hawk63 on Flickr

  11. Canary Cluster production cluster canary cluster #autocanary

  12. Canary Cluster production cluster canary cluster proxy #autocanary

  13. Canary Cluster production cluster canary cluster proxy #autocanary

  14. Canary Cluster v2 production cluster canary cluster proxy prod build

    candidate #autocanary
  15. Visual Pattern Matching #autocanary

  16. Tap Compare response A response B == #autocanary

  17. Machine Learning by clintadair on Unsplash #autocanary

  18. #statistics #autocanary

  19. “Does This Shard Look Like The Others?” #autocanary

  20. horizontally offset
 Ex: memory used maxPercentile MAD Do metrics that

    group 
 Ex: success rates, latencies Don’t median absolute deviation From NIST.gov (A) (B) (C)
  21. toleranceFactor DBSCAN oscillate but don’t group
 Ex: memory used Do

    metrics that group 
 Ex: success rates, latencies Don’t density-based spatial clustering of applications with noise from hdbscan docs #autocanary
  22. minSimilarShardsPercent HDBSCAN oscillate but don’t group
 Ex: memory used Do

    metrics that group 
 Ex: success rates, latencies Don’t hierarchical dbscan from hdbscan docs #autocanary
  23. tolerance Mann-Whitney U Test confidenceLevel direction oscillate but don’t group


    Ex: memory used Do metrics that group 
 Ex: success rates, latencies Don’t Kayenta #autocanary
  24. #autocanary

  25. Success Story #autocanary

  26. #autocanary

  27. #autocanary

  28. #Tips Simplify Configuration #autocanary

  29. #Tips Choosing Metrics #autocanary

  30. #Tips User Trust #autocanary

  31. …but you told me the build was fine?! #autocanary

  32. Per Pull Request Other Uses #autocanary

  33. Examples Kayenta Twitter #autocanary

  34. Twitter For more info on workflows see https://www.youtube.com/watch?v=w36TOkuyAVc (un)block deploy

    get metrics analyze clean
  35. Future Work better statistics no config scale to all metrics

    per pull request #autocanary
  36. Reclaim Your Time With Automated Canary Analysis safety confidence verification

    reliability Statistics #autocanary
  37. Thanks Dylan Dignan Rohit Khansili Chris Regado Akshay Thejaswi Ratheesh

    Vijayan Rich Vincelette @megankanne #autocanary http://bit.ly/autocanary