Reclaim your time: Automating Canary Analysis

Automating canary analysis can help you deploy more safely and with more confidence. You can reclaim the time you would have spent verifying deploys or handling incidents. Learn how to use statistical methods to automate the analysis of your canaries.

Transcript

1. Automating Canary Analysis @megankanne June 2018 Reclaim your time #autocanary • http://bit.ly/autocanary by cjaphotography on Flickr
2. Photo by Paul Fisher in SOMA
3. by quintanomedia on Flickr
5. Reclaim your time verification reliability safety confidence #autocanary
6. “We found out when customers complained” “We caught it before it caused any issues” 1 3 “We got alerted about it” 2 Is my build “healthy”? #autocanary
7. 1913 John Scott Haldane Wikipedia George McCaa, U.S. Bureau of Mines
8. definition A partial deployment of new code by quimby on Flickr by midom on Flickr
9. Portion of Production production canary #autocanary
10. by gt_hawk63 on Flickr
11. Canary Cluster production cluster canary cluster #autocanary
12. Canary Cluster production cluster canary cluster proxy #autocanary
13. Canary Cluster production cluster canary cluster proxy #autocanary
14. Canary Cluster v2 production cluster canary cluster proxy prod build candidate #autocanary
15. Visual Pattern Matching #autocanary
16. Tap Compare response A response B == #autocanary
17. Machine Learning by clintadair on Unsplash #autocanary
18. #statistics #autocanary
19. “Does This Shard Look Like The Others?” #autocanary
20. horizontally offset
 Ex: memory used maxPercentile MAD Do metrics that group 
 Ex: success rates, latencies Don’t median absolute deviation From NIST.gov (A) (B) (C)
21. toleranceFactor DBSCAN oscillate but don’t group
 Ex: memory used Do metrics that group 
 Ex: success rates, latencies Don’t density-based spatial clustering of applications with noise from hdbscan docs #autocanary
22. minSimilarShardsPercent HDBSCAN oscillate but don’t group
 Ex: memory used Do metrics that group 
 Ex: success rates, latencies Don’t hierarchical dbscan from hdbscan docs #autocanary
23. tolerance Mann-Whitney U Test confidenceLevel direction oscillate but don’t group
 Ex: memory used Do metrics that group 
 Ex: success rates, latencies Don’t Kayenta #autocanary
24. #autocanary
25. Success Story #autocanary
26. #autocanary
27. #autocanary
28. #Tips Simplify Configuration #autocanary
29. #Tips Choosing Metrics #autocanary
30. #Tips User Trust #autocanary
31. …but you told me the build was fine?! #autocanary
32. Per Pull Request Other Uses #autocanary
33. Examples Kayenta Twitter #autocanary
34. Twitter For more info on workflows see https://www.youtube.com/watch?v=w36TOkuyAVc (un)block deploy get metrics analyze clean
35. Future Work better statistics no config scale to all metrics per pull request #autocanary
36. Reclaim Your Time With Automated Canary Analysis safety confidence verification reliability Statistics #autocanary
37. Thanks Dylan Dignan Rohit Khansili Chris Regado Akshay Thejaswi Ratheesh Vijayan Rich Vincelette @megankanne #autocanary http://bit.ly/autocanary