CI/CD: More than Just Code

Presented at Think About! 2019

Transcript

1. Think About! 2019 Ryn Daniels @rynchantress they/them CI/CD: More than Just Code
2. RESILIENCE @rynchantress Think About! 2019
3. RESILIENCE IS NOT ROBUSTNESS @rynchantress Think About! 2019
4. Computer systems can be robust against a finite set of known problems. Only humans can be resilient against unknown problems. @rynchantress Think About! 2019
5. RESILIENCE @rynchantress Think About! 2019 - not of technical systems - of processes - of human systems
6. RESILIENCE @rynchantress Think About! 2019 OF OUTCOMES
7. Devops is a cultural and technical movement that enables resilience of outcomes. @rynchantress Think About! 2019
8. @rynchantress Think About! 2019 2 STORIES ABOUT RESILIENCE OF OUTCOMES
9. PUTTING THE DEV IN DEVOPS @rynchantress Think About! 2019 Or: Improving resilience of outcomes for (ops-related) code
10. (Infrastructure provisioning tooling at Etsy*) PROJECT INDIGO @rynchantress Think About! 2019
11. @rynchantress Think About! 2019
12. ConCnuous I...nsecurity about whether or not these changes were going to break something horribly. ConCnuous Delivery... of things that were broken in producCon that I promise worked on my machine, really. Yes, we have CI/CD! @rynchantress Think About! 2019
13. @rynchantress Think About! 2019 HERE BE DRAGONS
14. - Most commonly used workflow was completely untested - No versioning of any components - Lack of separate tesCng environment - Manual, error-prone deploy process - No tests (or even testable code) Project Indigo: Labeling the Dragons @rynchantress Think About! 2019
15. - Most commonly used workflow was completely untested - No versioning of any components - Lack of separate tesCng environment - Manual, error-prone deploy process - No tests (or even testable code) Project Indigo: Labeling the Dragons @rynchantress Think About! 2019
16. Untested Workflow: Unattended Mode @rynchantress Think About! 2019 - AOended mode: An engineer in the office runs Indigo by hand - UnaOended mode: The data center team boots up a new machine and it automaCcally netboots to Indigo
17. @rynchantress Think About! 2019
18. @rynchantress Think About! 2019
19. @rynchantress Think About! 2019
20. - Most commonly used workflow was completely untested - No versioning of any components - Lack of separate tesCng environment - Manual, error-prone deploy process - No tests (or even testable code) Project Indigo: Labeling the Dragons @rynchantress Think About! 2019
21. @rynchantress Think About! 2019
22. @rynchantress Think About! 2019
23. @rynchantress Think About! 2019
24. - Most commonly used workflow was completely untested - No versioning of any components - Lack of separate tes<ng environment - Manual, error-prone deploy process - No tests (or even testable code) Project Indigo: Labeling the Dragons @rynchantress Think About! 2019
25. @rynchantress Think About! 2019
26. @rynchantress Think About! 2019
27. @rynchantress Think About! 2019
28. - Most commonly used workflow was completely untested - No versioning of any components - Lack of separate tesCng environment - Manual, error-prone deploy process - No tests (or even testable code) Project Indigo: Labeling the Dragons @rynchantress Think About! 2019
29. AUTOMATE! @rynchantress Think About! 2019
30. - Most commonly used workflow was completely untested - No versioning of any components - Lack of separate tesCng environment - Manual, error-prone deploy process - No tests (or even testable code) Project Indigo: Labeling the Dragons @rynchantress Think About! 2019
31. @rynchantress Think About! 2019
32. @rynchantress Think About! 2019
33. @rynchantress Think About! 2019
34. - End-to-end tesCng of both workflows - Versioning of payload and API - Ability to create a tesCng environment for various components - Automated deploy process with Deployinator - Testable code with >0 unit tests Project Indigo: Slaying the Dragons @rynchantress Think About! 2019
35. - Reducing the set of unknowns by running tests against known problems - Decreasing surprises with smaller changes - Surfacing problems more quickly and automaCcally for quicker response CI/CD: Resilience of Outcomes for Code Changes @rynchantress Think About! 2019
36. Just one building block of resilience CI/CD: @rynchantress Think About! 2019
37. THE APACHE SNAFU @rynchantress Think About! 2019 Or: Improving resilience of outcomes for human incident response
38. @rynchantress Think About! 2019
39. @rynchantress Think About! 2019 WHOOPS.
40. - SomeCmes automaCon goes awry - SomeCmes broken automaCon saves you - SomeCmes an incident breaks all your incident monitoring/response tools - Always give people weird sweaters when they break things Lessons Learned, Part I @rynchantress Think About! 2019
41. CULTURE @rynchantress Think About! 2019
42. CULTURE @rynchantress Think About! 2019 Collec<on of norms, processes, values, and structures in a group or environment
43. - Understand when processes don't work - Encourage public communicaCon - IncenCvize people to help each other out - Budget in slack - Maintain psychological safety Lessons Learned, Part II @rynchantress Think About! 2019
44. Processes and documentation @rynchantress Think About! 2019 - Which processes are explicit versus implicit? - Do you know when people are working around processes that frustrate them? - How do processes get changed?
45. Communication and social scripts @rynchantress Think About! 2019 - What channels do people use for communicaCon? - Does communicaCon default to public or private? - What are the standards for how people interact with each other?
46. Motivation and incentive structures @rynchantress Think About! 2019 - What behaviors are explicitly encouraged in your organizaCon? - What is part of your career framework? - What behaviors are implicitly rewarded? - Are incenCves consistent across the org?
47. Prioritization and budgets @rynchantress Think About! 2019 - Who is available to help respond to incidents? - Are incident response and remediaCon items considered during planning and scheduling? - How much slack is in your schedules? - What happens when schedules slip?
48. Psychological safety @rynchantress Think About! 2019 - Do people feel comfortable asking for help? - Do people admit when they don't know something? - How are mistakes treated within your organizaCon?
49. - Processes and documentaCon - Social scripts - IncenCve structures - PrioriCes and budgets - Psychological safety Cultural Building Blocks @rynchantress Think About! 2019
50. @rynchantress Think About! 2019 Multiple ways to build resilience of outcomes
51. Technical and cultural building blocks @rynchantress Think About! 2019
52. You aren't just shipping code. You're also shipping everything that enables you to ship code. @rynchantress Think About! 2019
53. @rynchantress Think About! 2019 Devops: building blocks of resilience
54. THANK YOU! @rynchantress Think About! 2019