Wildfires, Firefighters and Sustainability Learnings from Mitigating Kubernetes Fires in the Community

Transcript

2. Nabarun Pal & Madhav Jivrajani, VMware Wildfires, Firefighters and Sustainability Learnings from Mitigating Kubernetes Fires in the Community
3. Code of Conduct Remember the Golden Rule: Treat others as you would want to be treated - with kindness and respect Scan the QR code to access and review the CNCF Code of Conduct:
4. Virtual Audience Closed Captioning Closed captioning for the virtual audience is available during each session through Wordly. The Wordly functionality can be found under the “Translations” tab on the session page. Wordly will default to English. If another language is needed, simply click the dropdown at the bottom of the “Translations” tab and choose from one of 26+ languages available so you don’t miss a beat from our presenters. *Note: Closed captioning is ONLY available during the scheduled live sessions and will not be available for the recordings on-demand within the virtual conference platform.
5. Who Are We? Madhav Jivrajani @MadhavJivrajani Kubernetes SIG ContribEx Technical Lead Nabarun Pal @theonlynabarun Kubernetes Steering Committee / SIG ContribEx Chair
6. Before We Start… @MadhavJivrajani & @theonlynabarun
7. registry.k8s.io is GA!🎉 🚨❄k8s.gcr.io is frozen❄🚨 More info on https://k8s.io/image-registry-redirect Also see: k8s.gcr.io Redirect to registry.k8s.io - What You Need to Know @MadhavJivrajani & @theonlynabarun
8. Agenda • Timeline of a Kubernetes Release • Introduction and Setting the context • Why were the releases delayed? • What went right? • What could be done better? • Takeaways @MadhavJivrajani & @theonlynabarun
9. Prelude: Timeline of a Kubernetes Release Cadence: Every ~4 months @MadhavJivrajani & @theonlynabarun
10. Prelude: Timeline of a Kubernetes Release Elaborate song and dance of People and Processes @MadhavJivrajani & @theonlynabarun
11. Prelude: Timeline of a Kubernetes Release Emeritus Adviser Release Lead Branch Manager Bug Triage CI Signal Comms Docs Enhancements Release Notes Release Lead Shadows Branch Manager Shadow Bug Triage Shadows CI Signal Shadows Comms Shadows Docs Shadows Enhancements Shadows Release Notes Shadows @MadhavJivrajani & @theonlynabarun
12. Prelude: Timeline of a Kubernetes Release @MadhavJivrajani & @theonlynabarun
13. Prelude: Timeline of a Kubernetes Release @MadhavJivrajani & @theonlynabarun
14. Prelude: Timeline of a Kubernetes Release @MadhavJivrajani & @theonlynabarun
15. Prelude: Timeline of a Kubernetes Release @MadhavJivrajani & @theonlynabarun
16. Prelude: Timeline of a Kubernetes Release @MadhavJivrajani & @theonlynabarun
17. Prelude: Timeline of a Kubernetes Release @MadhavJivrajani & @theonlynabarun
18. Prelude: Timeline of a Kubernetes Release @MadhavJivrajani & @theonlynabarun
19. Prelude: Timeline of a Kubernetes Release @MadhavJivrajani & @theonlynabarun
20. Prelude: Timeline of a Kubernetes Release @MadhavJivrajani & @theonlynabarun
21. Prelude: Timeline of a Kubernetes Release @MadhavJivrajani & @theonlynabarun
22. Wildfires, Firefighters and Sustainability Learnings from Mitigating Kubernetes Fires in the Community @MadhavJivrajani & @theonlynabarun
23. Wildfires, Firefighters and Sustainability Learnings from Mitigating Kubernetes Fires in the Community @MadhavJivrajani & @theonlynabarun
24. Wildfires, Firefighters and Sustainability Learnings from Mitigating Kubernetes Fires in the Community @MadhavJivrajani & @theonlynabarun
25. @MadhavJivrajani & @theonlynabarun
26. @MadhavJivrajani & @theonlynabarun
27. @MadhavJivrajani & @theonlynabarun
28. @MadhavJivrajani & @theonlynabarun
29. @MadhavJivrajani & @theonlynabarun
30. @MadhavJivrajani & @theonlynabarun
31. Wildfires, Firefighters and Sustainability Learnings from Mitigating Kubernetes Fires in the Community @MadhavJivrajani & @theonlynabarun
32. @MadhavJivrajani & @theonlynabarun
33. @MadhavJivrajani & @theonlynabarun
34. Usually, release-blockers tend to happen towards the end of a release, but not necessarily: @MadhavJivrajani & @theonlynabarun
35. Usually, release-blockers tend to happen towards the end of a release, but not necessarily: @MadhavJivrajani & @theonlynabarun
36. Wildfires, Firefighters and Sustainability Learnings from Mitigating Kubernetes Fires in the Community @MadhavJivrajani & @theonlynabarun
37. @MadhavJivrajani & @theonlynabarun
38. @MadhavJivrajani & @theonlynabarun
39. @MadhavJivrajani & @theonlynabarun
40. @MadhavJivrajani & @theonlynabarun
41. @MadhavJivrajani & @theonlynabarun
42. @MadhavJivrajani & @theonlynabarun
43. Typical Flow of Fighting A Wildfire @MadhavJivrajani & @theonlynabarun
44. Typical Flow of Fighting A Wildfire @MadhavJivrajani & @theonlynabarun
45. Typical Flow of Fighting A Wildfire @MadhavJivrajani & @theonlynabarun
46. Typical Flow of Fighting A Wildfire @MadhavJivrajani & @theonlynabarun
47. Typical Flow of Fighting A Wildfire @MadhavJivrajani & @theonlynabarun
48. Typical Flow of Fighting A Wildfire @MadhavJivrajani & @theonlynabarun
49. Typical Flow of Fighting A Wildfire @MadhavJivrajani & @theonlynabarun
50. Typical Flow of Fighting A Wildfire Data for release-blockers for releases 1.24 - 1.27 @MadhavJivrajani & @theonlynabarun
51. Wildfires, Firefighters and Sustainability Learnings from Mitigating Kubernetes Fires in the Community @MadhavJivrajani & @theonlynabarun
52. Sustainability @MadhavJivrajani & @theonlynabarun
53. Sustainability According to Elinor Ostrom, in her Nobel Prize winning work “Governing the Commons”: “[A system is sustainable] as long as the average rate of withdrawal does not exceed the average rate of replenishment” @MadhavJivrajani & @theonlynabarun
54. Sustainability @MadhavJivrajani & @theonlynabarun
55. Sustainability @MadhavJivrajani & @theonlynabarun
56. Sustainability @MadhavJivrajani & @theonlynabarun
57. Recapping… @MadhavJivrajani & @theonlynabarun
58. @MadhavJivrajani & @theonlynabarun
59. Fire Stories: Regressions and Heroics!!! @MadhavJivrajani & @theonlynabarun
60. Fire Stories: Regressions and Heroics!!! @MadhavJivrajani & @theonlynabarun
61. Fire Stories: Regressions and Heroics!!! @MadhavJivrajani & @theonlynabarun
62. Fire Stories: Regressions and Heroics!!! @MadhavJivrajani & @theonlynabarun
63. Fire Stories: Regressions and Heroics!!! @MadhavJivrajani & @theonlynabarun
64. Fire Stories: Regressions and Heroics!!! @MadhavJivrajani & @theonlynabarun
65. Fire Stories: Regressions and Heroics!!! Turn Around Time = ~1 day @MadhavJivrajani & @theonlynabarun
66. Fire Stories: Regressions and Heroics!!! Observations: • Detection possible due to consumption of latest version of Kubernetes @MadhavJivrajani & @theonlynabarun
67. Fire Stories: Regressions and Heroics!!! Observations: • Detection possible due to consumption of latest version of Kubernetes • Community Release Engineers and Triagers available around the globe @MadhavJivrajani & @theonlynabarun
68. Fire Stories: Regressions and Heroics!!! Observations: • Detection possible due to consumption of latest version of Kubernetes • Community Release Engineers and people with knowledge of machinery available around the globe Thank you Andy, dims, liggitt, Kubernetes Release Managers and Google Build Admins! @MadhavJivrajani & @theonlynabarun
69. Fire Stories: go1.18 Breaks CSR Validation Like most fires, we start with our CI looking like this: @MadhavJivrajani & @theonlynabarun
70. Fire Stories: go1.18 Breaks CSR Validation Like most fires, we start with our CI looking like this: @MadhavJivrajani & @theonlynabarun
71. Fire Stories: go1.18 Breaks CSR Validation Quick summary of what happened: • In go1.18 crypto/x509 started to reject certificates signed with SHA-1 hash function. • Problem was it also rejected CSRs while it should only have rejected certificates. • Due to this, CI remains red till we get a fix in the next minor Go version @MadhavJivrajani & @theonlynabarun
72. Fire Stories: go1.18 Breaks CSR Validation Triage @MadhavJivrajani & @theonlynabarun
73. Fire Stories: go1.18 Breaks CSR Validation Triage @MadhavJivrajani & @theonlynabarun
74. Fire Stories: go1.18 Breaks CSR Validation Triage Quick fix to unblock CI @MadhavJivrajani & @theonlynabarun
75. Fix: When the actual fix isn’t in our control, “fixing” includes charting the best course forward with what we can control. Fire Stories: go1.18 Breaks CSR Validation @MadhavJivrajani & @theonlynabarun
76. Fire Stories: go1.18 Breaks CSR Validation Fix: Watch and List @MadhavJivrajani & @theonlynabarun
77. Fire Stories: go1.18 Breaks CSR Validation “Subfires” @MadhavJivrajani & @theonlynabarun
78. Fire Stories: go1.18 Breaks CSR Validation From fighting this, we largely see the need for: • Folks with cross functional knowledge of the tooling and machinery of the project. • Folks with knowledge about policies of other open source communities and projects that we depend on (Go in this case). @MadhavJivrajani & @theonlynabarun
79. What went right? @MadhavJivrajani & @theonlynabarun
80. Dissecting issues into actionable chunks @MadhavJivrajani & @theonlynabarun
81. Correct Set of Tools @MadhavJivrajani & @theonlynabarun
82. @MadhavJivrajani & @theonlynabarun Correct Set of Tools
83. Correct Set of Tools @MadhavJivrajani & @theonlynabarun
84. Correct Set of Tools @MadhavJivrajani & @theonlynabarun
85. Correct Set of Tools @MadhavJivrajani & @theonlynabarun
86. Global Distribution of Contributors @MadhavJivrajani & @theonlynabarun
87. Employer Support For OSS Work Company Supported 3169 Independent 669 @MadhavJivrajani & @theonlynabarun
88. What Can Be Improved? We’ve seen what went right, let’s take a look at how we can potentially improve. @MadhavJivrajani & @theonlynabarun
89. Strategically Growing OWNERS @MadhavJivrajani & @theonlynabarun
90. Strategically Growing OWNERS • Growing OWNERS in the project is critical. Period. @MadhavJivrajani & @theonlynabarun
91. Strategically Growing OWNERS • Growing OWNERS in the project is critical. Period. • Looking back at our fire stories, we can get things back on track quicker if we have a geo distributed set of firefighters: @MadhavJivrajani & @theonlynabarun
92. Strategically Growing OWNERS • Growing OWNERS in the project is critical. Period. • Looking back at our fire stories, we can get things back on track quicker if we have a geo distributed set of firefighters: ◦ But is that enough? @MadhavJivrajani & @theonlynabarun
93. Strategically Growing OWNERS @MadhavJivrajani & @theonlynabarun
94. Strategically Growing OWNERS @MadhavJivrajani & @theonlynabarun
95. Strategically Growing OWNERS @MadhavJivrajani & @theonlynabarun
96. Strategically Growing OWNERS @MadhavJivrajani & @theonlynabarun
97. Strategically Growing OWNERS @MadhavJivrajani & @theonlynabarun
98. Strategically Growing OWNERS @MadhavJivrajani & @theonlynabarun
99. Strategically Growing OWNERS @MadhavJivrajani & @theonlynabarun
100. Strategically Growing OWNERS @MadhavJivrajani & @theonlynabarun
101. Strategically Growing OWNERS • Growing OWNERS in the project is critical. Period. • Looking back at our fire stories, we can get things back on track quicker if we have a geo distributed set of firefighters: ◦ But is that enough? ◦ Along with this, we also benefit from a geo distributed set of OWNERS ▪ Brings back things back on track faster (ex: unblocks CI faster) ▪ More time for CI to soak changes made by PRs (especially towards the end of a release) @MadhavJivrajani & @theonlynabarun
102. Reliability @MadhavJivrajani & @theonlynabarun
103. Reliability We don’t need firefighters if we don’t have fires @MadhavJivrajani & @theonlynabarun
104. Reliability Investing in the reliability of the project gives exponentially positive returns @MadhavJivrajani & @theonlynabarun
105. Reliability Investing in the reliability of the project gives exponentially positive returns: • There has been a great amount of work being put towards reliability of the Kubernetes project. @MadhavJivrajani & @theonlynabarun
106. Reliability Investing in the reliability of the project gives exponentially positive returns: • There has been a great amount of work being put towards reliability of the Kubernetes project. • This effort is largely owed to SIG Testing – thank you to everyone involved, but there is still a lot of help needed here. @MadhavJivrajani & @theonlynabarun
107. Reliability Investing in the reliability of the project gives exponentially positive returns: • There has been a great amount of work being put towards reliability of the Kubernetes project. • This effort is largely owed to SIG Testing – thank you to everyone involved, but there is still a lot of help needed here. ◦ If you are an end user or a vendor or someone who cares about Kubernetes, investing and funding folks to work on the Kubernetes project is critical for us as an ecosystem. @MadhavJivrajani & @theonlynabarun
108. Having More Firefighters @MadhavJivrajani & @theonlynabarun
109. Having More Firefighters According to Curto-Millet et al. in “The sustainability of open source commons”: “Not all participation is equal and projects and communities need to encourage positive social relations. This involves participants becoming core members through situated learning and identity construction.” @MadhavJivrajani & @theonlynabarun
110. Having More Firefighters • Undocumented context — one of the largest reasons we depend on a small number of project veterans. @MadhavJivrajani & @theonlynabarun
111. Having More Firefighters • Undocumented context — one of the largest reasons we depend on a small number of project veterans. ◦ As a first step, let’s start doing and publishing post mortems after each fire. @MadhavJivrajani & @theonlynabarun
112. Having More Firefighters • Undocumented context — one of the largest reasons we depend on a small number of project veterans. ◦ As a first step, let’s start doing and publishing post mortems after each fire. • Enable folks who are potential firefighters @MadhavJivrajani & @theonlynabarun
113. Having More Firefighters • Undocumented context — one of the largest reasons we depend on a small number of project veterans. ◦ As a first step, let’s start doing and publishing post mortems after each fire. • Enable folks who are potential firefighters ◦ When fires come up - having broken down, tangible descriptions and analyses enable potential firefighters. @MadhavJivrajani & @theonlynabarun
114. Having More Firefighters • Undocumented context — one of the largest reasons we depend on a small number of project veterans. ◦ As a first step, let’s start doing and publishing post mortems after each fire. • Enable folks who are potential firefighters ◦ When fires come up - having broken down, tangible descriptions and analyses enable potential firefighters. • We have amazing teams like the Release CI Signal who can be enabled to be the entry point of firefighting. @MadhavJivrajani & @theonlynabarun
115. Having More Firefighters Link to the video: YouTube @MadhavJivrajani & @theonlynabarun
116. Takeaways @MadhavJivrajani & @theonlynabarun
117. Takeaways Globally distributed contributors, with employer support, trained to triage and debug fires, with the right tools. @MadhavJivrajani & @theonlynabarun
118. Takeaways Globally distributed contributors, with employer support, trained to triage and debug fires, with the right tools. @MadhavJivrajani & @theonlynabarun
119. Takeaways Globally distributed contributors, with employer support, trained to triage and debug fires, with the right tools. @MadhavJivrajani & @theonlynabarun
120. Takeaways Globally distributed contributors, with employer support, trained to triage and debug fires, with the right tools. @MadhavJivrajani & @theonlynabarun
121. Takeaways Globally distributed contributors, with employer support, trained to triage and debug fires, with the right tools. @MadhavJivrajani & @theonlynabarun
122. Thank You! @MadhavJivrajani & @theonlynabarun
123. Come join us at the Kubernetes SIG Meet and Greet Tomorrow at 12.30PM at Europe Foyer 1, Ground Floor, Congress Centre. @MadhavJivrajani & @theonlynabarun
124. Please scan the QR Code above to leave feedback on this session @MadhavJivrajani & @theonlynabarun