Kaizen-inspired DevOps

High-performing teams sustain their momentum through the synergy of skilled individuals, efficient processes, and adaptable practices. The primary objective of performance tuning is to minimize inefficiencies and bolster a team's ability to withstand external disruptions.

This session will delve into a diverse array of techniques and processes aimed at reducing wastage and enhancing resilience. Our journey will commence with an exploration of the contemporary skill set essential for constructing swift and anti-fragile systems. Subsequently, we will delve into the development and management methodologies that underpin both the team and product in the face of a dynamic environment. Finally, we will investigate the tools available to support these methodologies effectively.

This session is designed primarily for engineering managers, particularly those with technical backgrounds. However, it also offers valuable insights to anyone with an interest in DevOps, team scaling, and agile methodologies.

Transcript

1. Kaizen-inspired DevOps Strategies for efficiency and resilience in a rapidly changing world
6. Kaizen (改善) • A “good change,” an improvement • Prefer improvement over innovation • Prefer process over creativity • Innovate only where you can automate
7. 10 principles of Kaizen 1. Stop assuming 2. Be proactive about solving problems 3. Status quo is not acceptable 4. Let go of perfectionism and take an attitude of iterative, adaptive change 5. Look for solutions as you find mistakes 6. Create an environment in which everyone feels empowered to contribute 7. Don't accept the obvious issue; ask "why" five times to get to the root cause 8. Cull information and opinions from multiple people 9. Use creativity to find low cost, small improvements 10. Never stop improving (GOTO 1)
8. Kaizen event Plan • Analyze current problem and conditions • Establish change objectives • Create processes to achieve solutions Do • Implement the plan • Test small changes • Gather data on the change’s effectiveness
9. Kaizen event Check • Evaluate data • Identify deviations between outcomes and planned objectives Act • Standardize • Review and define next issues
10. Increased performance by elimination of waste • Wasted time • Duplication of work • Wasted materials and resources in manufacturing industry • Wasted electricity, CPU, RAM, storage, bandwidth in IT
11. Can we measure performance?
12. (DORA, 2023)
13. High performers deploy more frequently.
14. Mean time between errors is irrelevant. Measure mean time to recover after failed deployment!
15. The 3 pillars of high performance • Technical capabilities • Lean processes • Generative culture
16. Technical capabilities Iterative improvement with the goal of automation and even commodification of DevOps. 1. Modern technology skillset 2. Update development practices 3. Normalize, standardize, expand 4. Automate 5. Commodify
17. Cut ties with the legacy technology 1. Unsupported runtime versions 2. Unsupported application versions 3. Redundant technologies 4. Inappropriate data models 5. Tightly coupled architectures 6. Root access installers 7. Stateful applications on disk
18. Technology stack does not matter. Architecture does!
19. Cloud-native design patterns 1. Loosely coupled architecture 2. Small batches 3. Continuous integration, delivery, and deployment 4. Immutable containers 5. APIs 6. Antifragile systems
20. Reduce complexity and waste through standardization.
21. Standardization What to standardize? Benefits of standardization 1. Operating systems 2. Development standards & practices (e.g. coding standards) 3. Testing practices 4. Deployment patterns 5. Operations practices 6. System configurations 1. Reduction in licencing costs 2. Hire for a smaller skillset 3. Create cross-functional teams 4. Lower risk 5. Required for automation
22. Reduce waste by automating as much as possible, with the aim of having production-like environments.
23. Infrastructure as code (IaC) • Manage and provision the infrastructure through code instead of through manual processes. • Track changes through version control. • Scale the infrastructure to handle varying workloads. • Deploying to production should not be your first production deploy. • Infrastructure security built in.
24. Commodify the toolstack by making it available on-demand, self-service, for anyone.
25. Lean processes Minimum of rituals necessary to ensure informed decision making. 1. Focus on users 2. Cross-functional initiatives 3. Teams enabled to make decisions and changes on software they work on 4. Centralized knowledge management 5. Efficient code review 6. Culture of feedback
26. Teams with strong user focus have 40% higher organizational performance. (DORA, 2023)
27. Cross-functional initiatives
28. Product Designer Frontend Engineer Product Manager Initiative: add new payment methods Component: Frontend Component: Billing System Comp. Maintainer Comp. Maintainer Team: Design VP, Design Team: Product VP, Product Initiative: design overhaul Billing System Engineer Billing System Engineer Product Manager Product Designer Frontend Engineer Frontend Engineer Frontend Engineer
29. Cross-functional initiatives • Teams formed around components and departments. Teams are being led by department heads or component maintainers. • Cross functional initiatives are a collaborative and strategically planned temporary effort that brings together different functional areas. • The initiatives are being led by the “Product Trio”: a product manager, a designer, and a lead engineer
30. Distribute decision making process Distributing the decision-making process increases efficiency and allows for innovation by leveraging diverse expertise and reducing bottlenecks in critical choices. It also greatly improves team morale, allowing each employee to shine. Executive decisions Strategic direction (Daft & Marcic, 2016) Resource allocation Tactical planning Resource distribution Policy implementation Execution and delivery Problem-solving Technical decisions Middle management decisions Team or project-level decisions
31. Cloud native DevOps driven change management Freedom to change the design of the code and infrastructure. Ability to test the infrastructure and code changes prior to release. Ability to release. All of it without communication and coordination outside the team. Live Code and Infra Development Code and Infra Staging Code and Infra Development Code and Infra Development Code and Infra … The team
32. Documentation is foundational Positive impact on outcomes, such as team performance, productivity, and job satisfaction. Increased risk of burnout with people identified as underrepresented. As the documentation quality, generative culture, and team stability increase, burnout also increases for people who identify as underrepresented.
33. Documentation is foundational www.writethedocs.org Society for Technical Communication Using a good documentation tool (Confluence, GitBook, ReadTheDocs, Sphinx, Guru…) helps: • Ease of use – minimize the overhead • Version control • Collaboration, feedback, comments • Search • Cross referencing
34. Efficient code review and culture of feedback • Increased technical capabilities facilitate reviews. Automation (CI/CD) reduces workload. Small batches and standardized coding style reduce cognitive load. • Documentation and context for the changes explain the purpose of the code, the problem it solves, and any trade-offs or considerations, thus drastically increasing the efficiency of reviews.
35. Efficient code review and culture of feedback • Checklists and automation guide reviewers through the process ensuring all the common issues get their attention. • Think about Kaizen and constantly improve the review and feedback process.
36. Generative culture “Teams with generative cultures have 30% higher organizational performance.” 1. Psychological safety 2. Open communication 3. Learning orientation 4. Continuous training 5. Data-driven decision making 6. Collaboration 7. Innovation 8. Adaptability 9. Shared values 10. Empowerment (DORA, 2023)
37. Westrum’s Organizational Typology 1. Pathological (power-oriented) organizations are characterized by large amounts of fear and threat. People often hoard information or withhold it for political reasons, or distort it to make themselves look better. 2. Bureaucratic (rule-oriented) organizations protect departments. Those in the department want to maintain their “turf,” insist on their own rules, and generally do things by the book—their book. 3. Generative (performance-oriented) organizations focus on the mission. How do we accomplish our goal? Everything is subordinated to good performance, to doing what we are supposed to do. (IT Revolution, 2021)
38. 61% reduction in burnout due to high levels of job security. (DORA, 2023)
39. Focus on mission before organization. (Wei-Skillern & Silver, 2013)
40. Manage through trust, not control. (Wei-Skillern & Silver, 2013)
41. Promote others, not yourself. (Wei-Skillern & Silver, 2013)
42. High performers are constellations, not stars. (Wei-Skillern & Silver, 2013)
43. Thank you! Branislav Bujisic Senior Director of Engineering branislav@platform.sh https://slides.bujisic.com
44. References Roser, C. (2016). PDCA Multi-Loop Diagram [Diagram]. Wikimedia Commons. https://commons.wikimedia.org/wiki/File:PDCA-Multi-Loop.png Forsgren, N. (2018, April 23). Secrets and surprises of high performance: What the data says [Slides]. Presented at O'Reilly Velocity Conference, California. Retrieved from https://conferences.oreilly.com/velocity/vl-ca-2018/cdn.oreillystatic.com/en/assets/1/e vent/270/Secrets%20and%20surprises%20of%20high%20performance_%20What%20 the%20data%20says%20Presentation.pdf Landsberg, M. (1996). The Tao of Coaching: Boost Your Effectiveness at Work by Inspiring and Developing Those Around You. Profile Books.
45. References Daft, R. L., & Marcic, D. (2016). Understanding Management. Cengage Learning. Wei-Skillern, J., & Silver, N. (2013). Four Network Principles for Collaboration Success. The Foundation Review, 5(1). https://doi.org/10.4087/FOUNDATIONREVIEW-D-12-00018.1 DORA (2023). State of DevOps Report 2023. Retrieved from https://services.google.com/fh/files/misc/2023_final_report_sodr.pdf IT Revolution. (2021). Westrum’s Organizational Model in Technology Organizations. https://itrevolution.com/articles/westrums-organizational-model-in-tech-orgs/