DevOps Culture Adoption: Lessons From Lost Battles

The DevOps Movement, although already consolidated a few years ago in the world, still crawls in Brazil and even under a lot of misunderstanding. Their values go hand in hand with Agile Methodologies, which implies a prior understanding of Agility and its Principles before adopting culture.

Being agile is a mandatory requirement to adopt DevOps satisfactorily, but it is not a sufficient condition. Without understanding the routine of your company's Development and Operations teams, conflicts and - especially - everyone involved, engaging in such a significant culture change becomes a risky move as well as making dynamite.

In this talk I will discuss how the adoption of DevOps may fail in your company, despite the proven benefits of this methodology.

Transcript

1. DevOps Culture Adoption: Lessons From Lost Battles Marcelo Pinheiro https://github.com/salizzar
3. $ whoami • Fireman / Problem Solver / Programmer since 2000 • Golang, Node.js, Erlang, Ruby, Python, Delphi, C++, C#, C, Java, PHP, ASP and Others • Fought, made coffee, negotiated deadlines • DevOps Engineer @ Work & Co
4. DevOps: What Is It? (Yes, another introduction.)
5. DevOps: What Is It? • TL;DR • Communication • Integration • Collaboration • A culture. Period.
6. DevOps: What Is It? • First appearing: Flickr @ Velocity 2009 • 10+ deploys per day • Netflix • Lot of tools developed and available at http:// netflix.github.io/ • Spotify, SoundCloud, Facebook, Hashicorp and others
7. DevOps: What Is It? • DevOps principles have their foundation into these categories: • Infrastructure as Code • Agile Infrastructure • Configuration Management • Continuous Integration / Delivery tools • Metrics, Logging and Monitoring solutions • Heavy use of Virtualization / Containerization tools and IaaS / PaaS / SaaS providers
8. DevOps: What Is It? • Continuous Integration • Use of Test Units, Integration Tests and Behavioral Tests combined in a manual pipeline for daily development • Developer needs to be disciplined to remember this (of course not works as we want sometimes) • Tools like Rake (Ruby) / make (father of all) helps you to automate it
9. DevOps: What Is It? • Continuous Delivery • Automation of your Continuous Integration plus Code Quality Measurement, Security Check and Artifact Generation • Deploys are responsibility of Operations Team (still manual) • Lot of options here: Jenkins, Thoughtworks Go, CircleCI, TravisCI and others
10. DevOps: What Is It? • Continuous Deployment • The Holy Grail of DevOps movement • In practice, a automated Continuous Delivery pipeline • Developers can deploy any application at any time, with zero downtime when possible • Definitively, not easy. You need to know some patterns of your team’s capacity, delivery quality and - of course - business requirements • You only automate things when you have a common pattern applicable to all systems. Period.
11. DevOps And Agility (Also: One thing requires another.)
12. DevOps And Agility • OK, you want to adopt DevOps… but: • How is your Operations Team routine? • How is your Development Team’s routine? • Are you Agile enough?
13. DevOps And Agility • Adopting DevOps without a correct Agile project management methodology is a mess. • Scrum, Kanban, Lean: look what method fits your expectations • Despite is hard to measure milestones and targets when using new technologies, HOLD ON • All teams must understand infrastructure / development demands and deliveries • We. Are. On. The. Same. Boat. • Our mission is to satisfy the customer.
14. DevOps And Agility • DevOps implies to handle Development and Operation Teams as customers each one demanding things to another. • Developers needs to understand Operation requirements (specially that related to SLA) • Sysops needs to be flexible about Development expectations (things will broken sometimes) • Business people, developers and system administrators must work together daily through the project. • :-P
15. DevOps And Agility • Adopting ANY DevOps tools and practices without changing people mindset is complicated. • A bureaucratic Operations Team routine with any DevOps tools being used is still a bureaucratic team; a Developer Team without any consideration about aspects of Operation’s work is a potential source of conflict • System administrators / operators needs to learn programming and developers needs to learn how Operations works. Period. • Promote self-organization and knowledge transfer • The best architectures, requirements and designs emerge from self-organized teams.
16. DevOps And Agility • Adopting Continuous Delivery without enforcing quality is risky. • A automated pipeline composed by bad code is a bad pipeline because it not attends a purpose to be a good one • Developers needs to embrace Operation’s requirements like Security Check and Behavioral Tests like “is my application up and running after I merge this PR on GitHub?” • Working software is the primary measure of progress.
17. DevOps And Agility • Automating a bad Continuos Delivery pipeline - consequently transforming it into a Continous Deployment - is EXTREMELY DANGEROUS. • Automatic deployment of bad code to production, whatever how complex is your pipeline and tools used, is just fashion • If anything goes wrong, you need to handle failures without bureaucracy • Deliver working software frequently, with a preference to the shorter timescale.
18. DevOps, Buzzwords & Black Swan (DON’T BE ENTHUSIASTIC)
19. DevOps, Buzzwords & Black Swan • At first moment, adopting DevOps is a kind of magical solution to a lot of problems related to delivery speed. But… • You know your job, your responsibilities, the expectations about your work? • You are prepared to change?
20. DevOps, Buzzwords & Black Swan • DevOps tools like Infrastructure As Code and Configuration Management requires knowledge of Infrastructure resources and RELATED COST. • Developers indiscriminately provisioning resources from scratch at any cloud provider using a Most-Awesome- DevOps-Tool is nice until you RECEIVE THE INVOICE. • Developers commonly configure servers to be more opened to the world than ideal, creating nightmares to Security Team • You need to take some things under control for the sake of sanity.
21. DevOps, Buzzwords & Black Swan • Continuous Integration requires good developers. • Your developers understand the valuable importance of Test Units? Code Quality reports? Refactoring? Behavioral, Integration Tests? A SCVS like Git? • Adopting DevOps without good programming techniques is TERRIBLE.
22. DevOps, Buzzwords & Black Swan • Continuous Delivery implies to measure how good are your deployments. • Rollbacks by bad code? • Rollbacks by operational mistakes? • Rollbacks by bad architecture? • You need to take these numbers in mind.
23. DevOps, Buzzwords & Black Swan • Continuous Deployment can - and will - automate a catastrophe when you don’t want. • Bad deploys are a reality, despite it means a exceptional situation, a human mistake or simply bad code • The first guy to be contacted in a Sunday’s midnight after a downtime will not be a developer, but a Operations person in a on-call scheduling, doesn’t matter if is a automatic or manual deployment • In Microservices Era, a bad deploy can propagate a dramatic turn of unhappy events • As fast as you can deploy, you need to have the same agility to rollback. Otherwise, good luck. Seriously.
24. DevOps, Buzzwords & Black Swan • Black Swan Theory • Any automation heavily impacts in human behavior. You need to do a consciousness work about how hard is to build a pipeline. • A automated pipeline is a group of simple steps that are composed by heavy effort each of them • There is no magic, is just a tool being rightly used to solve a specific problem. In other situations, can be a totally different approach • Dev / Sysadmin guys have the obligation to keep this in their hearts, not in their minds. Otherwise, you will have a SERIOUS expectation problem.
25. DevOps & Fear Of Change (Or: About People’s Factor.)
26. DevOps & Fear Of Change • Before adopting any tool, knows what your problem and how to solve it respecting their nature and people involved. • Using a DevOps tool just because Netflix, Spotify or anything else uses it is a TERRIBLE decision • Trying to solve your problem by adapting your reality to a tool workflow is not guarantee of anything • People must be trained to learn programming and infrastructure and not all have this ability without some effort or resistance • Map all needs and improvements before using anything. Period.
27. DevOps & Fear Of Change • In other way, avoid the “Everest Mountain Syndrome” • Don’t. Be. Enthusiastic • DevOps workflow really delivers a fantastic value, but keep your expectations aligned with the reality of your company • If you need to follow some compliance, think about revolutionize after doing your job • Continuous Deployment pipeline was a Delivery pipeline before being automated, that was a Integration at first. Baby steps.
28. DevOps & Fear Of Change • Rotten Apple’s Theory • One contaminates all basket. IT HAPPENS • If you need to change people, do it • Not all handles well their lifes, what to say about a culture shift. Is part of human behavior be resistant to changes
29. DevOps & Fear Of Change • In other way, avoid “DevOps Team Syndrome” • NO. NO. NO. • DevOps must be applied with developers AND operators keep near, having the support of a specialist if necessary due for knowledge gaps
30. DevOps & Fear Of Change • Management issues affects too. • How about your feedback process? • All directors understands all impacts of this culture change? • Firing people is a REAL possibility. How good is your leadership board? • Bad managers can negatively impacts any culture adoption • The best way is to do a hierarchical consciousness with all members of your company. You need to have some guarantee
31. DevOps & Fear Of Change • Egocentric Leadership can literally destroy your initiative. • As human beings that wants to grow, we follow mentors; tech leads are kinda too • Of course, for good and bad things • As a director / manager, support them to understand the value of DevOps by a effective HR Feedback methodology focused in social abilities • As a developer / sysadmin, be alert with any enthusiastic decision • Otherwise, if you don’t have power to change (or fire them in worst case)… forget it.
32. DevOps #fail: Lessons Learned (Also: try to BE)
33. DevOps #fail: Lessons Learned • Be POLITE. • Take up arms will cause more conflicts • Avoid any polemics, whatever it means • Empathy, empathy, empathy.
34. DevOps #fail: Lessons Learned • Be a POLITICAL BEAST. • In a good way, of course • Remember, you are a proxy between two conflicting teams PLUS customer needs • Identify gaps, negotiate as hell and see every challenge as a opportunity • Improve your diplomacy ASAP
35. DevOps #fail: Lessons Learned • Be PATIENT. • Implementing a cultural shift is a very frustrating job • You really embrace this herculean task, otherwise leave it • HOLD ON • Keep strong with what you believe
36. DevOps #fail: Lessons Learned • Be a PROBLEM-SOLVER. • From source code to human relationships • Most conflicts can be solved with empathy and a good conversation with coffee. Or drinks if you like • Avoid imperative approaches except in urgent situations • Read, read, read about human behavior. Read a lot
37. DevOps #fail: Lessons Learned • Be PRAGMATIC. • A incomplete Continuous Delivery pipeline is better than none • Manual operations are acceptable sometimes as a transitory step • Partial solutions can be adopted if attends most cases • A ideal world will NEVER happen. Deal with it.
38. DevOps #fail: Lessons Learned • Be a LEADER. • Inspire your colleagues being a excellent technical person AND a excellent manager • Charisma is not required but helps you a lot • Your attitudes will be a mirror of all members of Development and Operation Teams • Be thick as a brick and you will get resistance, doesn’t matter how best is your intentions
39. Questions? (As free as beer in a hypothetical world.)
40. Thank you. :) https://github.com/salizzar