Share
DevOps Culture Adoption: Lessons From Lost Battles
The DevOps Movement, although already consolidated a few years ago in the world, still crawls in Brazil and even under a lot of misunderstanding. Their values go hand in hand with Agile Methodologies, which implies a prior understanding of Agility and its Principles before adopting culture.
Being agile is a mandatory requirement to adopt DevOps satisfactorily, but it is not a sufficient condition. Without understanding the routine of your company's Development and Operations teams, conflicts and - especially - everyone involved, engaging in such a significant culture change becomes a risky move as well as making dynamite.
In this talk I will discuss how the adoption of DevOps may fail in your company, despite the proven benefits of this methodology.
Marcelo Pinheiro
Transcript
40 slides
-
$ whoami • Fireman / Problem Solver / Programmer since
2000 • Golang, Node.js, Erlang, Ruby, Python, Delphi, C++, C#, C, Java, PHP, ASP and Others • Fought, made coffee, negotiated deadlines • DevOps Engineer @ Work & Co -
DevOps: What Is It? • TL;DR • Communication • Integration
• Collaboration • A culture. Period. -
DevOps: What Is It? • First appearing: Flickr @ Velocity
2009 • 10+ deploys per day • Netflix • Lot of tools developed and available at http:// netflix.github.io/ • Spotify, SoundCloud, Facebook, Hashicorp and others -
DevOps: What Is It? • DevOps principles have their foundation
into these categories: • Infrastructure as Code • Agile Infrastructure • Configuration Management • Continuous Integration / Delivery tools • Metrics, Logging and Monitoring solutions • Heavy use of Virtualization / Containerization tools and IaaS / PaaS / SaaS providers -
DevOps: What Is It? • Continuous Integration • Use of
Test Units, Integration Tests and Behavioral Tests combined in a manual pipeline for daily development • Developer needs to be disciplined to remember this (of course not works as we want sometimes) • Tools like Rake (Ruby) / make (father of all) helps you to automate it -
DevOps: What Is It? • Continuous Delivery • Automation of
your Continuous Integration plus Code Quality Measurement, Security Check and Artifact Generation • Deploys are responsibility of Operations Team (still manual) • Lot of options here: Jenkins, Thoughtworks Go, CircleCI, TravisCI and others -
DevOps: What Is It? • Continuous Deployment • The Holy
Grail of DevOps movement • In practice, a automated Continuous Delivery pipeline • Developers can deploy any application at any time, with zero downtime when possible • Definitively, not easy. You need to know some patterns of your team’s capacity, delivery quality and - of course - business requirements • You only automate things when you have a common pattern applicable to all systems. Period. -
DevOps And Agility • OK, you want to adopt DevOps…
but: • How is your Operations Team routine? • How is your Development Team’s routine? • Are you Agile enough? -
DevOps And Agility • Adopting DevOps without a correct Agile
project management methodology is a mess. • Scrum, Kanban, Lean: look what method fits your expectations • Despite is hard to measure milestones and targets when using new technologies, HOLD ON • All teams must understand infrastructure / development demands and deliveries • We. Are. On. The. Same. Boat. • Our mission is to satisfy the customer. -
DevOps And Agility • DevOps implies to handle Development and
Operation Teams as customers each one demanding things to another. • Developers needs to understand Operation requirements (specially that related to SLA) • Sysops needs to be flexible about Development expectations (things will broken sometimes) • Business people, developers and system administrators must work together daily through the project. • :-P -
DevOps And Agility • Adopting ANY DevOps tools and practices
without changing people mindset is complicated. • A bureaucratic Operations Team routine with any DevOps tools being used is still a bureaucratic team; a Developer Team without any consideration about aspects of Operation’s work is a potential source of conflict • System administrators / operators needs to learn programming and developers needs to learn how Operations works. Period. • Promote self-organization and knowledge transfer • The best architectures, requirements and designs emerge from self-organized teams. -
DevOps And Agility • Adopting Continuous Delivery without enforcing quality
is risky. • A automated pipeline composed by bad code is a bad pipeline because it not attends a purpose to be a good one • Developers needs to embrace Operation’s requirements like Security Check and Behavioral Tests like “is my application up and running after I merge this PR on GitHub?” • Working software is the primary measure of progress. -
DevOps And Agility • Automating a bad Continuos Delivery pipeline
- consequently transforming it into a Continous Deployment - is EXTREMELY DANGEROUS. • Automatic deployment of bad code to production, whatever how complex is your pipeline and tools used, is just fashion • If anything goes wrong, you need to handle failures without bureaucracy • Deliver working software frequently, with a preference to the shorter timescale. -
DevOps, Buzzwords & Black Swan • At first moment, adopting
DevOps is a kind of magical solution to a lot of problems related to delivery speed. But… • You know your job, your responsibilities, the expectations about your work? • You are prepared to change? -
DevOps, Buzzwords & Black Swan • DevOps tools like Infrastructure
As Code and Configuration Management requires knowledge of Infrastructure resources and RELATED COST. • Developers indiscriminately provisioning resources from scratch at any cloud provider using a Most-Awesome- DevOps-Tool is nice until you RECEIVE THE INVOICE. • Developers commonly configure servers to be more opened to the world than ideal, creating nightmares to Security Team • You need to take some things under control for the sake of sanity. -
DevOps, Buzzwords & Black Swan • Continuous Integration requires good
developers. • Your developers understand the valuable importance of Test Units? Code Quality reports? Refactoring? Behavioral, Integration Tests? A SCVS like Git? • Adopting DevOps without good programming techniques is TERRIBLE. -
DevOps, Buzzwords & Black Swan • Continuous Delivery implies to
measure how good are your deployments. • Rollbacks by bad code? • Rollbacks by operational mistakes? • Rollbacks by bad architecture? • You need to take these numbers in mind. -
DevOps, Buzzwords & Black Swan • Continuous Deployment can -
and will - automate a catastrophe when you don’t want. • Bad deploys are a reality, despite it means a exceptional situation, a human mistake or simply bad code • The first guy to be contacted in a Sunday’s midnight after a downtime will not be a developer, but a Operations person in a on-call scheduling, doesn’t matter if is a automatic or manual deployment • In Microservices Era, a bad deploy can propagate a dramatic turn of unhappy events • As fast as you can deploy, you need to have the same agility to rollback. Otherwise, good luck. Seriously. -
DevOps, Buzzwords & Black Swan • Black Swan Theory •
Any automation heavily impacts in human behavior. You need to do a consciousness work about how hard is to build a pipeline. • A automated pipeline is a group of simple steps that are composed by heavy effort each of them • There is no magic, is just a tool being rightly used to solve a specific problem. In other situations, can be a totally different approach • Dev / Sysadmin guys have the obligation to keep this in their hearts, not in their minds. Otherwise, you will have a SERIOUS expectation problem. -
DevOps & Fear Of Change • Before adopting any tool,
knows what your problem and how to solve it respecting their nature and people involved. • Using a DevOps tool just because Netflix, Spotify or anything else uses it is a TERRIBLE decision • Trying to solve your problem by adapting your reality to a tool workflow is not guarantee of anything • People must be trained to learn programming and infrastructure and not all have this ability without some effort or resistance • Map all needs and improvements before using anything. Period. -
DevOps & Fear Of Change • In other way, avoid
the “Everest Mountain Syndrome” • Don’t. Be. Enthusiastic • DevOps workflow really delivers a fantastic value, but keep your expectations aligned with the reality of your company • If you need to follow some compliance, think about revolutionize after doing your job • Continuous Deployment pipeline was a Delivery pipeline before being automated, that was a Integration at first. Baby steps. -
DevOps & Fear Of Change • Rotten Apple’s Theory •
One contaminates all basket. IT HAPPENS • If you need to change people, do it • Not all handles well their lifes, what to say about a culture shift. Is part of human behavior be resistant to changes -
DevOps & Fear Of Change • In other way, avoid
“DevOps Team Syndrome” • NO. NO. NO. • DevOps must be applied with developers AND operators keep near, having the support of a specialist if necessary due for knowledge gaps -
DevOps & Fear Of Change • Management issues affects too.
• How about your feedback process? • All directors understands all impacts of this culture change? • Firing people is a REAL possibility. How good is your leadership board? • Bad managers can negatively impacts any culture adoption • The best way is to do a hierarchical consciousness with all members of your company. You need to have some guarantee -
DevOps & Fear Of Change • Egocentric Leadership can literally
destroy your initiative. • As human beings that wants to grow, we follow mentors; tech leads are kinda too • Of course, for good and bad things • As a director / manager, support them to understand the value of DevOps by a effective HR Feedback methodology focused in social abilities • As a developer / sysadmin, be alert with any enthusiastic decision • Otherwise, if you don’t have power to change (or fire them in worst case)… forget it. -
DevOps #fail: Lessons Learned • Be POLITE. • Take up
arms will cause more conflicts • Avoid any polemics, whatever it means • Empathy, empathy, empathy. -
DevOps #fail: Lessons Learned • Be a POLITICAL BEAST. •
In a good way, of course • Remember, you are a proxy between two conflicting teams PLUS customer needs • Identify gaps, negotiate as hell and see every challenge as a opportunity • Improve your diplomacy ASAP -
DevOps #fail: Lessons Learned • Be PATIENT. • Implementing a
cultural shift is a very frustrating job • You really embrace this herculean task, otherwise leave it • HOLD ON • Keep strong with what you believe -
DevOps #fail: Lessons Learned • Be a PROBLEM-SOLVER. • From
source code to human relationships • Most conflicts can be solved with empathy and a good conversation with coffee. Or drinks if you like • Avoid imperative approaches except in urgent situations • Read, read, read about human behavior. Read a lot -
DevOps #fail: Lessons Learned • Be PRAGMATIC. • A incomplete
Continuous Delivery pipeline is better than none • Manual operations are acceptable sometimes as a transitory step • Partial solutions can be adopted if attends most cases • A ideal world will NEVER happen. Deal with it. -
DevOps #fail: Lessons Learned • Be a LEADER. • Inspire
your colleagues being a excellent technical person AND a excellent manager • Charisma is not required but helps you a lot • Your attitudes will be a mirror of all members of Development and Operation Teams • Be thick as a brick and you will get resistance, doesn’t matter how best is your intentions