DEVOPS PITFALLS A story on how we failed to implement devops but saved our team from self destruction Rudy Gevaert – [email protected] - @rgevaert – http://www.webworm.be
devopsdays Cuba 2017 2 About me Rudy Gevaert Ghent – Belgium @rgevaert - http://www.webworm.be GNU/Linux user since 2000 Sustainable ICT for development in Cuba and Ethiopia Sysadmin at Ghent University Devopsdays Belgium and Cuba
devopsdays Cuba 2017 6 Start to devops 2008 start of configuration management 2009 first ever devopsdays in Ghent 2010 uptake of cfgmgt by some team members ... 2015: some developers start commiting into VCS first cross department multi functional team 2016: real commitment to use configuration management in Unix team Inter and extra team situation:
devopsdays Cuba 2017 7 Where are we now? 2016 Unix team with renewed spirit and motivation 2017 original cross functional team is not functioning anymore
devopsdays Cuba 2017 8 Some achievements ● Widespread adoption CI en CD ● Configuration management is mandatory ● CI en CD implemented for Puppet implemented ● Automated security updates are being rolled out ● Documentation plays key role in all new setups ● Kanban to visualize work ● Post mortem process is standard
devopsdays Cuba 2017 13 Cultural pitfall: failing to define devops Leads to: ● The developer will become sysadmin ● Devops ⇔ tool x && tool y ● Devops ⇒ agile ⇒ no planning ● Devops is about communication, but we have difficulty with it ➔ You need common understanding so nobody is left behind and expections can be met
devopsdays Cuba 2017 14 Cultural pitfall: expecting devops to fix bad leadership “By 2020, half of the CIOs who have not transformed their teams’ capabilities will be displaced from their organizations digital leadership teams.”1 Bottom up and bad leadership: very difficult and not fruitful in the long run ➔ We had to go through several crisis's to just reach where we are now ➔ Depending on the context: leadership will or can not be changed 1 Gartner Predicts: IT infrastructure & Operations
devopsdays Cuba 2017 15 Cultural pitfall: lack of skilled people with the correct mindset ● Sysadmins with the wrong mindset left the team ● Devs with the good mindset left the team ● New team members had the correct skills and mindset ➔ Building human capitial is frequently forgotten. Everybody (good or bad) can leave the team ➔ Enable running the race at their own pace
devopsdays Cuba 2017 16 Cultural pitfall: not defining your goal What is the goal of the enterprise? Make money? But what when it is something else? ➔Devops implementation should be aimed at reaching the goal
devopsdays Cuba 2017 18 Automation pitfall: it’s only about the tool Devops is more than using the latest tool It comes second to: ➔ Understand why to use a specific tool: efficiency, cost reduction, risk mitigation, and predictability ➔ Prevent sprawl of tools
devopsdays Cuba 2017 19 Automation pitfall: automate all of it now You don’t need to automate everything (at once). ➔ Document the process first ➔ We made documentation an integral and mandatory part of product development ➔ Automate predictable and error prone tasks ➔ Increase automation at the bottleneck
devopsdays Cuba 2017 20 Automation pitfall: failing to adhere to the core tenets The core tenets of automation in devops are ● automated testing, ● continuous integration and ● integrated configuration management Compromising will lead to a slippery slope.
devopsdays Cuba 2017 21 Metrics pitfall: set up different monitoring tools Sysadmin team had excellent monitoring tool, but: ● It was fully locked down ● Didn’t monitor development machines ● Only sent alerts to sysadmins Dev teams have their own monitoring solution... ➔ Sysadmin team is setting up fully open system ➔ Developers couldn’t wait and rolled v2 of their setup
devopsdays Cuba 2017 22 Metrics pitfall: nobody uses the metrics setup Because they don’t have any valuable metrics They aren’t accessible There is no awareness
devopsdays Cuba 2017 23 Sharing pitfall: private bug tracker Devs moved to central bug and task tracker. Devs manage their system with pride! Ops on different system that isn’t maintained and barely used. ➔ Ops shifted to central system ➔ Exponential increase in visibility of work ➔ Some teams have private projects, some of them for no good reason (secrecy?)
devopsdays Cuba 2017 24 Sharing pitfall: private chat channel open Jabber, but invite only channels Then, 2 Slack teams and more Little use by devs of Jabber ➔ Global Slack team, without official consent ➔ After two months, one system left More people than ever were using it ➔ Some suggested private Slack channels per team, but we did not allow ➔ After 8 months got management buy- in and moved to paid version
devopsdays Cuba 2017 25 Sharing pitfall: private repositories Configuration management system repository fully locked down is counter productive Because of: secrets, lack of trust Giving access doesn’t mean people will read and contribute ➔ Removed secrets from our repository ➔ Started an opt-in policy ➔ Devs are sending pull requests instead of change request
devopsdays Cuba 2017 26 Remember this Culture Transformational leadership Implement processes that enable developer productivity, reducing code deployment lead times and support more reliable infrastructure Work across organizational silos to achieve strategic alignment Automation The tools are important, but not the most important Sharing Move from closed to open systems Metrics Provide open systems and limit the number of diferent tools
devopsdayscuba
frogandcode
stephaniewalter
carmenintech
cromwellryan
tammielis
akosma
mthomps
rocio
chriscoyier
smashingmag
edds
trallard