DevOpsDays Cuba 2017: DEVOPS PITFALLS

Author: Rudy Gevaert
Summary: A story on how we failed to implement devops but saved our team from self destruction

1. DEVOPS PITFALLS A story on how we failed to implement devops but saved our team from self destruction Rudy Gevaert – rudy.gevaert@gmail.com - @rgevaert – http://www.webworm.be
2. devopsdays Cuba 2017 2 About me Rudy Gevaert Ghent – Belgium @rgevaert - http://www.webworm.be GNU/Linux user since 2000 Sustainable ICT for development in Cuba and Ethiopia Sysadmin at Ghent University Devopsdays Belgium and Cuba
3. devopsdays Cuba 2017 3 Table of contents Setting the scene Start to devops Where we are now Devops pitfalls Summary
4. devopsdays Cuba 2017 4 Setting the scene: Ghent University www.ugent.be established 1817 > 50.000 students 10.000 staf Campus in South-Korea
5. devopsdays Cuba 2017 5 Setting the scene... DICT: Department of ICT 120 staf Diferent offices Context: Unix-team vs diferent other teams
6. devopsdays Cuba 2017 6 Start to devops 2008 start of configuration management 2009 first ever devopsdays in Ghent 2010 uptake of cfgmgt by some team members ... 2015: some developers start commiting into VCS first cross department multi functional team 2016: real commitment to use configuration management in Unix team Inter and extra team situation:
7. devopsdays Cuba 2017 7 Where are we now? 2016 Unix team with renewed spirit and motivation 2017 original cross functional team is not functioning anymore
8. devopsdays Cuba 2017 8 Some achievements • Widespread adoption CI en CD • Configuration management is mandatory • CI en CD implemented for Puppet implemented • Automated security updates are being rolled out • Documentation plays key role in all new setups • Kanban to visualize work • Post mortem process is standard
9. devopsdays Cuba 2017 9 What is devops?
10. devopsdays Cuba 2017 10 What is devops?
11. devopsdays Cuba 2017 11 Key tipping points
12. devopsdays Cuba 2017 12 The not so definitive list of devops pitfalls
13. devopsdays Cuba 2017 13 Cultural pitfall: failing to define devops Leads to: • The developer will become sysadmin • Devops ⇔ tool x && tool y • Devops ⇒ agile ⇒ no planning • Devops is about communication, but we have difficulty with it ➔ You need common understanding so nobody is left behind and expections can be met
14. devopsdays Cuba 2017 14 Cultural pitfall: expecting devops to fix bad leadership “By 2020, half of the CIOs who have not transformed their teams’ capabilities will be displaced from their organizations digital leadership teams.”1 Bottom up and bad leadership: very difficult and not fruitful in the long run ➔ We had to go through several crisis's to just reach where we are now ➔ Depending on the context: leadership will or can not be changed 1 Gartner Predicts: IT infrastructure & Operations
15. devopsdays Cuba 2017 15 Cultural pitfall: lack of skilled people with the correct mindset • Sysadmins with the wrong mindset left the team • Devs with the good mindset left the team • New team members had the correct skills and mindset ➔ Building human capitial is frequently forgotten. Everybody (good or bad) can leave the team ➔ Enable running the race at their own pace
16. devopsdays Cuba 2017 16 Cultural pitfall: not defining your goal What is the goal of the enterprise? Make money? But what when it is something else? ➔Devops implementation should be aimed at reaching the goal
17. devopsdays Cuba 2017 17 Cultural pitfall: lack of
18. devopsdays Cuba 2017 18 Automation pitfall: it’s only about the tool Devops is more than using the latest tool It comes second to: ➔ Understand why to use a specific tool: efficiency, cost reduction, risk mitigation, and predictability ➔ Prevent sprawl of tools
19. devopsdays Cuba 2017 19 Automation pitfall: automate all of it now You don’t need to automate everything (at once). ➔ Document the process first ➔ We made documentation an integral and mandatory part of product development ➔ Automate predictable and error prone tasks ➔ Increase automation at the bottleneck
20. devopsdays Cuba 2017 20 Automation pitfall: failing to adhere to the core tenets The core tenets of automation in devops are • automated testing, • continuous integration and • integrated configuration management Compromising will lead to a slippery slope.
21. devopsdays Cuba 2017 21 Metrics pitfall: set up different monitoring tools Sysadmin team had excellent monitoring tool, but: • It was fully locked down • Didn’t monitor development machines • Only sent alerts to sysadmins Dev teams have their own monitoring solution... ➔ Sysadmin team is setting up fully open system ➔ Developers couldn’t wait and rolled v2 of their setup
22. devopsdays Cuba 2017 22 Metrics pitfall: nobody uses the metrics setup Because they don’t have any valuable metrics They aren’t accessible There is no awareness
23. devopsdays Cuba 2017 23 Sharing pitfall: private bug tracker Devs moved to central bug and task tracker. Devs manage their system with pride! Ops on different system that isn’t maintained and barely used. ➔ Ops shifted to central system ➔ Exponential increase in visibility of work ➔ Some teams have private projects, some of them for no good reason (secrecy?)
24. devopsdays Cuba 2017 24 Sharing pitfall: private chat channel open Jabber, but invite only channels Then, 2 Slack teams and more Little use by devs of Jabber ➔ Global Slack team, without official consent ➔ After two months, one system left More people than ever were using it ➔ Some suggested private Slack channels per team, but we did not allow ➔ After 8 months got management buy- in and moved to paid version
25. devopsdays Cuba 2017 25 Sharing pitfall: private repositories Configuration management system repository fully locked down is counter productive Because of: secrets, lack of trust Giving access doesn’t mean people will read and contribute ➔ Removed secrets from our repository ➔ Started an opt-in policy ➔ Devs are sending pull requests instead of change request
26. devopsdays Cuba 2017 26 Remember this Culture Transformational leadership Implement processes that enable developer productivity, reducing code deployment lead times and support more reliable infrastructure Work across organizational silos to achieve strategic alignment Automation The tools are important, but not the most important Sharing Move from closed to open systems Metrics Provide open systems and limit the number of diferent tools
27. devopsdays Cuba 2017 27 Suggestions for Free Software tools Config management Puppet Ansible Continuous integration and deployment Jenkins, Gitlab Repositories Gitlab Package management Aptly Pulp Monitoring Icinga2 Collaboration Mattermost Dokuwiki
28. devopsdays Cuba 2017 28 Further reading
29. devopsdays Cuba 2017 29 One last word
30. devopsdays Cuba 2017 30 Preguntas?