Upgrade to Pro — share decks privately, control downloads, hide ads and more …

DevOpsDays Cuba 2017: DEVOPS PITFALLS

DevOpsDays Cuba
October 26, 2017

DevOpsDays Cuba 2017: DEVOPS PITFALLS

Author: Rudy Gevaert
Summary: A story on how we failed to implement devops but saved our team from self destruction

DevOpsDays Cuba

October 26, 2017

More Decks by DevOpsDays Cuba


  1. DEVOPS PITFALLS A story on how we failed to implement

    devops but saved our team from self destruction Rudy Gevaert – [email protected] - @rgevaert – http://www.webworm.be
  2. devopsdays Cuba 2017 2 About me Rudy Gevaert Ghent –

    Belgium @rgevaert - http://www.webworm.be GNU/Linux user since 2000 Sustainable ICT for development in Cuba and Ethiopia Sysadmin at Ghent University Devopsdays Belgium and Cuba
  3. devopsdays Cuba 2017 3 Table of contents Setting the scene

    Start to devops Where we are now Devops pitfalls Summary
  4. devopsdays Cuba 2017 4 Setting the scene: Ghent University www.ugent.be

    established 1817 > 50.000 students 10.000 staf Campus in South-Korea
  5. devopsdays Cuba 2017 5 Setting the scene... DICT: Department of

    ICT 120 staf Diferent offices Context: Unix-team vs diferent other teams
  6. devopsdays Cuba 2017 6 Start to devops 2008 start of

    configuration management 2009 first ever devopsdays in Ghent 2010 uptake of cfgmgt by some team members ... 2015: some developers start commiting into VCS first cross department multi functional team 2016: real commitment to use configuration management in Unix team Inter and extra team situation:
  7. devopsdays Cuba 2017 7 Where are we now? 2016 Unix

    team with renewed spirit and motivation 2017 original cross functional team is not functioning anymore
  8. devopsdays Cuba 2017 8 Some achievements • Widespread adoption CI

    en CD • Configuration management is mandatory • CI en CD implemented for Puppet implemented • Automated security updates are being rolled out • Documentation plays key role in all new setups • Kanban to visualize work • Post mortem process is standard
  9. devopsdays Cuba 2017 13 Cultural pitfall: failing to define devops

    Leads to: • The developer will become sysadmin • Devops ⇔ tool x && tool y • Devops ⇒ agile ⇒ no planning • Devops is about communication, but we have difficulty with it ➔ You need common understanding so nobody is left behind and expections can be met
  10. devopsdays Cuba 2017 14 Cultural pitfall: expecting devops to fix

    bad leadership “By 2020, half of the CIOs who have not transformed their teams’ capabilities will be displaced from their organizations digital leadership teams.”1 Bottom up and bad leadership: very difficult and not fruitful in the long run ➔ We had to go through several crisis's to just reach where we are now ➔ Depending on the context: leadership will or can not be changed 1 Gartner Predicts: IT infrastructure & Operations
  11. devopsdays Cuba 2017 15 Cultural pitfall: lack of skilled people

    with the correct mindset • Sysadmins with the wrong mindset left the team • Devs with the good mindset left the team • New team members had the correct skills and mindset ➔ Building human capitial is frequently forgotten. Everybody (good or bad) can leave the team ➔ Enable running the race at their own pace
  12. devopsdays Cuba 2017 16 Cultural pitfall: not defining your goal

    What is the goal of the enterprise? Make money? But what when it is something else? ➔Devops implementation should be aimed at reaching the goal
  13. devopsdays Cuba 2017 18 Automation pitfall: it’s only about the

    tool Devops is more than using the latest tool It comes second to: ➔ Understand why to use a specific tool: efficiency, cost reduction, risk mitigation, and predictability ➔ Prevent sprawl of tools
  14. devopsdays Cuba 2017 19 Automation pitfall: automate all of it

    now You don’t need to automate everything (at once). ➔ Document the process first ➔ We made documentation an integral and mandatory part of product development ➔ Automate predictable and error prone tasks ➔ Increase automation at the bottleneck
  15. devopsdays Cuba 2017 20 Automation pitfall: failing to adhere to

    the core tenets The core tenets of automation in devops are • automated testing, • continuous integration and • integrated configuration management Compromising will lead to a slippery slope.
  16. devopsdays Cuba 2017 21 Metrics pitfall: set up different monitoring

    tools Sysadmin team had excellent monitoring tool, but: • It was fully locked down • Didn’t monitor development machines • Only sent alerts to sysadmins Dev teams have their own monitoring solution... ➔ Sysadmin team is setting up fully open system ➔ Developers couldn’t wait and rolled v2 of their setup
  17. devopsdays Cuba 2017 22 Metrics pitfall: nobody uses the metrics

    setup Because they don’t have any valuable metrics They aren’t accessible There is no awareness
  18. devopsdays Cuba 2017 23 Sharing pitfall: private bug tracker Devs

    moved to central bug and task tracker. Devs manage their system with pride! Ops on different system that isn’t maintained and barely used. ➔ Ops shifted to central system ➔ Exponential increase in visibility of work ➔ Some teams have private projects, some of them for no good reason (secrecy?)
  19. devopsdays Cuba 2017 24 Sharing pitfall: private chat channel open

    Jabber, but invite only channels Then, 2 Slack teams and more Little use by devs of Jabber ➔ Global Slack team, without official consent ➔ After two months, one system left More people than ever were using it ➔ Some suggested private Slack channels per team, but we did not allow ➔ After 8 months got management buy- in and moved to paid version
  20. devopsdays Cuba 2017 25 Sharing pitfall: private repositories Configuration management

    system repository fully locked down is counter productive Because of: secrets, lack of trust Giving access doesn’t mean people will read and contribute ➔ Removed secrets from our repository ➔ Started an opt-in policy ➔ Devs are sending pull requests instead of change request
  21. devopsdays Cuba 2017 26 Remember this Culture Transformational leadership Implement

    processes that enable developer productivity, reducing code deployment lead times and support more reliable infrastructure Work across organizational silos to achieve strategic alignment Automation The tools are important, but not the most important Sharing Move from closed to open systems Metrics Provide open systems and limit the number of diferent tools
  22. devopsdays Cuba 2017 27 Suggestions for Free Software tools Config

    management Puppet Ansible Continuous integration and deployment Jenkins, Gitlab Repositories Gitlab Package management Aptly Pulp Monitoring Icinga2 Collaboration Mattermost Dokuwiki