$30 off During Our Annual Pro Sale. View Details »

DevOpsDays Cuba 2017: DEVOPS PITFALLS

DevOpsDays Cuba
October 26, 2017
280

DevOpsDays Cuba 2017: DEVOPS PITFALLS

Author: Rudy Gevaert
Summary: A story on how we failed to implement devops but saved our team from self destruction

DevOpsDays Cuba

October 26, 2017
Tweet

More Decks by DevOpsDays Cuba

Transcript

  1. DEVOPS PITFALLS
    A story on how we failed to implement devops but saved our
    team from self destruction
    Rudy Gevaert – [email protected] - @rgevaert – http://www.webworm.be

    View Slide

  2. devopsdays Cuba 2017
    2
    About me
    Rudy Gevaert
    Ghent – Belgium
    @rgevaert - http://www.webworm.be
    GNU/Linux user since 2000
    Sustainable ICT for development in
    Cuba and Ethiopia
    Sysadmin at Ghent University
    Devopsdays Belgium and Cuba

    View Slide

  3. devopsdays Cuba 2017
    3
    Table of contents
    Setting the scene
    Start to devops
    Where we are now
    Devops pitfalls
    Summary

    View Slide

  4. devopsdays Cuba 2017
    4
    Setting the scene: Ghent University
    www.ugent.be
    established 1817
    > 50.000 students
    10.000 staf
    Campus in South-Korea

    View Slide

  5. devopsdays Cuba 2017
    5
    Setting the scene...
    DICT: Department of ICT
    120 staf
    Diferent offices
    Context:
    Unix-team vs diferent other
    teams

    View Slide

  6. devopsdays Cuba 2017
    6
    Start to devops
    2008
    start of configuration management
    2009
    first ever devopsdays in Ghent
    2010
    uptake of cfgmgt by some team members
    ...
    2015:
    some developers start commiting into VCS
    first cross department multi functional team
    2016:
    real commitment to use configuration
    management in Unix team
    Inter and extra team situation:

    View Slide

  7. devopsdays Cuba 2017
    7
    Where are we now?
    2016
    Unix team with renewed spirit and motivation
    2017
    original cross functional team is not functioning
    anymore

    View Slide

  8. devopsdays Cuba 2017
    8
    Some achievements

    Widespread adoption CI en CD

    Configuration management is mandatory

    CI en CD implemented for Puppet implemented

    Automated security updates are being rolled out

    Documentation plays key role in all new setups

    Kanban to visualize work

    Post mortem process is standard

    View Slide

  9. devopsdays Cuba 2017
    9
    What is devops?

    View Slide

  10. devopsdays Cuba 2017
    10
    What is devops?

    View Slide

  11. devopsdays Cuba 2017
    11
    Key tipping points

    View Slide

  12. devopsdays Cuba 2017
    12
    The not so definitive list of devops
    pitfalls

    View Slide

  13. devopsdays Cuba 2017
    13
    Cultural pitfall: failing to define devops
    Leads to:

    The developer will become sysadmin

    Devops ⇔ tool x && tool y

    Devops ⇒ agile ⇒ no planning

    Devops is about communication, but we have
    difficulty with it

    You need common understanding so nobody is left
    behind and expections can be met

    View Slide

  14. devopsdays Cuba 2017
    14
    Cultural pitfall: expecting devops to fix
    bad leadership
    “By 2020, half of the CIOs who have not transformed
    their teams’ capabilities will be displaced from their
    organizations digital leadership teams.”1
    Bottom up and bad leadership:
    very difficult and not fruitful in the long run

    We had to go through several crisis's to just reach
    where we are now

    Depending on the context:
    leadership will or can not be changed
    1 Gartner Predicts: IT infrastructure & Operations

    View Slide

  15. devopsdays Cuba 2017
    15
    Cultural pitfall: lack of skilled people
    with the correct mindset

    Sysadmins with the wrong mindset left the team

    Devs with the good mindset left the team

    New team members had the correct skills and
    mindset

    Building human capitial is frequently forgotten.
    Everybody (good or bad) can leave the team

    Enable running the race at their own pace

    View Slide

  16. devopsdays Cuba 2017
    16
    Cultural pitfall: not defining your goal
    What is the goal of the
    enterprise?
    Make money?
    But what when it is something
    else?
    ➔Devops implementation
    should be aimed at reaching
    the goal

    View Slide

  17. devopsdays Cuba 2017
    17
    Cultural pitfall: lack of

    View Slide

  18. devopsdays Cuba 2017
    18
    Automation pitfall:
    it’s only about the tool
    Devops is more than using the latest tool
    It comes second to:

    Understand why to use a specific tool:
    efficiency, cost reduction, risk mitigation, and
    predictability

    Prevent sprawl of tools

    View Slide

  19. devopsdays Cuba 2017
    19
    Automation pitfall:
    automate all of it now
    You don’t need to automate everything (at once).

    Document the process first

    We made documentation an integral and mandatory
    part of product development

    Automate predictable and error prone tasks

    Increase automation at the bottleneck

    View Slide

  20. devopsdays Cuba 2017
    20
    Automation pitfall: failing to adhere to
    the core tenets
    The core tenets of automation in
    devops are

    automated testing,

    continuous integration and

    integrated configuration
    management
    Compromising will lead to a
    slippery slope.

    View Slide

  21. devopsdays Cuba 2017
    21
    Metrics pitfall:
    set up different monitoring tools
    Sysadmin team had excellent monitoring tool, but:

    It was fully locked down

    Didn’t monitor development machines

    Only sent alerts to sysadmins
    Dev teams have their own monitoring solution...

    Sysadmin team is setting up fully open system

    Developers couldn’t wait and rolled v2 of their setup

    View Slide

  22. devopsdays Cuba 2017
    22
    Metrics pitfall:
    nobody uses the metrics setup
    Because they don’t have any valuable metrics
    They aren’t accessible
    There is no awareness

    View Slide

  23. devopsdays Cuba 2017
    23
    Sharing pitfall: private bug tracker
    Devs moved to central bug and task
    tracker.
    Devs manage their system with pride!
    Ops on different system that isn’t
    maintained and barely used.

    Ops shifted to central system

    Exponential increase in visibility of
    work

    Some teams have private projects,
    some of them for no good reason
    (secrecy?)

    View Slide

  24. devopsdays Cuba 2017
    24
    Sharing pitfall: private chat channel
    open Jabber, but invite only channels
    Then, 2 Slack teams and more
    Little use by devs of Jabber

    Global Slack team, without official
    consent

    After two months, one system left
    More people than ever were using it

    Some suggested private Slack
    channels per team, but we did not
    allow

    After 8 months got management buy-
    in and moved to paid version

    View Slide

  25. devopsdays Cuba 2017
    25
    Sharing pitfall: private repositories
    Configuration management system repository fully
    locked down is counter productive
    Because of: secrets, lack of trust
    Giving access doesn’t mean people will read and
    contribute

    Removed secrets from our repository

    Started an opt-in policy

    Devs are sending pull requests instead of change
    request

    View Slide

  26. devopsdays Cuba 2017
    26
    Remember this
    Culture
    Transformational leadership
    Implement processes that
    enable developer productivity,
    reducing code deployment lead
    times and support more reliable
    infrastructure
    Work across organizational silos
    to achieve strategic alignment
    Automation
    The tools are important, but
    not the most important
    Sharing
    Move from closed to open systems
    Metrics
    Provide open systems and limit
    the number of diferent tools

    View Slide

  27. devopsdays Cuba 2017
    27
    Suggestions for Free Software tools
    Config
    management
    Puppet
    Ansible
    Continuous
    integration and
    deployment
    Jenkins, Gitlab
    Repositories
    Gitlab
    Package
    management
    Aptly
    Pulp
    Monitoring
    Icinga2
    Collaboration
    Mattermost
    Dokuwiki

    View Slide

  28. devopsdays Cuba 2017
    28
    Further reading

    View Slide

  29. devopsdays Cuba 2017
    29
    One last word

    View Slide

  30. devopsdays Cuba 2017
    30
    Preguntas?

    View Slide