Upgrade to Pro — share decks privately, control downloads, hide ads and more …

DevOps for Dinosaurs

DevOps for Dinosaurs

My experience in applying DevOps at a large traditional enterprise.

The recorded video of this presentation can be found here: https://vimeo.com/62491557

For more details on this topic see my blog http://niek.bartholomeus.be.

Niek Bartholomeus

March 15, 2013
Tweet

More Decks by Niek Bartholomeus

Other Decks in Technology

Transcript

  1. DevOps
    For
    Dinosaurs

    View Slide

  2. Niek
    Developer
    Technical lead
    Software
    architect
    Software
    delivery
    I have a background as a developer, technical lead and software architect in traditional enterprises.
    In my last mission I was also responsible for the company-wide software delivery process, at least the development side, in
    a financial institution.

    View Slide

  3. Just released
    DevOps =
    Monitoring
    This is my first presentation to a public audience so this feels as my personal release into the wild.
    Because I want to be a good student of DevOps I have added some monitoring. Let us have a look at the results at the end
    of the presentation!

    View Slide

  4. Traditional enterprise
    Traditional
    software delivery
    In this presentation I will talk about my experience in improving the traditional software delivery process of this financial
    institution.
    First I will describe how the IT department was organized.
    Then I will give you a closer look at the biggest problems that existed.
    And finally I will explain the first steps we took to solve the problems.
    But first of all let me give you a bit of theory on my view.

    View Slide

  5. Theory

    View Slide

  6. Dev Ops
    Business
    First a bit of theory.
    Business seems to have conflicting requirements regarding software delivery:
    ! (click)
    ! they want to have their features implemented as fast as possible and for this they put pressure on the development
    teams
    ! and
    ! (click)
    ! they want to keep the existing environment as reliable as possible and for this they put pressure on the operations teams
    But this is not possible with the current software delivery process:
    ! if you want the features implemented fast this will have an impact on the stability of the environment
    ! if you want to keep a reliable environment you will have to slow down the changes
    (click)
    The solution is to redefine the software delivery process, to DevOps-ify it, so it is able to support both requirements.

    View Slide

  7. Dev Ops
    Fast
    Business
    First a bit of theory.
    Business seems to have conflicting requirements regarding software delivery:
    ! (click)
    ! they want to have their features implemented as fast as possible and for this they put pressure on the development
    teams
    ! and
    ! (click)
    ! they want to keep the existing environment as reliable as possible and for this they put pressure on the operations teams
    But this is not possible with the current software delivery process:
    ! if you want the features implemented fast this will have an impact on the stability of the environment
    ! if you want to keep a reliable environment you will have to slow down the changes
    (click)
    The solution is to redefine the software delivery process, to DevOps-ify it, so it is able to support both requirements.

    View Slide

  8. Dev Ops
    Fast
    Business
    Reliable
    First a bit of theory.
    Business seems to have conflicting requirements regarding software delivery:
    ! (click)
    ! they want to have their features implemented as fast as possible and for this they put pressure on the development
    teams
    ! and
    ! (click)
    ! they want to keep the existing environment as reliable as possible and for this they put pressure on the operations teams
    But this is not possible with the current software delivery process:
    ! if you want the features implemented fast this will have an impact on the stability of the environment
    ! if you want to keep a reliable environment you will have to slow down the changes
    (click)
    The solution is to redefine the software delivery process, to DevOps-ify it, so it is able to support both requirements.

    View Slide

  9. Dev Ops
    Fast
    Business
    Reliable
    First a bit of theory.
    Business seems to have conflicting requirements regarding software delivery:
    ! (click)
    ! they want to have their features implemented as fast as possible and for this they put pressure on the development
    teams
    ! and
    ! (click)
    ! they want to keep the existing environment as reliable as possible and for this they put pressure on the operations teams
    But this is not possible with the current software delivery process:
    ! if you want the features implemented fast this will have an impact on the stability of the environment
    ! if you want to keep a reliable environment you will have to slow down the changes
    (click)
    The solution is to redefine the software delivery process, to DevOps-ify it, so it is able to support both requirements.

    View Slide

  10. Process
    Tools
    Culture
    DevOps
    How should we redefine it?
    Let us have a look at the three levels that make up the software delivery process:
    The process itself
    ! The process should be as simple and consistent as possible to get the job done and should be clear for everyone
    The tools to implement it
    ! The process must be automated where possible, especially where things must happen fast or in high volume
    The culture - the people that use the process and the tools
    ! It is well known that people resist change and therefore this level is the most difficult and time consuming one.
    ! But it is also the most rewarding one:
    ! ! if you can convince them of the value that the change brings you will likely succeed
    ! ! if not even the best processes and tools won't get you anywhere
    ! !
    !

    View Slide

  11. Characteristics

    View Slide

  12. Buy before build
    OK let us see how we can apply the theory to the problems that existed in the software delivery process of the traditional
    enterprise I worked for.
    But first let me first give you some insight into how the IT department was organized.
    Let me start with the area of Strategy and Architecture.
    The general rule was to “buy before build”: first look on the market to buy a commercial application and only if nothing exists
    build it in-house.
    This rule had a big impact on the whole organization of the IT department.

    View Slide

  13. Heterogeneous
    environment
    Buy before build
    First of all this caused a heterogeneous environment with many different technologies, modern and not so modern.
    Although there were some rules to which these applications had to comply each application had its own technology stack,
    mode of installation, way of configuration, way of testing, ...

    View Slide

  14. Focus on integration
    Buy before build
    Heterogeneous
    environment
    The second consequence was that a lot of focus had to be put on the integration of these applications to make sure that they
    provide a consistent service to the business.
    This was done by development tools that specialize in ETL and EAI.
    A central data hub was created between the applications to simplify the integration problems.

    View Slide

  15. Business
    applications
    Data flows
    Here is an overview of the heterogeneous applications and their integrations through the data hub

    View Slide

  16. Top-down process
    frameworks
    Buy before build
    Heterogeneous
    environment
    Focus on integration
    The IT department was driven by “top-down” process frameworks like TOGAF, CMMi and ITIL.
    They were less aware of the more modern, lightweight, bottom-up processes like agile and DevOps.

    View Slide

  17. Heterogeneous dev
    and ops teams
    Buy before build
    Heterogeneous
    environment
    Focus on integration
    Top-down process
    frameworks
    In the area of development and operations the teams were quite heterogeneous.
    You had young people and older people, passionate and less passionate, experienced and newcomers
    Generally accepted practices like unit-testing and logging were certainly not used by all teams, let alone more advanced
    practices like continuous integration, feature flags, etc.

    View Slide

  18. Low frequency of
    releases
    Buy before build
    Heterogeneous
    environment
    Focus on integration
    Top-down process
    frameworks
    Heterogeneous dev
    and ops teams
    There were company-wide monthly (and for the core applications quarterly) releases with cold deployments during the
    weekend.

    View Slide

  19. Manual work
    Buy before build
    Heterogeneous
    environment
    Focus on integration
    Top-down process
    frameworks
    Heterogeneous dev
    and ops teams
    Low frequency of
    releases
    There was a lot of manual work involved in the software delivery process

    View Slide

  20. Version
    control
    Build
    automation
    Change
    management
    Deployment
    scripts
    Release
    management
    Testing
    Configuration
    management
    = automated process
    = manual process
    There existed a change management tool and shared version control tools and build and deployment scripts for the
    most used development technologies.
    But all the other activities were manual.
    Configuration management like knowing which versions of software component belonged together, which versions were
    installed in which environment, etc had to be tracked manually.
    The developers filled in the deployment requests in Word templates and mailed them to the release coordinator who did a
    validation and added it to the release plan. At the day of deployment he sent it to the ops teams for deployment.
    Testing was also a manual process which happened at the end of the process in a shared environment using a full copy of
    the production data.

    View Slide

  21. Attention points
    12 minutes

    View Slide

  22. Modern company
    Buy before build
    Heterogeneous
    environment
    Focus on integration
    Top-down process
    frameworks
    Heterogeneous dev
    and ops teams
    Manual work
    Low frequency of
    releases
    Lightweight bottom-
    up processes
    One or a few
    decoupled products
    Built with
    automation in mind
    Modern technologies
    More homogeneous
    community
    Higher frequency of
    releases
    One important thing that you may have noted is that traditional enterprises are very different from the modern, younger
    companies like Flickr, Amazon, Facebook etc who ....
    In modern companies automation is built in from the ground up

    View Slide

  23. Applying DevOps to a
    traditional enterprise
    Applying DevOps to a
    modern company
    And because of this difference it seems obvious that applying DevOps to a traditional company can not be done in the same
    way as how the modern companies do it.

    View Slide

  24. Modernized traditional enterprise
    The solution should take into account
    the existing structure
    In traditional companies we have to take into account the existing structure and gradually add the modern extensions to it
    and integrate them with the existing structure
    We should also be very cautious, starting with the biggest and easiest problems and each time verifying that what we are
    trying to do actually makes sense in our context. And this as evolve to a better situation.

    View Slide

  25. Software delivery has an impact on
    many people in many departments
    Making changes to the software delivery process is a huge undertaking because it covers many people and many
    departments, especially as a lot of activities are still very manual.

    View Slide

  26. The solution should be implemented
    slowly and in small steps
    So it is very important that we move in small steps and that we leave enough time for the people to get used to the changes
    before moving on.

    View Slide

  27. Problems
    15 minutes

    View Slide

  28. Release coordination:
    Takes more and more
    time
    Due to the dev and ops teams gradually growing over time and the increased complexity of technologies release
    coordination started taking more and more time.
    This in its turn put more and more pressure on the deployment window and the downstream processes

    View Slide

  29. Deployment instructions:
    Devs have difficulty to
    communicate with ops
    Takes more and more
    time
    Because the deployment instructions had to be written in free text they were very error prone:
    First of all they lacked standardization. Every this had a different way of requesting their deployments.
    Sometimes they were too vague for the ops team to understand, or steps were missing, or there was a small typo ...
    All of this caused a lot of friction between the teams, loss of time and higher risk for creating snowflake servers.

    View Slide

  30. Configuration management:
    Too vague and not
    reliable
    Takes more and more
    time
    Devs have difficulty to
    communicate with ops
    Configuration management was too vague:
    There was no clear list of all business applications and it was difficult to know which components belonged to which
    application.
    Configuration management had to be tracked manually and people regularly made mistakes or just forgot to update the
    information. This caused a lack of confidence in the information.

    View Slide

  31. Testing:
    Manual effort happening
    at the end of the process
    Takes more and more
    time
    Devs have difficulty to
    communicate with ops
    Too vague and not
    reliable
    Testing happened at the end of the process so any delays that occurred upstream immediately shortened the period that was
    foreseen for testing.
    Testing was manual so even with a full testing period it was impossible to do a full regression test.
    For these reasons it happened regularly that change requests could not be signed off by the testers and therefore had to be
    removed from the release. Because of all the integrations a lot of changes depended on one another so this usually caused a
    chain reaction.
    Worst of all: after the impacted changes were all removed there was no time left to retest the remaining environment.

    View Slide

  32. Solution - phase1
    20 minutes

    View Slide

  33. Focus on creating a reliable
    environment
    Start with configuration
    management
    Referring back to the theory of needing a fast and reliable environment, on short term there was a higher need for a more
    reliable environment than for delivering the changes faster so that’s what we had to focus on.
    And it was very clear to me that the solution should start with bringing the configuration management under control. It is the
    core of the system and a lot of building blocks were already available. With a small effort a big improvement could be
    made.

    View Slide

  34. Get mutual agreement on the
    structure of configuration
    management
    The first step was to get mutual agreement on the structure of configuration management.

    View Slide

  35. Business application
    Software component
    Source code
    We decided to create three levels of configuration items: business applications, software components and source code of
    which the changes had to be versioned.
    First a list of business applications was created which was shared between Enterprise Architecture and Service Desk. The
    teams that were responsible for the applications were also clearly listed.
    Change requests and deployment requests could only refer to one application.
    Then the application was statically or “slowly changingly” linked to the components it was made up of.
    And finally we had the source code that contained all files necessary to build the component.

    View Slide

  36. Build a configuration
    management system and
    software repository
    Secondly a configuration management tool was created to manage the store the information about the configuration items and
    a software repository was created to store the deployable files that are related to the software components

    View Slide

  37. Version
    control
    Build
    automation
    Change
    management
    Deployment
    scripts
    Release
    management
    Testing
    = automated process
    = manual process
    Configuration
    management
    In fact this was one tool that implemented both pieces.
    This tool was integrated with the build automations and received the built files (exe’s, dll’s, config files etc) as well as any
    relevant meta data like who executed the build, the commit messages including a reference to the change requests and the
    file diffs.

    View Slide

  38. Version
    control
    Build
    automation
    Change
    management
    Deployment
    scripts
    Release
    management
    Testing
    Configuration
    management
    Software
    repository
    NEW
    NEW
    = automated process
    = manual process
    In fact this was one tool that implemented both pieces.
    This tool was integrated with the build automations and received the built files (exe’s, dll’s, config files etc) as well as any
    relevant meta data like who executed the build, the commit messages including a reference to the change requests and the
    file diffs.

    View Slide

  39. The user interface looked as following.

    View Slide

  40. Solution - phase 2

    View Slide

  41. Implement a release
    management tool
    The second step involved the implementation of a release management tool that was acquired on the market (it is now part of
    BMC and is called RPM: Release Process Manager).
    This tool facilitated the developers with the creation of their deployment requests and the release coordinator with the planning
    and coordination of them.

    View Slide

  42. Version
    control
    Build
    automation
    Change
    management
    Deployment
    scripts
    Testing
    Configuration
    management
    Software
    repository
    = automated process
    = manual process
    Release
    management
    The tool was integrated with the configuration management tool to get the available components and with the change
    management tool to get the change requests

    View Slide

  43. Version
    control
    Build
    automation
    Change
    management
    Deployment
    scripts
    Testing
    Configuration
    management
    Software
    repository
    Release
    management
    NEW
    = automated process
    = manual process
    The tool was integrated with the configuration management tool to get the available components and with the change
    management tool to get the change requests

    View Slide

  44. The user interface looked as following.
    The developer selects an application and a release. Automatically a list of the applicable change requests is shown, including
    their statuses. This made it easier for the release coordinator to validate the deployment requests
    For each deployment of a component he appends a step and selects the applicable version number. Automatically the
    relevant deployment related information is shown.
    The tool was also able to show the versions of a component by environment, both currently as historically.

    View Slide

  45. Finally the configuration management tool was able to get the deployed versions from the release management tool and this
    allowed to provide overviews like these.
    It was also possible to do automated consistency checks like giving a warning when a certain change request was not
    implemented by any component or when a component implemented change requests for multiple releases.

    View Slide

  46. Next steps
    25 minutes

    View Slide

  47. Automate deployments
    Definitely!
    The project did not foresee in the automated deployment of the component, they was still the need for an ops person to
    execute the command of the deployment script. The reason for this was simply that the security requirements for the release
    management tool were way stricter if it needed a connection to all production servers.
    So this was delayed to a later moment.

    View Slide

  48. Automate testing
    Definitely!
    With automated testing we have moved from the relatively simple problems to the more complex ones.
    But the advantage here is that we can start small, focussing on the most complex and volatile areas and gradually extend
    the test coverage to the rest of the environment.
    Ideally this effort should also include the gradual automation of environment creation and server provisioning.
    But the code base is huge and there are many business applications and even development languages that have limited or
    no support for automated testing.
    So it will be a long and costly journey.

    View Slide

  49. More frequent releases ?
    Probably!
    As the software delivery process gets more and more controlled and automated I expect the release frequency to raise and
    two week releases should definitely become possible.
    It may also become more interesting to allow applications to be released on their own schedule as long as they are not too
    much integrated with other apps.

    View Slide

  50. Continuous delivery ?
    Maybe ...
    Applying continuous delivery in a traditional enterprise? I’m not so sure because this would require changing the core of the
    enterprise and the culture of the people.
    Maybe this is something that has to be built in from the start and adding it as an afterthought would take too much effort for
    what it delivers.

    View Slide

  51. We have a long and challenging
    journey ahead

    View Slide

  52. Can traditional enterprises
    adapt to the changing world or
    do they face extinction ?

    View Slide

  53. Just released
    DevOps =
    Measuring

    View Slide

  54. Niek Bartholomeus
    @niekbartho
    http://niek.bartholomeus.be
    [email protected]

    View Slide