Upgrade to Pro — share decks privately, control downloads, hide ads and more …

DevOpsDays Cuba 2016: DevOps from the cuban trenches

DevOpsDays Cuba 2016: DevOps from the cuban trenches

Author: Anesto del Toro Almenares
Summary:
We’ll share experiences during the implementation of our current software delivery system. Our initial goal was to achieve faster development and deployment cycles of quality products, in a reliable and sustainable way. The main challenges we faced and our achievements in terms of product planning, feature development, continuous integration, change and configuration management and continuous delivery will be exposed during the presentation.

DevOpsDays Cuba

October 19, 2016
Tweet

More Decks by DevOpsDays Cuba

Other Decks in Technology

Transcript

  1. DevOps from Cuban
    trenches

    View full-size slide

  2. We have challenges

    View full-size slide

  3. … configure to block access from your country

    View full-size slide

  4. Slooooooow internet access

    View full-size slide

  5. What can we achieve despite of this?

    View full-size slide

  6. DevOps from Cuban
    trenches

    View full-size slide

  7. DevOps from Cuban
    trenches

    View full-size slide

  8. DevOps from DATYS VC
    trenches

    View full-size slide

  9. 25
    Team
    Villa Clara Division

    View full-size slide

  10. 25
    Team
    1
    3
    Villa Clara Division
    1
    6
    Head of Production
    Team Lead
    Team members
    QA, Docs,…
    1
    6
    1
    6
    Ops*
    Dev

    View full-size slide

  11. About Me
    @anestodta
    [email protected]
    https://www.linkedin.com/in/anesto-del-toro
    • Head of Production, DATYS VC
    • Computer Sciences
    • Wearing multiple hats
    oProject Management
    o Business Analysis
    o Architecture
    o Technical Lead
    o Software Development (mostly backend)
    o Lean-Agile enthusiast
    o Eager learner
    • 16 years of experience
    o ~6 Researcher at UCLV (BE, PR & CV)
    Anesto del Toro Almenares

    View full-size slide

  12. About the TEAM

    View full-size slide

  13. About the TEAM
    • Disciplined
    • Collaboration culture
    • Automation mindset*

    View full-size slide

  14. About the TEAM
    @MarialinaBall
    @kikicarbonell
    @jcgomezcorrea1
    @DuvielRodriguez
    @dayron_aj
    @christian861220
    @genrylg
    @osmel_mora
    • Disciplined
    • Collaboration culture
    • Automation mindset*

    View full-size slide

  15. Up to 10 deploys/releases* per day

    View full-size slide

  16. Back to 2007 How bad could it be?

    View full-size slide

  17. manual software builDing

    View full-size slide

  18. manual software builDing
    manual Testing after development

    View full-size slide

  19. manual software builDing
    no artifacts Management
    manual Testing after development

    View full-size slide

  20. manual software builDing
    painful intEgrations
    no artifacts Management
    manual Testing after development

    View full-size slide

  21. manual proVisioning
    manual software builDing
    painful intEgrations
    no artifacts Management
    manual Testing after development

    View full-size slide

  22. manual proVisioning
    manual software builDing
    painful intEgrations
    no artifacts Management
    manual deplOyment
    manual Testing after development

    View full-size slide

  23. manual proVisioning
    poorLy controlled processes
    manual software builDing
    painful intEgrations
    no artifacts Management
    manual deplOyment
    manual Testing after development

    View full-size slide

  24. manual proVisioning
    manual software builDing
    painful intEgrations
    no artifacts Management
    manual deplOyment
    manual Testing after development
    infrequent Releases
    poorLy controlled processes

    View full-size slide

  25. manual proVisioning
    lOng feedback cycles
    manual software builDing
    painful intEgrations
    no artifacts Management
    manual deplOyment
    infrequent Releases
    manual Testing after development
    poorLy controlled processes

    View full-size slide

  26. manual proVisioning
    lOng feedback cycles
    manual software builDing
    painful intEgrations
    no artifacts Management
    manual deplOyment
    infrequent Releases
    manual Testing after development
    poorLy controlled processes

    View full-size slide

  27. At first sigh it looked scary

    View full-size slide

  28. Worsening effect

    View full-size slide

  29. Software Delivery System
    Achieve faster development and deployment cycles of
    quality products, in a reliable and sustainable way
    Enterprise Agility

    View full-size slide

  30. Jez Humble and David Farley
    Continuous Delivery
    A Maturity Assessment Model

    View full-size slide

  31. • Build management and Continuous Integration
    • Environment and deployments
    • Testing
    • Release management and compliance
    • Configuration management
    • Data management
    Continuous Delivery
    A Maturity Assessment Model

    View full-size slide

  32. "We can't solve problems by using the same kind
    of thinking we used when we created them."
    Albert Einstein

    View full-size slide

  33. Principles Practices

    View full-size slide

  34. Principles Practices Tools

    View full-size slide

  35. Principles Practices Tools
    Order matters

    View full-size slide

  36. Identify Evaluate
    Adopt /
    Discard
    Try
    Improve

    View full-size slide

  37. Identify Evaluate
    Adopt /
    Discard
    Try
    Improve
    Context matters

    View full-size slide

  38. Harry potter entrenando order de fenix
    Evangelism

    View full-size slide

  39. Don’t give up

    View full-size slide

  40. •Optimize the whole
    •Eliminate waste
    •Build Quality-In
    •Deliver Fast
    •Focus on Flow
    LEAN Principles
    • Defer committment
    • Amplified learning
    • Engage Everyone
    • Continuous Improvement
    Mary and Tom Poppendieck

    View full-size slide

  41. Removing waste
    Reducing time to
    customer
    feedback
    Accelerating
    software
    delivery
    Lean guiding transition to Agility (context creation)
    Lean-Agile

    View full-size slide

  42. Agile
    Practices
    focused on iteratively
    producing functionality
    Agility
    Business Value Increments
    not development cycles

    View full-size slide

  43. Technical and
    Engineering
    Practices

    View full-size slide

  44. Focus on optimize value creation
    Holistic view

    View full-size slide

  45. Value Stream
    Ideas
    Materialize
    MBIs
    Specify
    Prioritize Dev
    Review
    UAT Document Deploy
    Version

    View full-size slide

  46. Value Stream and Actors
    Ideas
    Materialize
    MBIs
    Specify
    Prioritize Dev
    Review
    UAT Document Deploy
    Version
    Head of
    Production
    Team
    Lead
    Dev
    Teams
    Ops
    Team
    QA&Docs
    Team

    View full-size slide

  47. Value Stream and Actors
    Ideas
    Materialize
    MBIs
    Specify
    Prioritize Dev
    Review
    UAT Document Deploy
    Version
    Head of
    Production
    Team
    Lead
    Dev
    Teams
    Ops
    Team
    QA&Docs
    Team
    • Workflow
    • Bottlenecks
    • Teamwork
    • …

    View full-size slide

  48. A system is never the sum of its parts. It is the
    product of the interactions of its parts
    Russell Ackoff

    View full-size slide

  49. Ideas
    Materialize
    MBIs
    Specify
    Prioritize Dev
    Review
    UAT Document Deploy
    Version

    View full-size slide

  50. Ideas
    Materialize
    MBIs
    Specify
    Prioritize Dev
    Review
    UAT Document Deploy
    Version
    Start
    End
    Cycle Time

    View full-size slide

  51. Ideas
    Materialize
    MBIs
    Specify
    Prioritize Dev
    Review
    UAT Document Deploy
    Version
    Reducing the Cycle Time requires the
    optimization of the whole Value Stream
    Start
    End
    Cycle Time

    View full-size slide

  52. Ideas
    Materialize
    MBIs
    Specify
    Prioritize Dev
    Review
    UAT Document Deploy
    Version
    Reducing the Cycle Time requires the
    optimization of the whole Value Stream
    Start
    End
    Cycle Time
    Optimize what
    we do?
    Remove delays
    between what
    we do?

    View full-size slide

  53. Ideas
    Materialize
    MBIs
    Specify
    Prioritize Dev
    Review
    UAT Document Deploy
    Version
    Speed
    without
    sacrificing
    quality
    Time To Market

    View full-size slide

  54. Ideas
    Materialize
    MBIs
    Specify
    Prioritize Dev
    Review
    UAT Document Deploy
    Version
    Speed
    without
    sacrificing
    quality
    Explicit policies
    =
    Definition of Done
    +
    Pull criteria

    View full-size slide

  55. Map the mess
    Empower the team to fix/improve it

    View full-size slide

  56. Increment
    Review
    User Acceptance
    Testing
    Documentation
    Development &
    Operations
    Management &
    Planning
    Knowledge sharing/retention
    Automation
    Self-services
    Code
    repository
    Platform as a
    Service
    Artifacts and dependencies repositories
    Stakeholders
    Internet
    Production System

    View full-size slide

  57. Increment
    Review
    User Acceptance
    Testing
    Documentation
    Development &
    Operations
    Management &
    Planning
    Knowledge sharing/retention
    Automation
    Self-services
    Code
    repository
    Platform as a
    Service
    Artifacts and dependencies repositories
    Stakeholders
    Internet
    Production System

    View full-size slide

  58. Business Value Increments
    Predictability

    View full-size slide

  59. LEVEL 1 LEVEL 2 LEVEL 3
    Milestone 01
    PROJECT XYZ
    Milestone 01
    PROJECT XYZ
    Milestone 01
    PROJECT XYZ
    High Level Goals MBIs Tasks
    ANALYSIS
    SPECIFICATION
    Tasks
    • Pull-System
    • Incremental & Iterative
    • Continuous Delivery
    Management and Planning

    View full-size slide

  60. • MBIs Mean resolution time: 15 days
    • 90 % MBIs takes at most 30 days
    • 98 % MBIs takes at most 34 días
    • …
    Predictability
    Analysis by Class of Service

    View full-size slide

  61. Increment
    Review
    User Acceptance
    Testing
    Documentation
    Development &
    Operations
    Management &
    Planning
    Knowledge sharing/retention
    Automation
    Self-services
    Code
    repository
    Platform as a
    Service
    Artifacts and dependencies repositories
    Stakeholders
    Internet
    Production System

    View full-size slide

  62. Technical agility
    Producing functionality

    View full-size slide

  63. Next reléase from this point
    forward
    Merge bugfix
    Start RC 1.0
    bugfixes
    Tag 0.1
    Tag 0.2
    Tag 1.0
    Features for Next
    release
    Bug fixed
    production: hotfix
    0.2
    Merge bugfixes from RC to
    develop
    Future feature
    Time
    Features branches Develop RC hotfixes master
    Practices
    • Branch per feature (short lived)
    • Sandboxing
    • Self-service CI & Deploy
    • Blessed repositories
    Development activity

    View full-size slide

  64. Minutes
    Weeks
    Production
    Staging
    UAT
    Security
    Increment
    deploy
    Unit
    Testing
    Acceptance
    tests
    Code Reviews
    Code metrics
    (*) Each cycle targets a
    different concern
    Feedback loops

    View full-size slide

  65. Ops Team as Enablers

    View full-size slide

  66. Deployment automation
    Infrastructure as Code
    Supporting services
    Monitoring
    Security

    TDI

    View full-size slide

  67. Collaboration materialized
    Improving efficiency
    Team happiness

    View full-size slide

  68. Increment
    Review
    User Acceptance
    Testing
    Documentation
    Development &
    Operations
    Management &
    Planning
    Knowledge sharing/retention
    Automation
    Self-services
    Code
    repository
    Platform as a
    Service
    Artifacts and dependencies repositories
    Stakeholders
    Internet
    Production System

    View full-size slide

  69. Build the thing Right

    View full-size slide

  70. Technical debt

    View full-size slide

  71. Maintainability
    Testability

    View full-size slide

  72. 99.74%
    Unit: 5086
    Integration: 3485
    ~ 25 min

    View full-size slide

  73. Increment
    Review
    User Acceptance
    Testing
    Documentation
    Development &
    Operations
    Management &
    Planning
    Knowledge sharing/retention
    Automation
    Self-services
    Code
    repository
    Platform as a
    Service
    Artifacts and dependencies repositories
    Stakeholders
    Internet
    Production System

    View full-size slide

  74. Did we built the
    Right product?

    View full-size slide

  75. Root cause analysis

    View full-size slide

  76. Learn from failures

    View full-size slide

  77. PRODUCTION SYSTEM
    Increment
    review
    User Acceptance
    Testing
    Documentation
    Development &
    Operations
    Management &
    Planning
    Knowledge sharing/retention
    Automation, Self-
    services,
    Code hosting
    Platform as a
    Service
    Artifact and dependencies repositories
    Stakeholders
    Internet

    View full-size slide

  78. Docs automation

    View full-size slide

  79. Docs as Code

    View full-size slide

  80. Docs and releases alignment
    Known workflow

    View full-size slide

  81. PRODUCTION SYSTEM
    Increment
    review
    User Acceptance
    Testing
    Documentation
    Development &
    Operations
    Management &
    Planning
    Knowledge sharing/retention
    Automation, Self-
    services,
    Code hosting
    Platform as a
    Service
    Artifact and dependencies repositories
    Stakeholders
    Internet

    View full-size slide

  82. (supporting services + products)

    View full-size slide

  83. Game changer
    Better resource utilization
    Simplified provisioning
    Uniform deployment
    Failure tolerance
    Scalability

    View full-size slide

  84. Bring Ops tools closer to Developers

    View full-size slide

  85. Increment
    Review
    User Acceptance
    Testing
    Documentation
    Development &
    Operations
    Management &
    Planning
    Knowledge sharing/retention
    Automation
    Self-services
    Code
    repository
    Platform as a
    Service
    Artifacts and dependencies repositories
    Stakeholders
    Internet
    Production System

    View full-size slide

  86. UAT
    Deploy
    Production
    like Deploy
    Unit
    tests
    Integration
    tests
    Code &
    Vulnerabilities
    Analysis
    Acceptance
    tests Load,
    Smoke,
    Security
    tests

    Exploratory
    Production
    deploy
    Deployment pipeline
    Up to 10 deploys/releases* per day

    View full-size slide

  87. Assessment
    2016

    View full-size slide

  88. Continuous Delivery: A Maturity Assessment Model

    View full-size slide

  89. Not designed a priori

    View full-size slide

  90. Not designed a priori
    Result of evolution

    View full-size slide

  91. Context matters

    View full-size slide

  92. Context matters
    Challenge principles,
    practices and tools

    View full-size slide

  93. Reality changes

    View full-size slide

  94. Stay constantly learning,
    adapting and evolving
    Reality changes

    View full-size slide

  95. Value creation
    People
    Culture
    Automation mindset
    Focus

    View full-size slide

  96. Results are not the solely point, but the development of People
    and Systems that allows us to achieve the results we expect

    View full-size slide

  97. Stay alert
    Endless trip

    View full-size slide

  98. It is supposed to be FUN!!!

    View full-size slide

  99. DevOps from Cuban
    trenches
    @anestodta
    [email protected]
    https://www.linkedin.com/in/anesto-del-toro
    Anesto
    del Toro Almenares
    Thank you

    View full-size slide