DevOpsDays Cuba 2017: Continuous Delivery with Gitlab Apache Mesos and Marathon

Author: Juan Carlos Gomez Correa
Summary: In the last few years software companies have been forced to shrink their development cycles. At Datys we start adopting agile practices in Development. Continuous integration allowed the development team to remove delays due to integration problems. But Operations were left behind… Adopting containers and automating our deployment process using open source technologies, allowed us to release applications and services multiple times a day. In this presentation we share our experiences in this journey. We will show how we set up Continuous Delivery pipelines with Gitlab, Apache Mesos and Marathon. We hope that the real-world story of a small company overcoming the challenges we faced can help others to continue in their DevOps journey.

Transcript

1. Continuous Delivery with
   Gitlab, Apache Mesos
   and Marathon
   Juan Carlos Gómez Correa
   @jcgomezcorrea1
   juan-carlos-gomez-correa
   Ops team at
   

   View Slide

2. About the team
   20 developers (2 dev teams + 1 ops)
   

   View Slide

3. …
   

   View Slide

4. What is Continuous Delivery?
   Continuous Delivery is the ability to get changes of all types, including
   new features, configuration changes, bug fixes and experiments, into
   production, or into the hands of users, safely and quickly in a
   sustainable way.
   https://continuousdelivery.com
   

   View Slide

5. What is Continuous Delivery?
   We achieve all this by ensuring our code is always in a deployable
   state […]
   https://continuousdelivery.com
   

   View Slide

6. The pipeline
   Unit Test
   Acceptance
   Test
   Deliver to
   Staging
   Integration
   Tests, Load,
   Performance,
   Security
   Deliver to
   Production
   Post Deploy
   Test
   Auto Manual Not yet
   

   View Slide

7. About Apache Mesos, Docker and Marathon
   Mesos
   Centralized fault-tolerant cluster manager designed for distributed
   computing environments to provide efficient resource isolation and
   management across a cluster
   Docker
   Package an application with its dependencies into a standardized unit
   Marathon
   Container orchestration platform on Mesos (PaAS)
   

   View Slide

8. Marathon user interface
   

   View Slide

9. Marathon REST API
   curl -H 'Content-Type: application/json' -X POST -d@
   {
   "id": "my-service",
   "instances": 2,
   "cpus": 1,
   "mem": 1024,
   "container": {
   docker": { "image": "my-app:1.0" … },
   …
   },
   …
   }
   

   View Slide

10. Gitlab
    Integrated product for the entire software development lifecycle
    • Features: Plan, Create, Verify, Package, Release, Configure, Monitor
    • Built-in CI/CD (gitlab-ci.yml)
    • Built-in docker registry
    

    View Slide

11. Gitlab CI/CD
    

    View Slide

12. Branch: issue-*
    Branch: master
    Gitlab CI/CD
    

    View Slide

13. Docker Registry
    

    View Slide

14. git push
    push
    Job: build
    

    View Slide

15. Job: deploy PUT app.json
    deployment monitoring
    

    View Slide

16. Source code structure
    

    View Slide

17. gitlab-ci.yml
    stages:
    - build
    - test
    - deploy
    before_script:
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD
    $CI_REGISTRY
    

    View Slide

18. gitlab-ci.yml
    variables:
    MARATHON_REST_URI: "http://mm.datys.cu:8080/v2"
    INSTANCES: "1"
    CPU: "1"
    MEMORY: "512"
    …
    

    View Slide

19. gitlab-ci.yml
    build:
    script:
    - /bin/bash .ci/build
    except:
    - tags
    stage: build
    

    View Slide

20. gitlab-ci.yml
    build:
    script:
    - /bin/bash .ci/build
    except:
    - tags
    stage: build
    [file: ci/build]
    image=$CI_REGISTRY/my_project/${CI_BUILD_REF:0:8}
    docker build -t $image .
    docker push $image
    

    View Slide

21. gitlab-ci.yml
    test:
    script:
    - /bin/bash .ci/test
    artifacts:
    paths:
    - coverage/
    except:
    - tags
    stage: test
    

    View Slide

22. gitlab-ci.yml
    review:
    script:
    - APP_ID=my_project/review_${CI_BUILD_REF_NAME}
    DATABASE=my_project_review_${CI_BUILD_REF_NAME}
    /bin/bash .ci/deploy
    only:
    - /^issue-.*$/
    stage: deploy
    

    View Slide

23. Reviews
    

    View Slide

24. gitlab-ci.yml
    staging:
    script:
    - APP_ID=my_project/staging
    DATABASE=my_project_staging
    /bin/bash .ci/deploy
    only:
    - master
    stage: deploy
    

    View Slide

25. Lessons we learned
    • Principles are more important than tools
    • Evolution vs. revolution
    • Ops team as enablers
    • Automation mindset
    • Stay constantly learning, adapting and evolving
    

    View Slide