DevOpsDays Cuba 2017: Continuous Delivery with Gitlab Apache Mesos and Marathon

DevOpsDays Cuba 2017: Continuous Delivery with Gitlab Apache Mesos and Marathon

Author: Juan Carlos Gomez Correa
Summary: In the last few years software companies have been forced to shrink their development cycles. At Datys we start adopting agile practices in Development. Continuous integration allowed the development team to remove delays due to integration problems. But Operations were left behind… Adopting containers and automating our deployment process using open source technologies, allowed us to release applications and services multiple times a day. In this presentation we share our experiences in this journey. We will show how we set up Continuous Delivery pipelines with Gitlab, Apache Mesos and Marathon. We hope that the real-world story of a small company overcoming the challenges we faced can help others to continue in their DevOps journey.

D5db2dc3cc883df3479797edb63b581b?s=128

DevOpsDays Cuba

October 26, 2017
Tweet

Transcript

  1. Continuous Delivery with Gitlab, Apache Mesos and Marathon Juan Carlos

    Gómez Correa @jcgomezcorrea1 juan-carlos-gomez-correa Ops team at
  2. About the team 20 developers (2 dev teams + 1

    ops)
  3. What is Continuous Delivery? Continuous Delivery is the ability to

    get changes of all types, including new features, configuration changes, bug fixes and experiments, into production, or into the hands of users, safely and quickly in a sustainable way. https://continuousdelivery.com
  4. What is Continuous Delivery? We achieve all this by ensuring

    our code is always in a deployable state […] https://continuousdelivery.com
  5. The pipeline Unit Test Acceptance Test Deliver to Staging Integration

    Tests, Load, Performance, Security Deliver to Production Post Deploy Test Auto Manual Not yet
  6. About Apache Mesos, Docker and Marathon Mesos Centralized fault-tolerant cluster

    manager designed for distributed computing environments to provide efficient resource isolation and management across a cluster Docker Package an application with its dependencies into a standardized unit Marathon Container orchestration platform on Mesos (PaAS)
  7. Marathon user interface

  8. Marathon REST API curl -H 'Content-Type: application/json' -X POST -d@

    { "id": "my-service", "instances": 2, "cpus": 1, "mem": 1024, "container": { docker": { "image": "my-app:1.0" … }, … }, … }
  9. Gitlab Integrated product for the entire software development lifecycle •

    Features: Plan, Create, Verify, Package, Release, Configure, Monitor • Built-in CI/CD (gitlab-ci.yml) • Built-in docker registry
  10. Gitlab CI/CD

  11. Branch: issue-* Branch: master Gitlab CI/CD

  12. Docker Registry

  13. git push push Job: build

  14. Job: deploy PUT app.json deployment monitoring

  15. Source code structure

  16. gitlab-ci.yml stages: - build - test - deploy before_script: -

    docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
  17. gitlab-ci.yml variables: MARATHON_REST_URI: "http://mm.datys.cu:8080/v2" INSTANCES: "1" CPU: "1" MEMORY: "512"

  18. gitlab-ci.yml build: script: - /bin/bash .ci/build except: - tags stage:

    build
  19. gitlab-ci.yml build: script: - /bin/bash .ci/build except: - tags stage:

    build [file: ci/build] image=$CI_REGISTRY/my_project/${CI_BUILD_REF:0:8} docker build -t $image . docker push $image
  20. gitlab-ci.yml test: script: - /bin/bash .ci/test artifacts: paths: - coverage/

    except: - tags stage: test
  21. gitlab-ci.yml review: script: - APP_ID=my_project/review_${CI_BUILD_REF_NAME} DATABASE=my_project_review_${CI_BUILD_REF_NAME} /bin/bash .ci/deploy only: -

    /^issue-.*$/ stage: deploy
  22. Reviews

  23. gitlab-ci.yml staging: script: - APP_ID=my_project/staging DATABASE=my_project_staging /bin/bash .ci/deploy only: -

    master stage: deploy
  24. Lessons we learned • Principles are more important than tools

    • Evolution vs. revolution • Ops team as enablers • Automation mindset • Stay constantly learning, adapting and evolving