DevOpsDays Cuba 2017: Continuous Delivery with Gitlab Apache Mesos and Marathon

Author: Juan Carlos Gomez Correa
Summary: In the last few years software companies have been forced to shrink their development cycles. At Datys we start adopting agile practices in Development. Continuous integration allowed the development team to remove delays due to integration problems. But Operations were left behind… Adopting containers and automating our deployment process using open source technologies, allowed us to release applications and services multiple times a day. In this presentation we share our experiences in this journey. We will show how we set up Continuous Delivery pipelines with Gitlab, Apache Mesos and Marathon. We hope that the real-world story of a small company overcoming the challenges we faced can help others to continue in their DevOps journey.

Transcript

1. Continuous Delivery with Gitlab, Apache Mesos and Marathon Juan Carlos Gómez Correa @jcgomezcorrea1 juan-carlos-gomez-correa Ops team at
2. About the team 20 developers (2 dev teams + 1 ops)
3. …
4. What is Continuous Delivery? Continuous Delivery is the ability to get changes of all types, including new features, configuration changes, bug fixes and experiments, into production, or into the hands of users, safely and quickly in a sustainable way. https://continuousdelivery.com
5. What is Continuous Delivery? We achieve all this by ensuring our code is always in a deployable state […] https://continuousdelivery.com
6. The pipeline Unit Test Acceptance Test Deliver to Staging Integration Tests, Load, Performance, Security Deliver to Production Post Deploy Test Auto Manual Not yet
7. About Apache Mesos, Docker and Marathon Mesos Centralized fault-tolerant cluster manager designed for distributed computing environments to provide efficient resource isolation and management across a cluster Docker Package an application with its dependencies into a standardized unit Marathon Container orchestration platform on Mesos (PaAS)
8. Marathon user interface
9. Marathon REST API curl -H 'Content-Type: application/json' -X POST -d@ { "id": "my-service", "instances": 2, "cpus": 1, "mem": 1024, "container": { docker": { "image": "my-app:1.0" … }, … }, … }
10. Gitlab Integrated product for the entire software development lifecycle • Features: Plan, Create, Verify, Package, Release, Configure, Monitor • Built-in CI/CD (gitlab-ci.yml) • Built-in docker registry
11. Gitlab CI/CD
12. Branch: issue-* Branch: master Gitlab CI/CD
13. Docker Registry
14. git push push Job: build
15. Job: deploy PUT app.json deployment monitoring
16. Source code structure
17. gitlab-ci.yml stages: - build - test - deploy before_script: - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
18. gitlab-ci.yml variables: MARATHON_REST_URI: "http://mm.datys.cu:8080/v2" INSTANCES: "1" CPU: "1" MEMORY: "512" …
19. gitlab-ci.yml build: script: - /bin/bash .ci/build except: - tags stage: build
20. gitlab-ci.yml build: script: - /bin/bash .ci/build except: - tags stage: build [file: ci/build] image=$CI_REGISTRY/my_project/${CI_BUILD_REF:0:8} docker build -t $image . docker push $image
21. gitlab-ci.yml test: script: - /bin/bash .ci/test artifacts: paths: - coverage/ except: - tags stage: test
22. gitlab-ci.yml review: script: - APP_ID=my_project/review_${CI_BUILD_REF_NAME} DATABASE=my_project_review_${CI_BUILD_REF_NAME} /bin/bash .ci/deploy only: - /^issue-.*$/ stage: deploy
23. Reviews
24. gitlab-ci.yml staging: script: - APP_ID=my_project/staging DATABASE=my_project_staging /bin/bash .ci/deploy only: - master stage: deploy
25. Lessons we learned • Principles are more important than tools • Evolution vs. revolution • Ops team as enablers • Automation mindset • Stay constantly learning, adapting and evolving