Modern Continuous Integration for Microservice Architecture

Transcript

1. CI for a modern age Concourse in action Strukov Anton От винта!
2. Strukov Anton tg: @strukov Tg channel штуки мне интересные(не факт что вам будет) http://bit.ly/32yKa3P Yandex cloud Solution Architect: Instance Groups, MongoDB, Serverless ex: VTB, Luxoft whoami
3. Opinions are my own Disclaimer
4. About what? • Current market state • Few words about • Jenkins • Bamboo • GitlabCI • Concourse CI • Rationale • Architecture • Yandex Cloud to the rescue • Quick start • How to speed-up almost any CI • Modern CI Rationale • CI with MSA • Metadata first approach • How to build products with MSA with infinite teams and so on
5. Part one: Old timers
6. Some people Current state report
7. • Jenkins • Travis CI • CircleCI • Atlassian Bamboo • Gitlab • Drone.io • GoCD • Concourse • Bitrise • Appveyor By stackshare
8. • Jenkins • Travis CI • CircleCI • Atlassian Bamboo • Gitlab • Drone.io • GoCD • Concourse • Bitrise • Appveyor By stackshare
9. Jenkins, cons • Hard to maintain • Hard to master • Easy to make mess in Jenkins, yet works • High availability? Master-Slave/Master-Master? • Think I’m wrong? Imaging managing 50 instances of Jenkins simultaneously? How about 500? 5000?
10. Jenkins, cons • HA through NFS
11. Jenkins, pros • A lot of documentation/SO/Forums • People knew how to do things in Jenkins • Build/Deploy/Delivery Pipeline as Code (PaC) • Main CI/CD system for OpenShift
12. Пятиминуточка БОЛИ с Jenkins • MGMT: О, привет, есть один проектик, надо ребятам сделать быстрее CI/CD(?) • ME: Окай, давайте там посмотрим, что там и как, прежде чем • MGMT: Не нас не пустят на нормальный аудит, там всё “нормас”, Jenkins, ты же знаешь Jenkins? Ну вот, “давай давай делай” • ME:…
13. Пятиминуточка БОЛИ с Jenkins • О, привет, есть один проектик, надо ребятам сделать быстрее CI/CD(?) • Окай, давайте там посмотрим, что там и как, прежде чем • Не нас не пустят на нормальный аудит, там всё “нормас”, Jenkins, ты же знаешь Jenkins? Ну вот, “давай давай делай” • …
14. Проблемы которые Я ожидал решать А вы хотели бы услышать • Groovy coding style • Caching best practices • Artifacts management • Blue/Green deploys • http://groovy-lang.org/style-guide.html
15. Пятиминуточка БОЛИ с Jenkins • Freestyle project запускает 8 мегабайтный bat файл
16. Пятиминуточка БОЛИ с Jenkins • Который работает 5 дней
17. Пятиминуточка БОЛИ с Jenkins • Который работает 5 дней • А потом может быть у тебя получится какой-то результат
18. Пятиминуточка БОЛИ с Jenkins • А там Freestyle project запускает 8 мегабайтный bat файл • Который работает 5 дней • А потом может быть у тебя получится какой-то результат • Но обычно нет, поэтому нас и позвали, потому что последние 180+ дней не было удачных билдов
19. Пятиминуточка БОЛИ с Jenkins • Окай, 2 месяца работы, распил “батничка”, около 150 тасочек, все на груви, добавляются в Jenkins через API, запускаются параллельно на всех воркерах что можем использовать
20. Пятиминуточка БОЛИ с Jenkins • Окай, 2 месяца работы, распил “батничка”, около 150 тасочек, все на груви, добавляются в Jenkins через API, запускаются параллельно на всех воркерах что можем использовать • Теперь это 7 часов а не 5 дней, ура?
21. Пятиминуточка БОЛИ с Jenkins • Окай, 2 месяца работы, распил “батничка”, около 150 тасочек, все на груви, добавляются в Jenkins через API, запускаются параллельно на всех воркерах что можем использовать • Теперь это 7 часов а не 5 дней, ура? • Ускорить больше не можем
22. Пятиминуточка БОЛИ с Jenkins • Покажите как что куда, ага ЯСНО
23. Пятиминуточка БОЛИ с Jenkins • Буквально через месяц +15 пайплайнов на фристайле с башом в инлайне
24. Пятиминуточка БОЛИ с Jenkins • С кучей плагинов на что угодно • Потому что так удобно
25. Две проблемки Jenkins с которыми я устал мирится • Новые версии Jenkins не добавляют функционал(обычно) • Но из-за того что вы обновились, некоторые плагины перестали работать так как работали на версии ранее • Пришёл из отпуска, весь Дженкинс в баше, в эти текстовые амбразуры понапихали всяких кастомов • Через месяц никто не знает как это работает • Не доверяем тулу, которым пользуемся
28. Jenkins law • Jenkins Law: If any task could be done by any shortcut and not through pipeline as code, some from your team will go for it eventually. And you can’t trace it(kinda), and can’t forbid(sorta), and can’t update(sometimes).
29. • Jenkins • Travis CI • CircleCI • Gitlab • Atlassian • Bamboo • Drone.io • GoCD • Concourse • Bitrise • Appveyor By stackshare
30. Atlassian bamboo, cons • 3 teams • ~100 pipelines • ~10 Deploy projects to multiple variables • Unable to upgrade at all pipelines something inside • We have some scripts to patch database on fly • Barely solves problem, due to stateful workers
31. Atlassian bamboo, cons • Cost a lot • But price model based on agents, not on working users • Beta-like PaC, fails a lot, you either need to learn Java, because yaml definitions aren’t covered in docs any good, and Java is more powerful • Doesn’t share bitbucket pipelines syntax(YARLY!)
32. Atlassian bamboo, pros • Best in class integration with Jira/Bitbucket • One of best teams segregation functionality • Easy to use CD • Strict difference between admin and regular user in Control
33. Any CI system fallacy • People • Wrong cache usage • Webhooks • Secrets in plain text
34. Increase control pressure
35. Summary • Pipeline as Code • Immutable definitions for build and deploy process, stored and versioned in Git • Code review • Able to template builds(team A, Appname, git, docker repo) • Able to parametrise builds(deploy to (dev, preprod, prod) • Same process for everyone, because same reusable parts used • DRY? SOLID? KISS? YAGNI? • Be simple/Flat learning curve • API First • Maybe nice GUI? • But restrict user to not change anything via GUI? • Auto register build agents • Build should be repeatable • Consistency through stateless, container-based builds • Integrate with external systems without plug-in hell • Able to extend with minimum efforts • Minimising efforts to maintain CI/CD (no more Jenkins Janitors) • OpenSource • Not Cryptic code from 1999, be able to contribute
37. Gitlab, cons • Pipeline as Code • Witch let BASH NINJAS in • All in one • Yet all tools are at-least “meh” state • Cannot be split off your repository • Building with triggers on external repos like github/bitbucket — known trickstery • Intermediate artifacts have to be defined and uploaded/downloaded for every job • Gitlab GOLD $99 per one user • Yeah that one witch have K8s support • At scale of 15k developers its $1 500 000
38. Gitlab, cons • Gitlab GOLD $99 per one user • Yeah that one witch have K8s support • At scale of 15k developers its $1 500 000
39. Gitlab, cons • Gitlab GOLD $99 per one user • Yeah that one witch have K8s support • At scale of 15k developers its $1 500 000
40. $1 500 000 PM
41. $1 500 000 CTO PM
42. $1 500 000 CFO CTO PM
43. $1 500 000 CEO CFO CTO PM
44. Gitlab, cons • How much is this mystical $1 500 000?
45. Gitlab, cons • How much is this mystical $1 500 000? • It’s 14 Tesla Model S Performance, $106990 each
46. Gitlab, cons • How much is this mystical $1 500 000? • It’s 7 Mercedes-AMG S 63 4MATIC+ Coupe, $201250 each
47. Gitlab, cons • How much is this mystical $1 500 000? • It’s 6 Panamera Turbo S E-Hybrid Executive, $225437 each
48. Gitlab, cons • PoC • Witch let BASH NINJAS in • All in one • Yet all tools are atleast “meh” state • Non reproducible builds • Cannot be split off your repository • Building with triggers on external repos like github/bitbucket — known trickstery • Intermediate artifacts have to be defined and uploaded/downloaded for every job • Gitlab GOLD $99 per one user • Yeah that one witch have K8s support • At scale of 15k developerrs its $1 500 000
49. Gitlab, pros • Simple • All in one • Best integration with
50. Gitlab, pros • Simple • All in one • Best integration with, SELF
51. Gitlab, pros • Simple • All in one • Best integration with, SELF • Pipeline as Code • Yet unable to take care of external repos • You can, but this is bash-snippet hell
52. Gitlab, pros • Simple • All in one • Best integration with, SELF • Pipeline as Code • Yet unable to take care of external repos • You can, but this is bash-snippet hell • But there gitlab have snippets for that
53. Gitlab, pros • Simple • All in one • Best integration with, SELF • Pipeline as Code • Yet unable to take care of external repos • You can, but this is bash-snippet hell • But there gitlab have snippets for that
54. Summary • Pipeline as Code • Immutable definitions for build and deploy process, stored and versioned in Git • Code review • Able to template builds(team A, Appname, git, docker repo) • Able to parametrise builds(deploy to (dev, preprod, prod) • Same process for everyone, because same reusable parts used • DRY? SOLID? KISS? YAGNI? • Be simple/Flat learning curve • API First • Maybe nice GUI? • But restrict user to not change anything via GUI? • Auto register build agents • Build should be repeatable • Consistency through stateless, container-based builds • Integrate with external systems without plug-in hell • Able to extend with minimum efforts • Minimising efforts to maintain CI/CD (no more Jenkins Janitors) • OpenSource • Not Cryptic code from 1999, be able to contribute
55. Part two: Concourse
56. Concourse, architecture
57. Concourse, architecture
58. Concourse, architecture
59. Concourse, architecture
60. Concourse, ATC • Main components, responsible for • API • nicest WEB GUI, you ever seen • Scheduler
61. Concourse, architecture
62. Concourse, TSA • Worker lifecycle • Register
63. Concourse, TSA • Worker lifecycle • Register • Deregister
64. Concourse, TSA • Worker lifecycle • Register • Deregister
65. Concourse, TSA • Worker lifecycle • Register • Deregister • Forward-worker used to reverse-tunnel a worker's addresses through the TSA and register the forwarded connections with the ATC. This allows workers running in arbitrary networks to register securely, so long as they can reach the TSA. This is much safer than opening the worker up to the outside world.
66. Concourse, ATC • Need PostgreSQL • For storing pipeline data, and build logs • Redis(with some resource, like git-multibranch) • For store commits metadata
67. Concourse, Workers • No state • Any task running in container • NO state • If we need transfer state from one task to another we need to declare it
68. Speeding up Any CI workers • Builds in docker • move out build data to ramdisk • Up to 5X faster on same workloads • Using Instances with huge ram
69. 64 CPU 384 GB RAM 91k Regular VM
70. 64 CPU 384 GB RAM 25k Preemptible VM ~3x price drops
71. 64 CPU 384 GB RAM 25k Regular vs Preemptible VM ~3x price drops 64 CPU 384 GB RAM 25k 64 CPU 384 GB RAM 91k
72. Yandex cloud to the rescue • And how Preemptible vm’s could help us?
73. Yandex cloud to the rescue • And how Preemptible vm’s could help us? • As single vm dies in next 24h and could be stopped at any moment?
74. Yandex cloud to the rescue • The answer is Instance Groups, which would take care about vm’s lifecycle, if any of Preemptible vm’s would be sacrificed, Instance Groups would try to create another one
75. Yandex cloud to the rescue • 128GB RAM on worker is enough for most cases
76. Instance groups • Ideal for stateless payload • Could use Load Balancer to reach terrabytes/per second traffic • Have health checks, if something goes wrong, try to heal instance
77. Deploy Concourse part I: Database in docker docker run --name concourse-db \ -v /opt/atc-db:/var/lib/postgresql/data/ \ --restart=always \ -h concourse-postgres \ -p 5432:5432 \ -e POSTGRES_USER=concourse \ -e POSTGRES_PASSWORD=concourse \ -e POSTGRES_DB=concourse \ -d postgres:12.0
78. Deploy Concourse part I: Managed database On concourse host: mkdir -p ~/.postgresql && \ wget "https://storage.yandexcloud.net/cloud-certs/CA.pem" -O ~/.postgresql/root.crt && \ chmod 0600 ~/.postgresql/root.crt On your laptop: https://cloud.yandex.ru/docs/managed-postgresql/operations/cluster- create#create-cluster yc managed-postgresql cluster create \ --name concourse \ --environment production \ --network-name default \ --resource-preset s1.nano \ --host zone-id=ru-central1-c,subnet-id=$(yc vpc network get -- name=default --format=json | jq -r .id) \ --disk-type network-ssd \ --disk-size 60 \ --user name=concourse,password=concourse \ --database name=concourse,owner=concourse
79. Deploy Concourse part II: ATC mkdir -p /opt/concourse/keys cd /opt/concourse/keys ssh-keygen -t rsa -q -N '' -f ./tsa_host_key ssh-keygen -t rsa -q -N '' -f ./worker_key ssh-keygen -t rsa -q -N '' -f ./session_signing_key #!/usr/bin/env bash cw=5.7.0-ubuntu docker run --name concourse-$cw -h concourse -p 2222:2222 -p 80:80 \ —detach --privileged --restart=always \ -v /opt/concourse/keys:/data/keys:ro \ -v /root/.postgresql/root.crt:/root/.postgresql/root.crt:ro \ -e CONCOURSE_CONTAINER_PLACEMENT_STRATEGY=volume-locality \ -e CONCOURSE_ADD_LOCAL_USER=concourserocks \ -e CONCOURSE_MAIN_TEAM_LOCAL_USER=concourserocks \ -e CONCOURSE_EXTERNAL_URL=http://your_ip_addresss8:80 \ -e CONCOURSE_BIND_PORT=80 \ -e CONCOURSE_POSTGRES_HOST=YOUR_MDB.mdb.yandexcloud.net \ -e CONCOURSE_POSTGRES_PORT=6432 \ -e CONCOURSE_POSTGRES_USER=concourse \ -e CONCOURSE_POSTGRES_DATABASE=concourse \ -e CONCOURSE_POSTGRES_PASSWORD=concourse \ -e CONCOURSE_POSTGRES_SSLMODE=verify \ -e CONCOURSE_POSTGRES_CA_CERT=/root/.postgresql/root.crt \ -e CONCOURSE_AUTH_DURATION=120h \ -e CONCOURSE_TSA_HEARTBEAT_INTERVAL=15s \ -e CONCOURSE_TSA_SESSION_SIGNING_KEY=/data/keys/session_signing_key \ -e CONCOURSE_TSA_HOST_KEY=/data/keys/tsa_host_key \ -e CONCOURSE_TSA_AUTHORIZED_KEYS=/data/keys/worker_key.pub \ -e CONCOURSE_ENCRYPTION_KEY=generate-some-key \ -e CONCOURSE_OLD_ENCRYPTION_KEY= \ concourse/concourse:$cw web
80. Deploy Concourse, part III: Workers • The script to bootstrap worker is too big
81. Deploy Concourse • Before you begin, think carefully, changing CI system is a tedious process
82. Modern CI rationale • Due to micro services, since all they basically • Java • maven • Typescript • npm
83. Modern CI rationale • Build process is same except, name of product, team, repository • Since its Code • TEMPLATE ALL THE THINGS
84. Concourse part is over, now to the CI with MSA • Build process is same except, name of product, team, repository • Since its Code • TEMPLATE ALL THE THINGS
85. Part three: CI with MSA
86. Continuous Integration Rationale •Git as source of truth • Bug fix on early stages costs less • Repeatable • Metadata around commit • Easy, same pipelines • Don’t trust single developer • Trust team
87. Fea e b a ch 6348200c92dec8848e55552f3e8039b3da95bd91 (HEAD -> a e , igi / a e , igi /HEAD) 10161a6ef3b1cfef89b61e94f048863b46b29b0b af791274aef9dc9afacf7fc759cd0f444999d5ea a e Ma e b a ch deadbeefdeadbeefdeadbeefdeadbeefdeadbeef e67f848abc35c5cf11861cf8be3b2c0c12ceace4 Microservices Development: Commit stages
88. Fea e b a ch 6348200c92dec8848e55552f3e8039b3da95bd91 (HEAD -> a e , igi / a e , igi /HEAD) 10161a6ef3b1cfef89b61e94f048863b46b29b0b af791274aef9dc9afacf7fc759cd0f444999d5ea a e Ma e b a ch deadbeefdeadbeefdeadbeefdeadbeefdeadbeef e67f848abc35c5cf11861cf8be3b2c0c12ceace4 Microservices Development: Commit stages
89. Fea e b a ch 6348200c92dec8848e55552f3e8039b3da95bd91 (HEAD -> a e , igi / a e , igi /HEAD) 10161a6ef3b1cfef89b61e94f048863b46b29b0b af791274aef9dc9afacf7fc759cd0f444999d5ea a e Ma e b a ch deadbeefdeadbeefdeadbeefdeadbeefdeadbeef e67f848abc35c5cf11861cf8be3b2c0c12ceace4 Microservices Development: Commit stages
90. Fea e b a ch 6348200c92dec8848e55552f3e8039b3da95bd91 10161a6ef3b1cfef89b61e94f048863b46b29b0b af791274aef9dc9afacf7fc759cd0f444999d5ea a e Ma e b a ch deadbeefdeadbeefdeadbeefdeadbeefdeadbeef d5c0829d613ef45dab1846e4e1a259908ee269fd (HEAD -> a e , igi / a e , igi /HEAD) e67f848abc35c5cf11861cf8be3b2c0c12ceace4 Microservices Development: Commit stages
91. Microservices Development: Commit stages Precious business Logic Frontend API adapter to SOAP Wrapper for some SOA RabbitMQ Precious business Logic Precious business Logic ScyllaDB Redis Clickhouse
92. Microservices Development: Commit stages Precious business Logic Frontend API adapter to Money Wrapper for some SOA RabbitMQ Top notch π divider Very important app ScyllaDB Redis Clickhouse Team B Team A Team C Team D Team E Team F
93. Microservices Development
94. Fea e b a ch 6348200c92dec8848e55552f3e8039b3da95bd91 10161a6ef3b1cfef89b61e94f048863b46b29b0b af791274aef9dc9afacf7fc759cd0f444999d5ea a e Ma e b a ch deadbeefdeadbeefdeadbeefdeadbeefdeadbeef d5c0829d613ef45dab1846e4e1a259908ee269fd (HEAD -> a e , igi / a e , igi /HEAD) e67f848abc35c5cf11861cf8be3b2c0c12ceace4 Microservices Development: Commit stages • We checking any our commit against other stable versions
95. Microservices Development: Commit stages • Linter • Dependency • Unit tests • Build • SonarQube Stage • Sonar code check • Code Smells • Potential bugs • Unit test run time (Y U COMMIT SKIP TESTS?) • Skip unit tests • Unit test failures • Maintainability of new code lower than B • Tech debt of new code • Security rating on new code • Uncovered conditions on new code • Code coverage • Security • Aquasecurity/Microscanner • Aquasecurity/Trivy(after push to Artifactory actually)
96. Microservices Development: Commit stages • Linter • Dependency • Unit tests • Build • SonarQube Stage • Code Smells • Potential bugs • Unit test run time (The junior cut off all unit tests?) • Skip unit tests • Unit test failures • Maintainability of new code lower than B • Tech debt of new code • Security rating on new code • Uncovered conditions on new code • Code coverage • Security • Aquasecurity/Microscanner • Aquasecurity/Trivy(after push to Artifactory actually) • Run integration tests on mockups • Spin DB • Spin nearest services(testcontainers) • Push image to artifactory • Prepare Kubernetes Environment
97. Microservices Development: Commit stages • Prepare Kubernetes Environment • Create NS, push secrets, configure all the things: • Create svc, ingress • Create network policies, pod disruption budget, limits, qos/priority • Deploy everything
98. Microservices Development: Commit stages
99. Microservices Development: Commit stages Precious business Logic Frontend API adapter to Money Wrapper for some SOA RabbitMQ Top notch π divider Very important app ScyllaDB Redis Clickhouse Team B Team A Team C Team D Team E Team F
100. Microservices Development: Commit stages • Prepare Kubernetes Environment • Create NS, push secrets to pull images, configure all the things: • Create svc, ingress • Create network policies, pod disruption budget, limits, qos/priority • Deploy everything • Deploy in team-appname-sha1commit-sidecar-ns tools to run • Starts from API tests • Ends with Selenium, running under Aerokube
101. Microservices Development: Commit stages • Prepare Kubernetes Environment • Create NS, push secrets to pull images, configure all the things: • Create svc, ingress • Create network policies, pod disruption budget, limits, qos/priority • Deploy everything • Deploy in team-appname-sha1commit-sidecar-ns tools to run • Starts from API tests • Ends with Selenium, running under Aerokube
102. Microservices Development: Commit stages • Test stages: • Unit tests, almost 0 cost • Integration tests running on nearest services with testcontainers • e2e tests, runs from sidecar-NS • Starts from API tests • Ends with Selenium, running under Aerokube
104. • Deploy Concourse • https://t.me/concourse_ru • Мой канальчик в тг Thanks! Any questions?