Jenkins as a Service

This is a story about success CI process improvement in large company (1K+ employees, over 700 Bitbucket repositories with .NET project on Windows).
When we started, company had Green-Red deployment process (joke: repository build only during deployment, on demand and as a result You never know it will be green or red).
1. Using shared Cake scripts and automation (like Travis CI or another public providers), we cover over 30% of repositories (all nuget packages) very fast (in a month) and started to provide CI as a Service.
2. On a second stage, instead of scale horizontally to other repositories, we focused on the terrible one: 8Gb of disk size monster and decrease build size from 40 min to 5 min and made it stable.
3. Jenkins functionality on stage of horizontal scaling (build over 400 repositories) was not enough to provide good user experience, and DevOps team integrate it with infrastructural products as Elastic and MySQL and manually created applications (using Go, Scala, Angular).
4. In a large scale (over 1 build per minute) we are using Kibana boards to analyze CI dynamically changed state and found possible issues. Jenkins + Elastic is most powerful tool.
5. Knowing about pet vs cattle principle, I will show how we made our 40 windows slaves provisioned dynamically, and looking forward to have Jenkins master to be created from scratch on demand.

Transcript

1. Jenkins as a Service Sergey Dzyuban, SBTech, Ukraine
2. Sergey Dzyuban DevOps Tech Lead at SBTech LAMP admin 2003 2007 2010 2014 2016 Web Developer .NET Developer Technical Account Manager Team Lead DevOps Tech Lead
3. R&D offices Datacenters 1000+ 2500 700 200 300 CPU/3Tb RAM 28 nodes 2 Tb 100K+ ~ 500 builds per day ~ 1 build per minute ~ 40 windows slaves
4. Part 1. Red-Green Deployment
5. Deploy on demand Developer Repository commit
6. Adding CI process Developer Repository test commit
7. But for another 699 repositories …
8. Bitbucket Jenkins CI became a time killing feature
9. Summary • Lot of repositories, managed separately • Repository-specific logic placed in each repository as scrips • 1++ jobs in Jenkins to process each repository OR none jobs at all • Common logic (unit test, nuget publish) need to be applied per each repository … as result • 90 % of all DevOps resources are taken to fix existing jobs and adding new one • Impossible to make common changes (adding Sonar, notifications) for all jobs at once • “Lost” and “phantom” jobs: build_xxx_repo_dev_temp_1
10. Part 2: The King of Nugets
11. Common Unique C# Solutions count File structure Scripts (even language) VS solution Projects type Git controlled Deploy type • Manage CI\CD logic in version control system • Add ability to minimize Jenkins jobs number by adding common jobs, which can process multiple repositories • Allow to easy use CI\CD scripts outside Jenkins • Allow to make centralized changes in whole CI\CD process as simply as possible. • Add support for create Nuget Packages Analyzing existing repositories
12. Repository as Interface Each repository should contain some part of internal scripts, which will have common interface to interact with repository in same way. Common Interface members • REPO HELP • REPO BUILD • REPO UNITTEST • REPO PACKAGE REPOSITORY help build test pack
13. Cake scripts cake build Repository build.cake src cake build.ps1 repo.sln tools Tasks deploy.cake tests.cake cake.bat REPOSITORY pack cake cake test cake package
14. Adding Master repository Master Repo build.cake cake build.ps1 Repository custom.cake src cake repo.sln cake.bat build.cake build.ps1 copy tools
15. Implementing CI pipeline hook cake master slave copy cake.exe addins push • build .NET Framework • build .NET Core • create Nuget packages • run unit tests • versioning
16. Decreasing jobs number after
17. Unexpected issue X1 X2 X3 X4 X1 X2 X3 X4 X1 X2 X3 X4 X3 X1 One job processing all repositories significantly decrease process clearance and visibility. Unclear: • Total number covered of repositories • Current CI state for given repository • Overall CI state The any final source of true is Bitbucket.
18. Dashboard hook cake master slave copy cake.exe addins push Dashboard
19. CI status dashboard
20. Build events & repos snapshots Repository Status Repository Branch Build Status
21. Build events & repos snapshots Repository Status cms success Repository Branch Build Status cms master success
22. Build events & repos snapshots Repository Status cms success mobile failure Repository Branch Build Status cms master success mobile master failure
23. Build events & repos snapshots Repository Status cms success mobile failure Repository Branch Build Status cms master success mobile master failure cms feature/test failure
24. Build events & repos snapshots Repository Status cms success mobile failure Repository Branch Build Status cms master success mobile master failure cms feature/test failure cms feature/test success
25. Build events & repos snapshots Repository Status cms success mobile success Repository Branch Build Status cms master success mobile master failure cms feature/test failure cms feature/test success mobile master success const
26. Cake covers 33% of repos Nuget packages .NET Core projects Microservices Cake Unknown Unknown
27. Part 3: The Monster
28. Large repository is a gap 8Gb repository can be a bottleneck because of clone and build time. And it is using submodules. Unfortunately, this is the most common used one. … but build pipeline is simple
29. Only 2 jobs can proceed CI Build master & release branches Build pull requests Clone Clone - Merge branches: source->target Build Build Unit Test Unit Test
30. Cake & Pipeline covers 66% of repos Nuget packages .NET projects Microservices Cake Pipeline Unknown • Job that adds repository to pipeline CI • Build settings and sln file as hook parameters • Notify Status callback to Bitbucket
31. Accumulate Statistics hook status When number of repositories > 200, make sense to start gathering statistics data. From analyzing standalone incidents we move to analyze overall CI system.
32. Everything in Kibana makes sense
33. Same data – different views
34. Pull/Push git & CI stats
35. Know Your Repositories
36. Pull/Push git & CI hook
37. Be consistent “We plan the work and work the plan” principle will do the results more predictable in case of scope and in case of time frames.
38. Part 4: Happy Ops and Angry Bot
39. Slack Bot for Nugets commit hook build Artifactory Slack BOT
40. Jenkins External Integrations • notify users • accumulate statistics • show gaps and issues • extend Jenkins functionality • improve visibility
41. Integrations v2
42. Integrations v3
43. Hosting Master Master Master
44. Happy Ops
45. Questions ?