Cooking the Cake for Nuget packages

How to manage CI\CD process for many git repositories (400+), when each of repository needs to be published as NuGet package? Combining Cake and Jenkins features allow building the generic pipeline for packages creation, versioning and publish. Based on C# code, source controlled and compiled in runtime, Cake gives perfect flexibility to build advanced CI\CD process. Large scale also required some tricks inside CI infrastructure and related services.
https://fwdays.com/en/event/dotnet-fwdays-2018/review/cooking-the-cake-for-nuget-packages

Transcript

1. Sergey Dzyuban, SBTech Cooking the Cake for Nuget packages
2. Part 1 Repository hell
3. Consider regular repository CI\CD DevOps Repository build tests deploy
4. … but what if company has more repos ?
6. … much more
9. … ok, actually 400+ repos and most of them are .NET solutions
10. The common issues • Lot of repositories, managed separately • Repository-specific logic placed in each repository as scrips • 1++ jobs in Jenkins to process each repository OR none jobs at all • Common logic (unit test, nuget publish) need to be applied per each repository … as result • 90 % of all DevOps resources are taken to fix existing jobs and adding new one • Impossible to make common changes (adding Sonar, notifications) for all jobs at once • “Lost” and “phantom” jobs: build_xxx_repo_dev_temp_1
11. Analyzing existing repositories Common Unique C# Solutions count File structure Scripts (even language) VS solution Projects type Git controlled Deploy type • Manage CI\CD logic in version control system • Add ability to minimize Jenkins jobs number by adding common jobs, which can process multiple repositories • Allow to easy use CI\CD scripts outside Jenkins • Allow to make centralized changes in whole CI\CD process as simply as possible. • Add support for create Nuget Packages
12. The main idea – Repository as Interface Each repository should contain some part of internal scripts, which will have common interface to interact with repository in same way. Common Interface members • REPO HELP • REPO BUILD • REPO UNITTEST • REPO PACKAGE REPOSITORY help build test pack
13. How it was
14. How it should be build test deploy
15. Part 2 Cake
16. https://cakebuild.net/
17. Why we are using cake ? • Cake is not Powershell • Many build-in integrations • Cake is C# and Roslyn • Cake can use PowerShell • Cake can use tools and add-ins as NuGet packages • Cake can reference another *.cake as Tasks • Simpsons already did it
18. Adding cake to existing repository Repository build.cake src cake build.ps1 repo.sln tools Tasks deploy.cake tests.cake
19. Best practices for single repository • Place cake in separate folder • Use separate cake tasks • Powershell is the best start
20. … but take a look at real call example ./cake/build.ps1 -ScriptArgs “- solution='.\src\log4net.sln' -source='src'" -Target build -Verbosity Verbose
21. … but take a look at real call example ./cake/build.ps1 -ScriptArgs “- solution='.\src\log4net.sln' -source='src'" -Target build -Verbosity Verbose
22. Adding batch file Repository build.cake src cake build.ps1 repo.sln tools Tasks deploy.cake tests.cake cake.bat cake build
23. Adding batch file Repository build.cake src cake build.ps1 repo.sln tools Tasks deploy.cake tests.cake cake.bat Batch script: • build.ps1 check-download • define solution file(s) name • hides default settings • makes call clear REPOSITORY help build test pack
24. Adding batch file Repository build.cake src cake build.ps1 repo.sln tools Tasks deploy.cake tests.cake cake.bat Batch script: • build.ps1 check-download • define solution file(s) name • hides default settings • makes call clear REPOSITORY help build test pack
25. Adding batch file Repository build.cake src cake build.ps1 repo.sln tools Tasks deploy.cake tests.cake cake.bat Batch script: • build.ps1 check-download • define solution file(s) name • hides default settings • makes call clear REPOSITORY help build test pack cake build cake cake test cake package
26. What about another repository ? … we have lot of them
27. Using contract Repository build.cake src cake build.ps1 repo.sln tools Tasks deploy.cake tests.cake cake.bat F# build.cake src cake build.ps1 repo.sln tools Tasks fsharp.cake gatling.cake cake.bat > cake > cake build > cake test > cake package
28. Adding Jenkins jobs build test deploy git clone cake build notify
29. Some issues on this stage • Lot of cake tasks are duplicated • Hard to make changes in cake tasks • Adding new repo by copy-paste • Bug fixes in cake requires changes of all repos
30. Adding Master repository Master Repo build.cake cake build.ps1 Repository custom.cake src cake repo.sln cake.bat build.cake build.ps1 copy tools
31. Master Repository allows • Keep all logic in one place • Fix bugs and implement features quickly • Avoid code duplication • Decrease cake script size committed in each repo • Easy to add a new repository
32. Deep dive into the build.cake
33. Dependency based programming package init clean set new version build pack publish
34. Dependency based programming package init clean set new version build pack publish Task("package") .IsDependentOn("init") .IsDependentOn("clean") .IsDependentOn("setnewversion") .IsDependentOn("build") .IsDependentOn("pack") .IsDependentOn("publish") .Does(() => { });
35. Dependency based programming package init clean set new version build pack publish before build msbuild unit test sonar begin sonar end clean nuget restore nunit ncover
36. … what about customization? Master Repo build.cake cake build.ps1 Repository custom.cake src cake repo.sln cake.bat
37. Tasks inheritance base.package init clean set new version build pack publish custom.cake package build.cake base.init base.clean base.version base.build base.pack base.publish
38. Tasks inheritance base.package init clean set new version build pack publish custom.cake package build.cake base.init base.clean base.version base.build base.pack base.publish Task("Information") .IsDependentOn("Base.Information") .Does(() => {}); Task("Clean") .IsDependentOn("Base.Clean") .Does(() => {}); Task("Build") .IsDependentOn("Base.Build") .Does(() => {}); Task("BeforeBuild") .Does(() => {}); Task("UnitTest") .IsDependentOn("Base.UnitTest") .Does(() => {});
39. Tasks inheritance base.package init clean set new version build pack publish custom.cake package build.cake base.init base.clean base.version base.build base.pack base.publish
40. Tasks inheritance base.package init clean set new version build pack publish custom.cake package build.cake base.init base.clean base.version base.build base.pack base.publish Task("Information") .IsDependentOn("Base.Information") .Does(() => {}); Task("Clean") //.IsDependentOn("Base.Clean") .Does(() => { StartProcess("rm -rf /"); }); Task("Build") .IsDependentOn("Base.Build") .Does(() => {}); Task("BeforeBuild") .Does(() => {}); Task("UnitTest") .IsDependentOn("Base.UnitTest") .Does(() => {});
41. Out of the box features We provide some list of features, available for all repositories by default
42. Build-in features • Run UnitTests • NUnit • XUnit • Code coverage • Sonar code analysis • Build .NET solutions • .NET Framework • .NET Core • Restore nuget packages • Create nuget packages • By csproj • By nuspec • By core csproj • Make commits\tags to git • Publish artifacts
43. How we maintain scripts 5% 95% CODE custom.cake build.cake 5% 55% 40% CODE custom.cake build.cake addons
44. How we maintain scripts Master Repo build.cake cake build.ps1 Repository custom.cake src cake repo.sln cake.bat build.cake build.ps1 tools src addon Class1.cs addon.dll
45. How we test scripts
46. How we test scripts Master repo master Repo 0 Repo 60 master master cake
47. How we test scripts Master repo master Repo 0 Repo 60 master master cake dev feature/_temp F# Repo master dev
48. How we test scripts Master repo master Repo 0 Repo 60 master master cake dev feature/_temp F# Repo master dev
49. The infrastructure Part 3
50. Infrastructure hook cake master slave copy cake.exe addins push
51. Infrastructure hook cake master slave copy cake.exe addins push
52. Jenkins by itself is good enough … but let’s add some additional features for static code analysis
53. Infrastructure hook cake master slave copy cake.exe addins push
54. Infrastructure hook cake master slave copy cake.exe addins push
55. Slack for build notifications
56. Infrastructure hook cake master slave copy cake.exe addins push
57. Infrastructure hook cake master slave copy cake.exe addins push
58. Elastic for great CI telemetry In high scale CI process single build makes no lot of sense, but Jenkins telemetry allows to show the whole picture.
59. Infrastructure hook cake master slave copy cake.exe addins push
60. Infrastructure hook cake master slave copy cake.exe addins push
61. Custom Dashboard … allows to aggregate events from all system parts and visualize status snapshot, actual on the current moment of time. It provide only the required information from large amount of data.
62. Infrastructure hook cake master slave copy cake.exe addins push Dashboard
63. Infrastructure hook cake master slave copy cake.exe addins push Dashboard
64. Infrastructure hook cake master slave copy cake.exe addins push Dashboard
65. Infrastructure hook cake master slave copy cake.exe addins push Dashboard
66. Custom Dashboard Powered By
67. Hosting for Dashboard Having DCOS cluster setup and running, it’s easy to add and maintain all required components.
68. Custom Dashboard Hosting
69. Custom Dashboard Hosting
70. Custom Dashboard Hosting
71. Q & A