.NET Day 2023: The symbiosis of Continuous Deployment and Stability

Our software ensures that all the parcels you order online find their way to your home. For every parcel that travels through the parcel centers of the Swiss Post, our services provide the next action to be taken. If these services are not running, the parcel truck soon begins to the roads because they cannot unload anymore. Even with these stakes, we deploy our mission-critical software services multiple times a day. Every commit to the master branch goes straight into production. Magic? Not at all! In this talk, we will show you how we do this and discuss our journey with a focus on quality assurance.

Transcript

1. The Symbiosis of Continuous Deployment and Stability Marc Sallin 29. August 2023
2. Table of Contents 2 1 Introduction & Context 10’ 2 DevOps & CD 5’ 3 Quality Culture & QA 15’ 4 Deployment 5’ 5 How .NET helps us 10’
3. Page 3
4. Page 4
5. Page 5
6. Page 6
7. Page 7
8. Facts and Figures 15 Developers 4 Business Analysts 3 Solution Architects 1 Scrum Master 1 Product Owner End-2-End Responsibility & 24/7 on call 99.8% Unit Test Coverage ~200k LoC C# 9 Deploys per Day 3 days commit lead time 3 Week Scrum Sprints 8
9. Page 9
10. Page 10
11. 11 Speed / Cost / Quality trade-off fallacy
12. Our Quality Culture Principles - .NET SDK + Docker = Clone & F5 - Development w/o application start - Trunk Based Development - Devs On Duty 24/7 - Rigorous PR based Code Reviews - Continuous Deployment - No Tolerance for Roslyn or Sonar Issues - Zero Errors & Warnings in the Logs - No End-2-End Tests (i.e., only test the application and the contracts) - Automation, automation, automation… 12
13. Functional Testing Strategy 13 Unit Tests User Story Tests Component Int. Tests System Integration Tests 100% Coverage Unit Test - We all know them - Every class is tested in isolution Component Integration Tests - Starts an instance of the application in memory - In-Memory mocks for external systems User Story Tests - Test a complete flow - Specified by a business analyst - Implemented by the developer System Integration Tests - Integration with the actual systems - Implemented as a normal unit test - Run on Kubernetes (temporary namespace)
14. User Story Tests 14 // Arrange await SetupStateAsync( masterDataBuilder => { masterDataBuilder.WithDeliveryPlan(WithCountryDestinationDe) .WithPropertyGroupCollection(WithPropertyGroupPrio) .WithProductCatalog(WithVelocityPriority) .WithNonEmptyProductCodesCatalog() .AddSortPlanRelease(WithSortingProductDePrioAndEco); }); ArrangeCoding( that => that .ForProcessingItemId("processingItemId") .Responds( transmitFinalResult => transmitFinalResult .WithAddress(address => address.WithCountryKey("DE").WithDeliveryPointType(DeliveryPointType.INTERNATIONAL)) .AddProduct("509"))); // Priority Product // Act await PublishSubmitMailpiece("processingItemId"); // Assert DestinationAssignedEvent[] sortedEvent = await WaitForDestinationAssignedEvents(); sortedEvent.Should().HaveCount(1); sortedEvent[0].ShouldNotSortToAnySortReasonOrSpecialSort(); sortedEvent[0].ShouldSortToOutletGroup("outletGroupDEPri");
15. System Integration Tests 15 Temporary namespace for each test run Kafka EMQX Mongo MSSQL MinIO Service Tests
17. Deployment Page 17
18. How .NET helps us 1. Open Source & Platform Independent 2. Performance & Resource usage 3. Easy Extensibility using Roslyn 4. Built-in Industry Standards 5. Possibilities for Testing 6. Silverback Messaging 18
19. OSS & Platform Independent 19
20. Performance & Resource Usage 20 Traffic 200k Kafka Messages per Hour 35k MQTT Messages per Hour 26ms Median for Executing the Sorting Pipeline Ressource Usage* - 2 vCPU - 300MB RAM *one replica, we have two
21. Custom Roslyn Analyzers 21
22. OpenTelemetry Built In 22
23. Integration Tests in ASP.NET Core 23
24. Silverback Messaging 24
25. Danke, Merci, Grazie, Thank you 31