How CNCF tools safeguard releases

This talk highlights the role of CNCF tools in enhancing the reliability and security of software releases. We will explore best practices for leveraging these tools to transform release processes, ensuring robust and secure releases that can mitigate recent well-known failures.

Transcript

1. How CNCF tools safeguard releases Edgaras Apšega CNCF Ambassador / Kubestronaut / Cloud Native Lithuania meetup co-organizer / SRE
2. 2 Software Supply Chain Security
4. 4 1. Comprehensive Testing 2. Integrity Verification 3. Staged Rollouts 4. Quick Rollback Mechanisms 5. Transparent Communication 6. Proper Key Management Critical components of a robust release process
5. 5 1. Comprehensive Testing 2. Integrity Verification 3. Staged Rollouts 4. Quick Rollback Mechanisms 5. Transparent Communication 6. Proper Key Management Critical components of a robust release process
6. 6 CNCF Projects
7. 7 in-toto a framework that provides end-to-end security and verification for software supply chains
9. 9 Software Supply Chain Source Build Package Deliver Consumer Test
10. 10 Layout Developer Jenkins Bob QA Project Owner steps: - name: "write-code" expected_materials: [] expected_products: - "main.go" pubkeys: ["developer_key"] expected_command: ["echo", "write code"] - name: "review-code" expected_materials: - "main.go" expected_products: - "main.go" pubkeys: ["reviewer_key"] expected_command: ["echo", "review code"]
11. 11 Links
12. 12 Delivered Product Verification End user or System Layout Link Link Link Link
13. 13 • End-to-End Supply Chain Security • Link Metadata • Policy Enforcement • Tamper Detection • Transparency and Compliance • Ease of Integration Key features of in-toto
14. 14 • Enforced Process: By defining a clear layout, in-toto ensures that all required steps in the release process are followed, reducing the chance of skipping crucial steps like thorough testing. • Verification of Inputs and Outputs: Each step’s materials and products are verified, ensuring that unexpected changes aren’t introduced during the process. • Authorized Actions: By specifying who can perform each action, in-toto reduces the risk of unauthorized or accidental changes to the software. • Auditability: The layout provides a clear record of what should happen in the release process, making it easier to audit and identify potential issues. • Final Product Verification: The inspection step ensures that the final product meets all specified criteria before release. How in-toto Layouts Could Help Prevent Incidents Like the CrowdStrike Issue
16. 16 Copa CLI tool for directly patching container images
17. 17 How does it work? Container image Vulnerability Report Trivy Container image Patched
18. 18 How does it work?
19. 19 Argo Rollouts Kubernetes controller that provides advanced deployment capabilities to Kubernetes
20. 20 • Integrates with ArgoCD • Provides blue-green, canary, canary analysis, experimentation, and progressive delivery • Supports canary analysis based on Prometheus, DataDog, New Relic, etc. metrics Key features of Argo Rollouts
21. 21 Analysis template with Prometheus metrics
22. 22 Automated rollbacks 1. New rollout started 2. Rollback initiated 3. Rollout with bug fix pass successfully Number of replicas, %
23. Thank you!
24. 24 Mentioned CNCF projects
25. Questions? Let's connect on LinkedIn Follow on X/Twitter
26. 26 • in-toto: Providing farm-to-table guarantees for bits and bytes | USENIX • Secure Publication of Datadog Agent Integrations With TUF and In-Toto • Automated container CVE and vulnerability patching using Trivy and Copacetic | by Edgaras Apšega | Jul, 2024 | Medium References and further reading