Malicious Compliance Automated: When You Have 4000 Vulnerabilities and only 24 Hours Before Release

You know that feeling when you think are done, but then you realize you are not even close and you don't have time to do anything about it?

In this talk we'll go on a journey with a developer, who just finished his application. He's happy because he's done early and there's still 24 hours before the application has to be released, but then he learns that he has 4000 vulnerabilities in his application and there's no way he'll be able to fix them all. He needs a miracle and whatever it is it needs to be automated.

We'll explore the good, the bad and the fun of minifying container images. We'll see the side effects of image minification on the existing vulnerability scanners and exploits and how they will be disrupted and broken. We'll also investigate a number of additional container obfuscation techniques that will make vulnerability scanners completely blind.

You will learn what it takes to build minimal container images and how to make sure you have only the components you need to reduce the attack surface of your containers. You will learn about what's truely necessary for your containers to function. You will also learn how it's possible to automate container image minification leveraging low level Linux kernel interfaces and application analysis.

The vulnerability scanner limitations exposed in this talk shouldn't be there. They are there because users don't demand something better. The call to action in this talk is to ask for better products and not to accept the current status quo. It's time for change!

Transcript

1. Malicious Compliance : When You Have 4000 Vulnerabilities and only 24 Hours Before Release
2. Who are we? Duffie Cooley @mauilion everywhere. Field CTO for Isovalent @ Cisco Member of Sig Honk and the CNCF TOC , fascinated with distributed systems and all things container security. Kyle Quest ( https://linktr.ee/kcqon ). The DockerSlim Guy :-) Founder @ AutonomousLayer @mauilion @kcqon
3. Malicious Compliance @mauilion @kcqon
4. The App: Ready to release? @mauilion @kcqon
5. There is gonna be trouble! @mauilion @kcqon
6. Yikes! @mauilion @kcqon
7. Kyle has also received a report @mauilion @kcqon
8. Let’s see what all vulnerability scanners report @mauilion @kcqon
9. Only 24 hours before release! What do we do? @mauilion @kcqon
10. github.com/mintoolkit/mint • Shrink fat container images • Build regular containers with multiple build engines (Docker, BuildKit, Podman, Depot, Simple) • Debug minimal and regular container images (Docker, Kubernetes, Podman, ContainerD) • Lots of examples : https://github.com/mintoolkit/examples • Looking for new contributors. Come to the KubeCon contrib session if you are interested! Introducing minToolkit (aka DockerSlim and SlimToolkit) @mauilion @kcqon
11. Step 1: We need a minimal container image @mauilion @kcqon
12. Ask AI: Explain the --http-probe-full flag @mauilion @kcqon
13. What do we get from vulnerability scanners now? @mauilion @kcqon
14. Reminder this is before the minification @mauilion @kcqon
15. What do we see after the minification?
16. What happened? Let’s investigate… @mauilion @kcqon
17. What happened to the OS and distro metadata? @mauilion @kcqon
18. What happened to the OS package metadata? • No /etc/os-release • No /var/lib/dpkg/* • No /var/lib/apt/* • No /var/log/apt/* • No /var/log/dpkg.log @mauilion @kcqon
19. We still have vulnerabilities! What do we do? @mauilion @kcqon
20. AI Reply: How can I hide the vulnerabilities? @mauilion @kcqon
21. Step 2: Hide what’s left @mauilion @kcqon
22. What do we get from vulnerability scanners now? @mauilion @kcqon
23. What do we get from the Scanners now?
24. All vulnerabilities are gone! But how?
25. Changed binary hashes (“node” binary example) @mauilion @kcqon
26. Changed package metadata @mauilion @kcqon
27. More fun mutating package metadata @mauilion @kcqon
28. Takeaways and food for thought… @mauilion @kcqon
29. Tools Used In the Talk • Mint / MinToolkit (aka DockerSlim aka SlimToolkit) - create minimal container images and also debug and inspect any container images you have - github.com/mintoolkit/mint ( https://github.com/mintoolkit/examples ) • Diffoci - compare container images - https://github.com/reproducible-containers/diffoci • Container-diff - compare container images - https://github.com/GoogleContainerTools/container-diff • https://ima.ge.cx - explore container images online • https://hexed.it - explore/edit binaries online • OSV scanner - https://github.com/google/osv-scanner • Docker Scout - https://docs.docker.com/scout/ • Snyk - https://docs.snyk.io/scan-using-snyk/snyk-container • Grype - https://github.com/anchore/grype • Trivy - https://github.com/aquasecurity/trivy • Syft - https://github.com/anchore/syft @mauilion @kcqon