Shades of crisis across an insecure software supply chain and the containers - 20122020

This is my presentation on software supply chain security and containers as a new beast impacting software supply chain security.

This was presented as part of OWASP Hyderabad Chapter December Meet 2020. https://www.meetup.com/hyderabad-owasp-meetup-group/events/275212537/

Description: Given the presumed state of Nation state actors conducting supply chain compromise against FireEye and US Govt targets, let's look at what is a software supply chain and what's its significance to enterprise cyber security and application security. Also will focus on what to do as a DevSecOps personnel in order to protect against such attacks and to understand cyber hygiene in this area.

Transcript

1. |PUBLIC| Shades of crisis across an insecure software supply chain, and the containers 20 December 2020 Presentation by: Natti | Nataraj Gurrapu OWASP Hyderabad Chapter Meet
2. |PUBLIC| @0xNatti 2 sts:GetCallerIdentity arn:aws:iam:sa-ap-south-1:*:user/natti Hi, I’m Nataraj aka Natti Cybersecurity Architect You can find me @0xNatti Cloud Containers Zero Trust IAM Crypto Cyber Intel Infrastructure Security Application Security
3. |PUBLIC| 1. What is a software supply chain? 2. Attacks and vulnerabilities in the wild 3. Existing Tooling 4. Containers and Supply chain security 5. Securing software supply chain 6. References Agenda @0xNatti 3
4. |PUBLIC| Questions 4 @0xNatti Do you perform any security review of your supply chain providers? Apart from checking vendor cyber hygiene, do you verify the software artifacts provided by them? What’s your patch management or software update frequency? Do you run patches on test beds first? Do you follow micro segmented networks? Any zero trust readiness? Have you heard of code provenance? Google BeyondProd Security Whitepaper
5. |PUBLIC| 5 @0xNatti Vulnerability and Patch Management
6. |PUBLIC| 6 @0xNatti Software supply chain
7. |PUBLIC| What is a software supply chain? 01 @0xNatti 7
8. |PUBLIC| 8 Farmer Buyer Distributor Customer Food supply chain @0xNatti
9. |PUBLIC| 9 Open-Source Developer Source Code Repository Application Developers End users and Customers Software supply chain @0xNatti
10. |PUBLIC| 10 @0xNatti Software supply chain
11. |PUBLIC| A software supply chain is anything that touches our code from development, through our CI/CD pipeline, until it gets deployed into production. 11 @0xNatti
12. |PUBLIC| A vast majority of code is open source We don’t actually know what we are running • We don’t know what dependencies we have • We don’t know what vulnerabilities we have introduced • We don’t know what licenses we are using @0xNatti 12 We’re all dependent on open source code
13. |PUBLIC| @0xNatti 13 Our code has lot of dependencies • 19 LOC • 191,155 LOC
14. |PUBLIC| @0xNatti 14 Our code has lot of dependencies @MayaKaczorowski
15. |PUBLIC| @0xNatti 15 Dependencies introduces vulnerabilities @MayaKaczorowski
16. |PUBLIC| Demo @0xNatti 16 http://npm.broofa.com/?q=express https://www.npmjs.com/
17. |PUBLIC| Attacks and vulnerabilities in the wild 02 @0xNatti 17
18. |PUBLIC| 18 @0xNatti
19. |PUBLIC| 19 @0xNatti
20. |PUBLIC| Every time you pip install, go get, or mvn fetch something, you’re doing the equivalent of plugging a thumb drive you found on the sidewalk into your production server. 20 https://medium.com/better-programming/getting-serious-about-open-source-security-1d15609478fa @0xNatti
21. |PUBLIC| High-level vulnerability trends @0xNatti 21 Source: State of Open Source Security Report 2020 by Snyk
22. |PUBLIC| Method • Malicious code, e.g., backdoor, malware, known vulnerability • Compromised build tool • Compromised signing keys • Compromised package manager • Compromised vulnerability reporting • Account takeover • Project takeover • Accidental, e.g., typo squatting • Deletion Goal • Backdoor, e.g., targeted, watering hole • Malware, e.g., crypto currency mining • Service disruption, e.g., deletion Supply Chain Attacks @0xNatti 22
23. |PUBLIC| Key Trends in Software supply chain attacks Abusing Trust in Code Signing Deep Impact from State Actors @0xNatti 23 Hijacking Software Updates Poisoning Open-Source Code Targeting App Stores
24. |PUBLIC| Cyber Hygiene Account/Project Takeover @0xNatti 24 Method: Phishing, Exploiting insecure hygiene Goal: Account/Project Takeover Source: Microsoft Digital Defense Report 2020
25. |PUBLIC| • Attackers compromised an update for the MEDoc software, a Ukranian tax program • Self-replicating infection, moved laterally • Attributed to Russian hacking group Sandworm Team NotPetya Destructive ransomware @0xNatti 25 Method: Compromised software update distribution server Goal: Ransomware, Service Disruption Maersk, FedEx, and Merck & Co., resulting in over $10 billion in damages • https://blog.talosintelligence.com/2017/07/the-medoc- connection.html • https://www.wired.com/story/notpetya-cyberattack- ukraine-russia-code-crashed-the-world/
26. |PUBLIC| • 17 images in docker123321 registry • Contained malware since at least July 2017 • Suspected malware, positively identified as part of a crypto mining botnet • Removed by Docker Hub in May 2018 docker123321 17 Docker Hub images @0xNatti 26 Method: Accidental Easy to type registry name Goal: Malware Mining ~$90k of Monero https://kromtech.com/blog/security-center/cryptojacking-in vades-cloud-how-modern-containerization-trend-is-exploit ed-by-attackers
27. |PUBLIC| • kik library maintainer approached by Kik.com for copyright reasons • Deletes 273 packages, including left-pad • Reverted within 2 hours left-pad 11-line npm library @0xNatti 27 Method: Deletion “I think I have the right of deleting all my stuff” Goal: Service disruption Major packages like React https://qz.com/646467/how-one-programmer-broke- the-internet-by-deleting-a-tiny-piece-of-code/
28. |PUBLIC| FireEye/SolarWinds Nation State Actors @0xNatti 28 Method: Compromised build pipeline Goal: Cyber Espionage, Data Exfiltration, Lateral Movement Solorigate, SUNBURST Backdoor, TearDrop, APT29?!
29. |PUBLIC| SolarWinds Supply Chain Attack @0xNatti 29
30. |PUBLIC| FireEye/SolarWinds Nation State Actors @0xNatti 30 Method: Compromised build pipeline Goal: Cyber Espionage, Data Exfiltration, Lateral Movement Solorigate, SUNBURST Backdoor, TearDrop, APT29?!
31. |PUBLIC| UNC2452 and SUNBURST @0xNatti 31
32. |PUBLIC| SolarWinds Supply Chain Attack | Timeline of events @0xNatti 32
33. |PUBLIC| UNC2452 and SUNBURST | Solorigate @0xNatti 33
34. |PUBLIC| SolarWinds Supply Chain Attack | Timeline of events @0xNatti 34
35. |PUBLIC| SolarWinds Supply Chain Attack @0xNatti 35
36. |PUBLIC| @0xNatti 36 10 years of Software supply chain attacks https://www.atlanticcouncil.or g/resources/breaking-trust- the-dataset/
37. |PUBLIC| @0xNatti 37
38. |PUBLIC| Tooling 03 @0xNatti 38 What security tools people use!
39. |PUBLIC| They Exist, You Just Need to Find Them SAST DAST VAPT SBOM MAST 39 For static code analysis and scan for vulnerabilities in code built internally. SAST For dynamic application security testing of applications in runtime. DAST For regular infrastructure vulnerability assessments. VA/PT For Free and Open Source (FOSS) libraries and packages. SBOM/Dependency Graph Container Security @0xNatti
40. |PUBLIC| Containers and Supply Chain Security 04 @0xNatti 40
41. |PUBLIC| @0xNatti 41 Containers (~new beast in town)
42. |PUBLIC| @0xNatti 42 How Containers are different from VMs? Containers and Virtual Machines have similar resource isolation and allocation benefits, but function differently because containers virtualize the operating system instead of hardware. Containers are more portable and efficient.
43. |PUBLIC| of container images 1 billion weekly downloads @0xNatti 43
44. |PUBLIC| @0xNatti 44 Source: https://snyk.io/open-source-security
45. |PUBLIC| @0xNatti 45 Source: https://snyk.io/open-source-security
46. |PUBLIC| of docker image vulnerabilities can be fixed with newer base images 44% @0xNatti 46
47. |PUBLIC| @0xNatti 47 Source: https://snyk.io/open-source-security
48. |PUBLIC| @0xNatti 48 Source: https://snyk.io/open-source-security
49. |PUBLIC| Containers and FOSS Container Images Official Images from DockerHub Images downloaded from official repos on DockerHub or CSP marketplace Images from trusted third parties Images provided by your trusted3rd parties/vendors FOSS images from Internet Images downloaded from DockerHub repos/Github or any other place on the Internet @0xNatti 49
50. |PUBLIC| @0xNatti 50 What are core components of container workloads? Images Registries Orchestrator Host OS Containers Dockerfile, K8s manifests, Application packages, Open-source libraries Base Images, Secure CI/CD Pipelines, Application Images Kubernetes distribution, Pods, Namespaces, Nodes, Admission Controllers Host Operating System, Shared Kernel Container Runtime, Secrets, Services, Logging and monitoring
51. |PUBLIC| @0xNatti 51 Core components and their security risks Image Risks: Image vulnerabilities Image configuration defects Embedded malware Embedded clear text secrets Use of untrusted images Registry Risks: Insecure connections to registries Stale images in registries Insufficient authentication and authorization restrictions Untrusted 3rd party images Orchestrator Risks: Unbounded administrative access Unauthorized access Poorly separated inter- container network traffic Mixing of workload sensitivity levels Orchestrator node trust Kubernetes attack surface Host OS Risks: Large attack surface Shared kernel Host OS vulnerabilities Improper user access rights Host OS file system tampering Container Risks: Vulnerabilities within the runtime software Unbounded network access from containers Insecure container runtime configurations App vulnerabilities Rogue containers Privileged containers Secrets in plain text Container escape attacks Supply Chain Security Risks Infrastructure Security Risks Runtime Security Risks
52. |PUBLIC| Securing software supply chain 05 @0xNatti 52
53. |PUBLIC| The issue is unpatched software @0xNatti 53
54. |PUBLIC| Vulnerabilities are still more common than compromises 85% of open-source vulnerabilities have a fix available at disclosure 54 @0xNatti
55. |PUBLIC| What can I do as a developer? Figure out your dependencies Declare them explicitly Only use security assessed packages and libraries Trust, but verify everytime Security audits Scan code, packages, e.g., npm audit Check checksums @0xNatti 55 Remove unnecessary dependencies Automate or update the dependencies or patch them cautiously!
56. |PUBLIC| @0xNatti 56 BeyondProd
57. |PUBLIC| @0xNatti 57 BeyondProd
58. |PUBLIC| @0xNatti 58 ATT&CK like matrix for k8s
59. |PUBLIC| @0xNatti 59 We’re as strong as our weakest link!
60. |PUBLIC| Questions 60 @0xNatti Do you perform any security review of your supply chain providers? Apart from checking vendor cyber hygiene, do you verify the software artifacts provided by them? What’s your patch management or software update frequency? Do you run patches on test beds first? Do you follow micro segmented networks? Any zero trust readiness? Have you heard of code provenance? Google BeyondProd Security Whitepaper
61. |PUBLIC| References 06 @0xNatti 61
62. |PUBLIC| References • Microsoft Digital Defense Report 2020: https://blogs.microsoft.com/on-the- issues/2020/09/29/microsoft-digital-defense-report-cyber-threats/ • Airtable of various attacks in the wild: https://airtable.com/shrsj26j1Ls1RV3uD/tblOFnErtdh2mx1JD?backgroundColor=yellow&vie wControls=on • Atlantic Council Software Supply Chain Security Report 2020: https://www.atlanticcouncil.org/programs/scowcroft-center-for-strategy-and- security/cyber-statecraft-initiative/breaking-trust/ • SANS Emergency Webcast: What you need to know about the SolarWinds Supply-Chain Attack: https://youtu.be/qP3LQNsjKWw • Unauthorized Access of FireEye Red Team Tools: https://www.fireeye.com/blog/threat- research/2020/12/unauthorized-access-of-fireeye-red-team-tools.html @0xNatti 62
63. |PUBLIC| References • Highly Evasive Attacker Leverages SolarWinds Supply Chain to Compromise Multiple Global Victims With SUNBURST Backdoor: https://www.fireeye.com/blog/threat- research/2020/12/evasive-attacker-leverages-solarwinds-supply-chain-compromises-with- sunburst-backdoor.html • FireEye Mandiant SunBurst Countermeasures: https://github.com/fireeye/sunburst_countermeasures • BeyondProd: A new approach to cloud-native security: https://cloud.google.com/security/beyondprod • NIST SP 800-190 - Application Container Security Guide: https://csrc.nist.gov/publications/detail/sp/800-190/final @0xNatti 63
64. |PUBLIC| Thank you ☺ Any questions! @0xNatti 64