DIY Patch Management

Slides for the talk by @shantycode and @ingobente from BSidesMunich 2018.

Installing software patches is a no brainer when it comes to security. Over the past years, several stories about major flaws have been covered in the media. Those who follow the coverage understand that being up to date with the latest MacOS or Windows patches has become a necessity. For consumer computer it is easily done: click a button, sometimes reboot, and you are good to go.

Yet, when it comes to larger IT infrastructures, promptly patching seems to be a hard problem. Otherwise, recent ransomware attacks that exploit vulnerabilities months after a patch was made available would not have such a great impact that makes everybody go into panic mode.

In this talk, we will tell the story of how we managed to fix our own patch management procedures. What used to be a manual, checklist based and thus inconsistent approach in a heterogenous environment, has now become largely automated. Daily, host-based scans search for software packages with known vulnerabilities. Results are logged to a central ELK stack. Dashboards provide insights for management. The actual patching is done by leveraging unattended-upgrades with some custom code. The result: an IT-infrastructure that keeps itself up-to-date. Bonus: no money was spent on commercial software.

Transcript

1. DIY Patch Management F L O R I A N J U N G E ( @ S H A N T YC O D E ) I N G O B E N T E ( @ I N G O B E N T E ) B S I D E S M U N I C H 2 0 1 8
2. Patching
 Isn’t that solved?
3. Nope, it’s not. R E M E M B E R T H O S E R A N S O M WA R E N E W S I N 2 0 1 7 ?
4. SINNERSCHRADER BSides Munich | DIY Patch Management | 2018 4 https://en.wikipedia.org/wiki/WannaCry_ransomware_attack
5. SINNERSCHRADER Example WannaCry • WannaCry hit the world on May 12 2017 • It spread via a vulnerability called EternalBlue • EternalBlue was fixed by Microsoft on March 14 2017
6. SINNERSCHRADER Example WannaCry • WannaCry hit the world on May 12 2017 • It spread via a vulnerability called EternalBlue • EternalBlue was fixed by Microsoft on March 14 2017
7. 2 months between patch and outbreak I . E . YO U CA N WO R RY L E S S A B O U T 0 DAYS : )
8. Patching is hard AT L E AST I N M A N Y R E A L WO R L D S C E N A R I O S.
9. But why? I M E A N I T WO R KS O N O U R FA M I LY M E M B E R S C O M P U T E R S, TO O.
10. Constraints L E GAC Y : E N D O F L I F E O S T H AT I S N OT PATC H A B L E .
11. Constraints AVA I L A B I L I T Y : H OW TO R E B O OT T H AT H Y P E RV I S O R C L U ST E R ?
12. Constraints M O N E Y : I T WO R KS W I T H O U T T H E PATC H , D O E S N ’ T I T ?
13. So what now? L E T ’ S TA K E A G L I M P S E I N TO O U R WO R L D.
14. SinnerSchrader Ecosystem
15. SINNERSCHRADER In a world where … • N tenants • M tech stacks • N x M requirements
16. SINNERSCHRADER In a world where … • Heterogenous infrastructure • OS-wise mostly Debian and Ubuntu • Packed into VMs and Containers • Yes, there is also some serverless stuff :)
17. In a world where … Inconsistent patch management N E V E R , S O M E T I M E S, R E G U L A R LY.
18. In a world where … Commercial scanners? N O B U D G E T. W E A R E N OT T H AT E N T E R P R I SY.
19. SINNERSCHRADER Lessons learned … so far • Installation is easy • Patching is hard • Knowing when to patch is even harder
20. Not cool S O W E WA N T E D TO C H A N G E T H AT.
21. SINNERSCHRADER Solution - the easy part • Manually scan for all the CVEs • Automate CVE scans (i.e. daily) • Gather all the logs
22. SINNERSCHRADER Solution - the tricky part • Dashboard everything • Get metrics that CXOs can understand • Take action (i.e. patch) and check the metrics • Lean back … for now
23. Building blocks
24. SINNERSCHRADER • CVE scanner to audit VMs • Integration to config management • Central logging and dashboarding
25. SINNERSCHRADER Spot the vuln - the audit • Vulnerability databases • Vulnerability scanner • Vulnerability subscriptions • Freemium pricing model • Nice people :)
26. SINNERSCHRADER Spot the vuln - the audit • nmap plugin • Burp plugin • getsploit • API
27. SINNERSCHRADER Spot the vuln - CVE scanner • Get installed packages • Audit each for CVEs • Get CVSS scores
28. Demo V U L N E R S. C O M A P I
29. SINNERSCHRADER Orchestration via config management • Do it! • Our solution: SaltStack • Codify your update strategy https://docs.saltstack.com/en/getstarted/overview.html
30. SINNERSCHRADER Orchestration • Define systems with formula • Minion matching • Template engine https://docs.saltstack.com/en/getstarted/overview.html
31. SINNERSCHRADER Three patch management flavours • Unattended upgrades • Orchestrated updates • Patch Day https://en.wikipedia.org/wiki/Neapolitan_ice_cream
32. SINNERSCHRADER Centralized Logging • Elasticsearch • Logstash • Kibana figure based on https://www.elastic.co/guide/en/logstash/current/introduction.html
33. SINNERSCHRADER
34. Demo K I B A N A . S Z O P S. D E
35. BSides Munich | DIY Patch Management | 2018 SINNERSCHRADER The big picture 35
36. BSides Munich | DIY Patch Management | 2018 SINNERSCHRADER The big picture 36
37. BSides Munich | DIY Patch Management | 2018 SINNERSCHRADER The big picture 37
38. Limitations
39. There are some limitations A K A ST U F F T H AT C O M P L I A N C E FO L KS L I K E LY WO N ’ T F I N D, B U T T H AT YO U K N OW I S T H E R E …
40. SINNERSCHRADER Tales from the kernel • Notice version encoded in package name • This confuses vulners (no CVEs) • As well as unattended upgrades (yep, no upgrades)
41. SINNERSCHRADER Tales from the kernel - the fix • Install the meta package linux-image-generic
42. BSides Munich | DIY Patch Management | 2018 SINNERSCHRADER Reboot hassle 42 • Running old kernel although newer one is installed
43. BSides Munich | DIY Patch Management | 2018 SINNERSCHRADER Reboot hassle - the fix 43 • Monitor uptime of your servers ;-) • The two metrics that matter for host security by Diogo Monica
44. Next steps
45. SINNERSCHRADER Next steps • Fix some quirks ;-) • Container checks with Claire • AWS integration • nsp check
46. SINNERSCHRADER Alternatives • OpenVAS / vuls • https://github.com/0x4D31/salt-scanner • Your-typical-Enterprise-Distribution-Mgmt-here • Reverse uptime & Golden image freshness
47. SINNERSCHRADER Kudos • Kirill Ermakov from Vulners.com(@isox_xx) • Christoph Trautwein (@trautw) & the S2 ops crew
48. Thanks!