Host level security with HIDS agents on 20,000+ hosts

Taeee Kim
LINE Infra Protection Team Security Engineer
https://linedevday.linecorp.com/2020/ja/sessions/0429
https://linedevday.linecorp.com/2020/en/sessions/0429

Transcript

2. Background
3. Agenda › Introduction › Provisioning › Architecture › Operation › Automation › Case Study
4. Agenda › Introduction › Provisioning › Architecture › Operation › Automation › Case Study
5. Host-Level security Necessity Limitation of Traditional Security Correlation analysis required › Process, logs, network, configuration etc. › Evolving Attack Technology and Types › Mainly only anti-malware agent installed to servers › New types of cyber threats that traditional anti-virus programs can't stop. (Ransomware) Move to Edge network from External › Lateral movement, Worm › Targeting attack
6. Network And Host Level Switch Switch Switch Backbone Servers Servers Servers Servers Servers Servers Servers Servers Servers Servers Servers Servers Encrypted Traffic › Need to decryption solutions to monitor › But volume of traffic is too large IDS IPS
7. Network And Host Level Servers Switch Switch Switch Servers Servers Servers Backbone Only monitoring purpose If a disorder occurs › All Server under can’t be monitored Servers Servers Servers Servers Servers Servers Servers Servers IDS IPS
8. Network And Host Level Servers Switch Switch Switch Servers Servers Servers Servers Servers Servers Servers Backbone If a disorder occurs at inline solutions › Bypass all traffic Servers Servers Servers Servers IDS IPS
9. Network And Host Level Switch Switch Switch Backbone Servers Servers Servers Servers Servers Servers Servers Servers Servers Servers Servers Servers No solution, no traffic can be monitored IDS IPS
10. Network And Host Level Switch Switch Switch Backbone Servers Servers Servers Servers Servers Servers Servers Servers Apply security policies › Servers under are applied same policy Servers Servers Servers Servers IDS IPS
11. Network And Host Level Lateral Movement › Need to monitor internal traffic › SMB Exploit (MS17-010 vulnerability) › No network firewall › No IDS/IPS Switch Server Server Server
12. LINE Host Level Security Hybrid Environment On Premises Public Cloud Private Cloud AWS / GCP / Azure Verda
13. Log Inspection IPS Application Control Firewall Integrity Monitoring Anti-malware HIDS Agent Modules Multiple Security Functions
14. Agenda › Introduction › Provisioning › Architecture › Operation › Automation › Case Study
15. Provision when Server Server Development in progress after server is shipped › Installing agents is big burden › Anxiety about service failure Installing when the server is shipped › Prevention of omission › No pressure on developers either
16. Provision Target Fintech 3,115 Verda Dev 17,389 Windows 658 As of September 2020
17. Provision On-promise Provision Fintech 3,115 › Initially IT Department handle installation › Use Installation script provided by Infra Protection Team › Separated by services › Store in Git repository › Git Watch - notify about changes › Move to internal YUM repository Periodic management Windows 658
18. Provision Private Cloud Provision Verda Dev 17,389 › Init-script › When instances launch, the script runs › Use Installation script provided by IPT › Separated by specifications (high / low) › Store in Git repository › Git Watch - notify about changes Automate Installation
19. Increase in number of agents The No of Installed HIDS agents (2014~2020) 0 5000 10000 15000 20000 25000 30000 2014 2015 2016 2017 2018 2019 2020 KR JP Total As of September 2020
20. Agenda › Introduction › Provisioning › Architecture › Operation › Automation › Case Study
21. Distributed Expansion Configuration › Fintech / Compliance / Production servers › Verda Dev › External (Non LINE Asset, AWS, etc.) By Infra High Ability › Configured at least two or more servers › Managers: Load Balancing › Databases: Failover › Databases are managed by DB Team
22. Distributed Expansion Configuration Taking advantage of internal resource › Verda Infra › Various Services and unpredictable › Quick and easy expansion / change
23. Centralized Log Aggregation Elasticsearch Log GW SIEM Connector Manager #2 Hadoop for archiving Analysis systems Store logs for 53 weeks Agents Manager #1 Long-term preservation Log monitoring Aggregation
24. Agenda › Introduction › Provisioning › Architecture › Operation › Automation › Case Study
25. Operation Access Control › HTTPS › Use domain account › RBAC › No common account › Rotate API account Access to Web Console Access to Servers › Only Administers can access to › PAM to restrict › Use domain account › Monitor behavior Manager Database Domain Controller PAM Web Console
26. Operation Groups & Policy LVC LINE Insurance LINE Securities LINE FRIENDS LINE Mobile Ads Internal systems Service Name
27. Operation Groups & Policy Fintech Private Public LVC LINE Insurance LINE Securities LINE FRIENDS LINE Mobile Ads Internal systems Groups 2nd Service Name › Exposure to external attacks › Easy to organize and manage › Fast to check current situation
28. Operation Groups & Policy Important Windows Linux DIST KMS Default Services Default Services Policy Flexible response to service expansion Actual way to detect/prevent actions › By OS Platform › From Default policy › Easy to send policies against vulnerability
29. Operation Groups & Policy Fintech Private Public LVC LINE Insurance LINE Securities LINE FRIENDS LINE Mobile Ads Internal systems Services Services Apply discrete policies by service Criterion of use › Active Directory › Databases › Web Applications › Analysis (YARN, HBase)
30. Agenda › Introduction › Provisioning › Architecture › Operation › Automation › Case Study
31. Automation What to be considered Deletion Agent status == offline Malfunction Disk insufficient Communication error Pattern update error Duplicated error Engine offline error Vaildation Installed or not Works properly Upgrade needed
32. Automation What to be considered Deletion Agent status == offline › Not in asset data › Unused computer Sure to delete?
33. Automation What to be considered Malfunction Disk insufficient Communication error Pattern update error Duplicated error Engine offline error › Who can handle these problems? › Disk problem? › Communication error? › Pattern, Duplicated, engine problems? Classify all cases
34. Automation What to be considered Malfunction Disk insufficient Communication error Pattern update error Duplicated error Engine offline error › Who can handle these problems? › Disk problem? › Sending e-mail to development operator › Communication error? › Agent-manager port open? (reinstall agent or restart service) › Pattern, Duplicated, engine problems? › Administrator handles Classify all cases
35. Automation What to be considered Vaildation Installed or not Works properly Upgrade needed › Inform of the status of the uninstalled › Communicate to schedule available date for installation / upgrade › Installation is conducted by whom? Sending scripts to? Fintech Server
36. Automation What to be considered Vaildation Installed or not Works properly Upgrade needed › Inform of the status of the uninstalled › Communicate to schedule available date for installation / upgrade › Sending e-mail every week automatically to each ITOPS › Installation is conducted by administrator Fintech Server
37. Automation Work Flow Deletion Malfunction Vaildation API Administrator ITOPS
38. Agenda › Introduction › Provisioning › Architecture › Operation › Automation › Case Study
39. Verda Kubernetes Service Case 1 Messenger # It appears that there is a security agent problem with the VMs used by Verda Kubernetes Service (VKS).Could you please confirm it? * Security agent somehow hooks system call and does Anti-malware * VKS may attempt to count from the kubernetes container process in a different namespace, but may fail * It works fine when you unload the security agent kernel module. * I'm thinking about whether or not to include the fs(tmpfs) that Kubernetes is going to count. Exception for tmpfs?
40. Verda Kubernetes Service Case 1 Node cAdvisor Kube Proxy Kubelet Pod Containers Pod Containers General Architecture of Kubernetes
41. Verda Kubernetes Service Case 1 Node Kubelet Pod Containers /tmpfs/.. › VKS is slightly different › Kubelet runs inside a container
42. Verda Kubernetes Service Case 1 Node /tmpfs/.. umount Kubelet › When a pod is removed, kubelet send umount call to detach volumes. › But..
43. Verda Kubernetes Service Case 1 Node /tmpfs/.. umount Kubelet › It doesn’t work out for some reason, just ignored.
44. Verda Kubernetes Service Case 1 Let me check message logs › Can see which sys calls are hooked by the driver › Found sys_mount /sys_umount call hooking log
45. Verda Kubernetes Service Case 1 › From dmesg-1585276062.log, the driver got a sys_umount event, then ignored it because the system call was not invoked from the host, but from a container. › By design, the anti-malware driver will track mount points in a host and will only deal with mount/umount system call events coming from the host, which means mount/unmount system call events will be ignored by the driver if they are from containers.
46. Verda Kubernetes Service Case 1 Application USER KERNEL SYSTABLE SYSCALL
47. Verda Kubernetes Service Case 1 Application USER KERNEL SYSTABLE SYSCALL Hook sys_open sys_close sys_execve CALLBACKS
48. Verda Kubernetes Service Case 1 Application USER KERNEL SYSTABLE SYSCALL Hook sys_open sys_close sys_execve pre-open post-close pre-execve CALLBACKS HANDLERS Event Eval Handle Scan
49. Verda Kubernetes Service Case 1 Application USER KERNEL SYSTABLE SYSCALL Hook sys_open sys_close sys_execve pre-open post-close pre-execve CALLBACKS HANDLERS Event Eval Handle IOCTL Scan Manager GET REPLY
50. Verda Kubernetes Service Case 1 Application USER KERNEL SYSTABLE SYSCALL Hook sys_open sys_close sys_execve pre-open post-close pre-execve CALLBACKS HANDLERS Event Eval Handle IOCTL Scan Manager GET REPLY /proc/driver/HOOK/ Configure
51. Case 1 Verda Kubernetes Service Over 4K nodes found › Deploy pod to continually monitor and change parameter within all nodes › Failed (root permission needed) Pod Monitor and Change Kubelet Pod Containers Nodes Parameter /proc/driver/HOOK/
52. Case 1 Verda Kubernetes Service Over 4K nodes found › Deploy pod to continually monitor and change parameter within all nodes › Failed (root permission needed) › Plan B › Use Cron to execute script › existing nodes (pod) › newly created nodes (init-script) › Docker process runs › kubelet exist inside
53. Verda Redis Case 2 Messenger # Hello. I have an inquiry regarding Verda redis. Can you tell me the coverage of security agent? Currently, we are simply testing performance in dev, stage-dev environments, but due to the influence of Security Agent, the performance difference is between 30 and 50%. Therefore, I would like to check the application scope of security agent and the possibility of future expansion.
54. Verda Redis Case 2
55. Verda Redis Case 2
56. Verda Redis Case 2 Total 25 rules applied to avoid monitoring unnecessary DB port traffic 1433, 1434 MSSQL 3306 10306, 20306, 13306, 13316 MySQL 1521-1526, 1725 Oracle 7000 Redis 27017, 27018, 27019 Mongo
57. Verda Redis Case 2 approximately 70% No rules applied Applied all rules
58. Verda Redis Case 2 No rules applied Applied all rules Applied DB port exception approximately 15%
59. Epilogue • Scalable architecture design • Integrated management • Establish proper standards • Let the robot do it • Compromise with Services
60. Thank you