Share
Embed
Host level security with HIDS agents on 20,000+ hosts
Taeee Kim
LINE Infra Protection Team Security Engineer
https://linedevday.linecorp.com/2020/ja/sessions/0429
https://linedevday.linecorp.com/2020/en/sessions/0429
LINE DevDay 2020
November 25, 2020
Transcript
60 slides
-
Agenda › Introduction › Provisioning › Architecture › Operation ›
Automation › Case Study -
Agenda › Introduction › Provisioning › Architecture › Operation ›
Automation › Case Study -
Host-Level security Necessity Limitation of Traditional Security Correlation analysis required
› Process, logs, network, configuration etc. › Evolving Attack Technology and Types › Mainly only anti-malware agent installed to servers › New types of cyber threats that traditional anti-virus programs can't stop. (Ransomware) Move to Edge network from External › Lateral movement, Worm › Targeting attack -
Network And Host Level Switch Switch Switch Backbone Servers Servers
Servers Servers Servers Servers Servers Servers Servers Servers Servers Servers Encrypted Traffic › Need to decryption solutions to monitor › But volume of traffic is too large IDS IPS -
Network And Host Level Servers Switch Switch Switch Servers Servers
Servers Backbone Only monitoring purpose If a disorder occurs › All Server under can’t be monitored Servers Servers Servers Servers Servers Servers Servers Servers IDS IPS -
Network And Host Level Servers Switch Switch Switch Servers Servers
Servers Servers Servers Servers Servers Backbone If a disorder occurs at inline solutions › Bypass all traffic Servers Servers Servers Servers IDS IPS -
Network And Host Level Switch Switch Switch Backbone Servers Servers
Servers Servers Servers Servers Servers Servers Servers Servers Servers Servers No solution, no traffic can be monitored IDS IPS -
Network And Host Level Switch Switch Switch Backbone Servers Servers
Servers Servers Servers Servers Servers Servers Apply security policies › Servers under are applied same policy Servers Servers Servers Servers IDS IPS -
Network And Host Level Lateral Movement › Need to monitor
internal traffic › SMB Exploit (MS17-010 vulnerability) › No network firewall › No IDS/IPS Switch Server Server Server -
LINE Host Level Security Hybrid Environment On Premises Public Cloud
Private Cloud AWS / GCP / Azure Verda -
Log Inspection IPS Application Control Firewall Integrity Monitoring Anti-malware HIDS
Agent Modules Multiple Security Functions -
Agenda › Introduction › Provisioning › Architecture › Operation ›
Automation › Case Study -
Provision when Server Server Development in progress after server is
shipped › Installing agents is big burden › Anxiety about service failure Installing when the server is shipped › Prevention of omission › No pressure on developers either -
Provision Target Fintech 3,115 Verda Dev 17,389 Windows 658 As
of September 2020 -
Provision On-promise Provision Fintech 3,115 › Initially IT Department handle
installation › Use Installation script provided by Infra Protection Team › Separated by services › Store in Git repository › Git Watch - notify about changes › Move to internal YUM repository Periodic management Windows 658 -
Provision Private Cloud Provision Verda Dev 17,389 › Init-script ›
When instances launch, the script runs › Use Installation script provided by IPT › Separated by specifications (high / low) › Store in Git repository › Git Watch - notify about changes Automate Installation -
Increase in number of agents The No of Installed HIDS
agents (2014~2020) 0 5000 10000 15000 20000 25000 30000 2014 2015 2016 2017 2018 2019 2020 KR JP Total As of September 2020 -
Agenda › Introduction › Provisioning › Architecture › Operation ›
Automation › Case Study -
Distributed Expansion Configuration › Fintech / Compliance / Production servers
› Verda Dev › External (Non LINE Asset, AWS, etc.) By Infra High Ability › Configured at least two or more servers › Managers: Load Balancing › Databases: Failover › Databases are managed by DB Team -
Distributed Expansion Configuration Taking advantage of internal resource › Verda
Infra › Various Services and unpredictable › Quick and easy expansion / change -
Centralized Log Aggregation Elasticsearch Log GW SIEM Connector Manager #2
Hadoop for archiving Analysis systems Store logs for 53 weeks Agents Manager #1 Long-term preservation Log monitoring Aggregation -
Agenda › Introduction › Provisioning › Architecture › Operation ›
Automation › Case Study -
Operation Access Control › HTTPS › Use domain account ›
RBAC › No common account › Rotate API account Access to Web Console Access to Servers › Only Administers can access to › PAM to restrict › Use domain account › Monitor behavior Manager Database Domain Controller PAM Web Console -
Operation Groups & Policy LVC LINE Insurance LINE Securities LINE
FRIENDS LINE Mobile Ads Internal systems Service Name -
Operation Groups & Policy Fintech Private Public LVC LINE Insurance
LINE Securities LINE FRIENDS LINE Mobile Ads Internal systems Groups 2nd Service Name › Exposure to external attacks › Easy to organize and manage › Fast to check current situation -
Operation Groups & Policy Important Windows Linux DIST KMS Default
Services Default Services Policy Flexible response to service expansion Actual way to detect/prevent actions › By OS Platform › From Default policy › Easy to send policies against vulnerability -
Operation Groups & Policy Fintech Private Public LVC LINE Insurance
LINE Securities LINE FRIENDS LINE Mobile Ads Internal systems Services Services Apply discrete policies by service Criterion of use › Active Directory › Databases › Web Applications › Analysis (YARN, HBase) -
Agenda › Introduction › Provisioning › Architecture › Operation ›
Automation › Case Study -
Automation What to be considered Deletion Agent status == offline
Malfunction Disk insufficient Communication error Pattern update error Duplicated error Engine offline error Vaildation Installed or not Works properly Upgrade needed -
Automation What to be considered Deletion Agent status == offline
› Not in asset data › Unused computer Sure to delete? -
Automation What to be considered Malfunction Disk insufficient Communication error
Pattern update error Duplicated error Engine offline error › Who can handle these problems? › Disk problem? › Communication error? › Pattern, Duplicated, engine problems? Classify all cases -
Automation What to be considered Malfunction Disk insufficient Communication error
Pattern update error Duplicated error Engine offline error › Who can handle these problems? › Disk problem? › Sending e-mail to development operator › Communication error? › Agent-manager port open? (reinstall agent or restart service) › Pattern, Duplicated, engine problems? › Administrator handles Classify all cases -
Automation What to be considered Vaildation Installed or not Works
properly Upgrade needed › Inform of the status of the uninstalled › Communicate to schedule available date for installation / upgrade › Installation is conducted by whom? Sending scripts to? Fintech Server -
Automation What to be considered Vaildation Installed or not Works
properly Upgrade needed › Inform of the status of the uninstalled › Communicate to schedule available date for installation / upgrade › Sending e-mail every week automatically to each ITOPS › Installation is conducted by administrator Fintech Server -
Agenda › Introduction › Provisioning › Architecture › Operation ›
Automation › Case Study -
Verda Kubernetes Service Case 1 Messenger # It appears that
there is a security agent problem with the VMs used by Verda Kubernetes Service (VKS).Could you please confirm it? * Security agent somehow hooks system call and does Anti-malware * VKS may attempt to count from the kubernetes container process in a different namespace, but may fail * It works fine when you unload the security agent kernel module. * I'm thinking about whether or not to include the fs(tmpfs) that Kubernetes is going to count. Exception for tmpfs? -
Verda Kubernetes Service Case 1 Node cAdvisor Kube Proxy Kubelet
Pod Containers Pod Containers General Architecture of Kubernetes -
Verda Kubernetes Service Case 1 Node Kubelet Pod Containers /tmpfs/..
› VKS is slightly different › Kubelet runs inside a container -
Verda Kubernetes Service Case 1 Node /tmpfs/.. umount Kubelet ›
When a pod is removed, kubelet send umount call to detach volumes. › But.. -
Verda Kubernetes Service Case 1 Node /tmpfs/.. umount Kubelet ›
It doesn’t work out for some reason, just ignored. -
Verda Kubernetes Service Case 1 Let me check message logs
› Can see which sys calls are hooked by the driver › Found sys_mount /sys_umount call hooking log -
Verda Kubernetes Service Case 1 › From dmesg-1585276062.log, the driver
got a sys_umount event, then ignored it because the system call was not invoked from the host, but from a container. › By design, the anti-malware driver will track mount points in a host and will only deal with mount/umount system call events coming from the host, which means mount/unmount system call events will be ignored by the driver if they are from containers. -
Verda Kubernetes Service Case 1 Application USER KERNEL SYSTABLE SYSCALL
Hook sys_open sys_close sys_execve CALLBACKS -
Verda Kubernetes Service Case 1 Application USER KERNEL SYSTABLE SYSCALL
Hook sys_open sys_close sys_execve pre-open post-close pre-execve CALLBACKS HANDLERS Event Eval Handle Scan -
Verda Kubernetes Service Case 1 Application USER KERNEL SYSTABLE SYSCALL
Hook sys_open sys_close sys_execve pre-open post-close pre-execve CALLBACKS HANDLERS Event Eval Handle IOCTL Scan Manager GET REPLY -
Verda Kubernetes Service Case 1 Application USER KERNEL SYSTABLE SYSCALL
Hook sys_open sys_close sys_execve pre-open post-close pre-execve CALLBACKS HANDLERS Event Eval Handle IOCTL Scan Manager GET REPLY /proc/driver/HOOK/ Configure -
Case 1 Verda Kubernetes Service Over 4K nodes found ›
Deploy pod to continually monitor and change parameter within all nodes › Failed (root permission needed) Pod Monitor and Change Kubelet Pod Containers Nodes Parameter /proc/driver/HOOK/ -
Case 1 Verda Kubernetes Service Over 4K nodes found ›
Deploy pod to continually monitor and change parameter within all nodes › Failed (root permission needed) › Plan B › Use Cron to execute script › existing nodes (pod) › newly created nodes (init-script) › Docker process runs › kubelet exist inside -
Verda Redis Case 2 Messenger # Hello. I have an
inquiry regarding Verda redis. Can you tell me the coverage of security agent? Currently, we are simply testing performance in dev, stage-dev environments, but due to the influence of Security Agent, the performance difference is between 30 and 50%. Therefore, I would like to check the application scope of security agent and the possibility of future expansion. -
Verda Redis Case 2 Total 25 rules applied to avoid
monitoring unnecessary DB port traffic 1433, 1434 MSSQL 3306 10306, 20306, 13306, 13316 MySQL 1521-1526, 1725 Oracle 7000 Redis 27017, 27018, 27019 Mongo -
Verda Redis Case 2 No rules applied Applied all rules
Applied DB port exception approximately 15% -
Epilogue • Scalable architecture design • Integrated management • Establish
proper standards • Let the robot do it • Compromise with Services