Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Host level security with HIDS agents on 20,000+ hosts

Host level security with HIDS agents on 20,000+ hosts

Eebedc2ee7ff95ffb9d9102c6d4a065c?s=128

LINE DevDay 2020

November 25, 2020
Tweet

Transcript

  1. None
  2. Background

  3. Agenda › Introduction › Provisioning › Architecture › Operation ›

    Automation › Case Study
  4. Agenda › Introduction › Provisioning › Architecture › Operation ›

    Automation › Case Study
  5. Host-Level security Necessity Limitation of Traditional Security Correlation analysis required

    › Process, logs, network, configuration etc. › Evolving Attack Technology and Types › Mainly only anti-malware agent installed to servers › New types of cyber threats that traditional anti-virus programs can't stop. (Ransomware) Move to Edge network from External › Lateral movement, Worm › Targeting attack
  6. Network And Host Level Switch Switch Switch Backbone Servers Servers

    Servers Servers Servers Servers Servers Servers Servers Servers Servers Servers Encrypted Traffic › Need to decryption solutions to monitor › But volume of traffic is too large IDS IPS
  7. Network And Host Level Servers Switch Switch Switch Servers Servers

    Servers Backbone Only monitoring purpose If a disorder occurs › All Server under can’t be monitored Servers Servers Servers Servers Servers Servers Servers Servers IDS IPS
  8. Network And Host Level Servers Switch Switch Switch Servers Servers

    Servers Servers Servers Servers Servers Backbone If a disorder occurs at inline solutions › Bypass all traffic Servers Servers Servers Servers IDS IPS
  9. Network And Host Level Switch Switch Switch Backbone Servers Servers

    Servers Servers Servers Servers Servers Servers Servers Servers Servers Servers No solution, no traffic can be monitored IDS IPS
  10. Network And Host Level Switch Switch Switch Backbone Servers Servers

    Servers Servers Servers Servers Servers Servers Apply security policies › Servers under are applied same policy Servers Servers Servers Servers IDS IPS
  11. Network And Host Level Lateral Movement › Need to monitor

    internal traffic › SMB Exploit (MS17-010 vulnerability) › No network firewall › No IDS/IPS Switch Server Server Server
  12. LINE Host Level Security Hybrid Environment On Premises Public Cloud

    Private Cloud AWS / GCP / Azure Verda
  13. Log Inspection IPS Application Control Firewall Integrity Monitoring Anti-malware HIDS

    Agent Modules Multiple Security Functions
  14. Agenda › Introduction › Provisioning › Architecture › Operation ›

    Automation › Case Study
  15. Provision when Server Server Development in progress after server is

    shipped › Installing agents is big burden › Anxiety about service failure Installing when the server is shipped › Prevention of omission › No pressure on developers either
  16. Provision Target Fintech 3,115 Verda Dev 17,389 Windows 658 As

    of September 2020
  17. Provision On-promise Provision Fintech 3,115 › Initially IT Department handle

    installation › Use Installation script provided by Infra Protection Team › Separated by services › Store in Git repository › Git Watch - notify about changes › Move to internal YUM repository Periodic management Windows 658
  18. Provision Private Cloud Provision Verda Dev 17,389 › Init-script ›

    When instances launch, the script runs › Use Installation script provided by IPT › Separated by specifications (high / low) › Store in Git repository › Git Watch - notify about changes Automate Installation
  19. Increase in number of agents The No of Installed HIDS

    agents (2014~2020) 0 5000 10000 15000 20000 25000 30000 2014 2015 2016 2017 2018 2019 2020 KR JP Total As of September 2020
  20. Agenda › Introduction › Provisioning › Architecture › Operation ›

    Automation › Case Study
  21. Distributed Expansion Configuration › Fintech / Compliance / Production servers

    › Verda Dev › External (Non LINE Asset, AWS, etc.) By Infra High Ability › Configured at least two or more servers › Managers: Load Balancing › Databases: Failover › Databases are managed by DB Team
  22. Distributed Expansion Configuration Taking advantage of internal resource › Verda

    Infra › Various Services and unpredictable › Quick and easy expansion / change
  23. Centralized Log Aggregation Elasticsearch Log GW SIEM Connector Manager #2

    Hadoop for archiving Analysis systems Store logs for 53 weeks Agents Manager #1 Long-term preservation Log monitoring Aggregation
  24. Agenda › Introduction › Provisioning › Architecture › Operation ›

    Automation › Case Study
  25. Operation Access Control › HTTPS › Use domain account ›

    RBAC › No common account › Rotate API account Access to Web Console Access to Servers › Only Administers can access to › PAM to restrict › Use domain account › Monitor behavior Manager Database Domain Controller PAM Web Console
  26. Operation Groups & Policy LVC LINE Insurance LINE Securities LINE

    FRIENDS LINE Mobile Ads Internal systems Service Name
  27. Operation Groups & Policy Fintech Private Public LVC LINE Insurance

    LINE Securities LINE FRIENDS LINE Mobile Ads Internal systems Groups 2nd Service Name › Exposure to external attacks › Easy to organize and manage › Fast to check current situation
  28. Operation Groups & Policy Important Windows Linux DIST KMS Default

    Services Default Services Policy Flexible response to service expansion Actual way to detect/prevent actions › By OS Platform › From Default policy › Easy to send policies against vulnerability
  29. Operation Groups & Policy Fintech Private Public LVC LINE Insurance

    LINE Securities LINE FRIENDS LINE Mobile Ads Internal systems Services Services Apply discrete policies by service Criterion of use › Active Directory › Databases › Web Applications › Analysis (YARN, HBase)
  30. Agenda › Introduction › Provisioning › Architecture › Operation ›

    Automation › Case Study
  31. Automation What to be considered Deletion Agent status == offline

    Malfunction Disk insufficient Communication error Pattern update error Duplicated error Engine offline error Vaildation Installed or not Works properly Upgrade needed
  32. Automation What to be considered Deletion Agent status == offline

    › Not in asset data › Unused computer Sure to delete?
  33. Automation What to be considered Malfunction Disk insufficient Communication error

    Pattern update error Duplicated error Engine offline error › Who can handle these problems? › Disk problem? › Communication error? › Pattern, Duplicated, engine problems? Classify all cases
  34. Automation What to be considered Malfunction Disk insufficient Communication error

    Pattern update error Duplicated error Engine offline error › Who can handle these problems? › Disk problem? › Sending e-mail to development operator › Communication error? › Agent-manager port open? (reinstall agent or restart service) › Pattern, Duplicated, engine problems? › Administrator handles Classify all cases
  35. Automation What to be considered Vaildation Installed or not Works

    properly Upgrade needed › Inform of the status of the uninstalled › Communicate to schedule available date for installation / upgrade › Installation is conducted by whom? Sending scripts to? Fintech Server
  36. Automation What to be considered Vaildation Installed or not Works

    properly Upgrade needed › Inform of the status of the uninstalled › Communicate to schedule available date for installation / upgrade › Sending e-mail every week automatically to each ITOPS › Installation is conducted by administrator Fintech Server
  37. Automation Work Flow Deletion Malfunction Vaildation API Administrator ITOPS

  38. Agenda › Introduction › Provisioning › Architecture › Operation ›

    Automation › Case Study
  39. Verda Kubernetes Service Case 1 Messenger # It appears that

    there is a security agent problem with the VMs used by Verda Kubernetes Service (VKS).Could you please confirm it? * Security agent somehow hooks system call and does Anti-malware * VKS may attempt to count from the kubernetes container process in a different namespace, but may fail * It works fine when you unload the security agent kernel module. * I'm thinking about whether or not to include the fs(tmpfs) that Kubernetes is going to count. Exception for tmpfs?
  40. Verda Kubernetes Service Case 1 Node cAdvisor Kube Proxy Kubelet

    Pod Containers Pod Containers General Architecture of Kubernetes
  41. Verda Kubernetes Service Case 1 Node Kubelet Pod Containers /tmpfs/..

    › VKS is slightly different › Kubelet runs inside a container
  42. Verda Kubernetes Service Case 1 Node /tmpfs/.. umount Kubelet ›

    When a pod is removed, kubelet send umount call to detach volumes. › But..
  43. Verda Kubernetes Service Case 1 Node /tmpfs/.. umount Kubelet ›

    It doesn’t work out for some reason, just ignored.
  44. Verda Kubernetes Service Case 1 Let me check message logs

    › Can see which sys calls are hooked by the driver › Found sys_mount /sys_umount call hooking log
  45. Verda Kubernetes Service Case 1 › From dmesg-1585276062.log, the driver

    got a sys_umount event, then ignored it because the system call was not invoked from the host, but from a container. › By design, the anti-malware driver will track mount points in a host and will only deal with mount/umount system call events coming from the host, which means mount/unmount system call events will be ignored by the driver if they are from containers.
  46. Verda Kubernetes Service Case 1 Application USER KERNEL SYSTABLE SYSCALL

  47. Verda Kubernetes Service Case 1 Application USER KERNEL SYSTABLE SYSCALL

    Hook sys_open sys_close sys_execve CALLBACKS
  48. Verda Kubernetes Service Case 1 Application USER KERNEL SYSTABLE SYSCALL

    Hook sys_open sys_close sys_execve pre-open post-close pre-execve CALLBACKS HANDLERS Event Eval Handle Scan
  49. Verda Kubernetes Service Case 1 Application USER KERNEL SYSTABLE SYSCALL

    Hook sys_open sys_close sys_execve pre-open post-close pre-execve CALLBACKS HANDLERS Event Eval Handle IOCTL Scan Manager GET REPLY
  50. Verda Kubernetes Service Case 1 Application USER KERNEL SYSTABLE SYSCALL

    Hook sys_open sys_close sys_execve pre-open post-close pre-execve CALLBACKS HANDLERS Event Eval Handle IOCTL Scan Manager GET REPLY /proc/driver/HOOK/ Configure
  51. Case 1 Verda Kubernetes Service Over 4K nodes found ›

    Deploy pod to continually monitor and change parameter within all nodes › Failed (root permission needed) Pod Monitor and Change Kubelet Pod Containers Nodes Parameter /proc/driver/HOOK/
  52. Case 1 Verda Kubernetes Service Over 4K nodes found ›

    Deploy pod to continually monitor and change parameter within all nodes › Failed (root permission needed) › Plan B › Use Cron to execute script › existing nodes (pod) › newly created nodes (init-script) › Docker process runs › kubelet exist inside
  53. Verda Redis Case 2 Messenger # Hello. I have an

    inquiry regarding Verda redis. Can you tell me the coverage of security agent? Currently, we are simply testing performance in dev, stage-dev environments, but due to the influence of Security Agent, the performance difference is between 30 and 50%. Therefore, I would like to check the application scope of security agent and the possibility of future expansion.
  54. Verda Redis Case 2

  55. Verda Redis Case 2

  56. Verda Redis Case 2 Total 25 rules applied to avoid

    monitoring unnecessary DB port traffic 1433, 1434 MSSQL 3306 10306, 20306, 13306, 13316 MySQL 1521-1526, 1725 Oracle 7000 Redis 27017, 27018, 27019 Mongo
  57. Verda Redis Case 2 approximately 70% No rules applied Applied

    all rules
  58. Verda Redis Case 2 No rules applied Applied all rules

    Applied DB port exception approximately 15%
  59. Epilogue • Scalable architecture design • Integrated management • Establish

    proper standards • Let the robot do it • Compromise with Services
  60. Thank you