Adventures in Hardening the Security of LINE's Infra

Adventures in Hardening the Security of LINE's Infra

ByoungYun Lee
LINE Plus GrayLab Security Engineer
https://linedevday.linecorp.com/jp/2019/sessions/F1-4

Be4518b119b8eb017625e0ead20f8fe7?s=128

LINE DevDay 2019

November 20, 2019
Tweet

Transcript

  1. 2019 DevDay Adventures in Hardening the Security of LINE's Infra

    > ByoungYun Lee > LINE Plus GrayLab Security Engineer
  2. Who Am I Security Risk Assessment @LINE > since 2018

    Offensive Security Research @GrayHash > for 3 years
  3. Scott ikeda, CPO Magazine, April 9, 2019 Eustance Huang, CNBC,

    May 4, 2019 THE PAPERS, 1 May, 2019
  4. Infrastructures An Attacker’s View

  5. Public Network Web Server API Server Database User

  6. Public Network Web Server API Server Database User www.example.com

  7. Public Network Web Server API Server Database User www.example.com api.example.com

  8. Public Network Web Server API Server Database User www.example.com api.example.com

    ?
  9. Public Network Web Server API Server Database Attacker

  10. Public Network Web Server API Server Database Attacker db.example.com database.example.com

  11. Public Network Web Server API Server Database Attacker

  12. Public Network Web Server API Server Database Attacker Developer

  13. Public Network Web Server API Server Database Attacker

  14. Public Network Web Server API Server Database Attacker Weak Password

  15. Public Network Web Server API Server Database Attacker Weak Password

  16. Public Network Web Server API Server Database Attacker Weak Password

    1, 1!, !@#, …
  17. Public Network Web Server API Server Database Attacker Port Scanning

  18. Public Network web pages etc ssh Web Server API Server

    Database Attacker Port Scanning
  19. Public Network Web Server API Server Database Attacker Exploit

  20. Public Network Web Server API Server Database Attacker Exploit

  21. Public Network Web Server API Server Database Attacker Exploit

  22. Public Network Web Server API Server Database Attacker Exploit

  23. Public Network Web Server API Server Database Attacker

  24. Public Network Web Server API Server Database Attacker Danger

  25. PRIVATE NETWORK Private Network Web Server API Server Database User

    Internal Servers
  26. Private Network PRIVATE NETWORK Web Server API Server Database User

    Internal Servers
  27. Private Network PRIVATE NETWORK Web Server API Server Database User

    Internal Servers
  28. Private Network PRIVATE NETWORK Web Server API Server Database Internal

    Servers Attacker
  29. Private Network PRIVATE NETWORK Internal Servers Users

  30. Private Network PRIVATE NETWORK Internal Servers Users

  31. Private Network PRIVATE NETWORK Internal Servers Users Attacker

  32. Lateral Movement Attack PRIVATE NETWORK Internal Servers Users Attacker

  33. Lateral Movement Attack PRIVATE NETWORK Internal Servers Users Attacker

  34. Lateral Movement Attack PRIVATE NETWORK Internal Servers Users Attacker

  35. Lateral Movement Attack PRIVATE NETWORK Internal Servers Users Attacker

  36. Lateral Movement Attack PRIVATE NETWORK Internal Servers Users Attacker Danger

  37. LINE’s Infrastructure PRIVATE NETWORK Internal Servers Users

  38. LINE’s Infrastructure PRIVATE NETWORK Internal Servers Users

  39. LINE’s Infrastructure PRIVATE NETWORK Internal Servers Users Attacker

  40. LINE’s Infrastructure PRIVATE NETWORK Internal Servers Users Attacker

  41. Goal PRIVATE NETWORK Breach SAFE SAFE SAFE

  42. Goal TCP/7000 10.0.0.1-10 TCP/5000 10.0.0.51-60 10.0.0.200

  43. Source IP Destination IP Destination Port Action 10.0.0.1-10 10.0.0.200 7000

    Allow Other 10.0.0.200 7000 Deny 10.0.0.51-60 10.0.0.200 5000 Allow Other 10.0.0.200 5000 Deny Goal
  44. Goal TCP/7000 10.0.0.1-10 TCP/5000 10.0.0.51-60 10.0.0.200 10.0.2.1-10

  45. Source IP Destination IP Destination Port Action 10.0.0.1-10 10.0.0.200 7000

    Allow 10.0.3.1-10 10.0.0.200 7000 Allow 10.0.4.1-10 10.0.0.200 7000 Allow 10.0.5.1-10 10.0.0.200 7000 Allow 10.0.100.50 10.0.0.200 5000 Allow 10.0.200.50 10.0.0.200 5000 Allow 10.0.300.50 10.0.0.200 5000 Allow 10.0.400.50 10.0.0.200 5000 Allow Goal
  46. Source IP Destination IP Destination Port Action 10.0.0.1-10 10.0.0.200 7000

    Allow 10.0.3.1-10 10.0.0.200 7000 Allow 10.0.4.1-10 10.0.0.200 7000 Allow 10.0.5.1-10 10.0.0.200 7000 Allow 10.0.100.50 10.0.0.200 5000 Allow 10.0.200.50 10.0.0.200 5000 Allow 10.0.300.50 10.0.0.200 5000 Allow 10.0.400.50 10.0.0.200 5000 Allow Goal
  47. Hardening Block unnecessary network access

  48. Tools

  49. LOW- MAINTENANCE Requirements CLOUD SUPPORT MULTI 
 REGIONAL PRE-EXISTING SERVICE

  50. LOW- MAINTENANCE Requirements CLOUD SUPPORT MULTI 
 REGIONAL PRE-EXISTING SERVICE

  51. Candidates We are aiming to block unnecessary network access iptables

    Network Firewall Host-based Firewall
  52. PROS iptables Easy-to-use Everyone knows how to use No cost

    No physical constraints, cloud-compatible
  53. PROS iptables Easy-to-use Everyone knows how to use No cost

    No physical constraints, cloud-compatible
  54. CONS iptables Easy-to-use Anyone can create security holes Not centralized

    = high maintenance
  55. CONS iptables Easy-to-use Anyone can create security holes Not centralized

    = high maintenance
  56. CONS iptables Easy-to-use Anyone can create security holes Not centralized

    = high maintenance
  57. PROs Network Firewall No resource overhead for hosts Already deployed:

    zero learning curve
  58. PROs Network Firewall No resource overhead for hosts Already deployed:

    zero learning curve
  59. CONs Network Firewall High cost Physical constraints: component proximity required

    Physical constraint: Incompatible with cloud infrastructure
  60. CONs Network Firewall High cost Physical constraints: component proximity required

    Physical constraint: Incompatible with cloud infrastructure
  61. Host-Based Firewall Install Firewall Apply Rules Make Rules

  62. Host-Based Firewall Install Firewall Apply Rules Make Rules

  63. Host-Based Firewall Install Firewall Apply Rules Make Rules

  64. PROs Host-Based Firewall No physical constraints, cloud-compatible Centralized administration ‘Detect

    Only’ mode
  65. PROs Host-Based Firewall No physical constraints, cloud-compatible Centralized administration ‘Detect

    Only’ mode
  66. PROs Host-Based Firewall No physical constraints, cloud-compatible Centralized administration ‘Detect

    Only’ mode
  67. So... iptables Network Firewall Host-based
 Firewall The Host-based Firewall Solution

    is the best option
  68. Hardening Steps Analyze target Design process Apply

  69. Service A's Database Servers about 700 hosts, 27 groups, 2,000

    clients Analyze Target Various Database Types • HBase • Redis • etc • Cassandra • MySQL
  70. Service A's Database Servers about 700 hosts, 27 groups, 2,000

    clients Analyze Target Various Database Types • HBase • Redis • etc • Cassandra • MySQL
  71. Service A's Database Servers about 700 hosts, 27 groups, 2,000

    clients Analyze Target Various Database Types • HBase • Redis • etc • Cassandra • MySQL
  72. Service A's Database Servers about 700 hosts, 27 groups, 2,000

    clients Analyze Target Various Database Types • HBase • Redis • etc • Cassandra • MySQL
  73. Service A's Database Servers about 700 hosts, 27 groups, 2,000

    clients Analyze Target Various Database Types • HBase • Redis • etc • Cassandra • MySQL
  74. Challenges Service Failure Analyze Target Because it’s one of the

    critical services No Service Downtime No
  75. Challenges Service Failure Analyze Target Because it’s one of the

    critical services No Service Downtime No
  76. Design Process Install Firewall Apply Rules Make Rules

  77. Design Process Install Firewall Apply Rules Make Rules

  78. Design Process Install Firewall Apply Rules Make Rules

  79. Design Process Install Firewall Apply Rules Make Rules

  80. Design Process Install Firewall Apply Rules Make Rules

  81. Design Process Install Firewall Apply Rules Make Rules

  82. Design Process Install Firewall Apply Rules Make Rules

  83. Design Process Install Firewall Apply Rules Make Rules

  84. Service Failure Challenges Because it’s one of the critical services

    No Service Downtime No
  85. Design Process Possible Problems > Omitted, misspelled info > Missed

    Infrastructure- related info Install Firewall Apply Rules Make Rules
  86. Design Process Possible Problems > Omitted, misspelled info > Missed

    Infrastructure- related info Install Firewall Apply Rules Make Rules
  87. Design Process Possible Problems > Kernel panic > Device driver

    conflict Install Firewall Apply Rules Make Rules
  88. Design Process Possible Problems > Kernel panic > Device driver

    conflict Install Firewall Apply Rules Make Rules
  89. Design Process Possible Problems > Degraded service performance > Other

    unexpected problems Install Firewall Apply Rules Make Rules
  90. Design Process Possible Problems > Degraded service performance > Other

    unexpected problems Install Firewall Apply Rules Make Rules
  91. Preventing Problems Design Process > ‘Detect only’ mode > Log

    vs. Rule analysis 1. Faulty Rules > Test one server with redundancy • Most services are redundant with High Availability 2. Unpredictable Failure
  92. Preventing Problems Design Process > ‘Detect only’ mode > Log

    vs. Rule analysis 1. Faulty Rules > Test one server with redundancy • Most services are redundant with High Availability 2. Unpredictable Failure
  93. Preventing Problems Design Process > ‘Detect only’ mode > Log

    vs. Rule analysis 1. Faulty Rules > Test one server with redundancy • Most services are redundant with High Availability 2. Unpredictable Failure
  94. Apply By Server Group Set ‘Deny’ mode Apply 1 host,

    
 1/2 hosts, 
 2/2 hosts Install agent Apply 1 host, 
 1/2 hosts, 
 2/2 hosts Update rule based on 
 'Detect' logs Apply rule Apply 1 host, 
 1/2 hosts, 
 2/2 hosts Make rules with
 ‘Detect Only’ Mode
  95. Apply By Server Group Set ‘Deny’ mode Apply 1 host,

    
 1/2 hosts, 
 2/2 hosts Install agent Apply 1 host, 
 1/2 hosts, 
 2/2 hosts Update rule based on 
 'Detect' logs Apply rule Apply 1 host, 
 1/2 hosts, 
 2/2 hosts Make rules with
 ‘Detect Only’ Mode
  96. Realtime Communication Service Metrics Dev & Infrastructure Team Firewall Event

    Security Team Report 
 
 Response
  97. Realtime Communication Service Metrics Dev & Infrastructure Team Firewall Event

    Security Team Report 
 
 Response
  98. Result Hosts 700 Failure 0 Groups 27

  99. Result Hosts 700 Failure 0 Groups 27

  100. Alert System

  101. Without registering firewalls Human Error Scale Out
 API Server API

    Server’s Access Denied OUTAGE
  102. Without registering firewalls Human Error Scale Out
 API Server API

    Server’s Access Denied OUTAGE
  103. Quickly (automatically) detect human errors and respond Improvement Detect ‘Deny’

    event Notify Dev/Sec teams
  104. Alert System Deny Detect Notify

  105. Alert System Deny Detect Notify

  106. Alert System Deny Detect Notify

  107. Scale Out Assign Deploy Provision Allocate

  108. Scale Out Assign Deploy Provision Allocate Detect

  109. Episode

  110. Episode

  111. Episode

  112. Inconvenience due to Improved Security Inconvenience Emergency deployment/extension requires security

    team’s support Detailed firewall policies are high-maintenance
  113. Inconvenience due to Improved Security Inconvenience Emergency deployment/extension requires security

    team’s support Detailed firewall policies are high-maintenance
  114. Inconvenience due to Improved Security Inconvenience Emergency deployment/extension requires security

    team’s support Detailed firewall policies are high-maintenance
  115. Root Cause Inconvenience due to Improved Security Emergency deployment/extension requires

    security team’s support Detailed firewall policies are high-maintenance Too Complex for Dev. Team To Understand Rules Difficult To Request a Rule Change
  116. Improvement Plan Build Firewall Control Service Hide Irrelevant Rules Provide

    Simple Rule Change UI Deployment-Group-Aware Auto-Configuration
  117. Improvement Plan Build Firewall Control Service Hide Irrelevant Rules Provide

    Simple Rule Change UI Deployment-Group-Aware Auto-Configuration
  118. Scale Out Assign Deploy Provision Allocate

  119. Scale Out Assign Deploy Provision Allocate Make Rule

  120. Scale Out Assign Deploy Provision Allocate Make Rule

  121. Scale Out Assign Deploy Provision Allocate Make Rule

  122. Epilogue

  123. Epilogue

  124. Thank You