Adventures in Hardening the Security of LINE's Infra

Be4518b119b8eb017625e0ead20f8fe7?s=47 LINE DevDay 2019
November 20, 2019
Technology
0
530

Adventures in Hardening the Security of LINE's Infra

ByoungYun Lee
LINE Plus GrayLab Security Engineer
https://linedevday.linecorp.com/jp/2019/sessions/F1-4

Be4518b119b8eb017625e0ead20f8fe7?s=128

LINE DevDay 2019

November 20, 2019
Tweet

More Decks by LINE DevDay 2019

See All by LINE DevDay 2019
Be4518b119b8eb017625e0ead20f8fe7?s=48 line_devday2019
3
1.4k
Be4518b119b8eb017625e0ead20f8fe7?s=48 line_devday2019
6
480
Be4518b119b8eb017625e0ead20f8fe7?s=48 line_devday2019
3
210
Be4518b119b8eb017625e0ead20f8fe7?s=48 line_devday2019
0
190
Be4518b119b8eb017625e0ead20f8fe7?s=48 line_devday2019
0
340
Be4518b119b8eb017625e0ead20f8fe7?s=48 line_devday2019
0
540
Be4518b119b8eb017625e0ead20f8fe7?s=48 line_devday2019
0
200
Be4518b119b8eb017625e0ead20f8fe7?s=48 line_devday2019
1
130
Be4518b119b8eb017625e0ead20f8fe7?s=48 line_devday2019
0
360

Other Decks in Technology

See All in Technology
A64ac825509279a770c13c6fc517f11a?s=48 kawaguti
0
130
88964b936e864ca7d326272eaa70fa9a?s=48 hgsgtk
0
1.5k
4b2f3a64637b51e81813accbe8a98083?s=48 miura55
0
260
A857277d74d4719f7f7751dca5ed553e?s=48 cmusudakeisuke
0
1.2k
5c3556b7c8c0ab6bc90a62161a8315f8?s=48 gracia
0
440
F098812e012aee9cadc37e398c193d1c?s=48 tsukaman
1
150
992ab21437c2ec7c70d9af79ed2a2273?s=48 7110
0
230
A64ac825509279a770c13c6fc517f11a?s=48 kawaguti
2
430
53850955f15249a1a9dc49df6113e400?s=48 line_developers
0
130
8a84268593355816432ceaf78777d585?s=48 dena_tech
0
2k
Cd6e67775ba3275c466e9927ede23080?s=48 nk35jk
2
870
Cd823abda454a7a2065fe6fe273ae84b?s=48 viva_tweet_x
2
260

Featured

See All Featured
5ce91fa49a613cbc3e20d5f96856473f?s=48 edds
55
9.2k
32de0bd2ba869609d26fd052a4622778?s=48 cromwellryan
99
5.7k
A9179349dd2bdc67f377719f56d85656?s=48 garrettdimon
283
110k
C620790ae5bf5b50c245b2e0ef95f338?s=48 deanohume
293
27k
7edec06b5435e0dac07a353b2cc26253?s=48 geeforr
330
29k
C9b18d9dff88a9dd2393364c2b3b21bd?s=48 colly
183
14k
433acaea1012b25d97ae66da9b998dad?s=48 malarkey
189
8.2k
1f74b13f1e5c6c69cb5d7fbaabb1e2cb?s=48 sferik
606
53k
0c6fd6a08d0f898159cda7cafcacf07f?s=48 afnizarnur
175
14k
1519587b1a0febb658c36c94f6272b0d?s=48 roundedbygravity
84
7.6k
06f8b41980eb4c577fa40c41d5030c19?s=48 keathley
12
430
E14f55d3f939977cecbf51b64ff6f861?s=48 keithpitt
398
20k

Transcript

  1. 2019 DevDay Adventures in Hardening the Security of LINE's Infra

    > ByoungYun Lee > LINE Plus GrayLab Security Engineer

  2. Who Am I Security Risk Assessment @LINE > since 2018

    Offensive Security Research @GrayHash > for 3 years

  3. Scott ikeda, CPO Magazine, April 9, 2019 Eustance Huang, CNBC,

    May 4, 2019 THE PAPERS, 1 May, 2019

  4. Infrastructures An Attacker’s View

  5. Public Network Web Server API Server Database User

  6. Public Network Web Server API Server Database User www.example.com

  7. Public Network Web Server API Server Database User www.example.com api.example.com

  8. Public Network Web Server API Server Database User www.example.com api.example.com

    ?

  9. Public Network Web Server API Server Database Attacker

  10. Public Network Web Server API Server Database Attacker db.example.com database.example.com

  11. Public Network Web Server API Server Database Attacker

  12. Public Network Web Server API Server Database Attacker Developer

  13. Public Network Web Server API Server Database Attacker

  14. Public Network Web Server API Server Database Attacker Weak Password

  15. Public Network Web Server API Server Database Attacker Weak Password

  16. Public Network Web Server API Server Database Attacker Weak Password

    1, 1!, !@#, …

  17. Public Network Web Server API Server Database Attacker Port Scanning

  18. Public Network web pages etc ssh Web Server API Server

    Database Attacker Port Scanning

  19. Public Network Web Server API Server Database Attacker Exploit

  20. Public Network Web Server API Server Database Attacker Exploit

  21. Public Network Web Server API Server Database Attacker Exploit

  22. Public Network Web Server API Server Database Attacker Exploit

  23. Public Network Web Server API Server Database Attacker

  24. Public Network Web Server API Server Database Attacker Danger

  25. PRIVATE NETWORK Private Network Web Server API Server Database User

    Internal Servers

  26. Private Network PRIVATE NETWORK Web Server API Server Database User

    Internal Servers

  27. Private Network PRIVATE NETWORK Web Server API Server Database User

    Internal Servers

  28. Private Network PRIVATE NETWORK Web Server API Server Database Internal

    Servers Attacker

  29. Private Network PRIVATE NETWORK Internal Servers Users

  30. Private Network PRIVATE NETWORK Internal Servers Users

  31. Private Network PRIVATE NETWORK Internal Servers Users Attacker

  32. Lateral Movement Attack PRIVATE NETWORK Internal Servers Users Attacker

  33. Lateral Movement Attack PRIVATE NETWORK Internal Servers Users Attacker

  34. Lateral Movement Attack PRIVATE NETWORK Internal Servers Users Attacker

  35. Lateral Movement Attack PRIVATE NETWORK Internal Servers Users Attacker

  36. Lateral Movement Attack PRIVATE NETWORK Internal Servers Users Attacker Danger

  37. LINE’s Infrastructure PRIVATE NETWORK Internal Servers Users

  38. LINE’s Infrastructure PRIVATE NETWORK Internal Servers Users

  39. LINE’s Infrastructure PRIVATE NETWORK Internal Servers Users Attacker

  40. LINE’s Infrastructure PRIVATE NETWORK Internal Servers Users Attacker

  41. Goal PRIVATE NETWORK Breach SAFE SAFE SAFE

  42. Goal TCP/7000 10.0.0.1-10 TCP/5000 10.0.0.51-60 10.0.0.200

  43. Source IP Destination IP Destination Port Action 10.0.0.1-10 10.0.0.200 7000

    Allow Other 10.0.0.200 7000 Deny 10.0.0.51-60 10.0.0.200 5000 Allow Other 10.0.0.200 5000 Deny Goal

  44. Goal TCP/7000 10.0.0.1-10 TCP/5000 10.0.0.51-60 10.0.0.200 10.0.2.1-10

  45. Source IP Destination IP Destination Port Action 10.0.0.1-10 10.0.0.200 7000

    Allow 10.0.3.1-10 10.0.0.200 7000 Allow 10.0.4.1-10 10.0.0.200 7000 Allow 10.0.5.1-10 10.0.0.200 7000 Allow 10.0.100.50 10.0.0.200 5000 Allow 10.0.200.50 10.0.0.200 5000 Allow 10.0.300.50 10.0.0.200 5000 Allow 10.0.400.50 10.0.0.200 5000 Allow Goal

  46. Source IP Destination IP Destination Port Action 10.0.0.1-10 10.0.0.200 7000

    Allow 10.0.3.1-10 10.0.0.200 7000 Allow 10.0.4.1-10 10.0.0.200 7000 Allow 10.0.5.1-10 10.0.0.200 7000 Allow 10.0.100.50 10.0.0.200 5000 Allow 10.0.200.50 10.0.0.200 5000 Allow 10.0.300.50 10.0.0.200 5000 Allow 10.0.400.50 10.0.0.200 5000 Allow Goal

  47. Hardening Block unnecessary network access

  48. Tools

  49. LOW- MAINTENANCE Requirements CLOUD SUPPORT MULTI 
 REGIONAL PRE-EXISTING SERVICE

  50. LOW- MAINTENANCE Requirements CLOUD SUPPORT MULTI 
 REGIONAL PRE-EXISTING SERVICE

  51. Candidates We are aiming to block unnecessary network access iptables

    Network Firewall Host-based Firewall

  52. PROS iptables Easy-to-use Everyone knows how to use No cost

    No physical constraints, cloud-compatible

  53. PROS iptables Easy-to-use Everyone knows how to use No cost

    No physical constraints, cloud-compatible

  54. CONS iptables Easy-to-use Anyone can create security holes Not centralized

    = high maintenance

  55. CONS iptables Easy-to-use Anyone can create security holes Not centralized

    = high maintenance

  56. CONS iptables Easy-to-use Anyone can create security holes Not centralized

    = high maintenance

  57. PROs Network Firewall No resource overhead for hosts Already deployed:

    zero learning curve

  58. PROs Network Firewall No resource overhead for hosts Already deployed:

    zero learning curve

  59. CONs Network Firewall High cost Physical constraints: component proximity required

    Physical constraint: Incompatible with cloud infrastructure

  60. CONs Network Firewall High cost Physical constraints: component proximity required

    Physical constraint: Incompatible with cloud infrastructure

  61. Host-Based Firewall Install Firewall Apply Rules Make Rules

  62. Host-Based Firewall Install Firewall Apply Rules Make Rules

  63. Host-Based Firewall Install Firewall Apply Rules Make Rules

  64. PROs Host-Based Firewall No physical constraints, cloud-compatible Centralized administration ‘Detect

    Only’ mode

  65. PROs Host-Based Firewall No physical constraints, cloud-compatible Centralized administration ‘Detect

    Only’ mode

  66. PROs Host-Based Firewall No physical constraints, cloud-compatible Centralized administration ‘Detect

    Only’ mode

  67. So... iptables Network Firewall Host-based
 Firewall The Host-based Firewall Solution

    is the best option

  68. Hardening Steps Analyze target Design process Apply

  69. Service A's Database Servers about 700 hosts, 27 groups, 2,000

    clients Analyze Target Various Database Types • HBase • Redis • etc • Cassandra • MySQL

  70. Service A's Database Servers about 700 hosts, 27 groups, 2,000

    clients Analyze Target Various Database Types • HBase • Redis • etc • Cassandra • MySQL

  71. Service A's Database Servers about 700 hosts, 27 groups, 2,000

    clients Analyze Target Various Database Types • HBase • Redis • etc • Cassandra • MySQL

  72. Service A's Database Servers about 700 hosts, 27 groups, 2,000

    clients Analyze Target Various Database Types • HBase • Redis • etc • Cassandra • MySQL

  73. Service A's Database Servers about 700 hosts, 27 groups, 2,000

    clients Analyze Target Various Database Types • HBase • Redis • etc • Cassandra • MySQL

  74. Challenges Service Failure Analyze Target Because it’s one of the

    critical services No Service Downtime No

  75. Challenges Service Failure Analyze Target Because it’s one of the

    critical services No Service Downtime No

  76. Design Process Install Firewall Apply Rules Make Rules

  77. Design Process Install Firewall Apply Rules Make Rules

  78. Design Process Install Firewall Apply Rules Make Rules

  79. Design Process Install Firewall Apply Rules Make Rules

  80. Design Process Install Firewall Apply Rules Make Rules

  81. Design Process Install Firewall Apply Rules Make Rules

  82. Design Process Install Firewall Apply Rules Make Rules

  83. Design Process Install Firewall Apply Rules Make Rules

  84. Service Failure Challenges Because it’s one of the critical services

    No Service Downtime No

  85. Design Process Possible Problems > Omitted, misspelled info > Missed

    Infrastructure- related info Install Firewall Apply Rules Make Rules

  86. Design Process Possible Problems > Omitted, misspelled info > Missed

    Infrastructure- related info Install Firewall Apply Rules Make Rules

  87. Design Process Possible Problems > Kernel panic > Device driver

    conflict Install Firewall Apply Rules Make Rules

  88. Design Process Possible Problems > Kernel panic > Device driver

    conflict Install Firewall Apply Rules Make Rules

  89. Design Process Possible Problems > Degraded service performance > Other

    unexpected problems Install Firewall Apply Rules Make Rules

  90. Design Process Possible Problems > Degraded service performance > Other

    unexpected problems Install Firewall Apply Rules Make Rules

  91. Preventing Problems Design Process > ‘Detect only’ mode > Log

    vs. Rule analysis 1. Faulty Rules > Test one server with redundancy • Most services are redundant with High Availability 2. Unpredictable Failure

  92. Preventing Problems Design Process > ‘Detect only’ mode > Log

    vs. Rule analysis 1. Faulty Rules > Test one server with redundancy • Most services are redundant with High Availability 2. Unpredictable Failure

  93. Preventing Problems Design Process > ‘Detect only’ mode > Log

    vs. Rule analysis 1. Faulty Rules > Test one server with redundancy • Most services are redundant with High Availability 2. Unpredictable Failure

  94. Apply By Server Group Set ‘Deny’ mode Apply 1 host,

    
 1/2 hosts, 
 2/2 hosts Install agent Apply 1 host, 
 1/2 hosts, 
 2/2 hosts Update rule based on 
 'Detect' logs Apply rule Apply 1 host, 
 1/2 hosts, 
 2/2 hosts Make rules with
 ‘Detect Only’ Mode

  95. Apply By Server Group Set ‘Deny’ mode Apply 1 host,

    
 1/2 hosts, 
 2/2 hosts Install agent Apply 1 host, 
 1/2 hosts, 
 2/2 hosts Update rule based on 
 'Detect' logs Apply rule Apply 1 host, 
 1/2 hosts, 
 2/2 hosts Make rules with
 ‘Detect Only’ Mode

  96. Realtime Communication Service Metrics Dev & Infrastructure Team Firewall Event

    Security Team Report 
 
 Response

  97. Realtime Communication Service Metrics Dev & Infrastructure Team Firewall Event

    Security Team Report 
 
 Response

  98. Result Hosts 700 Failure 0 Groups 27

  99. Result Hosts 700 Failure 0 Groups 27

  100. Alert System

  101. Without registering firewalls Human Error Scale Out
 API Server API

    Server’s Access Denied OUTAGE

  102. Without registering firewalls Human Error Scale Out
 API Server API

    Server’s Access Denied OUTAGE

  103. Quickly (automatically) detect human errors and respond Improvement Detect ‘Deny’

    event Notify Dev/Sec teams

  104. Alert System Deny Detect Notify

  105. Alert System Deny Detect Notify

  106. Alert System Deny Detect Notify

  107. Scale Out Assign Deploy Provision Allocate

  108. Scale Out Assign Deploy Provision Allocate Detect

  109. Episode

  110. Episode

  111. Episode

  112. Inconvenience due to Improved Security Inconvenience Emergency deployment/extension requires security

    team’s support Detailed firewall policies are high-maintenance

  113. Inconvenience due to Improved Security Inconvenience Emergency deployment/extension requires security

    team’s support Detailed firewall policies are high-maintenance

  114. Inconvenience due to Improved Security Inconvenience Emergency deployment/extension requires security

    team’s support Detailed firewall policies are high-maintenance

  115. Root Cause Inconvenience due to Improved Security Emergency deployment/extension requires

    security team’s support Detailed firewall policies are high-maintenance Too Complex for Dev. Team To Understand Rules Difficult To Request a Rule Change

  116. Improvement Plan Build Firewall Control Service Hide Irrelevant Rules Provide

    Simple Rule Change UI Deployment-Group-Aware Auto-Configuration

  117. Improvement Plan Build Firewall Control Service Hide Irrelevant Rules Provide

    Simple Rule Change UI Deployment-Group-Aware Auto-Configuration

  118. Scale Out Assign Deploy Provision Allocate

  119. Scale Out Assign Deploy Provision Allocate Make Rule

  120. Scale Out Assign Deploy Provision Allocate Make Rule

  121. Scale Out Assign Deploy Provision Allocate Make Rule

  122. Epilogue

  123. Epilogue

  124. Thank You