Adventures in Hardening the Security of LINE's Infra

ByoungYun Lee
LINE Plus GrayLab Security Engineer
https://linedevday.linecorp.com/jp/2019/sessions/F1-4

Transcript

1. 2019 DevDay Adventures in Hardening the Security of LINE's Infra > ByoungYun Lee > LINE Plus GrayLab Security Engineer
2. Who Am I Security Risk Assessment @LINE > since 2018 Offensive Security Research @GrayHash > for 3 years
3. Scott ikeda, CPO Magazine, April 9, 2019 Eustance Huang, CNBC, May 4, 2019 THE PAPERS, 1 May, 2019
4. Infrastructures An Attacker’s View
5. Public Network Web Server API Server Database User
6. Public Network Web Server API Server Database User www.example.com
7. Public Network Web Server API Server Database User www.example.com api.example.com
8. Public Network Web Server API Server Database User www.example.com api.example.com ?
9. Public Network Web Server API Server Database Attacker
10. Public Network Web Server API Server Database Attacker db.example.com database.example.com
11. Public Network Web Server API Server Database Attacker
12. Public Network Web Server API Server Database Attacker Developer
13. Public Network Web Server API Server Database Attacker
14. Public Network Web Server API Server Database Attacker Weak Password
15. Public Network Web Server API Server Database Attacker Weak Password
16. Public Network Web Server API Server Database Attacker Weak Password 1, 1!, !@#, …
17. Public Network Web Server API Server Database Attacker Port Scanning
18. Public Network web pages etc ssh Web Server API Server Database Attacker Port Scanning
19. Public Network Web Server API Server Database Attacker Exploit
20. Public Network Web Server API Server Database Attacker Exploit
21. Public Network Web Server API Server Database Attacker Exploit
22. Public Network Web Server API Server Database Attacker Exploit
23. Public Network Web Server API Server Database Attacker
24. Public Network Web Server API Server Database Attacker Danger
25. PRIVATE NETWORK Private Network Web Server API Server Database User Internal Servers
26. Private Network PRIVATE NETWORK Web Server API Server Database User Internal Servers
27. Private Network PRIVATE NETWORK Web Server API Server Database User Internal Servers
28. Private Network PRIVATE NETWORK Web Server API Server Database Internal Servers Attacker
29. Private Network PRIVATE NETWORK Internal Servers Users
30. Private Network PRIVATE NETWORK Internal Servers Users
31. Private Network PRIVATE NETWORK Internal Servers Users Attacker
32. Lateral Movement Attack PRIVATE NETWORK Internal Servers Users Attacker
33. Lateral Movement Attack PRIVATE NETWORK Internal Servers Users Attacker
34. Lateral Movement Attack PRIVATE NETWORK Internal Servers Users Attacker
35. Lateral Movement Attack PRIVATE NETWORK Internal Servers Users Attacker
36. Lateral Movement Attack PRIVATE NETWORK Internal Servers Users Attacker Danger
37. LINE’s Infrastructure PRIVATE NETWORK Internal Servers Users
38. LINE’s Infrastructure PRIVATE NETWORK Internal Servers Users
39. LINE’s Infrastructure PRIVATE NETWORK Internal Servers Users Attacker
40. LINE’s Infrastructure PRIVATE NETWORK Internal Servers Users Attacker
41. Goal PRIVATE NETWORK Breach SAFE SAFE SAFE
42. Goal TCP/7000 10.0.0.1-10 TCP/5000 10.0.0.51-60 10.0.0.200
43. Source IP Destination IP Destination Port Action 10.0.0.1-10 10.0.0.200 7000 Allow Other 10.0.0.200 7000 Deny 10.0.0.51-60 10.0.0.200 5000 Allow Other 10.0.0.200 5000 Deny Goal
44. Goal TCP/7000 10.0.0.1-10 TCP/5000 10.0.0.51-60 10.0.0.200 10.0.2.1-10
45. Source IP Destination IP Destination Port Action 10.0.0.1-10 10.0.0.200 7000 Allow 10.0.3.1-10 10.0.0.200 7000 Allow 10.0.4.1-10 10.0.0.200 7000 Allow 10.0.5.1-10 10.0.0.200 7000 Allow 10.0.100.50 10.0.0.200 5000 Allow 10.0.200.50 10.0.0.200 5000 Allow 10.0.300.50 10.0.0.200 5000 Allow 10.0.400.50 10.0.0.200 5000 Allow Goal
46. Source IP Destination IP Destination Port Action 10.0.0.1-10 10.0.0.200 7000 Allow 10.0.3.1-10 10.0.0.200 7000 Allow 10.0.4.1-10 10.0.0.200 7000 Allow 10.0.5.1-10 10.0.0.200 7000 Allow 10.0.100.50 10.0.0.200 5000 Allow 10.0.200.50 10.0.0.200 5000 Allow 10.0.300.50 10.0.0.200 5000 Allow 10.0.400.50 10.0.0.200 5000 Allow Goal
47. Hardening Block unnecessary network access
48. Tools
49. LOW- MAINTENANCE Requirements CLOUD SUPPORT MULTI 
 REGIONAL PRE-EXISTING SERVICE
50. LOW- MAINTENANCE Requirements CLOUD SUPPORT MULTI 
 REGIONAL PRE-EXISTING SERVICE
51. Candidates We are aiming to block unnecessary network access iptables Network Firewall Host-based Firewall
52. PROS iptables Easy-to-use Everyone knows how to use No cost No physical constraints, cloud-compatible
53. PROS iptables Easy-to-use Everyone knows how to use No cost No physical constraints, cloud-compatible
54. CONS iptables Easy-to-use Anyone can create security holes Not centralized = high maintenance
55. CONS iptables Easy-to-use Anyone can create security holes Not centralized = high maintenance
56. CONS iptables Easy-to-use Anyone can create security holes Not centralized = high maintenance
57. PROs Network Firewall No resource overhead for hosts Already deployed: zero learning curve
58. PROs Network Firewall No resource overhead for hosts Already deployed: zero learning curve
59. CONs Network Firewall High cost Physical constraints: component proximity required Physical constraint: Incompatible with cloud infrastructure
60. CONs Network Firewall High cost Physical constraints: component proximity required Physical constraint: Incompatible with cloud infrastructure
61. Host-Based Firewall Install Firewall Apply Rules Make Rules
62. Host-Based Firewall Install Firewall Apply Rules Make Rules
63. Host-Based Firewall Install Firewall Apply Rules Make Rules
64. PROs Host-Based Firewall No physical constraints, cloud-compatible Centralized administration ‘Detect Only’ mode
65. PROs Host-Based Firewall No physical constraints, cloud-compatible Centralized administration ‘Detect Only’ mode
66. PROs Host-Based Firewall No physical constraints, cloud-compatible Centralized administration ‘Detect Only’ mode
67. So... iptables Network Firewall Host-based
 Firewall The Host-based Firewall Solution is the best option
68. Hardening Steps Analyze target Design process Apply
69. Service A's Database Servers about 700 hosts, 27 groups, 2,000 clients Analyze Target Various Database Types • HBase • Redis • etc • Cassandra • MySQL
70. Service A's Database Servers about 700 hosts, 27 groups, 2,000 clients Analyze Target Various Database Types • HBase • Redis • etc • Cassandra • MySQL
71. Service A's Database Servers about 700 hosts, 27 groups, 2,000 clients Analyze Target Various Database Types • HBase • Redis • etc • Cassandra • MySQL
72. Service A's Database Servers about 700 hosts, 27 groups, 2,000 clients Analyze Target Various Database Types • HBase • Redis • etc • Cassandra • MySQL
73. Service A's Database Servers about 700 hosts, 27 groups, 2,000 clients Analyze Target Various Database Types • HBase • Redis • etc • Cassandra • MySQL
74. Challenges Service Failure Analyze Target Because it’s one of the critical services No Service Downtime No
75. Challenges Service Failure Analyze Target Because it’s one of the critical services No Service Downtime No
76. Design Process Install Firewall Apply Rules Make Rules
77. Design Process Install Firewall Apply Rules Make Rules
78. Design Process Install Firewall Apply Rules Make Rules
79. Design Process Install Firewall Apply Rules Make Rules
80. Design Process Install Firewall Apply Rules Make Rules
81. Design Process Install Firewall Apply Rules Make Rules
82. Design Process Install Firewall Apply Rules Make Rules
83. Design Process Install Firewall Apply Rules Make Rules
84. Service Failure Challenges Because it’s one of the critical services No Service Downtime No
85. Design Process Possible Problems > Omitted, misspelled info > Missed Infrastructure- related info Install Firewall Apply Rules Make Rules
86. Design Process Possible Problems > Omitted, misspelled info > Missed Infrastructure- related info Install Firewall Apply Rules Make Rules
87. Design Process Possible Problems > Kernel panic > Device driver conflict Install Firewall Apply Rules Make Rules
88. Design Process Possible Problems > Kernel panic > Device driver conflict Install Firewall Apply Rules Make Rules
89. Design Process Possible Problems > Degraded service performance > Other unexpected problems Install Firewall Apply Rules Make Rules
90. Design Process Possible Problems > Degraded service performance > Other unexpected problems Install Firewall Apply Rules Make Rules
91. Preventing Problems Design Process > ‘Detect only’ mode > Log vs. Rule analysis 1. Faulty Rules > Test one server with redundancy • Most services are redundant with High Availability 2. Unpredictable Failure
92. Preventing Problems Design Process > ‘Detect only’ mode > Log vs. Rule analysis 1. Faulty Rules > Test one server with redundancy • Most services are redundant with High Availability 2. Unpredictable Failure
93. Preventing Problems Design Process > ‘Detect only’ mode > Log vs. Rule analysis 1. Faulty Rules > Test one server with redundancy • Most services are redundant with High Availability 2. Unpredictable Failure
94. Apply By Server Group Set ‘Deny’ mode Apply 1 host, 
 1/2 hosts, 
 2/2 hosts Install agent Apply 1 host, 
 1/2 hosts, 
 2/2 hosts Update rule based on 
 'Detect' logs Apply rule Apply 1 host, 
 1/2 hosts, 
 2/2 hosts Make rules with
 ‘Detect Only’ Mode
95. Apply By Server Group Set ‘Deny’ mode Apply 1 host, 
 1/2 hosts, 
 2/2 hosts Install agent Apply 1 host, 
 1/2 hosts, 
 2/2 hosts Update rule based on 
 'Detect' logs Apply rule Apply 1 host, 
 1/2 hosts, 
 2/2 hosts Make rules with
 ‘Detect Only’ Mode
96. Realtime Communication Service Metrics Dev & Infrastructure Team Firewall Event Security Team Report 
 
 Response
97. Realtime Communication Service Metrics Dev & Infrastructure Team Firewall Event Security Team Report 
 
 Response
98. Result Hosts 700 Failure 0 Groups 27
99. Result Hosts 700 Failure 0 Groups 27
100. Alert System
101. Without registering firewalls Human Error Scale Out
 API Server API Server’s Access Denied OUTAGE
102. Without registering firewalls Human Error Scale Out
 API Server API Server’s Access Denied OUTAGE
103. Quickly (automatically) detect human errors and respond Improvement Detect ‘Deny’ event Notify Dev/Sec teams
104. Alert System Deny Detect Notify
105. Alert System Deny Detect Notify
106. Alert System Deny Detect Notify
107. Scale Out Assign Deploy Provision Allocate
108. Scale Out Assign Deploy Provision Allocate Detect
109. Episode
110. Episode
111. Episode
112. Inconvenience due to Improved Security Inconvenience Emergency deployment/extension requires security team’s support Detailed firewall policies are high-maintenance
113. Inconvenience due to Improved Security Inconvenience Emergency deployment/extension requires security team’s support Detailed firewall policies are high-maintenance
114. Inconvenience due to Improved Security Inconvenience Emergency deployment/extension requires security team’s support Detailed firewall policies are high-maintenance
115. Root Cause Inconvenience due to Improved Security Emergency deployment/extension requires security team’s support Detailed firewall policies are high-maintenance Too Complex for Dev. Team To Understand Rules Difficult To Request a Rule Change
116. Improvement Plan Build Firewall Control Service Hide Irrelevant Rules Provide Simple Rule Change UI Deployment-Group-Aware Auto-Configuration
117. Improvement Plan Build Firewall Control Service Hide Irrelevant Rules Provide Simple Rule Change UI Deployment-Group-Aware Auto-Configuration
118. Scale Out Assign Deploy Provision Allocate
119. Scale Out Assign Deploy Provision Allocate Make Rule
120. Scale Out Assign Deploy Provision Allocate Make Rule
121. Scale Out Assign Deploy Provision Allocate Make Rule
122. Epilogue
123. Epilogue
124. Thank You