LINE Infra Security Log Platform And Analysis

LINE Infra Security のLog Platform で、どのようなログをどのように活用しているかを紹介します。後半では、キーワードパターンによるスコアリングとElastic machine learningを活用した事例をお話します。

Seonmin Kim | Security Engineer | LINE株式会社
Hiroyuki Haruki | Security Engineer | LINE株式会社

Transcript

1. 1 Seonmin Kim , Hiroyuki Haruki Mar 2018, Security Log platform Monolith, LINE corporation Security Log platform Monolith How we control our Infra-Security.
2. 2 Bullet Title (Arial 24 pt) • Try to keep your use of bullet slides to a minimum • Be creative and think visually • If you need to source something copy and paste the text box at the bottom left onto your page Subtitle sentence case (Arial 18pt) Source: Gray Arial 10pt
3. 3 Bullet Title (Arial 24 pt) • Try to keep your use of bullet slides to a minimum • Be creative and think visually • If you need to source something copy and paste the text box at the bottom left onto your page Subtitle sentence case (Arial 18pt) Source: Gray Arial 10pt
4. 4 Games Web services Policies Network Applications Servers Incident response monitor APP Security Solutions Design Infra Protection Windows Mac
5. 5 5 Security Log platform Monolith How we control our Infra-Security. Infra Security Monolith 1 N-gram Pattern Matching 2
6. 6 Bullet Title (Arial 24 pt) • Try to keep your use of bullet slides to a minimum • Be creative and think visually • If you need to source something copy and paste the text box at the bottom left onto your page Subtitle sentence case (Arial 18pt) Source: Gray Arial 10pt
7. 7 Bullet Title (Arial 24 pt) • Try to keep your use of bullet slides to a minimum • Be creative and think visually • If you need to source something copy and paste the text box at the bottom left onto your page Subtitle sentence case (Arial 18pt) Source: Gray Arial 10pt
8. 8 Bullet Title (Arial 24 pt) • Try to keep your use of bullet slides to a minimum • Be creative and think visually • If you need to source something copy and paste the text box at the bottom left onto your page Subtitle sentence case (Arial 18pt) Source: Gray Arial 10pt
9. 9 Bullet Title (Arial 24 pt) • Try to keep your use of bullet slides to a minimum • Be creative and think visually • If you need to source something copy and paste the text box at the bottom left onto your page Subtitle sentence case (Arial 18pt) Source: Gray Arial 10pt
10. 10 Bullet Title (Arial 24 pt) • Try to keep your use of bullet slides to a minimum • Be creative and think visually • If you need to source something copy and paste the text box at the bottom left onto your page Subtitle sentence case (Arial 18pt) Source: Gray Arial 10pt
13. 13 Bullet Title (Arial 24 pt) • Try to keep your use of bullet slides to a minimum • Be creative and think visually • If you need to source something copy and paste the text box at the bottom left onto your page Subtitle sentence case (Arial 18pt) Source: Gray Arial 10pt
14. 14 Bullet Title (Arial 24 pt) • Try to keep your use of bullet slides to a minimum • Be creative and think visually • If you need to source something copy and paste the text box at the bottom left onto your page Subtitle sentence case (Arial 18pt) Source: Gray Arial 10pt
26. 26 26 Security Log platform Monolith How we control our Infra-Security. Infra Security Monolith N-gram Pattern Matching 1 2
27. 27 Feature engineering for Anomaly detection 1. Synchronizing or mirroring directories and transferring files. 2. It is normally operated by batch processes in a short time. Server Server
28. 28 Server Feature engineering for Anomaly detection Server
29. 29 Server Server Feature engineering for Anomaly detection
30. 30 • Purpose  Data transmission on server to server should be monitored to guard against internal and external threats.  The model should be operated in (near)real time.  Not just only a threat but also any abnormality, should be detected.  This model is intended to combine with other models to find advanced threats Feature engineering for Anomaly detection
31. 31 1. Target path(or files)  String attribute 2. Server IP(host name) 3. Time stamp 4. Received IP(host name) 5. Send bytes  Numeric attribute 6. Receive bytes  Numeric attribute • Log format Feature engineering for Anomaly detection
32. 32 1. Target path(or files)  String attribute 2. Server IP(host name) 3. Time stamp 4. Received IP(host name) 5. Send bytes  Numeric attribute 6. Receive bytes  Numeric attribute • Log format Feature engineering for Anomaly detection
33. 33 • This is what I expected! Feature engineering for Anomaly detection
34. 34 • In reality, I get this! Feature engineering for Anomaly detection
35. 35 • Human readable graph? Feature engineering for Anomaly detection
36. 36 • Another Viewpoint 1. Is it abnormal because of low or high transmission rate? 2. Is it normal because the pattern is within Confidence Interval? Feature engineering for Anomaly detection
37. 37 • How to analyze data Feature engineering for Anomaly detection
38. 38 • How to analyze data Feature engineering for Anomaly detection
39. 39 1. Target path(or files)  String attribute 2. Server IP(host name) 3. Time stamp 4. Received IP(host name) 5. Send bytes  Numeric attribute 6. Receive bytes  Numeric attribute • Log format Feature engineering for Anomaly detection
40. 40 • Focus on the full path Feature engineering for Anomaly detection INDEX SERVER ID PATH (= STRING ATTRIBUTE) 1 Id_1 security/strategy/../pattern1/filename.log 2 Id_1 security/strategy/../pattern2/filename1.log 3 Id_1 security/strategy/../pattern1/filename4.log 4 Id_2 Game/publishing/../name2/filename3.log 5 Id_2 Game/publishing/../name4/filename4.log 6 Id_2 Game/publishing/../name2/filename5.log
41. 41 • Focus on the full path INDEX SERVER ID PATH (= STRING ATTRIBUTE) 1 Id_1 security/strategy/../pattern1/filename.log 2 Id_1 security/strategy/../pattern2/filename1.log 3 Id_1 Future unknown path 4 Id_2 Game/publishing/../name2/filename3.log 5 Id_2 Game/publishing/../name4/filename4.log 6 Id_2 Future unknown path Feature engineering for Anomaly detection Detection Detection
42. 42 = () −+1 N-Score calculation • = count in c • = ℎ ℎ • = ∈ { } L I N E L I N E L I N E L I N E L I N E L I N E L I N E Unigram Bigram Trigram N-gram (N=4) Feature engineering for Anomaly detection
43. 43 = () −+1 N-Score calculation • = count in c • = ℎ ℎ • = ∈ { } L I N E L I N E L I N E L I N E L I N E L I N E L I N E Unigram Bigram Trigram N-gram (N=4) Feature engineering for Anomaly detection Equation
44. 44 = () −+1 N-Score calculation • = count in c • = ℎ ℎ • = ∈ { } L I N E L I N E L I N E L I N E L I N E L I N E L I N E Unigram Bigram Trigram N-gram (N=4) Reference Feature engineering for Anomaly detection
45. 45 Data process Score Reference pattern file Elastic ML Features () − + 1 Framework
46. 46 • Can you tell why it is abnormal? Feature engineering for Anomaly detection
47. 47 • Can you tell why it is abnormal? Feature engineering for Anomaly detection
48. 48 • Can you tell why it is abnormal? Feature engineering for Anomaly detection
49. 49 1. Elastic ML is a great algorithmic instrument, not a magic box 2. Choosing the right feature will give you that much better results. • Result Feature engineering for Anomaly detection
50. 50 Q & A