Save 37% off PRO during our Black Friday Sale! »

LINE Infra Security Log Platform And Analysis

Dd9d954997353b37b4c2684f478192d3?s=47 Elastic Co
December 14, 2017

LINE Infra Security Log Platform And Analysis

LINE Infra Security のLog Platform で、どのようなログをどのように活用しているかを紹介します。後半では、キーワードパターンによるスコアリングとElastic machine learningを活用した事例をお話します。

Seonmin Kim | Security Engineer | LINE株式会社
Hiroyuki Haruki | Security Engineer | LINE株式会社

Dd9d954997353b37b4c2684f478192d3?s=128

Elastic Co

December 14, 2017
Tweet

Transcript

  1. 1 Seonmin Kim , Hiroyuki Haruki Mar 2018, Security Log

    platform Monolith, LINE corporation Security Log platform Monolith How we control our Infra-Security.
  2. 2 Bullet Title (Arial 24 pt) • Try to keep

    your use of bullet slides to a minimum • Be creative and think visually • If you need to source something copy and paste the text box at the bottom left onto your page Subtitle sentence case (Arial 18pt) Source: Gray Arial 10pt
  3. 3 Bullet Title (Arial 24 pt) • Try to keep

    your use of bullet slides to a minimum • Be creative and think visually • If you need to source something copy and paste the text box at the bottom left onto your page Subtitle sentence case (Arial 18pt) Source: Gray Arial 10pt
  4. 4 Games Web services Policies Network Applications Servers Incident response

    monitor APP Security Solutions Design Infra Protection Windows Mac
  5. 5 5 Security Log platform Monolith How we control our

    Infra-Security. Infra Security Monolith 1 N-gram Pattern Matching 2
  6. 6 Bullet Title (Arial 24 pt) • Try to keep

    your use of bullet slides to a minimum • Be creative and think visually • If you need to source something copy and paste the text box at the bottom left onto your page Subtitle sentence case (Arial 18pt) Source: Gray Arial 10pt
  7. 7 Bullet Title (Arial 24 pt) • Try to keep

    your use of bullet slides to a minimum • Be creative and think visually • If you need to source something copy and paste the text box at the bottom left onto your page Subtitle sentence case (Arial 18pt) Source: Gray Arial 10pt
  8. 8 Bullet Title (Arial 24 pt) • Try to keep

    your use of bullet slides to a minimum • Be creative and think visually • If you need to source something copy and paste the text box at the bottom left onto your page Subtitle sentence case (Arial 18pt) Source: Gray Arial 10pt
  9. 9 Bullet Title (Arial 24 pt) • Try to keep

    your use of bullet slides to a minimum • Be creative and think visually • If you need to source something copy and paste the text box at the bottom left onto your page Subtitle sentence case (Arial 18pt) Source: Gray Arial 10pt
  10. 10 Bullet Title (Arial 24 pt) • Try to keep

    your use of bullet slides to a minimum • Be creative and think visually • If you need to source something copy and paste the text box at the bottom left onto your page Subtitle sentence case (Arial 18pt) Source: Gray Arial 10pt
  11. 11

  12. 12

  13. 13 Bullet Title (Arial 24 pt) • Try to keep

    your use of bullet slides to a minimum • Be creative and think visually • If you need to source something copy and paste the text box at the bottom left onto your page Subtitle sentence case (Arial 18pt) Source: Gray Arial 10pt
  14. 14 Bullet Title (Arial 24 pt) • Try to keep

    your use of bullet slides to a minimum • Be creative and think visually • If you need to source something copy and paste the text box at the bottom left onto your page Subtitle sentence case (Arial 18pt) Source: Gray Arial 10pt
  15. 15

  16. 16

  17. 17

  18. 18

  19. 19

  20. 20

  21. 21

  22. 22

  23. 23

  24. 24

  25. 25

  26. 26 26 Security Log platform Monolith How we control our

    Infra-Security. Infra Security Monolith N-gram Pattern Matching 1 2
  27. 27 Feature engineering for Anomaly detection 1. Synchronizing or mirroring

    directories and transferring files. 2. It is normally operated by batch processes in a short time. Server Server
  28. 28 Server Feature engineering for Anomaly detection Server

  29. 29 Server Server Feature engineering for Anomaly detection

  30. 30 • Purpose  Data transmission on server to server

    should be monitored to guard against internal and external threats.  The model should be operated in (near)real time.  Not just only a threat but also any abnormality, should be detected.  This model is intended to combine with other models to find advanced threats Feature engineering for Anomaly detection
  31. 31 1. Target path(or files)  String attribute 2. Server

    IP(host name) 3. Time stamp 4. Received IP(host name) 5. Send bytes  Numeric attribute 6. Receive bytes  Numeric attribute • Log format Feature engineering for Anomaly detection
  32. 32 1. Target path(or files)  String attribute 2. Server

    IP(host name) 3. Time stamp 4. Received IP(host name) 5. Send bytes  Numeric attribute 6. Receive bytes  Numeric attribute • Log format Feature engineering for Anomaly detection
  33. 33 • This is what I expected! Feature engineering for

    Anomaly detection
  34. 34 • In reality, I get this! Feature engineering for

    Anomaly detection
  35. 35 • Human readable graph? Feature engineering for Anomaly detection

  36. 36 • Another Viewpoint 1. Is it abnormal because of

    low or high transmission rate? 2. Is it normal because the pattern is within Confidence Interval? Feature engineering for Anomaly detection
  37. 37 • How to analyze data Feature engineering for Anomaly

    detection
  38. 38 • How to analyze data Feature engineering for Anomaly

    detection
  39. 39 1. Target path(or files)  String attribute 2. Server

    IP(host name) 3. Time stamp 4. Received IP(host name) 5. Send bytes  Numeric attribute 6. Receive bytes  Numeric attribute • Log format Feature engineering for Anomaly detection
  40. 40 • Focus on the full path Feature engineering for

    Anomaly detection INDEX SERVER ID PATH (= STRING ATTRIBUTE) 1 Id_1 security/strategy/../pattern1/filename.log 2 Id_1 security/strategy/../pattern2/filename1.log 3 Id_1 security/strategy/../pattern1/filename4.log 4 Id_2 Game/publishing/../name2/filename3.log 5 Id_2 Game/publishing/../name4/filename4.log 6 Id_2 Game/publishing/../name2/filename5.log
  41. 41 • Focus on the full path INDEX SERVER ID

    PATH (= STRING ATTRIBUTE) 1 Id_1 security/strategy/../pattern1/filename.log 2 Id_1 security/strategy/../pattern2/filename1.log 3 Id_1 Future unknown path 4 Id_2 Game/publishing/../name2/filename3.log 5 Id_2 Game/publishing/../name4/filename4.log 6 Id_2 Future unknown path Feature engineering for Anomaly detection Detection Detection
  42. 42 = () −+1 N-Score calculation • = count in

    c • = ℎ ℎ • = ∈ { } L I N E L I N E L I N E L I N E L I N E L I N E L I N E Unigram Bigram Trigram N-gram (N=4) Feature engineering for Anomaly detection
  43. 43 = () −+1 N-Score calculation • = count in

    c • = ℎ ℎ • = ∈ { } L I N E L I N E L I N E L I N E L I N E L I N E L I N E Unigram Bigram Trigram N-gram (N=4) Feature engineering for Anomaly detection Equation
  44. 44 = () −+1 N-Score calculation • = count in

    c • = ℎ ℎ • = ∈ { } L I N E L I N E L I N E L I N E L I N E L I N E L I N E Unigram Bigram Trigram N-gram (N=4) Reference Feature engineering for Anomaly detection
  45. 45 Data process Score Reference pattern file Elastic ML Features

    () − + 1 Framework
  46. 46 • Can you tell why it is abnormal? Feature

    engineering for Anomaly detection
  47. 47 • Can you tell why it is abnormal? Feature

    engineering for Anomaly detection
  48. 48 • Can you tell why it is abnormal? Feature

    engineering for Anomaly detection
  49. 49 1. Elastic ML is a great algorithmic instrument, not

    a magic box 2. Choosing the right feature will give you that much better results. • Result Feature engineering for Anomaly detection
  50. 50 Q & A