システムログ書式の構造に着目した システム異常検出手法の検討 /css2018-mizutani

2ca9e6e68b43a796a8add2bcb9bbad2e?s=47 Masayoshi Mizutani
October 24, 2018
Research
3
1.5k

システムログ書式の構造に着目した システム異常検出手法の検討 /css2018-mizutani

2ca9e6e68b43a796a8add2bcb9bbad2e?s=128

Masayoshi Mizutani

October 24, 2018
Tweet

More Decks by Masayoshi Mizutani

See All by Masayoshi Mizutani
2ca9e6e68b43a796a8add2bcb9bbad2e?s=48 mizutani
4
1.8k
2ca9e6e68b43a796a8add2bcb9bbad2e?s=48 mizutani
1
230
2ca9e6e68b43a796a8add2bcb9bbad2e?s=48 mizutani
1
3.6k
2ca9e6e68b43a796a8add2bcb9bbad2e?s=48 mizutani
3
1.8k
2ca9e6e68b43a796a8add2bcb9bbad2e?s=48 mizutani
4
1.7k
2ca9e6e68b43a796a8add2bcb9bbad2e?s=48 mizutani
3
2.3k
2ca9e6e68b43a796a8add2bcb9bbad2e?s=48 mizutani
0
280
2ca9e6e68b43a796a8add2bcb9bbad2e?s=48 mizutani
10
6k
2ca9e6e68b43a796a8add2bcb9bbad2e?s=48 mizutani
7
2.8k

Other Decks in Research

See All in Research
7aff8f547184534da3ca2e14e63a68a8?s=48 mgiraldo
0
130
0fd69f775497ec247d1daff4ee7dcde8?s=48 yasuhiromuraji
1
280
059fb717431a8cd2b509ffebc57d905a?s=48 trycycle
0
270
30a79337fdc6926a768e2f65139dc656?s=48 hijojo_partners
1
1.3k
A42dd3541cd40296dcd8a5e6b4a01bef?s=48 scatterlab
0
550
Dae1d4cc4599a65a5da6e1a1f9d96f05?s=48 n0bisuke
1
430
Bb6c3fc8c577710c72d03aeb4fa56bf6?s=48 masakat0
2
300
6325a9b34da54d5cbddb814c3987a2fe?s=48 chokkan
0
2.9k
0e20ab5db11ca70d818a4f10bfea9c9c?s=48 ymas0315
3
1.1k
83722380372c00bd75ac920f2089f6aa?s=48 zacky1972
2
130
6bd63bb363d2cb9933b263463f6305cf?s=48 matsui_528
0
2.1k
2c0947c6a28e7f771ebd9859ecf54e5c?s=48 shinyorke
0
1.2k

Featured

See All Featured
E43f8dfd01057f8304f5f8fb9a294d7d?s=48 jeffersonlam
325
14k
5b6a2bd86ef643786a381a212e0ef2f1?s=48 geoffreycrofte
11
280
5f83ef806155b403c3393160ce51f955?s=48 jlugia
215
16k
D36d2c1821af9249b69ff7f5ed60529b?s=48 tammielis
236
23k
7fa6101cd43067653cf4ac6c7ca54305?s=48 akmur
252
19k
766b9746b59e6f09e280cd33cf4ed419?s=48 skipperchong
4
490
Aebc2d07a50011e2a30d38435fde564a?s=48 jrom
113
7k
1177e050db6bafe62885362edf6e3537?s=48 lauravandoore
436
28k
8081b26e05bb4354f7d65ffc34cbbd67?s=48 chriscoyier
498
130k
5f2da528927a2ec9ba4fec2069cbc958?s=48 kneath
218
14k
465724d73fe3a92c0879fdfb43a3a6f3?s=48 philhawksworth
190
8.7k
20bfe76b3d6105641f879fe45cfc9272?s=48 bkeepers
319
53k

Transcript

  1. γεςϜϩάॻࣜͷߏ଄ʹண໨ͨ͠ γεςϜҟৗݕग़ख๏ͷݕ౼ ΫοΫύουגࣜձࣾ ਫ୩ਖ਼ܚ ίϯϐϡʔληΩϡϦςΟγϯϙδ΢Ϝ $߈ܸݕ஌

  2. ͸͡Ίʹ • ҟৗͳγεςϜϩάΛݕग़͢Δ͜ͱͰηΩϡϦςΟ Πϯγσϯτൃݟʹͭͳ͍͛ͨ • ϩάͷҟৗݕ஌͸ઌߦݚڀ͕ଟ਺͋Δ෼໺͕ͩ͋· Γ͏·͘ݱ৔Ͱӡ༻Ͱ͖͍ͯΔྫΛฉ͔ͳ͍ • ຊൃද͸࣮؀ڥͰܧଓతʹҟৗݕ஌Λճ͍ͯͨ͘͠ Ίཁ݅Λ੔ཧ͠ϓϩτλΠϓ͓Αͼ࣮ݧͯ͠Έͨ݁

    Ռͷใࠂ  

  3. എܠ • γεςϜϩάʹग़ݱ͢ΔʮҟৗʯΛൃݟ͍ͨ͠ • ҟৗɿ௨ৗ͸ग़ྗ͞Εͳ͍͸ͣͷϩά • ηΩϡϦςΟ؅ཧऀ͕ͨ·ͨ·ʮ໨ʯͰݟ͚ͭΔ͜ͱ ΋͋Δ͕͜ΕΛػցతʹݕग़͍ͨ͠ • ਖ਼ৗܥΛ͢΂ͯఆٛͰ͖Ε͹༰қ͕ͩ…

    • ѻ͏γεςϜͷछྨ͕ଟ͍ˍߏ੒มߋ΋ଟ͍ • ߏ੒มߋʹ൐ͬͯޡݕग़͞ΕΔͷ͸΍ΉΛಘͳ͍͕ม Խʹ௥ਵͯ͠΄͍͠  

  4. [Sun Sep 09 06:41:35.275151 2018] [:error] [pid 1517] [client 10.2.3.4:30626]

    script '/var/www/html/help.php' not found or unable to stat [Sun Sep 09 06:41:35.489685 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/java.php' not found or unable to stat [Sun Sep 09 06:41:35.699245 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/_query.php' not found or unable to stat [Sun Sep 09 06:41:35.903910 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/test.php' not found or unable to stat [Sun Sep 09 06:41:36.104791 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/db_cts.php' not found or unable to stat [Sun Sep 09 06:41:36.305791 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/db_pma.php' not found or unable to stat [Sun Sep 09 06:41:39.067416 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/logon.php' not found or unable to stat [Sun Sep 09 06:41:39.263892 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/help-e.php' not found or unable to stat [Sun Sep 09 06:41:39.469702 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/license.php' not found or unable to stat [Sun Sep 09 06:41:39.681337 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/log.php' not found or unable to stat [Sun Sep 09 06:41:39.875956 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/hell.php' not found or unable to stat [Sun Sep 09 06:41:40.072330 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/pmd_online.php' not found or unable to stat [Sun Sep 09 06:41:42.835995 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/x.php' not found or unable to stat [Sun Sep 09 06:41:43.050760 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/shell.php' not found or unable to stat [Sun Sep 09 06:41:43.248025 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/desktop.ini.php' not found or unable to stat [Sun Sep 09 06:41:43.450317 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/z.php' not found or unable to stat [Sun Sep 09 06:41:43.655400 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/lala.php' not found or unable to stat [Sun Sep 09 06:41:43.863698 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/lala-dpr.php' not found or unable to stat [Sun Sep 09 06:41:44.094330 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/wpo.php' not found or unable to stat [Sun Sep 09 06:41:47.003542 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/text.php' not found or unable to stat [Sun Sep 09 06:41:47.201627 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/wp-config.php' not found or unable to stat [Sun Sep 09 06:41:47.419065 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/muhstik.php' not found or unable to stat [Sun Sep 09 06:41:47.625909 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/muhstik2.php' not found or unable to stat [Sun Sep 09 06:41:47.836779 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/muhstiks.php' not found or unable to stat [Sun Sep 09 06:41:48.032147 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/muhstik-dpr.php' not found or unable to stat [Sun Sep 09 06:41:48.243196 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/lol.php' not found or unable to stat [Sun Sep 09 06:41:51.060698 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/uploader.php' not found or unable to stat [Sun Sep 09 06:41:51.279140 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/cmd.php' not found or unable to stat [Sun Sep 09 06:41:51.485373 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/cmx.php' not found or unable to stat [Sun Sep 09 06:41:51.696468 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/cmv.php' not found or unable to stat [Sun Sep 09 06:41:51.915964 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/cmdd.php' not found or unable to stat [Sun Sep 09 06:41:52.160011 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/knal.php' not found or unable to stat [Sun Sep 09 06:41:54.967426 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/cmd.php' not found or unable to stat [Sun Sep 09 06:41:54.967426 2018] [:error] [pid 1517] [client 192.168.1.2:12344] /bin/rm: cannot remove `/tmp/a.png’: No such file or directory [Sun Sep 09 06:41:55.161857 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/shell.php' not found or unable to stat [Sun Sep 09 06:41:55.396761 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/appserv.php' not found or unable to stat [Sun Sep 09 06:41:55.593244 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/wuwu11.php' not found or unable to stat [Sun Sep 09 06:41:55.787654 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/xw.php' not found or unable to stat [Sun Sep 09 06:41:55.989777 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/xw1.php' not found or unable to stat [Sun Sep 09 06:41:56.175298 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/9678.php' not found or unable to stat [Sun Sep 09 06:41:59.016329 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/wc.php' not found or unable to stat [Sun Sep 09 06:41:59.207458 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/xx.php' not found or unable to stat [Sun Sep 09 06:41:59.407406 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/s.php' not found or unable to stat [Sun Sep 09 06:41:59.606750 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/w.php' not found or unable to stat [Sun Sep 09 06:41:59.804259 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/sheep.php' not found or unable to stat [Sun Sep 09 06:41:59.992308 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/qaq.php' not found or unable to stat [Sun Sep 09 06:42:00.173038 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/db.init.php' not found or unable to stat [Sun Sep 09 06:42:02.956986 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/db_session.init.php' not found or unable to stat [Sun Sep 09 06:42:03.143240 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/db__.init.php' not found or unable to stat ͜ͷϩά͸ΠϝʔδͰ͢

  5. [Sun Sep 09 06:41:35.275151 2018] [:error] [pid 1517] [client 10.2.3.4:30626]

    script '/var/www/html/help.php' not found or unable to stat [Sun Sep 09 06:41:35.489685 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/java.php' not found or unable to stat [Sun Sep 09 06:41:35.699245 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/_query.php' not found or unable to stat [Sun Sep 09 06:41:35.903910 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/test.php' not found or unable to stat [Sun Sep 09 06:41:36.104791 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/db_cts.php' not found or unable to stat [Sun Sep 09 06:41:36.305791 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/db_pma.php' not found or unable to stat [Sun Sep 09 06:41:39.067416 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/logon.php' not found or unable to stat [Sun Sep 09 06:41:39.263892 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/help-e.php' not found or unable to stat [Sun Sep 09 06:41:39.469702 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/license.php' not found or unable to stat [Sun Sep 09 06:41:39.681337 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/log.php' not found or unable to stat [Sun Sep 09 06:41:39.875956 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/hell.php' not found or unable to stat [Sun Sep 09 06:41:40.072330 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/pmd_online.php' not found or unable to stat [Sun Sep 09 06:41:42.835995 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/x.php' not found or unable to stat [Sun Sep 09 06:41:43.050760 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/shell.php' not found or unable to stat [Sun Sep 09 06:41:43.248025 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/desktop.ini.php' not found or unable to stat [Sun Sep 09 06:41:43.450317 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/z.php' not found or unable to stat [Sun Sep 09 06:41:43.655400 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/lala.php' not found or unable to stat [Sun Sep 09 06:41:43.863698 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/lala-dpr.php' not found or unable to stat [Sun Sep 09 06:41:44.094330 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/wpo.php' not found or unable to stat [Sun Sep 09 06:41:47.003542 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/text.php' not found or unable to stat [Sun Sep 09 06:41:47.201627 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/wp-config.php' not found or unable to stat [Sun Sep 09 06:41:47.419065 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/muhstik.php' not found or unable to stat [Sun Sep 09 06:41:47.625909 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/muhstik2.php' not found or unable to stat [Sun Sep 09 06:41:47.836779 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/muhstiks.php' not found or unable to stat [Sun Sep 09 06:41:48.032147 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/muhstik-dpr.php' not found or unable to stat [Sun Sep 09 06:41:48.243196 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/lol.php' not found or unable to stat [Sun Sep 09 06:41:51.060698 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/uploader.php' not found or unable to stat [Sun Sep 09 06:41:51.279140 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/cmd.php' not found or unable to stat [Sun Sep 09 06:41:51.485373 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/cmx.php' not found or unable to stat [Sun Sep 09 06:41:51.696468 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/cmv.php' not found or unable to stat [Sun Sep 09 06:41:51.915964 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/cmdd.php' not found or unable to stat [Sun Sep 09 06:41:52.160011 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/knal.php' not found or unable to stat [Sun Sep 09 06:41:54.967426 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/cmd.php' not found or unable to stat [Sun Sep 09 06:41:54.967426 2018] [:error] [pid 1517] [client 192.168.1.2:12344] /bin/rm: cannot remove `/tmp/a.png’: No such file or directory [Sun Sep 09 06:41:55.161857 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/shell.php' not found or unable to stat [Sun Sep 09 06:41:55.396761 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/appserv.php' not found or unable to stat [Sun Sep 09 06:41:55.593244 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/wuwu11.php' not found or unable to stat [Sun Sep 09 06:41:55.787654 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/xw.php' not found or unable to stat [Sun Sep 09 06:41:55.989777 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/xw1.php' not found or unable to stat [Sun Sep 09 06:41:56.175298 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/9678.php' not found or unable to stat [Sun Sep 09 06:41:59.016329 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/wc.php' not found or unable to stat [Sun Sep 09 06:41:59.207458 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/xx.php' not found or unable to stat [Sun Sep 09 06:41:59.407406 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/s.php' not found or unable to stat [Sun Sep 09 06:41:59.606750 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/w.php' not found or unable to stat [Sun Sep 09 06:41:59.804259 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/sheep.php' not found or unable to stat [Sun Sep 09 06:41:59.992308 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/qaq.php' not found or unable to stat [Sun Sep 09 06:42:00.173038 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/db.init.php' not found or unable to stat [Sun Sep 09 06:42:02.956986 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/db_session.init.php' not found or unable to stat [Sun Sep 09 06:42:03.143240 2018] [:error] [pid 1517] [client 10.2.3.4:30626] script '/var/www/html/db__.init.php' not found or unable to stat  ͜ͷϩά͸ΠϝʔδͰ͢

  6. ϩάͷҟৗݕ஌ʹؔ͢Δઌߦݚڀ • ϩάͷҟৗݕ஌ख๏ • άϧʔϓԽ͞Εͨϩά͔Β஋ͷग़ݱύλʔϯΛ΋ͱʹݕग़[1,2] • ϩάͷू߹͔ΒҟৗͱͳΔཁૉΛݕग़͢Δ[3] • ࣌ܥྻҟৗݕ஌ •

    ϩάͷछྨʢ㲈ϑΥʔϚοτʣͷࣗಈ൑ఆख๏ • AprioriΞϧΰϦζϜͷԠ༻[4] • Shortest Edit ScriptΛ࢖ͬͨख๏[5]   [1] Min Du, Feifei Li, Guineng Zheng, and Vivek Srikumar. Deeplog: Anomaly detection and diagnosis from system logs through deep learning. In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Commu- nications Security, CCS ’17, pp. 1285–1298, New York, NY, USA, 2017. ACM. [2] Siadati, H., & Memon, N. (2017). Detecting Structurally Anomalous Logins Within Enterprise Networks. In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security - CCS ’17(pp. 1273–1284). [3] Xu, W., Huang, L., Fox, A., Patterson, D., & Jordan, M. (2009). Online system problem detection by mining patterns of console logs. In Proceedings - IEEE International Conference on Data Mining, ICDM. https://doi.org/10.1109/ICDM.2009.19 [4] Risto Vaarandi. A data clustering algorithm for mining patterns from event logs. in IEEE IPOM’03 Proceed- ings, pp. 119–126, 2003 [5] Min Du and Feifei Li. Spell: Streaming parsing of system event logs. In Proceedings - 16th IEEE International Conference on Data Mining, ICDM 2016, pp. 859–864, United States, 1 2017. Institute of Electrical and Electronics Engineers Inc.

  7. ࣮ӡ༻؀ڥͰͷར༻Λߟྀͨ͠ཁ݅ • આ໌Մೳੑͷߴ͍ݕग़ • ݕग़݁Ռͷղऍ͕ӡ༻ͷෛ୲ʹ • ηΩϡϦςΟ୲౰ऀ͕༰қʹཧղͰ͖ΔϩδοΫ͕ඞཁ • ஞ࣍తʹॲཧͰ͖ΔੑೳͷΑ͍ΞϧΰϦζϜ •

    ࣮࣌ؒੑͷ࣮ݱ • ࣌ؒܭࢉྔͷ཈੍ʢೖྗ਺Nʹରͯ͠O(N)͕ཧ૝ʣ • ۭؒܭࢉྔͷ཈੍ʢೖྗ਺NͰ੒௕͠ͳ͍ͷ͕ཧ૝ʣ • पظతʹൃੜ͢Δִ͕͕ؒ௕͍ϩά΋͋Δʢ೔ɺिɺ݄୯Ґʣͨ ΊόονॲཧʹΑΔݕग़͸ආ͚͍ͨ=iterativeʹॲཧ͍ͨ͠ • ܭࢉࢿݯͳͲίετͷ໰୊  

  8. Ξϓϩʔν 1. ϩάͷେ෦෼͸printfͳͲͷϑΥʔϚοτจʹ୯७ͳ ஋ΛຒΊࠐΉ͚ͩͱׂΓ੾ͬͯɺϩάΛߏ଄Խ͢Δ 2. ߏ଄Խ͞ΕͨϩάΛॱ࣍ΫϥελϦϯάͯ͠ҟৗݕ ग़ʹར༻͢Δ • ΫϥελΛ஝ੵ͢Δʮֶशʯ •

    طଘΫϥελͱൺֱͯ͠ҟৗͳϩάΛݟ͚ͭΔʮݕग़ʯ ɹˠ جຊతͳखॱ͸྆ํಉ͡  

  9. τʔΫϯ෼ׂͱΫϥελϦϯά   Dec 29 15:47:06 pylon sshd[27058]: Invalid user

    amavis from 111.205.93.154 m ʲೖྗʳ 0 < th < 1৽͍͠ΫϥελΛ࡞੒͢Δ͔Λܾఆ͢ΔͨΊͷ͖͍͠஋ mஞ࣍ೖྗ͞ΕΔϩάϝοηʔδ  ෼ׂ͞ΕͨτʔΫϯͱط ଘΫϥελͷத৺ͷϩάΛ ൺֱͯ͠Ұக཰ s Λܭࢉ ʢ࣍ϖʔδͰઆ໌ʣͯ͠ s ≧ th τʔΫϯ௕͕Ұக ͔ͭ࠷΋T͕େ͖͍Ϋϥελ ʹ౷߹  s < thͷ৔߹͸৽ ͨʹΫϥελΛ࡞੒ ݕग़ϑΣΠζͰ͸৽ͨ ͳΫϥελ࡞੒Λҟৗ ͳϩάͷݕग़ͱ͢Δ طଘΫϥελ ৽نΫϥελ ͦΕͧΕʹத৺ͱͳΔϩά͕͋Δ

  10. ϩάಉ࢜ͷҰக཰ͷܭࢉ   Dec 29 15:47:50 sshd pylon [ 34512

    ] : Invalid user bobber from 10.1.x.x Dec 29 15:47:06 pylon sshd[27058]: Invalid user amavis from 111.205.93.154 Dec 29 15:47:06 sshd pylon [ 27058 ] : Invalid user amavis from 192.168.x.x ϩάΛۭനจࣈɾه߸Λϕʔεʹ ϩάϝοηʔδΛτʔΫϯʹ෼ׂ m ೖྗ͞Εͨϩά ൺֱର৅ͷϩάʢΫϥελͷத৺ͷϩάʣ ෆҰக ෆҰக ෆҰக ෆҰக Ұகͨ͠ϩάͷτʔΫϯ਺ ϩάશମͷτʔΫϯ௕ = s ʢ্هྫ͸ҰகτʔΫϯ਺͕ɺશମͷτʔΫϯ௕ͳͷͰҰக཰s 㲈ʣ

  11. ࣮ݧ (1/3): ࣮૷͓ΑͼύϑΥʔϚϯεଌఆ • GoݴޠͰ࣮૷ • https://github.com/m-mizutani/logstruct • ύϑΥʔϚϯεධՁ •

    ۭؒܭࢉྔɾ࣌ؒܭࢉྔͱ΋ʹ ※ FʹϑΥʔϚοτͷू߹ɻϩάϝοηʔδ௕͸ඞͣҰఆҎԼʹͳΔલఏ   σʔληοτ छྨ ϩά݅਺ ຊख๏ )FΒͷख๏<> P1 #MVF(FOF-    ඵ ඵ P2 )%'4    ඵ ඵ [1] P.He,J.Zhu,P.Xu,Z.Zheng,andM.R.Lyu. A Directed Acyclic Graph Approach to Online Log Parsing. ArXiv e-prints, June 2018. https://arxiv.org/abs/1806.04356 O(|F|) <latexit sha1_base64="c1uD42CJ1CSa9YUdi+zVue/IDOM=">AAAB7XicbVBNSwMxEJ2tX7V+VT16CRahXsquCHosCuLNCvYD2qVk02wbm02WJCuUbf+DFw+KePX/ePPfmLZ70NYHA4/3ZpiZF8ScaeO6305uZXVtfSO/Wdja3tndK+4fNLRMFKF1IrlUrQBrypmgdcMMp61YURwFnDaD4fXUbz5RpZkUD2YUUz/CfcFCRrCxUuOuPL4Zn3aLJbfizoCWiZeREmSodYtfnZ4kSUSFIRxr3fbc2PgpVoYRTieFTqJpjMkQ92nbUoEjqv10du0EnVilh0KpbAmDZurviRRHWo+iwHZG2Az0ojcV//PaiQkv/ZSJODFUkPmiMOHISDR9HfWYosTwkSWYKGZvRWSAFSbGBlSwIXiLLy+TxlnFcyve/XmpepXFkYcjOIYyeHABVbiFGtSBwCM8wyu8OdJ5cd6dj3lrzslmDuEPnM8fwi2OlA==</latexit> <latexit sha1_base64="c1uD42CJ1CSa9YUdi+zVue/IDOM=">AAAB7XicbVBNSwMxEJ2tX7V+VT16CRahXsquCHosCuLNCvYD2qVk02wbm02WJCuUbf+DFw+KePX/ePPfmLZ70NYHA4/3ZpiZF8ScaeO6305uZXVtfSO/Wdja3tndK+4fNLRMFKF1IrlUrQBrypmgdcMMp61YURwFnDaD4fXUbz5RpZkUD2YUUz/CfcFCRrCxUuOuPL4Zn3aLJbfizoCWiZeREmSodYtfnZ4kSUSFIRxr3fbc2PgpVoYRTieFTqJpjMkQ92nbUoEjqv10du0EnVilh0KpbAmDZurviRRHWo+iwHZG2Az0ojcV//PaiQkv/ZSJODFUkPmiMOHISDR9HfWYosTwkSWYKGZvRWSAFSbGBlSwIXiLLy+TxlnFcyve/XmpepXFkYcjOIYyeHABVbiFGtSBwCM8wyu8OdJ5cd6dj3lrzslmDuEPnM8fwi2OlA==</latexit> <latexit sha1_base64="c1uD42CJ1CSa9YUdi+zVue/IDOM=">AAAB7XicbVBNSwMxEJ2tX7V+VT16CRahXsquCHosCuLNCvYD2qVk02wbm02WJCuUbf+DFw+KePX/ePPfmLZ70NYHA4/3ZpiZF8ScaeO6305uZXVtfSO/Wdja3tndK+4fNLRMFKF1IrlUrQBrypmgdcMMp61YURwFnDaD4fXUbz5RpZkUD2YUUz/CfcFCRrCxUuOuPL4Zn3aLJbfizoCWiZeREmSodYtfnZ4kSUSFIRxr3fbc2PgpVoYRTieFTqJpjMkQ92nbUoEjqv10du0EnVilh0KpbAmDZurviRRHWo+iwHZG2Az0ojcV//PaiQkv/ZSJODFUkPmiMOHISDR9HfWYosTwkSWYKGZvRWSAFSbGBlSwIXiLLy+TxlnFcyve/XmpepXFkYcjOIYyeHABVbiFGtSBwCM8wyu8OdJ5cd6dj3lrzslmDuEPnM8fwi2OlA==</latexit> <latexit sha1_base64="c1uD42CJ1CSa9YUdi+zVue/IDOM=">AAAB7XicbVBNSwMxEJ2tX7V+VT16CRahXsquCHosCuLNCvYD2qVk02wbm02WJCuUbf+DFw+KePX/ePPfmLZ70NYHA4/3ZpiZF8ScaeO6305uZXVtfSO/Wdja3tndK+4fNLRMFKF1IrlUrQBrypmgdcMMp61YURwFnDaD4fXUbz5RpZkUD2YUUz/CfcFCRrCxUuOuPL4Zn3aLJbfizoCWiZeREmSodYtfnZ4kSUSFIRxr3fbc2PgpVoYRTieFTqJpjMkQ92nbUoEjqv10du0EnVilh0KpbAmDZurviRRHWo+iwHZG2Az0ojcV//PaiQkv/ZSJODFUkPmiMOHISDR9HfWYosTwkSWYKGZvRWSAFSbGBlSwIXiLLy+TxlnFcyve/XmpepXFkYcjOIYyeHABVbiFGtSBwCM8wyu8OdJ5cd6dj3lrzslmDuEPnM8fwi2OlA==</latexit> ※ ܭଌ݁Ռͷখ਺఺ୈ3ҐҎԼ੾Γࣺͯ ※ ຊख๏ͷܭଌ؀ڥɿ CPU 2.9 GHz Intel Core i7 / ϝϞϦ16GB / MacBookPro (High Sierra 10.13.6) ࣮ࡍͷܭଌʹΑΓҰఆͷੑೳ͕࣮ݱͰ͖͍ͯΔ͜ͱΛ֬ೝ

  12. ࣮ݧ (2/3): ࣮؀ڥͰͷσʔλΛ༻͍࣮ͨݧ֓ཁ • ࣮ࡍͷϩάͰҙਤͨ͠ݕग़͕Ͱ͖Δ͔࣮ݧ • ໿1,000୆ͷAWS EC2Πϯελϯε͔Βsyslogʹग़ྗ͞ΕͨϩάΛར༻ • ͞Βʹ࿈ଓ͢Δ48࣌ؒͷϩάΛલޙ24࣌ؒͰ෼ׂ

    → D1, D2 • D1 Ͱϓϩηεछྨຖʹֶशͨ݁͠ՌΛ D2 ͷݕग़ʹར༻͠ҙਤͨ͠ݕग़ʹͳΔ͔Λݕূ • th = 0.65 ʹઃఆ   ݕ஌਺ ϓϩηε छྨ਺ %ͷϩά਺ ฏۉ஋ %ͷϩά਺ தԝ஋ ओͳϓϩηε໊      BQBDIF ECVT JSRCBMBODF NZTRME QPTUpY OQUE ODTE  QPTUHSFTRM STZTMPHE TZTUFNE [BCCJY ʙ      TTIE TTTE TV $30/ CBTI EPDLFSE DISPOZE DPOTVM  JOqVYE DMPVEJOJU ʙ       TVEP Ҏ্      LFSOFM EIDMJFOU KBWB EPDLFS ֓Ͷҙਤͨ݁͠Ռ͕ͩछྨͷϓϩηεͰϑΥʔϚοτ਺͕ऩଋͤͣ ऩଋ͍ͯ͠ͳ͍

  13. ࣮ݧ (3/3): ࣮؀ڥͰͷσʔλΛ༻͍࣮ͨݧ݁Ռྫ • sshd • Received SIGHUP; restarting. •

    sssd • Configuration file: /etc/sssd/sssd.conf does not exist. • dockerd • time="2018-08-14T06:11:26.411627122Z" level=debug msg="[BUILDER] Command to be executed: [/bin/sh -c curl -sS https://dl.yarnpkg.com/ debian/pubkey.gpg | apt-key add - && echo "deb https:// dl.yarnpkg.com/debian/ stable main\" | tee /etc/apt/sources.list.d/ yarn.list]" • chronyd •Initial frequency 9.204 ppm •Source 2600:2600::199 online • influxd •SELECT mean(****) FROM ***.*** WHERE time >= now() - 30d GROUP BY time(1h), milestone fill(none) service=query   ҟৗͳʢD1 ʹ͸ग़ݱͤͣD2 ʹ͸ग़ݱͨ͠ʣϩάͷྫ

  14. %FNP

  15. ߟ࡯ • ϑΥʔϚοτਪఆΞϧΰϦζϜͷվળ͓Αͼผख๏ ͱͷซ༻ͷݕ౼ • γϯϓϧͰӡ༻͠΍͍͢ΞϧΰϦζϜ͕ͩ೚ҙ௕ɾ೚ҙܗࣜͷϩάͷର Ԡ͕೉͍͠ • ಛఆܗࣜͷϩάΛআ֎͢ΔͳͲͷ޻෉͸ඞཁ •

    ݕग़݁Ռʹର͢ΔηΩϡϦςΟ৵֐ͷ൑ఆ • ҟৗͳϩάˠΠϯγσϯτͱ͸ݶΒͳ͍ • ผͷ৘ใͱͷ૊Έ߹ΘͤʹΑΔϦεΫ൑ఆͳͲߟ͑ΒΕΔ • ௕ظతͳӡ༻ʹ͓͚ΔϞσϧͷௐ੔ • ϑΥʔϚοτ಺ͷʮ೚ҙͷτʔΫϯʯ͕ଟ͘ͳΔͱҰக౓ܭࢉͷҙຯ͕ ͳ͘ͳΔ • Ұக౓ܭࢉͷࣜʹ޻෉͢Δ༨஍͕͋Δ  

  16. ·ͱΊ • େྔɾଟ༷ͳϩά͔Β࣮஍Ͱͷར༻Λ૝ఆͨ͠ҟৗݕग़ ख๏Λݕ౼ • ݁Ռͷઆ໌Մೳੑ΍ύϑΥʔϚϯεͳͲͷཁ݅੔ཧ • ౤ೖ͞ΕͨϩάͷϑΥʔϚοτΛਪఆ͠ɺͦͷϑΥʔϚοτ ͔Βҳ୤͢ΔϩάΛݕग़͢ΔγϯϓϧͳΞϧΰϦζϜͷݕ౼ •

    ΞϧΰϦζϜͷ࣮૷ͱ࣮ݧ • ύϑΥʔϚϯε͸ظ଴͢Δ݁Ռʹऩ·Δ • ҙਤ௨Γʹݕग़Ͱ͖ͨ΋ͷ΋͋Δ͕ɺద੾ʹϑΥʔϚοτԽ Ͱ͖ͳ͍छྨͷϩά΋͋ΔͨΊ໰୊ͷ෼ׂͳͲ͕ඞཁ  

  17. ຖ೔ͷྉཧΛָ͠Έʹ͢Δ Thank you