Norikra to realtime log analytics

Norikra meetup #2
2015-06-03

Transcript

1. Norikra to realtime log analytics harukasan / MICHII Shunsuke
2. Harukasan / MICHII Shunsuke - Infrastructure engineer in pixiv since 2012 - Develops contents distribution / convertor / storage - distributes up-to 16Gbps image traffic - Log collecting/analytics platform - Elasticsearch/Kibana - Fluentd
3. http://qiita.com/harukasan/items/957012833e5a361f7aa1
4. http://qiita.com/harukasan/items/a7c1dd1a11a61cd1ad75
5. Agenda - Log ecosystem - Batch processing vs. Stream processing - Getting started with Norikra - Norikra Deployment
6. Application Application Application Database Storage service HDFS RDB / Other  ʁ rsync syslog ssh custom script … Storage log storage
7. Application Application Application Database Storage service Fluentd HDFS RDB / Other  Storage log storage
8. Application Application Application Database Storage service Fluentd HDFS RDB / Other  Storage Google BigQuery Elasticsearch MongoDB log storage Treasure Data
9. Application Application Application Database Storage service HDFS RDB / Other  Storage Google BigQuery Elasticsearch MongoDB log storage Treasure Data Fluentd
10. Application Application Application Database Storage service HDFS RDB / Other  Storage Google BigQuery Elasticsearch MongoDB log storage Treasure Data Fluentd Kibana Spreadsheet HRForecast Tableau GrowthForecast Custom Script visualisation / analytics
11. Application Application Application Database Storage service HDFS RDB / Other  Storage Google BigQuery Elasticsearch MongoDB log storage Treasure Data Fluentd Kibana Spreadsheet HRForecast Tableau GrowthForecast Custom Script visualisation / analytics GAS
12. Application Application Application Database Storage service HDFS RDB / Other  Storage Google BigQuery Elasticsearch MongoDB log storage Treasure Data Fluentd Kibana Spreadsheet HRForecast Tableau GrowthForecast Custom Script visualisation / analytics Shib
13. Application Application Application Database Storage service HDFS RDB / Other  Storage Google BigQuery Elasticsearch MongoDB log storage Treasure Data Fluentd Kibana Spreadsheet HRForecast Tableau GrowthForecast Custom Script visualisation / analytics
14. Application Application Application Database Storage pixiv RDB / Other  Storage Google BigQuery Elasticsearch MongoDB log storage Fluentd Kibana HRForecast Tableau Custom Script visualisation / analytics Jenkins
15. Log ecosystem with Fluentd - Every log can stream to any type storages/queues - Every log are converted to structured data
16. Log Analytics Batch processing Ad-hoc analysis Offline analysis Stream processing
17. Batch processing Daily / Weekly / Monthly Reporting - page view - conversion count - num. of events デイリーレポート ================ - 2015/06/03更新 ▪ページビュー 2015/05/30 (水) 888888 PV 2015/05/30 (木) 888888 PV 2015/05/30 (金) 888888 PV 2015/05/30 (土) 888888 PV 2015/05/31 (日) 888888 PV ★過去最高 2015/06/01 (月) 888888 PV 2015/06/02 (火) 888888 PV 2015/06/03 (水) 888888 PV ▪新規登録数 2015/05/30 (水) 8888 人
18. Ad-hoc analysis - Kibana with Elasticsearch - BI Tools: Tableau, QlickView, Pentaho…
19. Offline Analysis - Excel is awesome - Analysis small data on laptops - Many techniques and know-how in Japan
20. Sometimes, Batch processes
 are too heavy Minutely Report - to know burst access - to know changes in the day Minutely Notification - to report error - to detect attacks
21. Stream Processing
 to realtime analytics - Process small data (almost case, in-memory) - High throughput - Low latency time window data stream 1 min.
22. Norikra - Streaming processing server - Schema-less - Use SQL-like query
23. Realtime Aggregation SELECT COUNT(1, status REGEXP '^2..$') AS count_2xx, COUNT(1, status REGEXP '^3..$') AS count_3xx, COUNT(1, status REGEXP '^4..$') AS count_4xx, COUNT(1, status REGEXP '^5..$') AS count_5xx FROM access_log.win:time_batch(1 min)
24. Fluentd Norikra
25. Output from fluent-plugin-norikra <source> type forward </source> <match log.**> # output to Norikra type norikra norikra localhost:26571 # specify norikra host (26571: default port) target_map_tag true # create target with tag </match>
26. Auto generated targets
27. Fluentd Norikra Elasticsearch GrowthForecast Idobata Google BigQuery ?
28. Fluentd Norikra Elasticsearch GrowthForecast Idobata Google BigQuery Fluentd
29. Sweep from Norikra <source> type norikra norikra localhost:26571 <fetch> method sweep # sweep output of query target gf # specify query group tag query_name # use query_name as tag tag_prefix norikra.gf # add tag prefix interval 10s </fetch> …
30. Sweep from Norikra … <fetch> method sweep # sweep output of query target idobata # specify query group tag query_name # use query_name as tag tag_prefix norikra.idobata # add tag prefix interval 10s </fetch> …
31. Sweep from Norikra … <fetch> method sweep # sweep output of query target es # specify query group tag query_name # use query_name as tag tag_prefix norikra.es # add tag prefix interval 10s </fetch> </source>
32. Output to GrowthForecast <match norikra.gf.**> type growthforecast remove_prefix norikra.gf name_key_pattern . gfapi_url http://localhost:5125/api/ graph_path norikra/${tag}/${key_name} </match>
33. Output to Idobata <match norikra.idobata.**> type idobata webhook_url #{put_your_hook_url_here} message_template <%= record['message'] %> </match>
34. Routing query to output
35. HTTP Status count SELECT COUNT(1, status REGEXP '^2..$') AS count_2xx, COUNT(1, status REGEXP '^3..$') AS count_3xx, COUNT(1, status REGEXP '^4..$') AS count_4xx, COUNT(1, status REGEXP '^5..$') AS count_5xx FROM access_log.win:time_batch(1 min) Name status_count Group gf Query
36. HTTP Status count SELECT COUNT(1, status REGEXP '^2..$') AS count_2xx, COUNT(1, status REGEXP '^3..$') AS count_3xx, COUNT(1, status REGEXP '^4..$') AS count_4xx, COUNT(1, status REGEXP '^5..$') AS count_5xx FROM access_log.win:time_batch(1 min) Name status_count Group mackerel Query
37. HTTP Status count SELECT "Notify: over 1000 access" AS message, COUNT(*) AS count FROM access_log.win:time_batch(1 min) WHERE count > 1000 Name notify_error Group idobata Query
38. Fluentd Norikra Elasticsearch GrowthForecast Idobata Google BigQuery Fluentd Output to anywhere with Fluentd
39. Norikra Deployment
40. Application Application Application Database Storage service Fluentd Active Fluentd Standby Computing node Norikra Fluentd GrowthForecast SPOF
41. Hardware structure - Norikra needs many memory (min. 8GB) - CPU cores are not so much required - Norikra is SPOF yet - Norikra can’t share query stats between active/standby
42. Build environment - Install JVM 1.7 by apt - Build JRuby by xbuild xbuild/ruby-install jruby-1.7.18 ~/local/jruby-1.7.18/
43. Install with Gemfile Gemfile: source "https://rubygems.org/" platforms :jruby do gem "norikra" end
44. Daemonize with Supervisord [program:norikra] command=/home/norikra/local/jruby-1.7.18/bin/norikra start \ --logdir=/var/log/norikra \ -s /home/norikra/norikra/norikra-stat.json \ --ui-context-path=/norikra \ -Xmx2048m … user=norikra directory=/home/norikra/norikra autostart=true autorestart=true environment=LANG=C
45. Conclusion - Use Norikra with Fluentd - Contribute to Norikra