Visualize Piwik Tracker logs on kibana through fluentd jp

Visualize Piwik Tracker logs on kibana through fluentd. Kibana used for visualization.

Transcript

1. ΞΫηεղੳιϑτ Piwik ͷ঺հͱ fluentd ʹΑΔԠ༻ฤ YAMAMOTO Takashi yamachan@piwikjapan.org @yamachan5593 Piwik Japan Team Feb 27th, 2016 at Open Source Conference Tokyo
2. ࣗݾ঺հ ▪ OpenSolaris Ϣʔβʔձʢ࠷ۙαϘΓؾຯʣ □ https://osdn.jp/projects/jposug/ ▪ Piwikjapan ೔ຊޠύον࡞੒/ OSC ౦ژͰͷൃද □ https://osdn.jp/projects/piwik-fluentd/ 2 of 47
3. ԿΛՄࢹԽ͢Δͷ͔ ▪ Piwik αʔόʔʹͷ͜Δ Piwik tracker ͔Βͷϩά 125.54.155.180 - - [21/Feb/2016:08:46:13 +0900] "GET /piwik.php?action_name=example.com%2F%E5%A0%B1%E5%91 ʢུ - snipʣ &idsite=1&rec=1&r=047899&h=23&m=46&s=16 &url=http%3A%2F%2Fjpvlad.com%2Findex.php%3Ftopic%3Deventresult_ &_id=4e5ded8520370239&_idts=1435710334&_idvc=387 &_idn=0&_refts=0&_viewts=1455979574&send_image=0 &pdf=1&qt=0&realp=1&wma=1&dir=1&fla=1&java=1&gears=0 &ag=1&cookie=1&res=1366x768 HTTP/1.1" 204 - "http://jpvlad.com/index.php?topic=eventresult_ja" "Mozilla/5.0 (WindowsNT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.63 Safari/537.36" ˢ͜ΜͳͷΛ elasticsearch ʹετΞͯ͠ kibana ͰݟͯΈ·͢ 3 of 47
4. ຊ೔͸͜ΕΛ໨ࢦ͠·͢ 4 of 47
5. Piwik Tracker ԿΛ Piwik ʹૹΔͷ͔ ▪ τϥοΧʔ͕ͳͯ͘΋ී௨ʹه࿥Ͱ͖Δ΋ͷ □ client ip addresses, user agent, referer ▪ Piwik Tracker ͕ૹ৴͢Δ΋ͷ □ idsite: Piwik Ͱूܭ͍ͯ͠ΔͲͷ Web αΠτͰ͔͢ □ action name: ݟΒΕͨ Web ϖʔδͷλΠτϧ □ id: ΫϥΠΞϯτݻ༗ ID □ res: ΫϥΠΞϯτ PC ͷը໘ղ૾౓ □ pdf: ΫϥΠΞϯτ Web ϒϥ΢βʹ pdf ϓϥάΠϯ͕ೖ͍ͬͯΔ? □ java: java ? □ fla: flash ? □ cookie: cookie Λαϙʔτ͍ͯ͠Δϒϥ΢β͔? □ viewts: લճ๚໰࣌ؒ ▪ ଞʹ΋͋Γ·͢ɻৄ͘͠͸ “Supported Query Parameters1” 1http://developer.piwik.org/api-reference/tracking-api 5 of 47
6. ՄࢹԽ·Ͱͷखॱ 1. Piwik, fluentd, elasticsearch, kibana ͷΠϯετʔϧ 2. ֤αΠτͷདྷ๚ঢ়گ͸ Piwik ͷτϥοΩϯάίʔυʹΑΓ Piwik αʔόʔʹू໿ □ Piwik αʔόʔͷ PHP Λ௚઀ୟ͚͘Ͳɺϩάʹ΋࢒Δ □ શͯ GET 3. Piwik αʔόʔͷϩάΛ fluentd ܦ༝Ͱ elasticsearch ʹετΞ □ elasticsearch ͸෼ࢄܕશจݕࡧαʔόͰ͢ □ fluentd ͰҰ෦ URL decode ΋ߦͬͯ͠·͍·͢ 4. kibana Ͱ elasticsearch ͷσʔλՄࢹԽ͢Δ 6 of 47
7. ߏ੒͸͜Μͳײ͡ʹͳΓ·͢ td-agent 送信 td-agent 受信 データ整形 Store 可視化 Apache アクセスログ tail no-sql データベース Piwik Tracker (JavaScript) Administrator Piwik server elasticsearch server forward （2 台に分ける場合） 7 of 47
8. ৚݅ ▪ RedHat7 ʢCentOS7, Scientific Linux 7ʣσϑΥϧτ □ RedHat6 ͕ผͷखॱʹͳΔ৔߹͸ʢRedHat6ʣͱ͢Δ □ ʢRedHat6ʣ· · · CentOS6, Scientific Linux 6 ▪ Piwik ͸͢ͰʹՔಇ͍ͯ͠Δ΋ͷͱ͠·͢ □ Piwik ೔ຊϢʔβʔձͷ Web αΠτΛ͝ཡ͍ͩ͘͞ 2 ▪ fluentd, elasticsearch, kibana ͸ಉ͡αʔόʔʹΠϯετʔϧ͠ ·͢ □ Piwik ʹ͍ͭͯ͸͜ΕΒͱಉډɺผډͷೋ௨Γઆ໌͠·͢ 2http://www.piwikjapan.org/ػೳઆ໌/3985 8 of 47
9. fluentd ͷΠϯετʔϧ ∼ ύοέʔδʢ1ʣ ▪ fluentd ͷϥούʔͰ͋Δ td-agent Λ࢖͏͜ͱʹ͢Δ ▪ td-agent ͸ 2.x ܥʹ͢Δʢ1.x ܥ͸αϙʔτऴྃʣ ▪ αʔόʔͷ ruby ͱ౰ͨΒͳ͍Α͏ʹ RPM ύοέʔδͰ౤ೖ͢Δ □ fluentd ͸ ruby Ͱߏ੒͞Ε͍ͯΔ □ RedHat6 ඪ४͸ ruby 1.9.3 □ RedHat7 ඪ४͸ ruby 2.0 □ Ұํ td-agent 2.x ͸ύοέʔδ࡞੒࣌ ruby 2.2 Λཁٻ ▪ fluentd ύοέʔδͷதʹ fluentd ͷ௥ՃϓϥάΠϯΛೖΕ͓ͯ͘ □ όΠφϦύοέʔδʢRPMʣ͸ଘࡏ͢Δ΋ͷͷ ྫ͑͹ elasticsearch ϓϥάΠϯ͸ؚ·Ε͍ͯͳ͍ □ ޙͰͲ͏΍ͬͯ௥Ճ͍͍͔ͯ͠Θ͔Βͳ͍ͷͰ࠷ॳ͔ΒೖΕ͓ͯ͘ □ ˢ͜Ε͕ύοέʔδΛ࡞Δཧ༝ 9 of 47
10. fluentd ͷΠϯετʔϧ ∼ ύοέʔδʢ2ʣ ▪ ·ͣ͸ ruby 2.2.4 ͷΠϯετʔϧ 1. ύοέʔδΛ࡞Δ ruby ͕Քಇ͍ͯ͠ͳ͍ RedHat ؀ڥΛ༻ҙ ▪ CentOS, Scientific Linux ΋ͪΖΜՄೳ ▪ 6 Ͱ΋ 7 Ͱ΋ 2. td-agent ͕ RPM ͰΠϯετʔϧ͞Ε͍ͯΕ͹࡟আ͓͖ͯ͠·͢ 3. SRPM ͔Β rpm Λ࡞੒͢ΔͨΊͷπʔϧΛ౤ೖ $ sudo yum groupinstall "Development tools" 4. “CentOS 6 Ͱ ruby ͷ RPM ύοέʔδΛ࡞Δ 3” ΑΓ ruby223.spec Λμ΢ϯϩʔυ 5. RPM Λ࡞ΔͨΊͷඞཁͳσΟϨΫτϦΛ࡞Δ ˠ Ctrl+C ͰࢭΊΔ $ rpmbuild -bp ruby223.spec ʢCtrl+C ͰࢭΊΔʣ ʢ~/rpmbuild σΟϨΫτϦ͕Ͱ͖͍ͯΔ͸ͣʣ $ mv ruby223.spec rpmbuild/SPECS/ruby224.spec ʢ224 ʹมߋʣ 3http://www.torutk.com/projects/swe/wiki/CentOS 6 Ͱ ruby ͷ RPM ύο έʔδΛ࡞Δ 10 of 47
11. fluentd ͷΠϯετʔϧ ∼ ύοέʔδʢ3ʣ ▪ ruby 2.2.4 ͷΠϯετʔϧʢଓ͖ʣ 1. ˜/rpmbuild/SPECS/ruby224.spec Ұߦ໨Λมߋ͢Δ %define rubyver 2.2.4 2. “Ruby 2.2.4 ϦϦʔε 4” ΑΓ ruby-2.2.4.tar.bz2 Λμ΢ϯϩʔυ 3. ruby-2.2.4.tar.bz2 Λ ˜/rpmbuild/SOURCES ʹҠಈ 4. RPM ϑΝΠϧΛ࡞Δ $ cd ~/rpmbuild/SPECS $ rpmbuild -ba ruby224.spec ʢུʣ $ sudo rpm -ivh \ʢ࣍ͷߦͱ͋Θͤͯʣ ~/rpmbuild/RPMS/x86_64/ruby-2.2.4-1.el7.x86_64.rpm ʢRedHat6ʣ el6 ʹͳΓ·͢ ʢུʣ $ ruby -v ruby 2.2.4p230 (2015-12-16 revision 53155) [x86_64-linux] 4https://www.ruby-lang.org/ja/news/2015/12/16/ruby-2-2-4-released/ 11 of 47
12. fluentd ͷΠϯετʔϧ ∼ ύοέʔδʢ4ʣ ▪ ඞཁύοέʔδͷΠϯετʔϧ 1. epel ϨϙδτϦΛࢀর͢ΔΑ͏ʹ͓͍͍ͯͯͩ͘͠͞ ▪ ҰߦͰ $ sudo yum install \ http://ftp-srv2.kddilabs.jp/Linux/distributions/ \ fedora/epel/7/x86 64/e/epel-release-7-5.noarch.rpm ▪ ʢRedHat6ʣҰߦͰ $ sudo yum install \ http://ftp-srv2.kddilabs.jp/Linux/distributions/ \ fedora/epel/6/x86 64/epel-release-6-8.noarch.rpm 2. Πϯετʔϧ $ sudo yum install gecode gecode-devel fakeroot 12 of 47
13. fluentd ͷΠϯετʔϧ ∼ ύοέʔδʢ5ʣ 1. ʢRedHat6ʣgit ͷΞοϓσʔτ $ wget http://dl.marmotte.net/rpms/redhat/el6/x86 64/\ git-1.8.3.1-3.el6/git-1.8.3.1-3.el6.src.rpm $ cp ~/rpmbuild/SRPMS/git-1.8.3.1-3.el6.src.rpm $ rpmbuild --rebuild \ ~/rpmbuild/SRPMS/git-1.8.3.1-3.el6.src.rpm $ sudo yum install perl-TermReadKey $ sudo rpm -ivh \ ~/rpmbuild/RPMS/x86 64/git-1.8.3.1-3.el6.x86_64.rpm □ git 1.8 ୆ʹ͠ͳ͍ͱ “-c” Φϓγϣϯ͕ͳ͍ͨΊϏϧυͰ͖ͳ͍ ▪ ެࣜʹ͸ git 1.8 ୆͸ଘࡏ͠ͳ͍ ▪ ͍Ζ͍Ζ͋ΔதͰɺґଘύοέʔδ͕ epel Ͱऩ·Δ΋ͷΛબ୒ 13 of 47
14. fluentd ͷΠϯετʔϧ ∼ ύοέʔδʢ6ʣ ▪ ruby ؀ڥߏஙɺtd-agent ιʔεͷμ΢ϯϩʔυ 1. bundle ΛΠϯετʔϧ $ sudo gem install bundler 2. github ͔Β clone $ cd ~ $ git clone \ʢ࣍ͷߦͱ͋Θͤͯʣ git@github.com:treasure-data/omnibus-td-agent.git $ cd ~/omnibus-td-agent 3. treasure-data/omnibus-td-agent5 ͷ௨Γʹ࣮ߦ͠·͕͢ɺґଘੑͷղ ܾʹࣦഊ͢ΔͷͰ Gemfile ͷ్தʹҰߦ௥Ճ͠·͢ʢ࣍ϖʔδʣ ɻ 5https://github.com/treasure-data/omnibus-td-agent 14 of 47
15. fluentd ͷΠϯετʔϧ ∼ ύοέʔδʢ7ʣ ▪ ιʔεΛमਖ਼͍͖ͯ͠·͢ □ ґଘੑ໰୊Λճආ □ ˜/omnibus-td-agent/Gemfile ʹ gem ’pedump’ · · · ͷҰߦΛ௥Ճ 6 source ’https://rubygems.org’ # Use Berkshelf for resolving cookbook dependencies gem ’berkshelf’, ’~> 3.0’ gem ’pedump’, git: ’https://github.com/ksubrama/pedump’, branch: ’patch-1’ # ্͔ΒҰߦͰ௥Ճ # Install omnibus software #gem ’omnibus’, ’~> 5.0’ ʢҎԼུʣ 6https://github.com/piwikjapan/omnibus-td-agent/blob/master/Gemfile 15 of 47
16. fluentd ͷΠϯετʔϧ ∼ ύοέʔδʢ8ʣ ▪ elasticsearch, record-reformer, norikra ϓϥάΠϯΛ RPM ʹ૊Έ ࠐΉ □ ࠓ೔͸ norikra ʹ͍ͭͯ͸ݴٴ͠·ͤΜ ▪ ˜/omnibus-td-agent/plugin gems.rb ͷ࠷ޙʹࡾߦ௥Ճ download "fluent-plugin-norikra", "0.2.2" download "fluent-plugin-elasticsearch", "1.3.0" download "fluent-plugin-record-reformer", "0.8.0" 16 of 47
17. fluentd ͷΠϯετʔϧ ∼ ύοέʔδʢ9ʣ ▪ norikra ϓϥάΠϯʹؔ࿈͢ΔϓϥάΠϯΛ௥Ճ □ ࠓ೔͸ norikra ʹ͍ͭͯ͸ݴٴ͠·ͤΜ □ norikra-client ͕Ҿ͖ࠐΉ msgpack-rpc-over-http ͕ rack Λཁٻ͢Δ ͕ɺ2.x ࠷৽൛ΛಡΈࠐΜͰΤϥʔ͕ग़ΔͷͰͻͱͭલͷ 1.6.4 Λڧ ੍తʹ࢖͏ ▪ ˜/omnibus-td-agent/core gems.rb ͷ࠷ޙʹೋߦ௥Ճ download "rack", "1.6.4" download "norikra-client", "1.3.1" 17 of 47
18. fluentd ͷΠϯετʔϧ ∼ ύοέʔδʢ10ʣ ▪ ϫʔΫσΟϨΫτϦΛ࡞Γ·͢ 7ɻ $ sudo mkdir -p /opt/td-agent /var/cache/omnibus $ sudo chown yamachan:yamachan /opt/td-agent $ sudo chown yamachan:yamachan/var/cache/omnibus □ yamachan:yamachan ͸ࣗ෼ͷ id ͱάϧʔϓʹஔ͖׵͍͑ͯͩ͘͞ 7https://github.com/treasure-data/omnibus-td-agent 18 of 47
19. fluentd ͷΠϯετʔϧ ∼ ύοέʔδʢ11:ऴʣ 1. खॱ௨Γʹ 8 $ cd ~/omnibus-td-agent $ bundle install --binstubs ʢུɺ్தͰ sudo ͢ΔͷͰύεϫʔυΛೖྗʣ $ bin/gem_downloader core_gems.rb ʢུʣ $ bin/gem_downloader plugin_gems.rb ʢུʣ $ bin/omnibus build td-agent2 ʢུʣ 8https://github.com/treasure-data/omnibus-td-agent 19 of 47
20. fluentd ͷΠϯετʔϧ ∼ Πϯετʔϧ 1. ग़དྷ্͕ͬͨύοέʔδ͸ pkg ҎԼʹೖΓ·͢ $ cd ~/omnibus-td-agent/pkg $ sudo yum install td-agent-2.3.1-0.el7.x86 64.rpm 2. ʢRedHat6ʣtd-agent-2.3.1-0.el6.x86 64.rpm 20 of 47
21. elasticsearch ͷΠϯετʔϧ 1. RedHat7, RedHat6 ڞ௨Ͱ͢ɻҰߦͰ $ sudo yum install \ https://download.elasticsearch.org/elasticsearch/\ release/org/elasticsearch/distribution/\ rpm/elasticsearch/2.2.0/elasticsearch-2.2.0.rpm 2. ೔ຊޠߏจղੳ kuromoji ϓϥάΠϯΛೖΕ·͢ɻҰߦͰ $ sudo /usr/share/elasticsearch/bin/plugin \ install analysis-kuromoji 21 of 47
22. kibana ͷΠϯετʔϧ 1. ύοέʔδ͕ͳ͍ͷͰ࡞Γ·͢ $ cd ~ $ git clone git@github.com:piwikjapan/kibana-rpm-packaging.git $ cd kibana-rpm-packaging $ cp kibana.sysconfig kibana.service ~/rpmbuild/SOURCES $ cp kibana.spec ~/rpmbuild/SPECS $ wget -P ~/rpmbuild/SOURCES \ https://download.elastic.co/kibana/kibana/\ kibana-4.4.1-linux-x64.tar.gz $ rpmbuild -ba ~/rpmbuild/SPECS/kibana.spec 2. Πϯετʔϧ͠·͢ $ sudo rpm -ivh ~rpmbuild/RPMS/x86_64/\ kibana-4.4.1-1.x86_64.rpm 22 of 47
23. ʢRedHat6ʣkibana ͷΠϯετʔϧ ▪ ͜ͷ΁Μ “kibana4 ηοτΞοϓ 9” ΛΈ͍ͯͩ͘͞ɻ ▪ ىಈεΫϦϓτ΋ܝࡌ͞Ε͍ͯ·͢ɻ ▪ ؾ͕޲͍ͨΒύοέʔδεΫϦϓτ࡞ͬͯΈ·͢ɻ 9http://qiita.com/nagomu1985/items/82e699dde4f99b2ce417 23 of 47
24. ϑΝΠΞʔ΢Υʔϧͷઃఆ 1. norikraʢ26578/tcpʣʹ͍ͭͯ͸ࠓճ͸ݴٴ͠·ͤΜ $ sudo firewall-cmd --zone=public \ --add-port=26578/tcp --permanent # norikra web $ sudo firewall-cmd --zone=public \ --add-port=5651/tcp --permanent # kibana web $ sudo firewall-cmd --zone=public \ --add-port=24224/udp --permanent # fluentd heatbeat $ sudo firewall-cmd --zone=public \ --add-port=24224/tcp --permanent # fluentd data 2. ൓ө͠·͢ $ sudo firewall-cmd --reload 24 of 47
25. ʢRedHat6ʣϑΝΠΞʔ΢Υʔϧͷઃఆ 1. norikraʢ26578/tcpʣʹ͍ͭͯ͸ࠓճ͸ݴٴ͠·ͤΜ 2. /etc/sysconfig/iptables ͷ -A INPUT -m state –state ESTABLISHED,RELATED -j ACCEPT ͷԼʹ௥Ճ͍ͯͩ͘͠͞ -A INPUT -m multiport -p tcp -m tcp \ʢҰߦͰʣ --dports 26578,5651,24224 -j ACCEPT -A INPUT -m multiport -p udp -m udp --dports 24224 -j ACCEPT 3. ൓ө͠·͢ $ sudo service iptables reload 25 of 47
26. td-agent ͷઃఆ ▪ Piwik ͕ elasticsearch, kibana ͱ 1. ผډʢPiwik server ͱ elasticsearch server ͕ผʣ 2. ಉډʢPiwik server ͱ elasticsearch server ͕ಉ͡ɺforward ͕ͳ͍ʣ td-agent 送信 td-agent 受信 データ整形 Store 可視化 Apache アクセスログ tail no-sql データベース Piwik Tracker (JavaScript) Administrator Piwik server elasticsearch server forward （2 台に分ける場合） 26 of 47
27. td-agent ͷઃఆ ∼ Piwik ͱผډʢ1ʣ ▪ Piwik ͱ elasticsearch ͕ผαʔόʔ □ ྆ํʹ td-agent ΛೖΕͯϙʔτΛۭ͚͓͖ͯ·͢ɻ □ ର৅ઃఆϑΝΠϧ/etc/td-agent/td-agent.conf ▪ ࣍ϖʔδҎ߱ͷྫΛॱ൪ʹͭͳ͛ͯͻͱͭʹ͍ͯͩ͘͠͞ ▪ ը໘ʹೖΓ͖Βͳ͍ͱ͖͸୺ંΓ·͢ɻ □ ׬શ൛͸ “Piwik ͷτϥοΩϯάσʔλΛ elasticsearch Ͱऩूͯ͠Έ Δ 10” Λࢀর͍ͯͩ͘͠͞ɻ 10https://osdn.jp/projects/piwik-fluentd/wiki/FrontPage 27 of 47
28. td-agent ͷઃఆ ∼ Piwik ͱผډʢ2ʣ ▪ Piwik αʔόʔ □ Piwik ͷΞΫηεϩάΛ fluentd Ͱٵ্͍͛·͢ □ tag piwiktracker.apache.access Ͱ࣍ͷॲཧΛҾ͖ܧ͗ <source> type tail format apache time_format %d/%b/%Y:%H:%M:%S %z pos_file /var/log/td-agent/access_log.pos path /var/log/httpd/access_log tag piwiktracker.apache.access </source> 28 of 47
29. td-agent ͷઃఆ ∼ Piwik ͱผډʢ3ʣ ▪ Piwik αʔόʔ □ host Ͱࢦఆ͢ΔαʔόʔʹϩάΛసૹ͠·͢ɻ <match piwiktracker.apache.access> type forward send_timeout 60s recover_wait 300s heartbeat_interval 1s phi_threshold 16 hard_timeout 60s <server> name fruentd host your_elsticsearch_server i.e. 10.x.x.x port 24224 weight 100 </server> </match> 29 of 47
30. td-agent ͷઃఆ ∼ Piwik ͱผډʢ4ʣ ▪ elasticsearch αʔόʔ □ Tracker ͷϩάͷΈΛநग़͠·͢ 1. Piwik ͷ؅ཧը໘ʹΞΫηεͨ͠ͱ͖ͷϩά 2. Piwik ͷ API Λ͍ͨͨͨͱ͖ͷϩά 3. filter ͷ࣍ʹ match piwiktracker.apache.access ʹભҠ͠·͢ <filter piwiktracker.apache.access> type grep regexp1 path /piwik\.php\?action name=.*\&idsite=\d+ </filter> <match piwiktracker.apache.access> type record_reformer tag piwiktracker.apache.access.urldecode ʢུɺ࣍ϖʔδͰʣ 30 of 47
31. td-agent ͷઃఆ ∼ Piwik ͱผډʢ5ʣ ▪ elasticsearch αʔόʔ □ ύʔεจࣈྻΛ fluentd ͷม਺ʹ෼ղɻͦΕͧΕͷҙຯ͸ “Supported Query Parameters11” Λࢀর □ ઌ಄ʹΞϯμʔείΞ “ ” ͸ېࢭɻ·ͨม਺ “id” ͸ෆՄɻ □ ෼ղͨ͠Β piwiktracker.apache.access.urldecode ΁ <match piwiktracker.apache.access> type record_reformer tag piwiktracker.apache.access.urldecode ʢҎԼ 29 ݸத 3 ͚ͭͩʣ idsite ${path[/piwik\.php\? action name=.*\&idsite=(\d+)/,1]} ˡ αΠτ ID piwikid ${path[/piwik\.php\?action name= .*\& id=([a-z\d]+)/,1]} ˡ ϢχʔΫ ID fla ${path[/piwik\.php\?action name= ˡ flash ΞυΦϯ? .*\&fla=(\d+)/,1] == "1" ? "true" : "false" } </match> 11http://developer.piwik.org/api-reference/tracking-api 31 of 47
32. td-agent ͷઃఆ ∼ Piwik ͱผډʢ6ʣ ▪ elasticsearch αʔόʔ □ fluentd ͷม਺தʹ url encode ͕͔͔͍ͬͯΔͷͰಡΊΔΑ͏ʹ͢Δ □ ऴΘͬͨΒ piwiktracker.apache.access.store ʹॲཧΛҠ͠·͢ <match piwiktracker.apache.access.urldecode> type uri_decode tag piwiktracker.apache.access.store key_names action_name,ref,url,urlref </match> 32 of 47
33. td-agent ͷઃఆ ∼ Piwik ͱผډʢ7:ऴʣ ▪ elasticsearch αʔόʔ □ store Λෳ਺࢖ͬͯ elasticsearch Ҏ֎ʹ΋σʔλసૹՄೳ <match piwiktracker.apache.access.store> type copy <store> type elasticsearch type_name access_log host 127.0.0.1 port 9200 logstash_format true logstash_prefix apache-log logstash_dateformat %Y%m%d include_tag_key true tag_key @log_name flush_interval 10s </store> </match> 33 of 47
34. td-agent ͷઃఆ ∼ Piwik ͱผډʢ1ʣ ▪ Piwik ͱ elasticsearch ͕ಉ͡αʔόʔ □ td-agent ΛೖΕͯϙʔτΛۭ͚͓͖ͯ·͢ɻ □ ର৅ઃఆϑΝΠϧ /etc/td-agent/td-agent.conf ▪ جຊ “ผډ” ࣌ͷೋ୆ͷαʔόʔͷઃఆΛͻͱͭʹ·ͱΊΔ͚ͩͰ͢ ▪ λά͚ͩࣔ͠·͢ɻ □ ׬શ൛͸ “Piwik ͷτϥοΩϯάσʔλΛ elasticsearch Ͱऩूͯ͠Έ Δ 12” Λࢀর͍ͯͩ͘͠͞ɻ 12https://osdn.jp/projects/piwik-fluentd/wiki/FrontPage 34 of 47
35. td-agent ͷઃఆ ∼ Piwik ͱผډʢ2:ऴʣ ▪ Piwik ͱ elasticsearch ͕ಉ͡αʔόʔ □ λά͚ͩࣔ͠·͢ɻத਎͸ “ผډ” ࣌ͱಉ͡Ͱ͢ɻ ▪ ͨͩ͠ɺ “ผډ” ࣌ͷ Piwik αʔόʔ forward ͕͋Γ·ͤΜɻ <source> tag piwiktracker.apache.access </source> <match piwiktracker.apache.access> tag piwiktracker.apache.access.urldecode </match> <match piwiktracker.apache.access.urldecode> tag piwiktracker.apache.access.store </match> <match piwiktracker.apache.access.store> </match> 35 of 47
36. elasticsearch ͷϑΟʔϧυܕʢ1ʣ ▪ ͜͜Ͱ fluentd ͱ elasticsearch Λ্ཱͪ͛Ε͹ࣗಈతʹ elasticsearch ଆʹλΠϓʢςʔϒϧʣ͕࡞੒͞Εɺσʔλ͕֨ೲ ͞ΕΔΘ͚Ͱ͕͢ɺ ▪ λΠϓதͷϑΟʔϧυʢΧϥϜʣ͸͢΂ͯ string ʹͳΓ·͢ɻ ▪ ͦ͜Ͱɺ֤λΠϓͷܕΛఆٛ͠·͢ɻ 36 of 47
37. elasticsearch ͷϑΟʔϧυܕʢ2ʣ∼ छྨ ▪ Elasticsearch supports the following simple field datatypes13: □ String: string □ Whole number: byte, short, integer, long □ Floating-point: float, double □ Boolean: boolean □ Date: date 13https://www.elastic.co/guide/en/elasticsearch/guide/current/mapping- intro.html 37 of 47
38. elasticsearch ͷϑΟʔϧυܕʢ3ʣ∼ ఆٛ ▪ Json ͰϚοϐϯάఆٛϑΝΠϧΛ࡞ΓɺಛఆͷΠϯσοΫε 14 ໊ͱλΠϓ໊ͰϑΟʔϧυͷܕΛࢦఆ͠·͢ 15ɻ ▪ ཁૉͷΈઆ໌͠·͢ɻશ෦ʹ͍ͭͯ͸ “elasticsearch ͷ mapping ઃఆ 16” ʹ͓͍͓͖ͯ·͢ɻ 14MySQL ͷϚϧνςʔϒϧΈ͍ͨͳ΋ͷɻ elasticsearch ͸ͻͱͭͷςʔϒϧΛ ೔෇ผͳͲͷϧʔϧʹΑΓ෼ׂ֨ೲͰ͖Δ 15σϑΥϧτͷৼΔ෣͍΋ఆٛͰ͖·͢ 16https://osdn.jp/projects/piwik-fluentd/wiki/ elasticsearch#h2-elasticsearch.20.E3.81.AE.20mapping.20.E8.A8.AD.E5.AE.9A 38 of 47
39. elasticsearch ͷϑΟʔϧυܕʢ4ʣ∼ Json ৄࡉ ▪ ”template”: ”apache-log-*”, ͲͷΠϯσοΫε 17 ͷ mapping ͳͷ͔ td-agent.conf ͷ logstash prefix apache-log ͱҰகͤ͞Δɻ ΞελϦεΫ͕͍͍ͭͯΔͷ͸ɺlogstash dateformat %Y%m%d ʹΑΓ “apache-log-೔෇” ͱͳࣉ಺א෩ΓΠϯσο Ϋε͸೔෇Ͱ෼ׂ͞Εɺ͜ΕΒΛશ෦र͏ͨΊɻ ▪ ”settings”: { Ұ෦ʹ೔ຊޠ͕૝ఆ͞ΕΔจࣈྻͷݕࡧ index Λ࡞੒͢Δࡍʹ kuromoji Λಋೖ͢ΔͨΊͷఆٛɻ“Elasticsearch ͱ kuromoji Ͱ ͪΌΜͱͨ͠೔ຊޠશจݕࡧΛ΍ΔϝϞ 18” ͷͱ͓Γʹ΍ͬͯΈ ·ͨ͠ɻ 17ϦϨʔγϣφϧ DB ͷςʔϒϧͷΑ͏ͳ΋ͷ 18http://tech.gmo-media.jp/post/70245090007/elasticsearch-kuromoji- japanese-fulltext-search 39 of 47
40. elasticsearch ͷϑΟʔϧυܕʢ5ʣ∼ Json ৄࡉ ▪ ”mappings”: { ”access log”: { ”access log” ͸ɺλΠϓͷ໊લͰ td-agent.conf ͷ type name access log ͱҰகͤ͞·͢ 19ɻҎԼλΠϓͷܕΛͦΕͧΕఆٛ͠ ͍͖ͯ·͢ɻ 19“ default ” ͸͢΂ͯͷλΠϓʹҰக͠·͢ 40 of 47
41. elasticsearch ͷϑΟʔϧυܕʢ6ʣ∼ Json ৄࡉ ▪ λΠϓͷϑΟʔϧυܕఆٛʢσϑΥϧτఆٛϑΟʔϧυʣ □ source ͱ all Λແޮʹͯ͠ΠϯσοΫεͷ༰ྔΛݮΒ͠·͢ɻ "mappings": { "access log": { ˡ λΠϓͷ໊લ " source": { ˡ σϑΥϧτͰશ෦ͷϑΟʔϧυͷ஋͕ೖΔ "enabled": "false" ˡ ඞཁͳ͠ɺσϑΥϧτ͸ true }, " all": { ˡ ͜Ε΋σϑΥϧτͰશ෦ͷϑΟʔϧυͷ஋͕ೖΔ "enabled": "false" ˡ ඞཁͳ͠ɺσϑΥϧτ͸ true }, 41 of 47
42. elasticsearch ͷϑΟʔϧυܕʢ7ʣ∼ Json ৄࡉ ▪ λΠϓͷϑΟʔϧυܕఆٛ □ ࣍ʹ֤ϑΟʔϧυͷఆٛͰ͢ "mappings": { "access log": { ʢུɺҰຕલͰઆ໌ʣ "properties": { "@log name": { ˠ ϑΟʔϧυ໊ʢsee td-agent.confʣ "type": "string", ˠ จࣈྻͰ͢ "store": "true", ˠ ੜσʔλΛه࿥͠·͢ "index": "not analyzed" ˠ ߏจղੳແΠϯσοΫε }, ▪ See “Mapping parameters20”. 20https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping- params.html 42 of 47
43. elasticsearch ͷϑΟʔϧυܕʢ8ʣ∼ Json ৄࡉ ▪ λΠϓͷϑΟʔϧυܕఆٛ □ ʢଓ͖ɺൈਮʣ֤ϑΟʔϧυͷఆٛ "ref": { ˡ ϑΟʔϧυ໊ʢtd-agent.conf Ͱఆٛʣ "type": "multi field", "fields": { ˡ ྆ํͷΠϯσοΫεΛ࡞Δ "ref": { "type": "string", "index": "analyzed", ˡ Ԥจߏจղੳ "store": "true" }, "full": { "type": "string", "index": "not analyzed", ˡ ߏจղੳແ "store": "true" } } }, 43 of 47
44. elasticsearch ͷϑΟʔϧυܕʢ9:ऴʣ∼ Json ৄࡉ ▪ λΠϓͷϑΟʔϧυܕఆٛ □ ʢଓ͖ɺൈਮʣ֤ϑΟʔϧυͷఆٛ "action_name": { "type": "string", "analyzer": "kuromoji analyzer", ˡ ೔ຊޠߏจղੳ "store": "true" }, 44 of 47
45. elasticsearch ͷςϯϓϨʔτొ࿥ 1. ˜/piwik-template.json Λ “elasticsearch ͷ mapping ઃఆ 21” ΑΓ ίϐϖͯ͠࡞੒͠·͢ɻ 2. elasticsearch Λ্ཱͪ͛·͢ɻ $ sudo service elasticsearch start 3. ςϯϓϨʔτ໊ piwik-tracker ͱͯ͠ಥͬࠐΈ·͢ʢҰߦͰʣ ɻ $ curl -XPUT localhost:9200/_template/piwik-tracker \ -d "‘cat ~/piwik-template.json‘" 21https://osdn.jp/projects/piwik-fluentd/wiki/elasticsearch#h2- elasticsearch.20.E3.81.AE.20mapping.20.E8.A8.AD.E5.AE.9A 45 of 47
46. ্ཱͪ͛·͢ ▪ ผډͷͱ͖͸ td-agent Λ྆αʔόʔڞ্ཱͪ͛·͢ɻ $ sudo service td-agent start $ sudo service kibana start ▪ kibana ͷը໘ http://your elasticserach server:5601/ 46 of 47
47. ͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠ 47 of 47