Lock in $30 Savings on PRO—Offer Ends Soon! ⏳

Visualize Piwik Tracker logs on kibana through ...

Visualize Piwik Tracker logs on kibana through fluentd jp

Visualize Piwik Tracker logs on kibana through fluentd. Kibana used for visualization.

YAMAMOTO Takashi

February 27, 2016
Tweet

More Decks by YAMAMOTO Takashi

Other Decks in Technology

Transcript

  1. ԿΛՄࢹԽ͢Δͷ͔ ▪ Piwik αʔόʔʹͷ͜Δ Piwik tracker ͔Βͷϩά 125.54.155.180 - -

    [21/Feb/2016:08:46:13 +0900] "GET /piwik.php?action_name=example.com%2F%E5%A0%B1%E5%91 ʢུ - snipʣ &idsite=1&rec=1&r=047899&h=23&m=46&s=16 &url=http%3A%2F%2Fjpvlad.com%2Findex.php%3Ftopic%3Deventresult_ &_id=4e5ded8520370239&_idts=1435710334&_idvc=387 &_idn=0&_refts=0&_viewts=1455979574&send_image=0 &pdf=1&qt=0&realp=1&wma=1&dir=1&fla=1&java=1&gears=0 &ag=1&cookie=1&res=1366x768 HTTP/1.1" 204 - "http://jpvlad.com/index.php?topic=eventresult_ja" "Mozilla/5.0 (WindowsNT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.63 Safari/537.36" ˢ͜ΜͳͷΛ elasticsearch ʹετΞͯ͠ kibana ͰݟͯΈ·͢ 3 of 47
  2. Piwik Tracker ԿΛ Piwik ʹૹΔͷ͔ ▪ τϥοΧʔ͕ͳͯ͘΋ී௨ʹه࿥Ͱ͖Δ΋ͷ □ client ip

    addresses, user agent, referer ▪ Piwik Tracker ͕ૹ৴͢Δ΋ͷ □ idsite: Piwik Ͱूܭ͍ͯ͠ΔͲͷ Web αΠτͰ͔͢ □ action name: ݟΒΕͨ Web ϖʔδͷλΠτϧ □ id: ΫϥΠΞϯτݻ༗ ID □ res: ΫϥΠΞϯτ PC ͷը໘ղ૾౓ □ pdf: ΫϥΠΞϯτ Web ϒϥ΢βʹ pdf ϓϥάΠϯ͕ೖ͍ͬͯΔ? □ java: java ? □ fla: flash ? □ cookie: cookie Λαϙʔτ͍ͯ͠Δϒϥ΢β͔? □ viewts: લճ๚໰࣌ؒ ▪ ଞʹ΋͋Γ·͢ɻৄ͘͠͸ “Supported Query Parameters1” 1http://developer.piwik.org/api-reference/tracking-api 5 of 47
  3. ՄࢹԽ·Ͱͷखॱ 1. Piwik, fluentd, elasticsearch, kibana ͷΠϯετʔϧ 2. ֤αΠτͷདྷ๚ঢ়گ͸ Piwik

    ͷτϥοΩϯάίʔυʹΑΓ Piwik αʔόʔʹू໿ □ Piwik αʔόʔͷ PHP Λ௚઀ୟ͚͘Ͳɺϩάʹ΋࢒Δ □ શͯ GET 3. Piwik αʔόʔͷϩάΛ fluentd ܦ༝Ͱ elasticsearch ʹετΞ □ elasticsearch ͸෼ࢄܕશจݕࡧαʔόͰ͢ □ fluentd ͰҰ෦ URL decode ΋ߦͬͯ͠·͍·͢ 4. kibana Ͱ elasticsearch ͷσʔλՄࢹԽ͢Δ 6 of 47
  4. ߏ੒͸͜Μͳײ͡ʹͳΓ·͢ td-agent 送信 td-agent 受信 データ整形 Store 可視化 Apache アクセスログ

    tail no-sql データベース Piwik Tracker (JavaScript) Administrator Piwik server elasticsearch server forward (2 台に分ける場合) 7 of 47
  5. ৚݅ ▪ RedHat7 ʢCentOS7, Scientific Linux 7ʣσϑΥϧτ □ RedHat6 ͕ผͷखॱʹͳΔ৔߹͸ʢRedHat6ʣͱ͢Δ

    □ ʢRedHat6ʣ· · · CentOS6, Scientific Linux 6 ▪ Piwik ͸͢ͰʹՔಇ͍ͯ͠Δ΋ͷͱ͠·͢ □ Piwik ೔ຊϢʔβʔձͷ Web αΠτΛ͝ཡ͍ͩ͘͞ 2 ▪ fluentd, elasticsearch, kibana ͸ಉ͡αʔόʔʹΠϯετʔϧ͠ ·͢ □ Piwik ʹ͍ͭͯ͸͜ΕΒͱಉډɺผډͷೋ௨Γઆ໌͠·͢ 2http://www.piwikjapan.org/ػೳઆ໌/3985 8 of 47
  6. fluentd ͷΠϯετʔϧ ∼ ύοέʔδʢ1ʣ ▪ fluentd ͷϥούʔͰ͋Δ td-agent Λ࢖͏͜ͱʹ͢Δ ▪

    td-agent ͸ 2.x ܥʹ͢Δʢ1.x ܥ͸αϙʔτऴྃʣ ▪ αʔόʔͷ ruby ͱ౰ͨΒͳ͍Α͏ʹ RPM ύοέʔδͰ౤ೖ͢Δ □ fluentd ͸ ruby Ͱߏ੒͞Ε͍ͯΔ □ RedHat6 ඪ४͸ ruby 1.9.3 □ RedHat7 ඪ४͸ ruby 2.0 □ Ұํ td-agent 2.x ͸ύοέʔδ࡞੒࣌ ruby 2.2 Λཁٻ ▪ fluentd ύοέʔδͷதʹ fluentd ͷ௥ՃϓϥάΠϯΛೖΕ͓ͯ͘ □ όΠφϦύοέʔδʢRPMʣ͸ଘࡏ͢Δ΋ͷͷ ྫ͑͹ elasticsearch ϓϥάΠϯ͸ؚ·Ε͍ͯͳ͍ □ ޙͰͲ͏΍ͬͯ௥Ճ͍͍͔ͯ͠Θ͔Βͳ͍ͷͰ࠷ॳ͔ΒೖΕ͓ͯ͘ □ ˢ͜Ε͕ύοέʔδΛ࡞Δཧ༝ 9 of 47
  7. fluentd ͷΠϯετʔϧ ∼ ύοέʔδʢ2ʣ ▪ ·ͣ͸ ruby 2.2.4 ͷΠϯετʔϧ 1.

    ύοέʔδΛ࡞Δ ruby ͕Քಇ͍ͯ͠ͳ͍ RedHat ؀ڥΛ༻ҙ ▪ CentOS, Scientific Linux ΋ͪΖΜՄೳ ▪ 6 Ͱ΋ 7 Ͱ΋ 2. td-agent ͕ RPM ͰΠϯετʔϧ͞Ε͍ͯΕ͹࡟আ͓͖ͯ͠·͢ 3. SRPM ͔Β rpm Λ࡞੒͢ΔͨΊͷπʔϧΛ౤ೖ $ sudo yum groupinstall "Development tools" 4. “CentOS 6 Ͱ ruby ͷ RPM ύοέʔδΛ࡞Δ 3” ΑΓ ruby223.spec Λμ΢ϯϩʔυ 5. RPM Λ࡞ΔͨΊͷඞཁͳσΟϨΫτϦΛ࡞Δ ˠ Ctrl+C ͰࢭΊΔ $ rpmbuild -bp ruby223.spec ʢCtrl+C ͰࢭΊΔʣ ʢ~/rpmbuild σΟϨΫτϦ͕Ͱ͖͍ͯΔ͸ͣʣ $ mv ruby223.spec rpmbuild/SPECS/ruby224.spec ʢ224 ʹมߋʣ 3http://www.torutk.com/projects/swe/wiki/CentOS 6 Ͱ ruby ͷ RPM ύο έʔδΛ࡞Δ 10 of 47
  8. fluentd ͷΠϯετʔϧ ∼ ύοέʔδʢ3ʣ ▪ ruby 2.2.4 ͷΠϯετʔϧʢଓ͖ʣ 1. ˜/rpmbuild/SPECS/ruby224.spec

    Ұߦ໨Λมߋ͢Δ %define rubyver 2.2.4 2. “Ruby 2.2.4 ϦϦʔε 4” ΑΓ ruby-2.2.4.tar.bz2 Λμ΢ϯϩʔυ 3. ruby-2.2.4.tar.bz2 Λ ˜/rpmbuild/SOURCES ʹҠಈ 4. RPM ϑΝΠϧΛ࡞Δ $ cd ~/rpmbuild/SPECS $ rpmbuild -ba ruby224.spec ʢུʣ $ sudo rpm -ivh \ʢ࣍ͷߦͱ͋Θͤͯʣ ~/rpmbuild/RPMS/x86_64/ruby-2.2.4-1.el7.x86_64.rpm ʢRedHat6ʣ el6 ʹͳΓ·͢ ʢུʣ $ ruby -v ruby 2.2.4p230 (2015-12-16 revision 53155) [x86_64-linux] 4https://www.ruby-lang.org/ja/news/2015/12/16/ruby-2-2-4-released/ 11 of 47
  9. fluentd ͷΠϯετʔϧ ∼ ύοέʔδʢ4ʣ ▪ ඞཁύοέʔδͷΠϯετʔϧ 1. epel ϨϙδτϦΛࢀর͢ΔΑ͏ʹ͓͍͍ͯͯͩ͘͠͞ ▪

    ҰߦͰ $ sudo yum install \ http://ftp-srv2.kddilabs.jp/Linux/distributions/ \ fedora/epel/7/x86 64/e/epel-release-7-5.noarch.rpm ▪ ʢRedHat6ʣҰߦͰ $ sudo yum install \ http://ftp-srv2.kddilabs.jp/Linux/distributions/ \ fedora/epel/6/x86 64/epel-release-6-8.noarch.rpm 2. Πϯετʔϧ $ sudo yum install gecode gecode-devel fakeroot 12 of 47
  10. fluentd ͷΠϯετʔϧ ∼ ύοέʔδʢ5ʣ 1. ʢRedHat6ʣgit ͷΞοϓσʔτ $ wget http://dl.marmotte.net/rpms/redhat/el6/x86

    64/\ git-1.8.3.1-3.el6/git-1.8.3.1-3.el6.src.rpm $ cp ~/rpmbuild/SRPMS/git-1.8.3.1-3.el6.src.rpm $ rpmbuild --rebuild \ ~/rpmbuild/SRPMS/git-1.8.3.1-3.el6.src.rpm $ sudo yum install perl-TermReadKey $ sudo rpm -ivh \ ~/rpmbuild/RPMS/x86 64/git-1.8.3.1-3.el6.x86_64.rpm □ git 1.8 ୆ʹ͠ͳ͍ͱ “-c” Φϓγϣϯ͕ͳ͍ͨΊϏϧυͰ͖ͳ͍ ▪ ެࣜʹ͸ git 1.8 ୆͸ଘࡏ͠ͳ͍ ▪ ͍Ζ͍Ζ͋ΔதͰɺґଘύοέʔδ͕ epel Ͱऩ·Δ΋ͷΛબ୒ 13 of 47
  11. fluentd ͷΠϯετʔϧ ∼ ύοέʔδʢ6ʣ ▪ ruby ؀ڥߏஙɺtd-agent ιʔεͷμ΢ϯϩʔυ 1. bundle

    ΛΠϯετʔϧ $ sudo gem install bundler 2. github ͔Β clone $ cd ~ $ git clone \ʢ࣍ͷߦͱ͋Θͤͯʣ [email protected]:treasure-data/omnibus-td-agent.git $ cd ~/omnibus-td-agent 3. treasure-data/omnibus-td-agent5 ͷ௨Γʹ࣮ߦ͠·͕͢ɺґଘੑͷղ ܾʹࣦഊ͢ΔͷͰ Gemfile ͷ్தʹҰߦ௥Ճ͠·͢ʢ࣍ϖʔδʣ ɻ 5https://github.com/treasure-data/omnibus-td-agent 14 of 47
  12. fluentd ͷΠϯετʔϧ ∼ ύοέʔδʢ7ʣ ▪ ιʔεΛमਖ਼͍͖ͯ͠·͢ □ ґଘੑ໰୊Λճආ □ ˜/omnibus-td-agent/Gemfile

    ʹ gem ’pedump’ · · · ͷҰߦΛ௥Ճ 6 source ’https://rubygems.org’ # Use Berkshelf for resolving cookbook dependencies gem ’berkshelf’, ’~> 3.0’ gem ’pedump’, git: ’https://github.com/ksubrama/pedump’, branch: ’patch-1’ # ্͔ΒҰߦͰ௥Ճ # Install omnibus software #gem ’omnibus’, ’~> 5.0’ ʢҎԼུʣ 6https://github.com/piwikjapan/omnibus-td-agent/blob/master/Gemfile 15 of 47
  13. fluentd ͷΠϯετʔϧ ∼ ύοέʔδʢ8ʣ ▪ elasticsearch, record-reformer, norikra ϓϥάΠϯΛ RPM

    ʹ૊Έ ࠐΉ □ ࠓ೔͸ norikra ʹ͍ͭͯ͸ݴٴ͠·ͤΜ ▪ ˜/omnibus-td-agent/plugin gems.rb ͷ࠷ޙʹࡾߦ௥Ճ download "fluent-plugin-norikra", "0.2.2" download "fluent-plugin-elasticsearch", "1.3.0" download "fluent-plugin-record-reformer", "0.8.0" 16 of 47
  14. fluentd ͷΠϯετʔϧ ∼ ύοέʔδʢ9ʣ ▪ norikra ϓϥάΠϯʹؔ࿈͢ΔϓϥάΠϯΛ௥Ճ □ ࠓ೔͸ norikra

    ʹ͍ͭͯ͸ݴٴ͠·ͤΜ □ norikra-client ͕Ҿ͖ࠐΉ msgpack-rpc-over-http ͕ rack Λཁٻ͢Δ ͕ɺ2.x ࠷৽൛ΛಡΈࠐΜͰΤϥʔ͕ग़ΔͷͰͻͱͭલͷ 1.6.4 Λڧ ੍తʹ࢖͏ ▪ ˜/omnibus-td-agent/core gems.rb ͷ࠷ޙʹೋߦ௥Ճ download "rack", "1.6.4" download "norikra-client", "1.3.1" 17 of 47
  15. fluentd ͷΠϯετʔϧ ∼ ύοέʔδʢ10ʣ ▪ ϫʔΫσΟϨΫτϦΛ࡞Γ·͢ 7ɻ $ sudo mkdir

    -p /opt/td-agent /var/cache/omnibus $ sudo chown yamachan:yamachan /opt/td-agent $ sudo chown yamachan:yamachan/var/cache/omnibus □ yamachan:yamachan ͸ࣗ෼ͷ id ͱάϧʔϓʹஔ͖׵͍͑ͯͩ͘͞ 7https://github.com/treasure-data/omnibus-td-agent 18 of 47
  16. fluentd ͷΠϯετʔϧ ∼ ύοέʔδʢ11:ऴʣ 1. खॱ௨Γʹ 8 $ cd ~/omnibus-td-agent

    $ bundle install --binstubs ʢུɺ్தͰ sudo ͢ΔͷͰύεϫʔυΛೖྗʣ $ bin/gem_downloader core_gems.rb ʢུʣ $ bin/gem_downloader plugin_gems.rb ʢུʣ $ bin/omnibus build td-agent2 ʢུʣ 8https://github.com/treasure-data/omnibus-td-agent 19 of 47
  17. fluentd ͷΠϯετʔϧ ∼ Πϯετʔϧ 1. ग़དྷ্͕ͬͨύοέʔδ͸ pkg ҎԼʹೖΓ·͢ $ cd

    ~/omnibus-td-agent/pkg $ sudo yum install td-agent-2.3.1-0.el7.x86 64.rpm 2. ʢRedHat6ʣtd-agent-2.3.1-0.el6.x86 64.rpm 20 of 47
  18. elasticsearch ͷΠϯετʔϧ 1. RedHat7, RedHat6 ڞ௨Ͱ͢ɻҰߦͰ $ sudo yum install

    \ https://download.elasticsearch.org/elasticsearch/\ release/org/elasticsearch/distribution/\ rpm/elasticsearch/2.2.0/elasticsearch-2.2.0.rpm 2. ೔ຊޠߏจղੳ kuromoji ϓϥάΠϯΛೖΕ·͢ɻҰߦͰ $ sudo /usr/share/elasticsearch/bin/plugin \ install analysis-kuromoji 21 of 47
  19. kibana ͷΠϯετʔϧ 1. ύοέʔδ͕ͳ͍ͷͰ࡞Γ·͢ $ cd ~ $ git clone

    [email protected]:piwikjapan/kibana-rpm-packaging.git $ cd kibana-rpm-packaging $ cp kibana.sysconfig kibana.service ~/rpmbuild/SOURCES $ cp kibana.spec ~/rpmbuild/SPECS $ wget -P ~/rpmbuild/SOURCES \ https://download.elastic.co/kibana/kibana/\ kibana-4.4.1-linux-x64.tar.gz $ rpmbuild -ba ~/rpmbuild/SPECS/kibana.spec 2. Πϯετʔϧ͠·͢ $ sudo rpm -ivh ~rpmbuild/RPMS/x86_64/\ kibana-4.4.1-1.x86_64.rpm 22 of 47
  20. ʢRedHat6ʣkibana ͷΠϯετʔϧ ▪ ͜ͷ΁Μ “kibana4 ηοτΞοϓ 9” ΛΈ͍ͯͩ͘͞ɻ ▪ ىಈεΫϦϓτ΋ܝࡌ͞Ε͍ͯ·͢ɻ

    ▪ ؾ͕޲͍ͨΒύοέʔδεΫϦϓτ࡞ͬͯΈ·͢ɻ 9http://qiita.com/nagomu1985/items/82e699dde4f99b2ce417 23 of 47
  21. ϑΝΠΞʔ΢Υʔϧͷઃఆ 1. norikraʢ26578/tcpʣʹ͍ͭͯ͸ࠓճ͸ݴٴ͠·ͤΜ $ sudo firewall-cmd --zone=public \ --add-port=26578/tcp --permanent

    # norikra web $ sudo firewall-cmd --zone=public \ --add-port=5651/tcp --permanent # kibana web $ sudo firewall-cmd --zone=public \ --add-port=24224/udp --permanent # fluentd heatbeat $ sudo firewall-cmd --zone=public \ --add-port=24224/tcp --permanent # fluentd data 2. ൓ө͠·͢ $ sudo firewall-cmd --reload 24 of 47
  22. ʢRedHat6ʣϑΝΠΞʔ΢Υʔϧͷઃఆ 1. norikraʢ26578/tcpʣʹ͍ͭͯ͸ࠓճ͸ݴٴ͠·ͤΜ 2. /etc/sysconfig/iptables ͷ -A INPUT -m state

    –state ESTABLISHED,RELATED -j ACCEPT ͷԼʹ௥Ճ͍ͯͩ͘͠͞ -A INPUT -m multiport -p tcp -m tcp \ʢҰߦͰʣ --dports 26578,5651,24224 -j ACCEPT -A INPUT -m multiport -p udp -m udp --dports 24224 -j ACCEPT 3. ൓ө͠·͢ $ sudo service iptables reload 25 of 47
  23. td-agent ͷઃఆ ▪ Piwik ͕ elasticsearch, kibana ͱ 1. ผډʢPiwik

    server ͱ elasticsearch server ͕ผʣ 2. ಉډʢPiwik server ͱ elasticsearch server ͕ಉ͡ɺforward ͕ͳ͍ʣ td-agent 送信 td-agent 受信 データ整形 Store 可視化 Apache アクセスログ tail no-sql データベース Piwik Tracker (JavaScript) Administrator Piwik server elasticsearch server forward (2 台に分ける場合) 26 of 47
  24. td-agent ͷઃఆ ∼ Piwik ͱผډʢ1ʣ ▪ Piwik ͱ elasticsearch ͕ผαʔόʔ

    □ ྆ํʹ td-agent ΛೖΕͯϙʔτΛۭ͚͓͖ͯ·͢ɻ □ ର৅ઃఆϑΝΠϧ/etc/td-agent/td-agent.conf ▪ ࣍ϖʔδҎ߱ͷྫΛॱ൪ʹͭͳ͛ͯͻͱͭʹ͍ͯͩ͘͠͞ ▪ ը໘ʹೖΓ͖Βͳ͍ͱ͖͸୺ંΓ·͢ɻ □ ׬શ൛͸ “Piwik ͷτϥοΩϯάσʔλΛ elasticsearch Ͱऩूͯ͠Έ Δ 10” Λࢀর͍ͯͩ͘͠͞ɻ 10https://osdn.jp/projects/piwik-fluentd/wiki/FrontPage 27 of 47
  25. td-agent ͷઃఆ ∼ Piwik ͱผډʢ2ʣ ▪ Piwik αʔόʔ □ Piwik

    ͷΞΫηεϩάΛ fluentd Ͱٵ্͍͛·͢ □ tag piwiktracker.apache.access Ͱ࣍ͷॲཧΛҾ͖ܧ͗ <source> type tail format apache time_format %d/%b/%Y:%H:%M:%S %z pos_file /var/log/td-agent/access_log.pos path /var/log/httpd/access_log tag piwiktracker.apache.access </source> 28 of 47
  26. td-agent ͷઃఆ ∼ Piwik ͱผډʢ3ʣ ▪ Piwik αʔόʔ □ host

    Ͱࢦఆ͢ΔαʔόʔʹϩάΛసૹ͠·͢ɻ <match piwiktracker.apache.access> type forward send_timeout 60s recover_wait 300s heartbeat_interval 1s phi_threshold 16 hard_timeout 60s <server> name fruentd host your_elsticsearch_server i.e. 10.x.x.x port 24224 weight 100 </server> </match> 29 of 47
  27. td-agent ͷઃఆ ∼ Piwik ͱผډʢ4ʣ ▪ elasticsearch αʔόʔ □ Tracker

    ͷϩάͷΈΛநग़͠·͢ 1. Piwik ͷ؅ཧը໘ʹΞΫηεͨ͠ͱ͖ͷϩά 2. Piwik ͷ API Λ͍ͨͨͨͱ͖ͷϩά 3. filter ͷ࣍ʹ match piwiktracker.apache.access ʹભҠ͠·͢ <filter piwiktracker.apache.access> type grep regexp1 path /piwik\.php\?action name=.*\&idsite=\d+ </filter> <match piwiktracker.apache.access> type record_reformer tag piwiktracker.apache.access.urldecode ʢུɺ࣍ϖʔδͰʣ 30 of 47
  28. td-agent ͷઃఆ ∼ Piwik ͱผډʢ5ʣ ▪ elasticsearch αʔόʔ □ ύʔεจࣈྻΛ

    fluentd ͷม਺ʹ෼ղɻͦΕͧΕͷҙຯ͸ “Supported Query Parameters11” Λࢀর □ ઌ಄ʹΞϯμʔείΞ “ ” ͸ېࢭɻ·ͨม਺ “id” ͸ෆՄɻ □ ෼ղͨ͠Β piwiktracker.apache.access.urldecode ΁ <match piwiktracker.apache.access> type record_reformer tag piwiktracker.apache.access.urldecode ʢҎԼ 29 ݸத 3 ͚ͭͩʣ idsite ${path[/piwik\.php\? action name=.*\&idsite=(\d+)/,1]} ˡ αΠτ ID piwikid ${path[/piwik\.php\?action name= .*\& id=([a-z\d]+)/,1]} ˡ ϢχʔΫ ID fla ${path[/piwik\.php\?action name= ˡ flash ΞυΦϯ? .*\&fla=(\d+)/,1] == "1" ? "true" : "false" } </match> 11http://developer.piwik.org/api-reference/tracking-api 31 of 47
  29. td-agent ͷઃఆ ∼ Piwik ͱผډʢ6ʣ ▪ elasticsearch αʔόʔ □ fluentd

    ͷม਺தʹ url encode ͕͔͔͍ͬͯΔͷͰಡΊΔΑ͏ʹ͢Δ □ ऴΘͬͨΒ piwiktracker.apache.access.store ʹॲཧΛҠ͠·͢ <match piwiktracker.apache.access.urldecode> type uri_decode tag piwiktracker.apache.access.store key_names action_name,ref,url,urlref </match> 32 of 47
  30. td-agent ͷઃఆ ∼ Piwik ͱผډʢ7:ऴʣ ▪ elasticsearch αʔόʔ □ store

    Λෳ਺࢖ͬͯ elasticsearch Ҏ֎ʹ΋σʔλసૹՄೳ <match piwiktracker.apache.access.store> type copy <store> type elasticsearch type_name access_log host 127.0.0.1 port 9200 logstash_format true logstash_prefix apache-log logstash_dateformat %Y%m%d include_tag_key true tag_key @log_name flush_interval 10s </store> </match> 33 of 47
  31. td-agent ͷઃఆ ∼ Piwik ͱผډʢ1ʣ ▪ Piwik ͱ elasticsearch ͕ಉ͡αʔόʔ

    □ td-agent ΛೖΕͯϙʔτΛۭ͚͓͖ͯ·͢ɻ □ ର৅ઃఆϑΝΠϧ /etc/td-agent/td-agent.conf ▪ جຊ “ผډ” ࣌ͷೋ୆ͷαʔόʔͷઃఆΛͻͱͭʹ·ͱΊΔ͚ͩͰ͢ ▪ λά͚ͩࣔ͠·͢ɻ □ ׬શ൛͸ “Piwik ͷτϥοΩϯάσʔλΛ elasticsearch Ͱऩूͯ͠Έ Δ 12” Λࢀর͍ͯͩ͘͠͞ɻ 12https://osdn.jp/projects/piwik-fluentd/wiki/FrontPage 34 of 47
  32. td-agent ͷઃఆ ∼ Piwik ͱผډʢ2:ऴʣ ▪ Piwik ͱ elasticsearch ͕ಉ͡αʔόʔ

    □ λά͚ͩࣔ͠·͢ɻத਎͸ “ผډ” ࣌ͱಉ͡Ͱ͢ɻ ▪ ͨͩ͠ɺ “ผډ” ࣌ͷ Piwik αʔόʔ forward ͕͋Γ·ͤΜɻ <source> tag piwiktracker.apache.access </source> <match piwiktracker.apache.access> tag piwiktracker.apache.access.urldecode </match> <match piwiktracker.apache.access.urldecode> tag piwiktracker.apache.access.store </match> <match piwiktracker.apache.access.store> </match> 35 of 47
  33. elasticsearch ͷϑΟʔϧυܕʢ1ʣ ▪ ͜͜Ͱ fluentd ͱ elasticsearch Λ্ཱͪ͛Ε͹ࣗಈతʹ elasticsearch ଆʹλΠϓʢςʔϒϧʣ͕࡞੒͞Εɺσʔλ͕֨ೲ

    ͞ΕΔΘ͚Ͱ͕͢ɺ ▪ λΠϓதͷϑΟʔϧυʢΧϥϜʣ͸͢΂ͯ string ʹͳΓ·͢ɻ ▪ ͦ͜Ͱɺ֤λΠϓͷܕΛఆٛ͠·͢ɻ 36 of 47
  34. elasticsearch ͷϑΟʔϧυܕʢ2ʣ∼ छྨ ▪ Elasticsearch supports the following simple field

    datatypes13: □ String: string □ Whole number: byte, short, integer, long □ Floating-point: float, double □ Boolean: boolean □ Date: date 13https://www.elastic.co/guide/en/elasticsearch/guide/current/mapping- intro.html 37 of 47
  35. elasticsearch ͷϑΟʔϧυܕʢ3ʣ∼ ఆٛ ▪ Json ͰϚοϐϯάఆٛϑΝΠϧΛ࡞ΓɺಛఆͷΠϯσοΫε 14 ໊ͱλΠϓ໊ͰϑΟʔϧυͷܕΛࢦఆ͠·͢ 15ɻ ▪

    ཁૉͷΈઆ໌͠·͢ɻશ෦ʹ͍ͭͯ͸ “elasticsearch ͷ mapping ઃఆ 16” ʹ͓͍͓͖ͯ·͢ɻ 14MySQL ͷϚϧνςʔϒϧΈ͍ͨͳ΋ͷɻ elasticsearch ͸ͻͱͭͷςʔϒϧΛ ೔෇ผͳͲͷϧʔϧʹΑΓ෼ׂ֨ೲͰ͖Δ 15σϑΥϧτͷৼΔ෣͍΋ఆٛͰ͖·͢ 16https://osdn.jp/projects/piwik-fluentd/wiki/ elasticsearch#h2-elasticsearch.20.E3.81.AE.20mapping.20.E8.A8.AD.E5.AE.9A 38 of 47
  36. elasticsearch ͷϑΟʔϧυܕʢ4ʣ∼ Json ৄࡉ ▪ ”template”: ”apache-log-*”, ͲͷΠϯσοΫε 17 ͷ

    mapping ͳͷ͔ td-agent.conf ͷ logstash prefix apache-log ͱҰகͤ͞Δɻ ΞελϦεΫ͕͍͍ͭͯΔͷ͸ɺlogstash dateformat %Y%m%d ʹΑΓ “apache-log-೔෇” ͱͳࣉ಺א෩ΓΠϯσο Ϋε͸೔෇Ͱ෼ׂ͞Εɺ͜ΕΒΛશ෦र͏ͨΊɻ ▪ ”settings”: { Ұ෦ʹ೔ຊޠ͕૝ఆ͞ΕΔจࣈྻͷݕࡧ index Λ࡞੒͢Δࡍʹ kuromoji Λಋೖ͢ΔͨΊͷఆٛɻ“Elasticsearch ͱ kuromoji Ͱ ͪΌΜͱͨ͠೔ຊޠશจݕࡧΛ΍ΔϝϞ 18” ͷͱ͓Γʹ΍ͬͯΈ ·ͨ͠ɻ 17ϦϨʔγϣφϧ DB ͷςʔϒϧͷΑ͏ͳ΋ͷ 18http://tech.gmo-media.jp/post/70245090007/elasticsearch-kuromoji- japanese-fulltext-search 39 of 47
  37. elasticsearch ͷϑΟʔϧυܕʢ5ʣ∼ Json ৄࡉ ▪ ”mappings”: { ”access log”: {

    ”access log” ͸ɺλΠϓͷ໊લͰ td-agent.conf ͷ type name access log ͱҰகͤ͞·͢ 19ɻҎԼλΠϓͷܕΛͦΕͧΕఆٛ͠ ͍͖ͯ·͢ɻ 19“ default ” ͸͢΂ͯͷλΠϓʹҰக͠·͢ 40 of 47
  38. elasticsearch ͷϑΟʔϧυܕʢ6ʣ∼ Json ৄࡉ ▪ λΠϓͷϑΟʔϧυܕఆٛʢσϑΥϧτఆٛϑΟʔϧυʣ □ source ͱ all

    Λແޮʹͯ͠ΠϯσοΫεͷ༰ྔΛݮΒ͠·͢ɻ "mappings": { "access log": { ˡ λΠϓͷ໊લ " source": { ˡ σϑΥϧτͰશ෦ͷϑΟʔϧυͷ஋͕ೖΔ "enabled": "false" ˡ ඞཁͳ͠ɺσϑΥϧτ͸ true }, " all": { ˡ ͜Ε΋σϑΥϧτͰશ෦ͷϑΟʔϧυͷ஋͕ೖΔ "enabled": "false" ˡ ඞཁͳ͠ɺσϑΥϧτ͸ true }, 41 of 47
  39. elasticsearch ͷϑΟʔϧυܕʢ7ʣ∼ Json ৄࡉ ▪ λΠϓͷϑΟʔϧυܕఆٛ □ ࣍ʹ֤ϑΟʔϧυͷఆٛͰ͢ "mappings": {

    "access log": { ʢུɺҰຕલͰઆ໌ʣ "properties": { "@log name": { ˠ ϑΟʔϧυ໊ʢsee td-agent.confʣ "type": "string", ˠ จࣈྻͰ͢ "store": "true", ˠ ੜσʔλΛه࿥͠·͢ "index": "not analyzed" ˠ ߏจղੳແΠϯσοΫε }, ▪ See “Mapping parameters20”. 20https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping- params.html 42 of 47
  40. elasticsearch ͷϑΟʔϧυܕʢ8ʣ∼ Json ৄࡉ ▪ λΠϓͷϑΟʔϧυܕఆٛ □ ʢଓ͖ɺൈਮʣ֤ϑΟʔϧυͷఆٛ "ref": {

    ˡ ϑΟʔϧυ໊ʢtd-agent.conf Ͱఆٛʣ "type": "multi field", "fields": { ˡ ྆ํͷΠϯσοΫεΛ࡞Δ "ref": { "type": "string", "index": "analyzed", ˡ Ԥจߏจղੳ "store": "true" }, "full": { "type": "string", "index": "not analyzed", ˡ ߏจղੳແ "store": "true" } } }, 43 of 47
  41. elasticsearch ͷϑΟʔϧυܕʢ9:ऴʣ∼ Json ৄࡉ ▪ λΠϓͷϑΟʔϧυܕఆٛ □ ʢଓ͖ɺൈਮʣ֤ϑΟʔϧυͷఆٛ "action_name": {

    "type": "string", "analyzer": "kuromoji analyzer", ˡ ೔ຊޠߏจղੳ "store": "true" }, 44 of 47
  42. elasticsearch ͷςϯϓϨʔτొ࿥ 1. ˜/piwik-template.json Λ “elasticsearch ͷ mapping ઃఆ 21”

    ΑΓ ίϐϖͯ͠࡞੒͠·͢ɻ 2. elasticsearch Λ্ཱͪ͛·͢ɻ $ sudo service elasticsearch start 3. ςϯϓϨʔτ໊ piwik-tracker ͱͯ͠ಥͬࠐΈ·͢ʢҰߦͰʣ ɻ $ curl -XPUT localhost:9200/_template/piwik-tracker \ -d "‘cat ~/piwik-template.json‘" 21https://osdn.jp/projects/piwik-fluentd/wiki/elasticsearch#h2- elasticsearch.20.E3.81.AE.20mapping.20.E8.A8.AD.E5.AE.9A 45 of 47
  43. ্ཱͪ͛·͢ ▪ ผډͷͱ͖͸ td-agent Λ྆αʔόʔڞ্ཱͪ͛·͢ɻ $ sudo service td-agent start

    $ sudo service kibana start ▪ kibana ͷը໘ http://your elasticserach server:5601/ 46 of 47