Visualize Piwik Tracker logs on kibana through fluentd. Kibana used for visualization.
ΞΫηεղੳιϑτ Piwik ͷհͱ fluentd ʹΑΔԠ༻ฤYAMAMOTO Takashi[email protected]@yamachan5593Piwik Japan TeamFeb 27th, 2016at Open Source ConferenceTokyo
View Slide
ࣗݾհ■ OpenSolaris Ϣʔβʔձʢ࠷ۙαϘΓؾຯʣ□ https://osdn.jp/projects/jposug/■ Piwikjapan ຊޠύον࡞/ OSC ౦ژͰͷൃද□ https://osdn.jp/projects/piwik-fluentd/2 of 47
ԿΛՄࢹԽ͢Δͷ͔■ Piwik αʔόʔʹͷ͜Δ Piwik tracker ͔Βͷϩά125.54.155.180 - - [21/Feb/2016:08:46:13 +0900] "GET/piwik.php?action_name=example.com%2F%E5%A0%B1%E5%91ʢུ - snipʣ&idsite=1&rec=1&r=047899&h=23&m=46&s=16&url=http%3A%2F%2Fjpvlad.com%2Findex.php%3Ftopic%3Deventresult_&_id=4e5ded8520370239&_idts=1435710334&_idvc=387&_idn=0&_refts=0&_viewts=1455979574&send_image=0&pdf=1&qt=0&realp=1&wma=1&dir=1&fla=1&java=1&gears=0&ag=1&cookie=1&res=1366x768 HTTP/1.1" 204 -"http://jpvlad.com/index.php?topic=eventresult_ja""Mozilla/5.0 (WindowsNT 6.1) AppleWebKit/537.36(KHTML, like Gecko) Chrome/28.0.1500.63 Safari/537.36"ˢ͜ΜͳͷΛ elasticsearch ʹετΞͯ͠ kibana ͰݟͯΈ·͢3 of 47
ຊ͜ΕΛࢦ͠·͢4 of 47
Piwik Tracker ԿΛ Piwik ʹૹΔͷ͔■τϥοΧʔ͕ͳͯ͘ී௨ʹهͰ͖Δͷ□ client ip addresses, user agent, referer■ Piwik Tracker ͕ૹ৴͢Δͷ□ idsite: Piwik Ͱूܭ͍ͯ͠ΔͲͷ Web αΠτͰ͔͢□ action name: ݟΒΕͨ Web ϖʔδͷλΠτϧ□ id: ΫϥΠΞϯτݻ༗ ID□ res: ΫϥΠΞϯτ PC ͷը໘ղ૾□ pdf: ΫϥΠΞϯτ Web ϒϥβʹ pdf ϓϥάΠϯ͕ೖ͍ͬͯΔ?□ java: java ?□ fla: flash ?□ cookie: cookie Λαϙʔτ͍ͯ͠Δϒϥβ͔?□ viewts: લճ๚࣌ؒ■ଞʹ͋Γ·͢ɻৄ͘͠ “Supported Query Parameters1”1http://developer.piwik.org/api-reference/tracking-api5 of 47
ՄࢹԽ·Ͱͷखॱ1. Piwik, fluentd, elasticsearch, kibana ͷΠϯετʔϧ2. ֤αΠτͷདྷ๚ঢ়گ Piwik ͷτϥοΩϯάίʔυʹΑΓ Piwikαʔόʔʹू□ Piwik αʔόʔͷ PHP Λୟ͚͘ͲɺϩάʹΔ□ શͯ GET3. Piwik αʔόʔͷϩάΛ fluentd ܦ༝Ͱ elasticsearch ʹετΞ□ elasticsearch ࢄܕશจݕࡧαʔόͰ͢□ fluentd ͰҰ෦ URL decode ߦͬͯ͠·͍·͢4. kibana Ͱ elasticsearch ͷσʔλՄࢹԽ͢Δ6 of 47
ߏ͜Μͳײ͡ʹͳΓ·͢td-agent送信td-agent受信データ整形Store可視化Apacheアクセスログtailno-sqlデータベースPiwik Tracker(JavaScript)AdministratorPiwik server elasticsearch serverforward（2 台に分ける場合）7 of 47
݅■ RedHat7 ʢCentOS7, Scientific Linux 7ʣσϑΥϧτ□ RedHat6 ͕ผͷखॱʹͳΔ߹ʢRedHat6ʣͱ͢Δ□ ʢRedHat6ʣ· · · CentOS6, Scientific Linux 6■ Piwik ͢ͰʹՔಇ͍ͯ͠Δͷͱ͠·͢□ Piwik ຊϢʔβʔձͷ Web αΠτΛ͝ཡ͍ͩ͘͞ 2■ fluentd, elasticsearch, kibana ಉ͡αʔόʔʹΠϯετʔϧ͠·͢□ Piwik ʹ͍ͭͯ͜ΕΒͱಉډɺผډͷೋ௨Γઆ໌͠·͢2http://www.piwikjapan.org/ػೳઆ໌/39858 of 47
fluentd ͷΠϯετʔϧ ∼ ύοέʔδʢ1ʣ■ fluentd ͷϥούʔͰ͋Δ td-agent Λ͏͜ͱʹ͢Δ■ td-agent 2.x ܥʹ͢Δʢ1.x ܥαϙʔτऴྃʣ■αʔόʔͷ ruby ͱͨΒͳ͍Α͏ʹ RPM ύοέʔδͰೖ͢Δ□ fluentd ruby Ͱߏ͞Ε͍ͯΔ□ RedHat6 ඪ४ ruby 1.9.3□ RedHat7 ඪ४ ruby 2.0□ Ұํ td-agent 2.x ύοέʔδ࡞࣌ ruby 2.2 Λཁٻ■ fluentd ύοέʔδͷதʹ fluentd ͷՃϓϥάΠϯΛೖΕ͓ͯ͘□ όΠφϦύοέʔδʢRPMʣଘࡏ͢Δͷͷྫ͑ elasticsearch ϓϥάΠϯؚ·Ε͍ͯͳ͍□ ޙͰͲ͏ͬͯՃ͍͍͔ͯ͠Θ͔Βͳ͍ͷͰ࠷ॳ͔ΒೖΕ͓ͯ͘□ ˢ͜Ε͕ύοέʔδΛ࡞Δཧ༝9 of 47
fluentd ͷΠϯετʔϧ ∼ ύοέʔδʢ2ʣ■·ͣ ruby 2.2.4 ͷΠϯετʔϧ1. ύοέʔδΛ࡞Δ ruby ͕Քಇ͍ͯ͠ͳ͍ RedHat ڥΛ༻ҙ■ CentOS, Scientific Linux ͪΖΜՄೳ■ 6 Ͱ 7 Ͱ2. td-agent ͕ RPM ͰΠϯετʔϧ͞Ε͍ͯΕআ͓͖ͯ͠·͢3. SRPM ͔Β rpm Λ࡞͢ΔͨΊͷπʔϧΛೖ$ sudo yum groupinstall "Development tools"4. “CentOS 6 Ͱ ruby ͷ RPM ύοέʔδΛ࡞Δ 3” ΑΓ ruby223.specΛμϯϩʔυ5. RPM Λ࡞ΔͨΊͷඞཁͳσΟϨΫτϦΛ࡞Δ ˠ Ctrl+C ͰࢭΊΔ$ rpmbuild -bp ruby223.spec ʢCtrl+C ͰࢭΊΔʣʢ~/rpmbuild σΟϨΫτϦ͕Ͱ͖͍ͯΔͣʣ$ mv ruby223.spec rpmbuild/SPECS/ruby224.spec ʢ224 ʹมߋʣ3http://www.torutk.com/projects/swe/wiki/CentOS 6 Ͱ ruby ͷ RPM ύοέʔδΛ࡞Δ10 of 47
fluentd ͷΠϯετʔϧ ∼ ύοέʔδʢ3ʣ■ ruby 2.2.4 ͷΠϯετʔϧʢଓ͖ʣ1. ˜/rpmbuild/SPECS/ruby224.spec ҰߦΛมߋ͢Δ%define rubyver 2.2.42. “Ruby 2.2.4 ϦϦʔε 4” ΑΓ ruby-2.2.4.tar.bz2 Λμϯϩʔυ3. ruby-2.2.4.tar.bz2 Λ ˜/rpmbuild/SOURCES ʹҠಈ4. RPM ϑΝΠϧΛ࡞Δ$ cd ~/rpmbuild/SPECS$ rpmbuild -ba ruby224.specʢུʣ$ sudo rpm -ivh \ʢ࣍ͷߦͱ͋Θͤͯʣ~/rpmbuild/RPMS/x86_64/ruby-2.2.4-1.el7.x86_64.rpmʢRedHat6ʣ el6 ʹͳΓ·͢ʢུʣ$ ruby -vruby 2.2.4p230 (2015-12-16 revision 53155) [x86_64-linux]4https://www.ruby-lang.org/ja/news/2015/12/16/ruby-2-2-4-released/11 of 47
fluentd ͷΠϯετʔϧ ∼ ύοέʔδʢ4ʣ■ඞཁύοέʔδͷΠϯετʔϧ1. epel ϨϙδτϦΛࢀর͢ΔΑ͏ʹ͓͍͍ͯͯͩ͘͠͞■ ҰߦͰ$ sudo yum install \http://ftp-srv2.kddilabs.jp/Linux/distributions/ \fedora/epel/7/x86 64/e/epel-release-7-5.noarch.rpm■ ʢRedHat6ʣҰߦͰ$ sudo yum install \http://ftp-srv2.kddilabs.jp/Linux/distributions/ \fedora/epel/6/x86 64/epel-release-6-8.noarch.rpm2. Πϯετʔϧ$ sudo yum install gecode gecode-devel fakeroot12 of 47
fluentd ͷΠϯετʔϧ ∼ ύοέʔδʢ5ʣ1. ʢRedHat6ʣgit ͷΞοϓσʔτ$ wget http://dl.marmotte.net/rpms/redhat/el6/x86 64/\git-1.8.3.1-3.el6/git-1.8.3.1-3.el6.src.rpm$ cp ~/rpmbuild/SRPMS/git-1.8.3.1-3.el6.src.rpm$ rpmbuild --rebuild \~/rpmbuild/SRPMS/git-1.8.3.1-3.el6.src.rpm$ sudo yum install perl-TermReadKey$ sudo rpm -ivh \~/rpmbuild/RPMS/x86 64/git-1.8.3.1-3.el6.x86_64.rpm□ git 1.8 ʹ͠ͳ͍ͱ “-c” Φϓγϣϯ͕ͳ͍ͨΊϏϧυͰ͖ͳ͍■ ެࣜʹ git 1.8 ଘࡏ͠ͳ͍■ ͍Ζ͍Ζ͋ΔதͰɺґଘύοέʔδ͕ epel Ͱऩ·ΔͷΛબ13 of 47
fluentd ͷΠϯετʔϧ ∼ ύοέʔδʢ6ʣ■ ruby ڥߏஙɺtd-agent ιʔεͷμϯϩʔυ1. bundle ΛΠϯετʔϧ$ sudo gem install bundler2. github ͔Β clone$ cd ~$ git clone \ʢ࣍ͷߦͱ͋Θͤͯʣ[email protected]:treasure-data/omnibus-td-agent.git$ cd ~/omnibus-td-agent3. treasure-data/omnibus-td-agent5 ͷ௨Γʹ࣮ߦ͠·͕͢ɺґଘੑͷղܾʹࣦഊ͢ΔͷͰ Gemfile ͷ్தʹҰߦՃ͠·͢ʢ࣍ϖʔδʣɻ5https://github.com/treasure-data/omnibus-td-agent14 of 47
fluentd ͷΠϯετʔϧ ∼ ύοέʔδʢ7ʣ■ιʔεΛमਖ਼͍͖ͯ͠·͢□ ґଘੑΛճආ□ ˜/omnibus-td-agent/Gemfile ʹ gem ’pedump’ · · · ͷҰߦΛՃ 6source ’https://rubygems.org’# Use Berkshelf for resolving cookbook dependenciesgem ’berkshelf’, ’~> 3.0’gem ’pedump’, git: ’https://github.com/ksubrama/pedump’,branch: ’patch-1’ # ্͔ΒҰߦͰՃ# Install omnibus software#gem ’omnibus’, ’~> 5.0’ʢҎԼུʣ6https://github.com/piwikjapan/omnibus-td-agent/blob/master/Gemfile15 of 47
fluentd ͷΠϯετʔϧ ∼ ύοέʔδʢ8ʣ■ elasticsearch, record-reformer, norikra ϓϥάΠϯΛ RPM ʹΈࠐΉ□ ࠓ norikra ʹ͍ͭͯݴٴ͠·ͤΜ■ ˜/omnibus-td-agent/plugin gems.rb ͷ࠷ޙʹࡾߦՃdownload "fluent-plugin-norikra", "0.2.2"download "fluent-plugin-elasticsearch", "1.3.0"download "fluent-plugin-record-reformer", "0.8.0"16 of 47
fluentd ͷΠϯετʔϧ ∼ ύοέʔδʢ9ʣ■ norikra ϓϥάΠϯʹؔ࿈͢ΔϓϥάΠϯΛՃ□ ࠓ norikra ʹ͍ͭͯݴٴ͠·ͤΜ□ norikra-client ͕Ҿ͖ࠐΉ msgpack-rpc-over-http ͕ rack Λཁٻ͢Δ͕ɺ2.x ࠷৽൛ΛಡΈࠐΜͰΤϥʔ͕ग़ΔͷͰͻͱͭલͷ 1.6.4 Λڧ੍తʹ͏■ ˜/omnibus-td-agent/core gems.rb ͷ࠷ޙʹೋߦՃdownload "rack", "1.6.4"download "norikra-client", "1.3.1"17 of 47
fluentd ͷΠϯετʔϧ ∼ ύοέʔδʢ10ʣ■ϫʔΫσΟϨΫτϦΛ࡞Γ·͢ 7ɻ$ sudo mkdir -p /opt/td-agent /var/cache/omnibus$ sudo chown yamachan:yamachan /opt/td-agent$ sudo chown yamachan:yamachan/var/cache/omnibus□ yamachan:yamachan ࣗͷ id ͱάϧʔϓʹஔ͖͍͑ͯͩ͘͞7https://github.com/treasure-data/omnibus-td-agent18 of 47
fluentd ͷΠϯετʔϧ ∼ ύοέʔδʢ11:ऴʣ1. खॱ௨Γʹ 8$ cd ~/omnibus-td-agent$ bundle install --binstubsʢུɺ్தͰ sudo ͢ΔͷͰύεϫʔυΛೖྗʣ$ bin/gem_downloader core_gems.rbʢུʣ$ bin/gem_downloader plugin_gems.rbʢུʣ$ bin/omnibus build td-agent2ʢུʣ8https://github.com/treasure-data/omnibus-td-agent19 of 47
fluentd ͷΠϯετʔϧ ∼ Πϯετʔϧ1. ग़དྷ্͕ͬͨύοέʔδ pkg ҎԼʹೖΓ·͢$ cd ~/omnibus-td-agent/pkg$ sudo yum install td-agent-2.3.1-0.el7.x86 64.rpm2. ʢRedHat6ʣtd-agent-2.3.1-0.el6.x86 64.rpm20 of 47
elasticsearch ͷΠϯετʔϧ1. RedHat7, RedHat6 ڞ௨Ͱ͢ɻҰߦͰ$ sudo yum install \https://download.elasticsearch.org/elasticsearch/\release/org/elasticsearch/distribution/\rpm/elasticsearch/2.2.0/elasticsearch-2.2.0.rpm2. ຊޠߏจղੳ kuromoji ϓϥάΠϯΛೖΕ·͢ɻҰߦͰ$ sudo /usr/share/elasticsearch/bin/plugin \install analysis-kuromoji21 of 47
kibana ͷΠϯετʔϧ1. ύοέʔδ͕ͳ͍ͷͰ࡞Γ·͢$ cd ~$ git clone [email protected]:piwikjapan/kibana-rpm-packaging.git$ cd kibana-rpm-packaging$ cp kibana.sysconfig kibana.service ~/rpmbuild/SOURCES$ cp kibana.spec ~/rpmbuild/SPECS$ wget -P ~/rpmbuild/SOURCES \https://download.elastic.co/kibana/kibana/\kibana-4.4.1-linux-x64.tar.gz$ rpmbuild -ba ~/rpmbuild/SPECS/kibana.spec2. Πϯετʔϧ͠·͢$ sudo rpm -ivh ~rpmbuild/RPMS/x86_64/\kibana-4.4.1-1.x86_64.rpm22 of 47
ʢRedHat6ʣkibana ͷΠϯετʔϧ■͜ͷΜ “kibana4 ηοτΞοϓ 9” ΛΈ͍ͯͩ͘͞ɻ■ىಈεΫϦϓτܝࡌ͞Ε͍ͯ·͢ɻ■ؾ͕͍ͨΒύοέʔδεΫϦϓτ࡞ͬͯΈ·͢ɻ9http://qiita.com/nagomu1985/items/82e699dde4f99b2ce41723 of 47
ϑΝΠΞʔΥʔϧͷઃఆ1. norikraʢ26578/tcpʣʹ͍ͭͯࠓճݴٴ͠·ͤΜ$ sudo firewall-cmd --zone=public \--add-port=26578/tcp --permanent # norikra web$ sudo firewall-cmd --zone=public \--add-port=5651/tcp --permanent # kibana web$ sudo firewall-cmd --zone=public \--add-port=24224/udp --permanent # fluentd heatbeat$ sudo firewall-cmd --zone=public \--add-port=24224/tcp --permanent # fluentd data2. ө͠·͢$ sudo firewall-cmd --reload24 of 47
ʢRedHat6ʣϑΝΠΞʔΥʔϧͷઃఆ1. norikraʢ26578/tcpʣʹ͍ͭͯࠓճݴٴ͠·ͤΜ2. /etc/sysconfig/iptables ͷ-A INPUT -m state –state ESTABLISHED,RELATED -j ACCEPTͷԼʹՃ͍ͯͩ͘͠͞-A INPUT -m multiport -p tcp -m tcp \ʢҰߦͰʣ--dports 26578,5651,24224 -j ACCEPT-A INPUT -m multiport -p udp -m udp --dports 24224 -j ACCEPT3. ө͠·͢$ sudo service iptables reload25 of 47
td-agent ͷઃఆ■ Piwik ͕ elasticsearch, kibana ͱ1. ผډʢPiwik server ͱ elasticsearch server ͕ผʣ2. ಉډʢPiwik server ͱ elasticsearch server ͕ಉ͡ɺforward ͕ͳ͍ʣtd-agent送信td-agent受信データ整形Store可視化Apacheアクセスログtailno-sqlデータベースPiwik Tracker(JavaScript)AdministratorPiwik server elasticsearch serverforward（2 台に分ける場合）26 of 47
td-agent ͷઃఆ ∼ Piwik ͱผډʢ1ʣ■ Piwik ͱ elasticsearch ͕ผαʔόʔ□ ྆ํʹ td-agent ΛೖΕͯϙʔτΛۭ͚͓͖ͯ·͢ɻ□ ରઃఆϑΝΠϧ/etc/td-agent/td-agent.conf■ ࣍ϖʔδҎ߱ͷྫΛॱ൪ʹͭͳ͛ͯͻͱͭʹ͍ͯͩ͘͠͞■ ը໘ʹೖΓ͖Βͳ͍ͱ͖ંΓ·͢ɻ□ શ൛ “Piwik ͷτϥοΩϯάσʔλΛ elasticsearch Ͱऩूͯ͠ΈΔ 10” Λࢀর͍ͯͩ͘͠͞ɻ10https://osdn.jp/projects/piwik-fluentd/wiki/FrontPage27 of 47
td-agent ͷઃఆ ∼ Piwik ͱผډʢ2ʣ■ Piwik αʔόʔ□ Piwik ͷΞΫηεϩάΛ fluentd Ͱٵ্͍͛·͢□ tag piwiktracker.apache.access Ͱ࣍ͷॲཧΛҾ͖ܧ͗type tailformat apachetime_format %d/%b/%Y:%H:%M:%S %zpos_file /var/log/td-agent/access_log.pospath /var/log/httpd/access_logtag piwiktracker.apache.access28 of 47
td-agent ͷઃఆ ∼ Piwik ͱผډʢ3ʣ■ Piwik αʔόʔ□ host Ͱࢦఆ͢ΔαʔόʔʹϩάΛసૹ͠·͢ɻtype forwardsend_timeout 60srecover_wait 300sheartbeat_interval 1sphi_threshold 16hard_timeout 60sname fruentdhost your_elsticsearch_server i.e. 10.x.x.xport 24224weight 10029 of 47
td-agent ͷઃఆ ∼ Piwik ͱผډʢ4ʣ■ elasticsearch αʔόʔ□ Tracker ͷϩάͷΈΛநग़͠·͢1. Piwik ͷཧը໘ʹΞΫηεͨ͠ͱ͖ͷϩά2. Piwik ͷ API Λ͍ͨͨͨͱ͖ͷϩά3. filter ͷ࣍ʹ match piwiktracker.apache.access ʹભҠ͠·͢type grepregexp1 path /piwik\.php\?action name=.*\&idsite=\d+type record_reformertag piwiktracker.apache.access.urldecodeʢུɺ࣍ϖʔδͰʣ30 of 47
td-agent ͷઃఆ ∼ Piwik ͱผډʢ5ʣ■ elasticsearch αʔόʔ□ ύʔεจࣈྻΛ fluentd ͷมʹղɻͦΕͧΕͷҙຯ“Supported Query Parameters11” Λࢀর□ ઌ಄ʹΞϯμʔείΞ “ ” ېࢭɻ·ͨม “id” ෆՄɻ□ ղͨ͠Β piwiktracker.apache.access.urldecode type record_reformertag piwiktracker.apache.access.urldecodeʢҎԼ 29 ݸத 3 ͚ͭͩʣidsite ${path[/piwik\.php\?action name=.*\&idsite=(\d+)/,1]} ˡ αΠτ IDpiwikid ${path[/piwik\.php\?action name=.*\& id=([a-z\d]+)/,1]} ˡ ϢχʔΫ IDfla ${path[/piwik\.php\?action name= ˡ flash ΞυΦϯ?.*\&fla=(\d+)/,1] == "1" ? "true" : "false" }11http://developer.piwik.org/api-reference/tracking-api31 of 47
td-agent ͷઃఆ ∼ Piwik ͱผډʢ6ʣ■ elasticsearch αʔόʔ□ fluentd ͷมதʹ url encode ͕͔͔͍ͬͯΔͷͰಡΊΔΑ͏ʹ͢Δ□ ऴΘͬͨΒ piwiktracker.apache.access.store ʹॲཧΛҠ͠·͢type uri_decodetag piwiktracker.apache.access.storekey_names action_name,ref,url,urlref32 of 47
td-agent ͷઃఆ ∼ Piwik ͱผډʢ7:ऴʣ■ elasticsearch αʔόʔ□ store Λෳͬͯ elasticsearch Ҏ֎ʹσʔλసૹՄೳtype copytype elasticsearchtype_name access_loghost 127.0.0.1port 9200logstash_format truelogstash_prefix apache-loglogstash_dateformat %Y%m%dinclude_tag_key truetag_key @log_nameflush_interval 10s33 of 47
td-agent ͷઃఆ ∼ Piwik ͱผډʢ1ʣ■ Piwik ͱ elasticsearch ͕ಉ͡αʔόʔ□ td-agent ΛೖΕͯϙʔτΛۭ͚͓͖ͯ·͢ɻ□ ରઃఆϑΝΠϧ /etc/td-agent/td-agent.conf■ جຊ “ผډ” ࣌ͷೋͷαʔόʔͷઃఆΛͻͱͭʹ·ͱΊΔ͚ͩͰ͢■ λά͚ͩࣔ͠·͢ɻ□ શ൛ “Piwik ͷτϥοΩϯάσʔλΛ elasticsearch Ͱऩूͯ͠ΈΔ 12” Λࢀর͍ͯͩ͘͠͞ɻ12https://osdn.jp/projects/piwik-fluentd/wiki/FrontPage34 of 47
td-agent ͷઃఆ ∼ Piwik ͱผډʢ2:ऴʣ■ Piwik ͱ elasticsearch ͕ಉ͡αʔόʔ□ λά͚ͩࣔ͠·͢ɻத “ผډ” ࣌ͱಉ͡Ͱ͢ɻ■ ͨͩ͠ɺ “ผډ” ࣌ͷ Piwik αʔόʔ forward ͕͋Γ·ͤΜɻtag piwiktracker.apache.accesstag piwiktracker.apache.access.urldecodetag piwiktracker.apache.access.store35 of 47
elasticsearch ͷϑΟʔϧυܕʢ1ʣ■͜͜Ͱ fluentd ͱ elasticsearch Λ্ཱͪ͛Εࣗಈతʹelasticsearch ଆʹλΠϓʢςʔϒϧʣ͕࡞͞Εɺσʔλ͕֨ೲ͞ΕΔΘ͚Ͱ͕͢ɺ■λΠϓதͷϑΟʔϧυʢΧϥϜʣͯ͢ string ʹͳΓ·͢ɻ■ͦ͜Ͱɺ֤λΠϓͷܕΛఆٛ͠·͢ɻ36 of 47
elasticsearch ͷϑΟʔϧυܕʢ2ʣ∼ छྨ■ Elasticsearch supports the following simple field datatypes13:□ String: string□ Whole number: byte, short, integer, long□ Floating-point: float, double□ Boolean: boolean□ Date: date13https://www.elastic.co/guide/en/elasticsearch/guide/current/mapping-intro.html37 of 47
elasticsearch ͷϑΟʔϧυܕʢ3ʣ∼ ఆٛ■ Json ͰϚοϐϯάఆٛϑΝΠϧΛ࡞ΓɺಛఆͷΠϯσοΫε 14໊ͱλΠϓ໊ͰϑΟʔϧυͷܕΛࢦఆ͠·͢ 15ɻ■ཁૉͷΈઆ໌͠·͢ɻશ෦ʹ͍ͭͯ “elasticsearch ͷ mappingઃఆ 16” ʹ͓͍͓͖ͯ·͢ɻ14MySQL ͷϚϧνςʔϒϧΈ͍ͨͳͷɻ elasticsearch ͻͱͭͷςʔϒϧΛผͳͲͷϧʔϧʹΑΓׂ֨ೲͰ͖Δ15σϑΥϧτͷৼΔ͍ఆٛͰ͖·͢16https://osdn.jp/projects/piwik-fluentd/wiki/elasticsearch#h2-elasticsearch.20.E3.81.AE.20mapping.20.E8.A8.AD.E5.AE.9A38 of 47
elasticsearch ͷϑΟʔϧυܕʢ4ʣ∼ Json ৄࡉ■ ”template”: ”apache-log-*”,ͲͷΠϯσοΫε 17 ͷ mapping ͳͷ͔ td-agent.conf ͷlogstash prefix apache-log ͱҰகͤ͞ΔɻΞελϦεΫ͕͍͍ͭͯΔͷɺlogstash dateformat%Y%m%d ʹΑΓ “apache-log-” ͱͳࣉא෩ΓΠϯσοΫεͰׂ͞Εɺ͜ΕΒΛશ෦र͏ͨΊɻ■ ”settings”: {Ұ෦ʹຊޠ͕ఆ͞ΕΔจࣈྻͷݕࡧ index Λ࡞͢Δࡍʹkuromoji Λಋೖ͢ΔͨΊͷఆٛɻ“Elasticsearch ͱ kuromoji ͰͪΌΜͱͨ͠ຊޠશจݕࡧΛΔϝϞ 18” ͷͱ͓ΓʹͬͯΈ·ͨ͠ɻ17ϦϨʔγϣφϧ DB ͷςʔϒϧͷΑ͏ͳͷ18http://tech.gmo-media.jp/post/70245090007/elasticsearch-kuromoji-japanese-fulltext-search39 of 47
elasticsearch ͷϑΟʔϧυܕʢ5ʣ∼ Json ৄࡉ■ ”mappings”: { ”access log”: {”access log” ɺλΠϓͷ໊લͰ td-agent.conf ͷ type nameaccess log ͱҰகͤ͞·͢ 19ɻҎԼλΠϓͷܕΛͦΕͧΕఆ͍͖ٛͯ͠·͢ɻ19“ default ” ͯ͢ͷλΠϓʹҰக͠·͢40 of 47
elasticsearch ͷϑΟʔϧυܕʢ6ʣ∼ Json ৄࡉ■λΠϓͷϑΟʔϧυܕఆٛʢσϑΥϧτఆٛϑΟʔϧυʣ□ source ͱ all Λແޮʹͯ͠ΠϯσοΫεͷ༰ྔΛݮΒ͠·͢ɻ"mappings": {"access log": { ˡ λΠϓͷ໊લ" source": { ˡ σϑΥϧτͰશ෦ͷϑΟʔϧυͷ͕ೖΔ"enabled": "false" ˡ ඞཁͳ͠ɺσϑΥϧτ true}," all": { ˡ ͜ΕσϑΥϧτͰશ෦ͷϑΟʔϧυͷ͕ೖΔ"enabled": "false" ˡ ඞཁͳ͠ɺσϑΥϧτ true},41 of 47
elasticsearch ͷϑΟʔϧυܕʢ7ʣ∼ Json ৄࡉ■λΠϓͷϑΟʔϧυܕఆٛ□ ࣍ʹ֤ϑΟʔϧυͷఆٛͰ͢"mappings": {"access log": {ʢུɺҰຕલͰઆ໌ʣ"properties": {"@log name": { ˠ ϑΟʔϧυ໊ʢsee td-agent.confʣ"type": "string", ˠ จࣈྻͰ͢"store": "true", ˠ ੜσʔλΛه͠·͢"index": "not analyzed" ˠ ߏจղੳແΠϯσοΫε},■ See “Mapping parameters20”.20https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-params.html42 of 47
elasticsearch ͷϑΟʔϧυܕʢ8ʣ∼ Json ৄࡉ■λΠϓͷϑΟʔϧυܕఆٛ□ ʢଓ͖ɺൈਮʣ֤ϑΟʔϧυͷఆٛ"ref": { ˡ ϑΟʔϧυ໊ʢtd-agent.conf Ͱఆٛʣ"type": "multi field","fields": { ˡ ྆ํͷΠϯσοΫεΛ࡞Δ"ref": {"type": "string","index": "analyzed", ˡ Ԥจߏจղੳ"store": "true"},"full": {"type": "string","index": "not analyzed", ˡ ߏจղੳແ"store": "true"}}},43 of 47
elasticsearch ͷϑΟʔϧυܕʢ9:ऴʣ∼ Json ৄࡉ■λΠϓͷϑΟʔϧυܕఆٛ□ ʢଓ͖ɺൈਮʣ֤ϑΟʔϧυͷఆٛ"action_name": {"type": "string","analyzer": "kuromoji analyzer", ˡ ຊޠߏจղੳ"store": "true"},44 of 47
elasticsearch ͷςϯϓϨʔτొ1. ˜/piwik-template.json Λ “elasticsearch ͷ mapping ઃఆ 21” ΑΓίϐϖͯ͠࡞͠·͢ɻ2. elasticsearch Λ্ཱͪ͛·͢ɻ$ sudo service elasticsearch start3. ςϯϓϨʔτ໊ piwik-tracker ͱͯ͠ಥͬࠐΈ·͢ʢҰߦͰʣɻ$ curl -XPUT localhost:9200/_template/piwik-tracker \-d "‘cat ~/piwik-template.json‘"21https://osdn.jp/projects/piwik-fluentd/wiki/elasticsearch#h2-elasticsearch.20.E3.81.AE.20mapping.20.E8.A8.AD.E5.AE.9A45 of 47
্ཱͪ͛·͢■ผډͷͱ͖ td-agent Λ྆αʔόʔڞ্ཱͪ͛·͢ɻ$ sudo service td-agent start$ sudo service kibana start■ kibana ͷը໘ http://your elasticserach server:5601/46 of 47
͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠47 of 47
yamachan
prathamesh
th0x0472
imsaku
tomzoh
mongolyy
yanzm
nktamago
yoshiitaka
gaelvaroquaux
murahigeas
yellowshippo
bumptakayuki
hatefulcrawdad
sonatard
addyosmani
chriscoyier
paulrobertlloyd
morganepeng
davidbonilla
phodgson
jmmastey
lauravandoore
cassininazir