Getting Started with Elastic Cloud and Beats for Log Analytics

Transcript

1. ‹#› Kosho Owa, Solutions Architect, Elastic October 2016 Elastic CloudͱBeatsͰ࢝ΊΔ

   ϩάͷՄࢹԽͱ෼ੳ

2. 2 Elastic Cloud Security X-Pack Kibana User Interface Elasticsearch Store,

   Index,
 & Analyze Ingest Logstash Beats + Elastic Stack Introducing the Elastic Stack, X-Pack, and Cloud Alerting Monitoring Reporting Graph

3. Elasticserach: σʔλετΞɺΠϯσοΫεɺ෼ੳ 3 ෼ࢄܕͰ
 εέʔϥϒϧ ճ෮ੑ͕͋ΓߴՄ༻ੑɺεέʔϧΞ΢τΛલఏ ͱͨ͠੡඼σβΠϯ
   ߏ଄ɺඇߏ଄σʔλΛΠϯσοΫε ։ൃऀ
 ϑϨϯυϦʔ

   εΩʔϚϨε
   Ϛϧνςφϯτ
   ๛෋ͳΫϥΠΞϯτϥΠϒϥϦ ݕࡧͱ෼ੳ ϦΞϧλΠϜ
   શจݕࡧ
   (FP
   "HHSFHBUJPO
   ଟݴޠʹରԠ

4. Kibana: ՄࢹԽͱ୳ࡧ 4 ൃݟͱಎ࡯ σʔλΛ୳ࡧɺنଇੑΛൃݟ
   ͲͷΑ͏ͳϨϕϧ ΁΋υϦϧμ΢ϯ
   &MBTUJDTFBSDIͷύϫϑϧͳ෼ੳػೳΛར༻
   ߏ଄ɺඇߏ଄σʔλ ΧελϚΠζ ͦͯ͠ڞ༗

   όʔνϟʔτɺંΕઢάϥϑɺ෼෍ਤɺ஍ਤɺ ώετάϥϜ
   μογϡϘʔυΛΛγΣΞ͠ɺӡ༻ϫʔΫϑϩ ʔʹ૊ΈࠐΈ Elastic Stack ΁ͷೖΓޱ ՄࢹԽͷͨΊͷ౷Ұతͳ6*
   &MBTUJD
   4UBDL
   ͷӡ༻؅ཧ
   ϓϥάΠϯՄೳͳΞʔΩςΫνϟͰɺಠࣗͷΞ ϓϦέʔγϣϯ͕࡞੒Մೳ

5. Beats: ElasticsearchͷͨΊͷσʔλγούʔ 5 Filebeat ϩάऩूΤʔδΣϯτͷ࣍ੈ୅ελϯμʔυ Winlogbeat 8JOEPXT
   γεςϜɺΞϓϦέʔγϣϯɺηΩ ϡϦςΟϩά Metricbeat "QBDIF

   
   .POHP%#
   .Z42-
   /HJOY
   3FEJT
   ;PPLFFQFSɺ04ͳͲͷϝτϦοΫ Packetbeat )551
   .Z42-
   $BTTBOESB
   %/4ͳͲͷωοτ ϫʔΫύέοτΛϦΞϧλΠϜͰղੳ Libbeat ΧελϜzCFBUTz։ൃ༻ϥΠϒϥϦʔ

6. ੒ޭ͢Δϩά෼ੳ σʔλऩू BeatsͰσʔλऩूͱElasticsearch΁ͷ౤ೖ μογϡϘʔυͷςϯϓϨʔτΛಉࠝ JSONߏ଄ԽϩάΛFilebeatͰऩूɺΠϯσοΫε ΠϯετʔϧͱηοτΞοϓ Elastic CloudͰΫϥελʔΛ਺ΫϦοΫͰల։ ৗʹ࠷৽൛ɺΞοϓάϨʔυ΋؆୯ʹ ӡ༻

   X-PackΛ׆༻ͯ͠ɺಛఆͷΠϕϯτʹରͯ͠Ξϥʔτ σʔλͷΞΫηε੍ޚ ElasticsearchΫϥελʔࣗ਎΋ϞχλϦϯά 6

7. 7 Performance Metrics Application Logs Filebeat ϩάऩू Packetbeat ύέοτ؂ࢹ Elasticsearch

   σʔλετΞ ݕࡧΤϯδϯ Kibana ՄࢹԽ Network Interfaces Metricbeat ϝτϦοΫऩू

8. JSONߏ଄ԽϩΪϯά - Apache 8 LogFormat "{ \"clientip\": \"%h\", \"ident\": \"%l\",

   \"auth\": \"%u\", \"timestamp\": \"%{%FT%T%z}t\", \"verb\": \"%m\", \"request\": \"%U%q\", \"httpversion\": \"%H\", \"response\": %>s, \"bytes\": %b, \"referer\": \"% {Referer}i\", \"agent\": \"%{User-agent}i\" }" combinedjson CustomLog logs/access_log.js combinedjson - input_type: log paths: - /var/log/httpd/access_log.js document_type: apache json.keys_under_root: true json.add_error_key: true httpd.conf filebeat.yml

9. JSONߏ଄ԽϩΪϯά - Squid 9 logformat combinedjson { "clientip": "%>a", "ident":

   "%ui", "uname": "%un", "timestamp": "%{%FT%T%z}tg", "verb": "%rm", "request": "%ru", "httpversion": "HTTP/%rv", "response": %>Hs, "bytes": %<st, "referer": "%{Referer}>h", "agent": "%{User-Agent}>h", "request_status": "%Sh", "hierarchy_status": "%Sh" } access_log /var/log/squid/access_log.js combinedjson - input_type: log paths: - /var/log/squid/access_log.js document_type: squid json.keys_under_root: true json.add_error_key: true squid.conf filebeat.yml

10. Metricbeat - OSɺΞϓϦέʔγϣϯͷϝτϦοΫऩू 10 ϦΞϧλΠϜϞχλϦϯά • OS΍αʔϏεͷϝτϦοΫΛϞχλʔ αʔϏεͷύϑΥʔϚϯε෼ੳ • System:

    CPU, load, IO, filesystem, memory, network, process • Apache, HAProxy, MongoDB, MySQL, Nginx, Redis, ZookeeperʹରԠ ElasticsearchʹసૹɺKibanaͰՄࢹԽ • αϯϓϧDashboardͰ͙͢ʹར༻Մೳ

11. Packetbeat - ωοτϫʔΫύέοοτͷղੳͱऩू 11 ϦΞϧλΠϜϞχλϦϯά • ΞϓϦέʔγϣϯͷ஗ԆɺΤϥʔɺԠ ౴࣌ؒͳͲΛϞχλʔ ωοτϫʔΫτϥϑΟοΫͷݕࡧͱ෼ੳ •

    ICMP, DNS, HTTP, AMQP, Cassandra, MySQL, PostgreSQL, Redis, Thrift- RPC, MongoDB, MemcacheʹରԠ ElasticsearchʹసૹɺKibanaͰՄࢹԽ • αϯϓϧDashboardͰ͙͢ʹར༻Մೳ

12. Hosted Elasticsearch & Kibana on AWS • Elasticͷ੡඼܈ͱಉظͨ͠࠷৽൛ͷఏڙ • εέʔϧΞ΢τɾΞοϓάϨʔυΛΫϦοΫૢ࡞

    Ͱ • ແྉͷKibanaΠϯελϯεͱ30෼͝ͱͷόοΫΞ οϓ • X-Packػೳ (Security, Alerting, Monitoring, Reporting) • ݄ʑ45USD͔Β • SLAϕʔεͷαϙʔτΦϓγϣϯ 12 Elastic Stackͷ։ൃऀʹΑΔ།Ұͷ ެࣜ Elasticsearch as a Service

13. X-Pack: Elastic Stackͷ෇ՃՁ஋ػೳ 13 \ ηΩϡϦςΟ෼ੳ ϩά෼ੳ ϝτϦοΫε ෼ੳ ӡ༻෼ੳ

    υΩϡϝϯτݕࡧ ΞϓϦέʔγϣϯ ݕࡧ ϩοΫμ΢ϯͱ ΞΫηε؂ࢹ σʔλͷมߋʹ
 ର͢Δ௨஌ Elasticsearch
 Ϋϥελͷ؂ࢹ σʔλ͔Βҙຯͷ
 ͋Δؔ܎Λൃݟ PDFΛ࡞੒ͯ͠ ൃݟΛγΣΞ Security Alerting Monitoring Graph Analytics Reporting

14. X-Pack: Security - ҉߸ԽͱϩʔϧϕʔεͷΞΫηε੍ޚ 14 ҉߸Խ • KibanaɺElasticsearchͷΤϯυϙΠ ϯτ΁ͷHTTPS௨৴ •

    Ϋϥελʔ಺ͷ௨৴ ΞΫηε੍ޚ • ID/PasswordʹΑΔϢʔβೝূ • ωΠςΟϒɺLDAPɺAD࿈ܞ • KibanaͷϩάΠϯμΠΞϩά • ϩʔϧ͝ͱʹΠϯσοΫεɺAPI΁ͷ ΞΫηεΛ੍ݶ

15. X-Pack: Alerting - σʔλͷมԽΛ௨஌ 15 εέδϡʔϧ • ಛఆͷ࣌ؒɺΠϯλʔόϧɺ Crontabॻࣜ ίϯσΟγϣϯ

    • Elasticsearchͷ͢΂ͯͷΫΤϦʔͱ ΞάϦήʔγϣϯΛαϙʔτ • ෳ਺ͷιʔεΛ૊Έ߹Θͤ ΞΫγϣϯ • ΠϯσοΫεɺϩάɺϝʔϧɺ΢Σ ϒϑοΫͳͲ

16. Monitoring - ΫϥελʔɺϊʔυɺΠϯσοΫεͷ؂ࢹ • ElasticsearchΫϥελʔɺϊʔυɺ ΠϯσοΫεͷϝτϦοΫΛϦΞϧ λΠϜͰ؂ࢹ • ӡ༻্ͷ܏޲Λ೺Ѳɺ໰୊Λൃݟ •

    ΫϥελʔɺΞϓϦέʔγϣϯͷ࠷ దԽ • ΩϟύγςΟϓϥχϯά 16

17. X-Pack: Graph - σʔλؒͷؔ܎ΛՄࢹԽ 17 • Elasticsearchͷsearch΍relevancyͷػ ೳΛ࢖༻ͯ͠ҙຯͷ͋Δؔ܎Λൃݟ • طଘͷΠϯσοΫεΛར༻

    • ϦΞϧλΠϜ͔ͭεέʔϥϒϧ

18. X-Pack: Reporting - DashboardΛΤΫεϙʔτ 18 Earthquake - Depth Timeseries Earthquake

    - Heatmap Earthquake — Sun, Jan 1, 2006 12:00 AM to Fri, Sep 2, 2016 5:54 AM • PDF΋͘͠͸CSVΛੜ੒ • ඇKibanaϢʔβͱڞ༗ • खಈɺ΋͘͠͸Alertingͱͷ૊Έ߹Θ ͤͰεέδϡʔϧɺ΋͘͠͸ಛఆͷΠ ϕϯτ͕ൃੜͨ͠৔߹ʹ࡞੒ N ew in V5

19. elastic.co/jp: ೔ຊޠ৘ใ΋͝ར༻Լ͍͞ 19 • ੡඼৘ใ • αϒεΫϦϓγϣϯ • ಋೖࣄྫ •

    ύʔτφʔ • ϋϯζΦϯϫʔΫγϣοϓ • ϒϩά • νϡʔτϦΞϧϏσΦ • ͓໰͍߹Θͤ