Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Getting Started with Elastic Cloud and Beats for Log Analytics

34cbde72de5f384380d5489543294dc5?s=47 Kosho Owa
October 07, 2016

Getting Started with Elastic Cloud and Beats for Log Analytics

情報セキュリティワークショップ in 越後湯沢 2016

34cbde72de5f384380d5489543294dc5?s=128

Kosho Owa

October 07, 2016
Tweet

Transcript

  1. ‹#› Kosho Owa, Solutions Architect, Elastic October 2016 Elastic CloudͱBeatsͰ࢝ΊΔ

    ϩάͷՄࢹԽͱ෼ੳ
  2. 2 Elastic Cloud Security X-Pack Kibana User Interface Elasticsearch Store,

    Index,
 & Analyze Ingest Logstash Beats + Elastic Stack Introducing the Elastic Stack, X-Pack, and Cloud Alerting Monitoring Reporting Graph
  3. Elasticserach: σʔλετΞɺΠϯσοΫεɺ෼ੳ 3 ෼ࢄܕͰ
 εέʔϥϒϧ ճ෮ੑ͕͋ΓߴՄ༻ੑɺεέʔϧΞ΢τΛલఏ ͱͨ͠੡඼σβΠϯ ߏ଄ɺඇߏ଄σʔλΛΠϯσοΫε ։ൃऀ
 ϑϨϯυϦʔ

    εΩʔϚϨε Ϛϧνςφϯτ ๛෋ͳΫϥΠΞϯτϥΠϒϥϦ ݕࡧͱ෼ੳ ϦΞϧλΠϜ શจݕࡧ (FP "HHSFHBUJPO ଟݴޠʹରԠ
  4. Kibana: ՄࢹԽͱ୳ࡧ 4 ൃݟͱಎ࡯ σʔλΛ୳ࡧɺنଇੑΛൃݟͲͷΑ͏ͳϨϕϧ ΁΋υϦϧμ΢ϯ &MBTUJDTFBSDIͷύϫϑϧͳ෼ੳػೳΛར༻ ߏ଄ɺඇߏ଄σʔλ ΧελϚΠζ ͦͯ͠ڞ༗

    όʔνϟʔτɺંΕઢάϥϑɺ෼෍ਤɺ஍ਤɺ ώετάϥϜ μογϡϘʔυΛΛγΣΞ͠ɺӡ༻ϫʔΫϑϩ ʔʹ૊ΈࠐΈ Elastic Stack ΁ͷೖΓޱ ՄࢹԽͷͨΊͷ౷Ұతͳ6* &MBTUJD4UBDLͷӡ༻؅ཧ ϓϥάΠϯՄೳͳΞʔΩςΫνϟͰɺಠࣗͷΞ ϓϦέʔγϣϯ͕࡞੒Մೳ
  5. Beats: ElasticsearchͷͨΊͷσʔλγούʔ 5 Filebeat ϩάऩूΤʔδΣϯτͷ࣍ੈ୅ελϯμʔυ Winlogbeat 8JOEPXTγεςϜɺΞϓϦέʔγϣϯɺηΩ ϡϦςΟϩά Metricbeat "QBDIF

    .POHP%# .Z42- /HJOY 3FEJT  ;PPLFFQFSɺ04ͳͲͷϝτϦοΫ Packetbeat )551 .Z42- $BTTBOESB %/4ͳͲͷωοτ ϫʔΫύέοτΛϦΞϧλΠϜͰղੳ Libbeat ΧελϜzCFBUTz։ൃ༻ϥΠϒϥϦʔ
  6. ੒ޭ͢Δϩά෼ੳ σʔλऩू BeatsͰσʔλऩूͱElasticsearch΁ͷ౤ೖ μογϡϘʔυͷςϯϓϨʔτΛಉࠝ JSONߏ଄ԽϩάΛFilebeatͰऩूɺΠϯσοΫε ΠϯετʔϧͱηοτΞοϓ Elastic CloudͰΫϥελʔΛ਺ΫϦοΫͰల։ ৗʹ࠷৽൛ɺΞοϓάϨʔυ΋؆୯ʹ ӡ༻

    X-PackΛ׆༻ͯ͠ɺಛఆͷΠϕϯτʹରͯ͠Ξϥʔτ σʔλͷΞΫηε੍ޚ ElasticsearchΫϥελʔࣗ਎΋ϞχλϦϯά 6
  7. 7 Performance Metrics Application Logs Filebeat ϩάऩू Packetbeat ύέοτ؂ࢹ Elasticsearch

    σʔλετΞ ݕࡧΤϯδϯ Kibana ՄࢹԽ Network Interfaces Metricbeat ϝτϦοΫऩू
  8. JSONߏ଄ԽϩΪϯά - Apache 8 LogFormat "{ \"clientip\": \"%h\", \"ident\": \"%l\",

    \"auth\": \"%u\", \"timestamp\": \"%{%FT%T%z}t\", \"verb\": \"%m\", \"request\": \"%U%q\", \"httpversion\": \"%H\", \"response\": %>s, \"bytes\": %b, \"referer\": \"% {Referer}i\", \"agent\": \"%{User-agent}i\" }" combinedjson CustomLog logs/access_log.js combinedjson - input_type: log paths: - /var/log/httpd/access_log.js document_type: apache json.keys_under_root: true json.add_error_key: true httpd.conf filebeat.yml
  9. JSONߏ଄ԽϩΪϯά - Squid 9 logformat combinedjson { "clientip": "%>a", "ident":

    "%ui", "uname": "%un", "timestamp": "%{%FT%T%z}tg", "verb": "%rm", "request": "%ru", "httpversion": "HTTP/%rv", "response": %>Hs, "bytes": %<st, "referer": "%{Referer}>h", "agent": "%{User-Agent}>h", "request_status": "%Sh", "hierarchy_status": "%Sh" } access_log /var/log/squid/access_log.js combinedjson - input_type: log paths: - /var/log/squid/access_log.js document_type: squid json.keys_under_root: true json.add_error_key: true squid.conf filebeat.yml
  10. Metricbeat - OSɺΞϓϦέʔγϣϯͷϝτϦοΫऩू 10 ϦΞϧλΠϜϞχλϦϯά • OS΍αʔϏεͷϝτϦοΫΛϞχλʔ αʔϏεͷύϑΥʔϚϯε෼ੳ • System:

    CPU, load, IO, filesystem, memory, network, process • Apache, HAProxy, MongoDB, MySQL, Nginx, Redis, ZookeeperʹରԠ ElasticsearchʹసૹɺKibanaͰՄࢹԽ • αϯϓϧDashboardͰ͙͢ʹར༻Մೳ
  11. Packetbeat - ωοτϫʔΫύέοοτͷղੳͱऩू 11 ϦΞϧλΠϜϞχλϦϯά • ΞϓϦέʔγϣϯͷ஗ԆɺΤϥʔɺԠ ౴࣌ؒͳͲΛϞχλʔ ωοτϫʔΫτϥϑΟοΫͷݕࡧͱ෼ੳ •

    ICMP, DNS, HTTP, AMQP, Cassandra, MySQL, PostgreSQL, Redis, Thrift- RPC, MongoDB, MemcacheʹରԠ ElasticsearchʹసૹɺKibanaͰՄࢹԽ • αϯϓϧDashboardͰ͙͢ʹར༻Մೳ
  12. Hosted Elasticsearch & Kibana on AWS • Elasticͷ੡඼܈ͱಉظͨ͠࠷৽൛ͷఏڙ • εέʔϧΞ΢τɾΞοϓάϨʔυΛΫϦοΫૢ࡞

    Ͱ • ແྉͷKibanaΠϯελϯεͱ30෼͝ͱͷόοΫΞ οϓ • X-Packػೳ (Security, Alerting, Monitoring, Reporting) • ݄ʑ45USD͔Β • SLAϕʔεͷαϙʔτΦϓγϣϯ 12 Elastic Stackͷ։ൃऀʹΑΔ།Ұͷ ެࣜ Elasticsearch as a Service
  13. X-Pack: Elastic Stackͷ෇ՃՁ஋ػೳ 13 \ ηΩϡϦςΟ෼ੳ ϩά෼ੳ ϝτϦοΫε ෼ੳ ӡ༻෼ੳ

    υΩϡϝϯτݕࡧ ΞϓϦέʔγϣϯ ݕࡧ ϩοΫμ΢ϯͱ ΞΫηε؂ࢹ σʔλͷมߋʹ
 ର͢Δ௨஌ Elasticsearch
 Ϋϥελͷ؂ࢹ σʔλ͔Βҙຯͷ
 ͋Δؔ܎Λൃݟ PDFΛ࡞੒ͯ͠ ൃݟΛγΣΞ Security Alerting Monitoring Graph Analytics Reporting
  14. X-Pack: Security - ҉߸ԽͱϩʔϧϕʔεͷΞΫηε੍ޚ 14 ҉߸Խ • KibanaɺElasticsearchͷΤϯυϙΠ ϯτ΁ͷHTTPS௨৴ •

    Ϋϥελʔ಺ͷ௨৴ ΞΫηε੍ޚ • ID/PasswordʹΑΔϢʔβೝূ • ωΠςΟϒɺLDAPɺAD࿈ܞ • KibanaͷϩάΠϯμΠΞϩά • ϩʔϧ͝ͱʹΠϯσοΫεɺAPI΁ͷ ΞΫηεΛ੍ݶ
  15. X-Pack: Alerting - σʔλͷมԽΛ௨஌ 15 εέδϡʔϧ • ಛఆͷ࣌ؒɺΠϯλʔόϧɺ Crontabॻࣜ ίϯσΟγϣϯ

    • Elasticsearchͷ͢΂ͯͷΫΤϦʔͱ ΞάϦήʔγϣϯΛαϙʔτ • ෳ਺ͷιʔεΛ૊Έ߹Θͤ ΞΫγϣϯ • ΠϯσοΫεɺϩάɺϝʔϧɺ΢Σ ϒϑοΫͳͲ
  16. Monitoring - ΫϥελʔɺϊʔυɺΠϯσοΫεͷ؂ࢹ • ElasticsearchΫϥελʔɺϊʔυɺ ΠϯσοΫεͷϝτϦοΫΛϦΞϧ λΠϜͰ؂ࢹ • ӡ༻্ͷ܏޲Λ೺Ѳɺ໰୊Λൃݟ •

    ΫϥελʔɺΞϓϦέʔγϣϯͷ࠷ దԽ • ΩϟύγςΟϓϥχϯά 16
  17. X-Pack: Graph - σʔλؒͷؔ܎ΛՄࢹԽ 17 • Elasticsearchͷsearch΍relevancyͷػ ೳΛ࢖༻ͯ͠ҙຯͷ͋Δؔ܎Λൃݟ • طଘͷΠϯσοΫεΛར༻

    • ϦΞϧλΠϜ͔ͭεέʔϥϒϧ
  18. X-Pack: Reporting - DashboardΛΤΫεϙʔτ 18 Earthquake - Depth Timeseries Earthquake

    - Heatmap Earthquake — Sun, Jan 1, 2006 12:00 AM to Fri, Sep 2, 2016 5:54 AM • PDF΋͘͠͸CSVΛੜ੒ • ඇKibanaϢʔβͱڞ༗ • खಈɺ΋͘͠͸Alertingͱͷ૊Έ߹Θ ͤͰεέδϡʔϧɺ΋͘͠͸ಛఆͷΠ ϕϯτ͕ൃੜͨ͠৔߹ʹ࡞੒ N ew in V5
  19. elastic.co/jp: ೔ຊޠ৘ใ΋͝ར༻Լ͍͞ 19 • ੡඼৘ใ • αϒεΫϦϓγϣϯ • ಋೖࣄྫ •

    ύʔτφʔ • ϋϯζΦϯϫʔΫγϣοϓ • ϒϩά • νϡʔτϦΞϧϏσΦ • ͓໰͍߹Θͤ