Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Getting Started with Elastic Cloud and Beats for Log Analytics

Kosho Owa
October 07, 2016

Getting Started with Elastic Cloud and Beats for Log Analytics

情報セキュリティワークショップ in 越後湯沢 2016

Kosho Owa

October 07, 2016
Tweet

More Decks by Kosho Owa

Other Decks in Technology

Transcript

  1. 2 Elastic Cloud Security X-Pack Kibana User Interface Elasticsearch Store,

    Index,
 & Analyze Ingest Logstash Beats + Elastic Stack Introducing the Elastic Stack, X-Pack, and Cloud Alerting Monitoring Reporting Graph
  2. Kibana: ՄࢹԽͱ୳ࡧ 4 ൃݟͱಎ࡯ σʔλΛ୳ࡧɺنଇੑΛൃݟͲͷΑ͏ͳϨϕϧ ΁΋υϦϧμ΢ϯ &MBTUJDTFBSDIͷύϫϑϧͳ෼ੳػೳΛར༻ ߏ଄ɺඇߏ଄σʔλ ΧελϚΠζ ͦͯ͠ڞ༗

    όʔνϟʔτɺંΕઢάϥϑɺ෼෍ਤɺ஍ਤɺ ώετάϥϜ μογϡϘʔυΛΛγΣΞ͠ɺӡ༻ϫʔΫϑϩ ʔʹ૊ΈࠐΈ Elastic Stack ΁ͷೖΓޱ ՄࢹԽͷͨΊͷ౷Ұతͳ6* &MBTUJD4UBDLͷӡ༻؅ཧ ϓϥάΠϯՄೳͳΞʔΩςΫνϟͰɺಠࣗͷΞ ϓϦέʔγϣϯ͕࡞੒Մೳ
  3. Beats: ElasticsearchͷͨΊͷσʔλγούʔ 5 Filebeat ϩάऩूΤʔδΣϯτͷ࣍ੈ୅ελϯμʔυ Winlogbeat 8JOEPXTγεςϜɺΞϓϦέʔγϣϯɺηΩ ϡϦςΟϩά Metricbeat "QBDIF

    .POHP%# .Z42- /HJOY 3FEJT  ;PPLFFQFSɺ04ͳͲͷϝτϦοΫ Packetbeat )551 .Z42- $BTTBOESB %/4ͳͲͷωοτ ϫʔΫύέοτΛϦΞϧλΠϜͰղੳ Libbeat ΧελϜzCFBUTz։ൃ༻ϥΠϒϥϦʔ
  4. 7 Performance Metrics Application Logs Filebeat ϩάऩू Packetbeat ύέοτ؂ࢹ Elasticsearch

    σʔλετΞ ݕࡧΤϯδϯ Kibana ՄࢹԽ Network Interfaces Metricbeat ϝτϦοΫऩू
  5. JSONߏ଄ԽϩΪϯά - Apache 8 LogFormat "{ \"clientip\": \"%h\", \"ident\": \"%l\",

    \"auth\": \"%u\", \"timestamp\": \"%{%FT%T%z}t\", \"verb\": \"%m\", \"request\": \"%U%q\", \"httpversion\": \"%H\", \"response\": %>s, \"bytes\": %b, \"referer\": \"% {Referer}i\", \"agent\": \"%{User-agent}i\" }" combinedjson CustomLog logs/access_log.js combinedjson - input_type: log paths: - /var/log/httpd/access_log.js document_type: apache json.keys_under_root: true json.add_error_key: true httpd.conf filebeat.yml
  6. JSONߏ଄ԽϩΪϯά - Squid 9 logformat combinedjson { "clientip": "%>a", "ident":

    "%ui", "uname": "%un", "timestamp": "%{%FT%T%z}tg", "verb": "%rm", "request": "%ru", "httpversion": "HTTP/%rv", "response": %>Hs, "bytes": %<st, "referer": "%{Referer}>h", "agent": "%{User-Agent}>h", "request_status": "%Sh", "hierarchy_status": "%Sh" } access_log /var/log/squid/access_log.js combinedjson - input_type: log paths: - /var/log/squid/access_log.js document_type: squid json.keys_under_root: true json.add_error_key: true squid.conf filebeat.yml
  7. Metricbeat - OSɺΞϓϦέʔγϣϯͷϝτϦοΫऩू 10 ϦΞϧλΠϜϞχλϦϯά • OS΍αʔϏεͷϝτϦοΫΛϞχλʔ αʔϏεͷύϑΥʔϚϯε෼ੳ • System:

    CPU, load, IO, filesystem, memory, network, process • Apache, HAProxy, MongoDB, MySQL, Nginx, Redis, ZookeeperʹରԠ ElasticsearchʹసૹɺKibanaͰՄࢹԽ • αϯϓϧDashboardͰ͙͢ʹར༻Մೳ
  8. Packetbeat - ωοτϫʔΫύέοοτͷղੳͱऩू 11 ϦΞϧλΠϜϞχλϦϯά • ΞϓϦέʔγϣϯͷ஗ԆɺΤϥʔɺԠ ౴࣌ؒͳͲΛϞχλʔ ωοτϫʔΫτϥϑΟοΫͷݕࡧͱ෼ੳ •

    ICMP, DNS, HTTP, AMQP, Cassandra, MySQL, PostgreSQL, Redis, Thrift- RPC, MongoDB, MemcacheʹରԠ ElasticsearchʹసૹɺKibanaͰՄࢹԽ • αϯϓϧDashboardͰ͙͢ʹར༻Մೳ
  9. Hosted Elasticsearch & Kibana on AWS • Elasticͷ੡඼܈ͱಉظͨ͠࠷৽൛ͷఏڙ • εέʔϧΞ΢τɾΞοϓάϨʔυΛΫϦοΫૢ࡞

    Ͱ • ແྉͷKibanaΠϯελϯεͱ30෼͝ͱͷόοΫΞ οϓ • X-Packػೳ (Security, Alerting, Monitoring, Reporting) • ݄ʑ45USD͔Β • SLAϕʔεͷαϙʔτΦϓγϣϯ 12 Elastic Stackͷ։ൃऀʹΑΔ།Ұͷ ެࣜ Elasticsearch as a Service
  10. X-Pack: Elastic Stackͷ෇ՃՁ஋ػೳ 13 \ ηΩϡϦςΟ෼ੳ ϩά෼ੳ ϝτϦοΫε ෼ੳ ӡ༻෼ੳ

    υΩϡϝϯτݕࡧ ΞϓϦέʔγϣϯ ݕࡧ ϩοΫμ΢ϯͱ ΞΫηε؂ࢹ σʔλͷมߋʹ
 ର͢Δ௨஌ Elasticsearch
 Ϋϥελͷ؂ࢹ σʔλ͔Βҙຯͷ
 ͋Δؔ܎Λൃݟ PDFΛ࡞੒ͯ͠ ൃݟΛγΣΞ Security Alerting Monitoring Graph Analytics Reporting
  11. X-Pack: Security - ҉߸ԽͱϩʔϧϕʔεͷΞΫηε੍ޚ 14 ҉߸Խ • KibanaɺElasticsearchͷΤϯυϙΠ ϯτ΁ͷHTTPS௨৴ •

    Ϋϥελʔ಺ͷ௨৴ ΞΫηε੍ޚ • ID/PasswordʹΑΔϢʔβೝূ • ωΠςΟϒɺLDAPɺAD࿈ܞ • KibanaͷϩάΠϯμΠΞϩά • ϩʔϧ͝ͱʹΠϯσοΫεɺAPI΁ͷ ΞΫηεΛ੍ݶ
  12. X-Pack: Alerting - σʔλͷมԽΛ௨஌ 15 εέδϡʔϧ • ಛఆͷ࣌ؒɺΠϯλʔόϧɺ Crontabॻࣜ ίϯσΟγϣϯ

    • Elasticsearchͷ͢΂ͯͷΫΤϦʔͱ ΞάϦήʔγϣϯΛαϙʔτ • ෳ਺ͷιʔεΛ૊Έ߹Θͤ ΞΫγϣϯ • ΠϯσοΫεɺϩάɺϝʔϧɺ΢Σ ϒϑοΫͳͲ
  13. X-Pack: Reporting - DashboardΛΤΫεϙʔτ 18 Earthquake - Depth Timeseries Earthquake

    - Heatmap Earthquake — Sun, Jan 1, 2006 12:00 AM to Fri, Sep 2, 2016 5:54 AM • PDF΋͘͠͸CSVΛੜ੒ • ඇKibanaϢʔβͱڞ༗ • खಈɺ΋͘͠͸Alertingͱͷ૊Έ߹Θ ͤͰεέδϡʔϧɺ΋͘͠͸ಛఆͷΠ ϕϯτ͕ൃੜͨ͠৔߹ʹ࡞੒ N ew in V5
  14. elastic.co/jp: ೔ຊޠ৘ใ΋͝ར༻Լ͍͞ 19 • ੡඼৘ใ • αϒεΫϦϓγϣϯ • ಋೖࣄྫ •

    ύʔτφʔ • ϋϯζΦϯϫʔΫγϣοϓ • ϒϩά • νϡʔτϦΞϧϏσΦ • ͓໰͍߹Θͤ