Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Elastic Stack Hands-on Workshop

Elastic Stack Hands-on Workshop

Elastic が実施する Hands-on Workshop の資料です。

Kosho Owa

May 01, 2016
Tweet

More Decks by Kosho Owa

Other Decks in Technology

Transcript

  1. 3 Elastic Cloud Security X-Pack Kibana User Interface Elasticsearch Store,

    Index,
 & Analyze Ingest Logstash Beats + Introducing the Elastic Stack, X-Pack, and Cloud Alerting Monitoring Reporting Graph &MBTUJD4UBDL
  2. &MBTUJDTFBSDI อଘɺΠϯσοΫεɺղੳ  w ճ෮ੑ͕͋ΓɺεέʔϧΞ ΢τΛલఏͱͨ͠੡඼σβ Πϯ w ߴՄ༻ੑ w

    Ϛϧνςφϯτ w ߏ଄ɺඇߏ଄σʔλ ෼ࢄܕͰ
 εέʔϥϒϧ ։ൃऀ ϑϨϯυϦʔ ݕࡧͱ෼ੳ w εΩʔϚϨε w +40/ w ΫϥΠΞϯτϥΠϒϥϦ w "QBDIF-VDFOF w ϦΞϧλΠϜ w શจݕࡧ w "HHSFHBUJPO w (FP w ଟݴޠ
  3. ,JCBOB ՄࢹԽͱ୳ࡧ  w σʔλΛ୳ࡧɺنଇੑΛൃ ݟͲͷΑ͏ͳϨϕϧ΁΋ υϦϧμ΢ϯ w &MBTUJDTFBSDIͷύϫϑϧ ͳ෼ੳػೳΛར༻

    ൃݟͱಎ࡯ ΧελϚΠζ ͦͯ͠ڞ༗ &MBTUJD4UBDL ΁ͷೖΓޱ w όʔνϟʔτɺંΕઢά ϥϑɺ෼෍ਤɺ஍ਤɺώ ετάϥϜ w μογϡϘʔυΛΛγΣ Ξ͠ɺӡ༻ϫʔΫϑϩʔ ʹ૊ΈࠐΈ w ՄࢹԽͷͨΊͷ౷Ұతͳ 6* w &MBTUJD4UBDLͷӡ༻؅ཧ w ϓϥάΠϯՄೳͳΞʔΩς ΫνϟͰɺಠࣗͷΞϓϦέ ʔγϣϯ͕࡞੒Մೳ
  4. *OHFTU σʔλͷ౤ೖ  w σʔλऩूͱՃ޻छ ྨҎ্ͷϓϥάΠϯ w ࣍ੈ୅ͷύΠϓϥΠϯϚ ΠΫϩόονͰΠϕϯτ܈ Λॲཧ

    &4)BEPPQ w ܰྔσʔλʔγούʔͷ ։ൃϓϥοτϑΥʔϜ w ϗετϕʔεͰϝτϦο ΫΛऩू͠ɺ &MBTUJDTFBSDIʹ౤ೖ 1BDLFUCFBU 5PQCFBU  'JMFCFBUBOE 8JOMPHCFBU w )%'4 4QBSL  .BQ3FEVDFͳͲͷͨΊ ͷ૒ํ޲ίωΫλʔ w )BEPPQʹετΞ͞Εͨ σʔλͷϦΞϧλΠϜݕࡧ ΫΤϦʔΛՄೳʹ͢Δ
  5. 7 Security for the Elastic Stack (Shield) Security Monitoring for

    the Elastic Stack (Marvel) Monitoring Notifications for the Elastic Stack (Watcher) Alerting Security X-Pack Alerting Monitoring Reporting Graph Automated reporting for the Elastic Stack Reporting Real-time graph analytics for the Elastic Stack Graph A Single Extension
  6. 8 Simply Secure the Elastic Stack • Username/password protection Advanced

    Security When Needed • LDAP/AD integration • Role-based access control • IP filtering • Field and document level security • Encrypted communications • Audit logging • Kibana plugin for login and session management Security (Shield) External Authentication (optional)
  7. 9 Setup Alerts • Create Watches based on data •

    Trigger automatic notifications • Setup chained inputs Notify and Integrate • Slack, Hipchat, JIRA, Pagerduty • Email • Elastic Monitoring (Marvel) • Other Alerting (Watcher)
  8. 10 Monitor Elasticsearch • Real-time statistics and metrics for all

    clusters and nodes Diagnose Issues • Analyze historical or real-time data for root cause analyses Optimize Performance • Utilize in-depth analyses to improve cluster performance Monitoring (Marvel)
  9. 11 Query and Visualize Relationships • Use relevance as a

    guide to uncover and explore new relationships in all your data stored in Elasticsearch • Interact with Graph via a Kibana plugin or use the Graph API to integrate with your applications • Enable new use cases – behavioral analysis, fraud, cybersecurity, drug discovery, and recommendations Graph Analytics
  10. 12 Generate and share reports • Export PDF’s of dashboards

    and visualizations with a click • Use alerting features to email reports ‒ Time-based (weekly) ‒ Event-based (when X happens, send me a picture of the dashboard) • Export to CSV Reporting
  11. 13 The only Elasticsearch as a Service offering powered by

    the creators of the Elastic Stack • Always runs on the latest software • One-click to scale/upgrade with no downtime • Free Kibana and backups every 30 minutes • Dedicated, SLA-based support • Easily add X-Pack features: security (Shield), alerting (Watcher), and monitoring (Marvel) • Pricing starts at $45 a month Hosted Elasticsearch Search Analytics Logging
  12. ݕࡧ ϩά ෼ੳ Ϣʔεέʔε  • ΠϯϓοτωοτϫʔΫػثɺαʔϏ εɺΞϓϦέʔγϣϯϩάɺηϯαʔͷΞ ΢τϓοτͷอଘɺ෼ੳɺΞϥʔτ •

    ωοτϫʔΫύέοτͷղੳ • &MBTUJDTFBSDI -PHTUBTI ,JCBOB #FBUT • ։ൃ޻਺খɺத • ϊʔυ਺ • υΩϡϝϯτɺ΢ΣϒαΠτͷݕࡧ • ஍ཧ৘ใΞϓϦέʔγϣϯαʔϏεͷόο ΫΤϯυ • &MBTUJDTFBSDI -PHTUBTI • ։ൃ޻਺େ • ϊʔυ਺
  13. )BOETPO؀ڥ • Ծ૝Ϛγϯ - VMWare - VirtualBox - Amazon EC2

    • ΦϖϨʔςΟϯάγεςϜ - Redhat Enterprise Linux 6 - CentOS 6.x - Amazon Linux AMI 2016.03.1 • ϝϞϦ - 4GB Ҏ্  • Support Matrix - https://www.elastic.co/support/matrix • ωοτϫʔΫ - Πϯλʔωοτ઀ଓ͕Մೳͳ͜ͱ - 9200/tcp, 5601/tcp ΁ͷ઀ଓΛڐՄ • +BWBO3VOUJNF - Oracle Java SE 1.7 Ҏ্ - OpenJDK 1.7 Ҏ্
  14. -BCϝχϡʔ  &MBTUJDTFBSDIͷΠϯετʔϧ  5PQCFBU 'JMFCFBUͷΠϯετʔϧ  ,JCBOBͷΠϯετʔϧ  5PQCFBU

    'JMFCFBUͷσʔλͷ֬ೝ  "QBDIFͷϩάͷऔΓࠐΈ MPHTUBTI  $36%ͱݕࡧ  ϞχλϦϯάઐ༻Ϋϥελʔͷߏ੒ 
  15. -BCԋशखॱ   &MBTUJDTFBSDIΛ31.ͰΠϯετʔϧ  .BSWFMϓϥάΠϯΛΠϯετʔϧ  αʔϏεͱͯ͠ొ࿥͠ɺىಈ͢Δ  ಈ࡞֬ೝ

     σʔλσΟϨΫτϦΛ֬ೝ͢Δ • Downloads > Marvel - https://www.elastic.co/downloads/marvel • Downloads > Elasticsearch - https://www.elastic.co/downloads/elasticsearch
  16. &MBTUJDTFBSDIΛ31.ͰΠϯετʔϧ  $ sudo rpm -i https://download.elastic.co/elasticsearch/release/org/elasticsearch/ distribution/rpm/elasticsearch/2.3.2/elasticsearch-2.3.2.rpm Creating elasticsearch

    group... OK Creating elasticsearch user... OK ### NOT starting on installation, please execute the following statements to configure elasticsearch service to start automatically using chkconfig $ sudo chkconfig --add elasticsearch ### You can start elasticsearch service by executing
  17. 1MVHJOͷΠϯετʔϧ  $ cd /usr/share/elasticsearch/ $ sudo bin/plugin install license

    -> Installing license... $ sudo bin/plugin install marvel-agent -> Installing marvel-agent... @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: plugin requires additional permissions @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ * java.lang.RuntimePermission setFactory * javax.net.ssl.SSLPermission setHostnameVerifier See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html for descriptions of what these permissions allow and the associated risks. Continue with installation? [y/N]y • Docs > Marvel Documentation > Installing Marvel > Installing Marvel on Offline Machines - https:// www.elastic.co/guide/en/marvel/current/installing-marvel.html#offline-installation
  18. &MBTUJDTFBSDIͷىಈͱ֬ೝ  $ sudo service elasticsearch start $ curl localhost:9200

    { "name" : "Juggernaut", "cluster_name" : "elasticsearch", "version" : { "number" : "2.3.2", "build_hash" : "b9e4a6acad4008027e4038f6abed7f7dba346f94", "build_timestamp" : "2016-04-21T16:03:47Z", "build_snapshot" : false, "lucene_version" : "5.5.0" }, "tagline" : "You Know, for Search" } $ cd /var/lib/elasticsearch $ ls elasticsearch
  19. -BCԋशखॱ   5PQCFBU 'JMFCFBUΛΠϯετʔϧ͢Δ  /var/log/*Λऔಘ͢ΔΑ͏ʹઃఆ  &MBTUJDTFBSDIʹϝτϦοΫΛૹΔΑ͏ʹઃఆ 

    ىಈ͢Δ • Downloads | Topbeat - https://www.elastic.co/downloads/beats/topbeat • Downloads | Filbeat - https://www.elastic.co/downloads/beats/filebeat
  20. 5PQCFBUͷΠϯετʔϧͱىಈ  $ sudo rpm -i https://download.elastic.co/beats/topbeat/topbeat-1.2.2-x86_64.rpm $ grep hosts

    /etc/topbeat/topbeat.yml hosts: [“localhost:9200"] $ curl -XPUT 'http://localhost:9200/_template/topbeat' -d@/etc/topbeat/topbeat.template.json {"acknowledged":true} $ sudo service topbeat start Starting topbeat: [ OK ]
  21. 'JMFCFBUͷΠϯετʔϧͱىಈ  $ sudo rpm -i https://download.elastic.co/beats/filebeat/filebeat-1.2.2-x86_64.rpm $ less /etc/filebeat/filebeat.yml

    … paths: - /var/log/*.log … hosts: ["localhost:9200"] … $ curl -XPUT 'http://localhost:9200/_template/filebeat' -d@/etc/filebeat/ filebeat.template.json {"acknowledged":true} $ sudo service filebeat start Starting filebeat: [ OK ]
  22. -BCԋशखॱ   ,JCBOBΛΠϯετʔϧ͢Δ  .BSWFM 4FOTFϓϥάΠϯΛΠϯετʔϧ͢Δ  ىಈ͢Δ •

    Downloads | Kibana - https://www.elastic.co/downloads/kibana • Downloads | Marvel - https://www.elastic.co/downloads/marvel • Sense Documentation » Installing Sense - https://www.elastic.co/guide/en/sense/current/installing.html
  23. ,JCBOBͷΠϯετʔϧ  $ sudo rpm -i https://download.elastic.co/kibana/kibana/kibana-4.5.0-1.x86_64.rpm $ cd /opt/kibana

    $ sudo bin/kibana plugin --install elasticsearch/marvel Installing marvel … Plugin installation complete $ sudo bin/kibana plugin --install elastic/sense Installing sense … Plugin installation complete $ sudo chown -R kibana:root /opt/kibana/optimize $ sudo service kibana start kibana started • Docs > Marvel Documentation > Installing Marvel > Installing Marvel on Offline Machines - https:// www.elastic.co/guide/en/marvel/current/installing-marvel.html#offline-installation • Docs > Sense Documentation » Installing Sense > Manual Download - https://www.elastic.co/guide/en/ sense/current/installing.html#manual_download
  24. ΠϯσοΫεͷҰཡ  $ curl localhost:9200/_cat/indices yellow open megacorp 5 1

    3 0 11.9kb 11.9kb yellow open .marvel-es-1-2016.04.30 1 1 6783 0 2.9mb 2.9mb yellow open topbeat-2016.04.30 5 1 24399 0 5.4mb 5.4mb yellow open filebeat-2016.04.30 5 1 1290 0 354.8kb 354.8kb yellow open .marvel-es-data-1 1 1 3 1 8.1kb 8.1kb yellow open .kibana 1 1 4 0 32.2kb 32.2kb
  25. -PHTUBTIͷΠϯετʔϧͱϑΝΠϧͷ४උ  $ sudo rpm -i https://download.elastic.co/logstash/logstash/packages/centos/ logstash-2.3.2-1.noarch.rpm $ mkdir

    Apache_ElasticStack_Example $ cd Apache_ElasticStack_Example $ wget https://raw.githubusercontent.com/elastic/examples/master/ElasticStack_apache/ apache_logstash.conf $ wget https://raw.githubusercontent.com/elastic/examples/master/ElasticStack_apache/ apache_template.json $ wget https://raw.githubusercontent.com/elastic/examples/master/ElasticStack_apache/ apache_kibana.json $ wget https://raw.githubusercontent.com/elastic/examples/master/ElasticStack_apache/
  26. ϩάͷऔΓࠐΈ  $ cat apache_logs | /opt/logstash/bin/logstash -f apache_logstash.conf Settings:

    Default pipeline workers: 2 Pipeline main started Pipeline main has been shutdown stopping pipeline {:id=>"main"} curl localhost:9200/apache_elk_example/_count?pretty { "count" : 10000, "_shards" : { "total" : 5, "successful" : 5, "failed" : 0 } }
  27. -PHTUBTIͷઃఆϑΝΠϧJOQVU GJMUFS  input { stdin { } } filter

    { grok { match => { "message" => '%{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] "%{WORD:verb} %{DATA:reques t} HTTP/%{NUMBER:httpversion}" %{NUMBER:response:int} (?:-|%{NUMBER:bytes:int}) %{QS:referrer} %{QS:agent}’ # ϩάͷύʔε } } date { # λΠϜελϯϓͱͯ͠࠾༻͢ΔϑΟʔϧυ match => [ "timestamp", "dd/MMM/YYYY:HH:mm:ss Z" ] locale => en } geoip { source => "clientip" } # IPΞυϨε͔Β஍ཧ৘ใͷల։ useragent { # User-agent ͷղੳ source => "agent" target => "useragent" } }
  28. -PHTUBTIͷઃఆϑΝΠϧPVUQVU  output { stdout { codec => plain }

    # ඪ४ग़ྗ elasticsearch { hosts => “http://localhost:9200” # ग़ྗઌ index => “apache_elk_example" # ࡞੒͢ΔΠϯσοΫε template => “./apache_template.json" # Ϛοϐϯάͷઃఆ template_name => “apache_elk_example" # ϚοϐϯάΛఆٛ͢ΔςϯϓϨʔτͷ໊শ template_overwrite => true } }
  29. -BCԋशखॱ   4FOTFΛ࢖ͬͯemployee 1, 2, 3Λ࡞੒͢Δ ‒ https://www.elastic.co/guide/en/elasticsearch/guide/current/_indexing_employee_documents.html 

    υΩϡϝϯτͷऔಘΛߦ͏  υΩϡϝϯτΛݕࡧ͢Δ  υΩϡϝϯτͷߋ৽ɺ࡟আΛߦ͏ • Elasticsearch: The Definitive Guide [2.x] » Getting Started » You Know, for Search… » Retrieving a Document - https://www.elastic.co/guide/en/elasticsearch/guide/current/_retrieving_a_document.html • Elasticsearch: The Definitive Guide [2.x] » Getting Started » You Know, for Search… » Search Lite - https://www.elastic.co/guide/en/elasticsearch/guide/current/_search_lite.html • Elasticsearch: The Definitive Guide [2.x] » Getting Started » Data In, Data Out » Updating a Whole Document - https://www.elastic.co/guide/en/elasticsearch/guide/current/update-doc.html • Elasticsearch: The Definitive Guide [2.x] » Getting Started » Data In, Data Out » Deleting a Document - https://www.elastic.co/guide/en/elasticsearch/guide/current/delete-doc.html
  30. υΩϡϝϯτͷ࡞੒  PUT /megacorp/employee/1 { "first_name" : "John", "last_name" :

    "Smith", "age" : 25, "about" : "I love to go rock climbing", "interests": [ "sports", "music" ] } { "_index": "megacorp", "_type": "employee", "_id": "1", "_version": 1, "_shards": { "total": 2, "successful": 1, "failed": 0 }, "created": true }   ϦΫΤετͱͯ͠ೖྗ FNQMPZFF ΋ಉ༷ʹ࡞੒͢Δ Ϩεϙϯε
  31. υΩϡϝϯτͷऔಘ  GET/megacorp/employee/1 { "_index": "megacorp", "_type": "employee", "_id": "1",

    "_version": 1, "found": true, "_source": { "first_name": "John", "last_name": "Smith", "age": 25, "about": "I love to go rock climbing", "interests": [ "sports", "music" ] } }
  32. υΩϡϝϯτͷݕࡧ  GET /megacorp/employee/_search?q=last_name:Smith { "took": 30, "timed_out": false, "_shards":

    { "total": 5, "successful": 5, "failed": 0 }, "hits": { "total": 2, "max_score": 0.30685282, "hits": [ { "_index": "megacorp", "_type": "employee", "_id": "2", "_score": 0.30685282, "_source": { "first_name": "Jane", ҎԼলུ
  33. υΩϡϝϯτͷߋ৽  PUT /megacorp/employee/3 { "first_name" : "Richard", "last_name" :

    "Roe" } { "_index": "megacorp", "_type": "employee", "_id": "3", "_version": 2, "_shards": { "total": 2, "successful": 1, "failed": 0 }, "created": false } GET /megacorp/employee/3 { "_index": "megacorp", "_type": "employee", "_id": "3", "_version": 2, "found": true, "_source": { "first_name": "Richard", "last_name": "Roe" } } υΩϡϝϯτ͸׬શʹߋ৽͞ΕΔ ෦෼తʹΞοϓσʔτ͢Δʹ͸@VQEBUF"1*Λ࢖༻͢Δ 
  34. σϓϩΠϝϯτϞσϧ 62 Cluster “elasticsearch” ES node marvel-agent Monitoring Cluster “es-monitor”

    ES node marvel-agent Kibana marvel-ui # config/elasticsearch.yml marvel.agent.exporters: id1: type: http host: [“es-mon-1:9200”,…]
  35. -BCԋशखॱ  • Marvel Documentation > Installing Marvel > Setting

    up a Separate Monitoring Cluster: https:// www.elastic.co/guide/en/marvel/current/installing-marvel.html#monitoring-cluster • Downloads | Elasticsearch - https://www.elastic.co/downloads/elasticsearch • Downloads | Marvel - https://www.elastic.co/downloads/marvel • Downloads | Kibana - https://www.elastic.co/downloads/kibana 1. “es-monitor” Ϋϥελʔ ͱ Kibana ΛηοτΞοϓ • ৽ͨʹ Elasticsearch ͷϊʔυΛΠϯετʔϧ͢Δ • Ϋϥελʔ໊͸ “es-monitor” ͱ͢Δ • ϙʔτ9001Λ࢖༻͢Δ • “es-monitor” Ϋϥελʔʹ઀ଓ͢ΔΑ͏ Kibana ΛηοτΞοϓ͢Δ 2. طଘͷΫϥελʔͷઃఆมߋ • Marvel ͷϝτϦοΫΛ “es-monitor” ΫϥελʔʹૹΔ 3. ֬ೝ
  36. lFTNPOJUPSzΫϥελʔͷηοτΞοϓ 64 $ cd $ mkdir es-monitor $ cd es-monitor

    $ curl https://download.elastic.co/elasticsearch/release/org/elasticsearch/distribution/tar/ elasticsearch/2.3.2/elasticsearch-2.3.2.tar.gz | tar zxf - $ cd elasticsearch-2.3.2 $ bin/plugin install license $ bin/plugin install marvel-agent $ vi config/elasticsearch.yml cluster.name: es-monitor http.port: 9201 $ bin/elasticsearch