Share
Embed
Transcript
67 slides
-
3 Elastic Cloud Security X-Pack Kibana User Interface Elasticsearch Store,
Index, & Analyze Ingest Logstash Beats + Introducing the Elastic Stack, X-Pack, and Cloud Alerting Monitoring Reporting Graph &MBTUJD4UBDL -
&MBTUJDTFBSDI อଘɺΠϯσοΫεɺղੳ w ճ෮ੑ͕͋ΓɺεέʔϧΞ τΛલఏͱͨ͠σβ Πϯ w ߴՄ༻ੑ w
Ϛϧνςφϯτ w ߏɺඇߏσʔλ ࢄܕͰ εέʔϥϒϧ ։ൃऀ ϑϨϯυϦʔ ݕࡧͱੳ w εΩʔϚϨε w +40/ w ΫϥΠΞϯτϥΠϒϥϦ w "QBDIF-VDFOF w ϦΞϧλΠϜ w શจݕࡧ w "HHSFHBUJPO w (FP w ଟݴޠ -
,JCBOB ՄࢹԽͱ୳ࡧ w σʔλΛ୳ࡧɺنଇੑΛൃ ݟͲͷΑ͏ͳϨϕϧ υϦϧμϯ w &MBTUJDTFBSDIͷύϫϑϧ ͳੳػೳΛར༻
ൃݟͱಎ ΧελϚΠζ ͦͯ͠ڞ༗ &MBTUJD4UBDL ͷೖΓޱ w όʔνϟʔτɺંΕઢά ϥϑɺਤɺਤɺώ ετάϥϜ w μογϡϘʔυΛΛγΣ Ξ͠ɺӡ༻ϫʔΫϑϩʔ ʹΈࠐΈ w ՄࢹԽͷͨΊͷ౷Ұతͳ 6* w &MBTUJD4UBDLͷӡ༻ཧ w ϓϥάΠϯՄೳͳΞʔΩς ΫνϟͰɺಠࣗͷΞϓϦέ ʔγϣϯ͕࡞Մೳ -
*OHFTU σʔλͷೖ w σʔλऩूͱՃछ ྨҎ্ͷϓϥάΠϯ w ࣍ੈͷύΠϓϥΠϯϚ ΠΫϩόονͰΠϕϯτ܈ Λॲཧ
&4)BEPPQ w ܰྔσʔλʔγούʔͷ ։ൃϓϥοτϑΥʔϜ w ϗετϕʔεͰϝτϦο ΫΛऩू͠ɺ &MBTUJDTFBSDIʹೖ 1BDLFUCFBU 5PQCFBU 'JMFCFBUBOE 8JOMPHCFBU w )%'4 4QBSL .BQ3FEVDFͳͲͷͨΊ ͷํίωΫλʔ w )BEPPQʹετΞ͞Εͨ σʔλͷϦΞϧλΠϜݕࡧ ΫΤϦʔΛՄೳʹ͢Δ -
7 Security for the Elastic Stack (Shield) Security Monitoring for
the Elastic Stack (Marvel) Monitoring Notifications for the Elastic Stack (Watcher) Alerting Security X-Pack Alerting Monitoring Reporting Graph Automated reporting for the Elastic Stack Reporting Real-time graph analytics for the Elastic Stack Graph A Single Extension -
8 Simply Secure the Elastic Stack • Username/password protection Advanced
Security When Needed • LDAP/AD integration • Role-based access control • IP filtering • Field and document level security • Encrypted communications • Audit logging • Kibana plugin for login and session management Security (Shield) External Authentication (optional) -
9 Setup Alerts • Create Watches based on data •
Trigger automatic notifications • Setup chained inputs Notify and Integrate • Slack, Hipchat, JIRA, Pagerduty • Email • Elastic Monitoring (Marvel) • Other Alerting (Watcher) -
10 Monitor Elasticsearch • Real-time statistics and metrics for all
clusters and nodes Diagnose Issues • Analyze historical or real-time data for root cause analyses Optimize Performance • Utilize in-depth analyses to improve cluster performance Monitoring (Marvel) -
11 Query and Visualize Relationships • Use relevance as a
guide to uncover and explore new relationships in all your data stored in Elasticsearch • Interact with Graph via a Kibana plugin or use the Graph API to integrate with your applications • Enable new use cases – behavioral analysis, fraud, cybersecurity, drug discovery, and recommendations Graph Analytics -
12 Generate and share reports • Export PDF’s of dashboards
and visualizations with a click • Use alerting features to email reports ‒ Time-based (weekly) ‒ Event-based (when X happens, send me a picture of the dashboard) • Export to CSV Reporting -
13 The only Elasticsearch as a Service offering powered by
the creators of the Elastic Stack • Always runs on the latest software • One-click to scale/upgrade with no downtime • Free Kibana and backups every 30 minutes • Dedicated, SLA-based support • Easily add X-Pack features: security (Shield), alerting (Watcher), and monitoring (Marvel) • Pricing starts at $45 a month Hosted Elasticsearch Search Analytics Logging -
ݕࡧ ϩά ੳ Ϣʔεέʔε • ΠϯϓοτωοτϫʔΫػثɺαʔϏ εɺΞϓϦέʔγϣϯϩάɺηϯαʔͷΞ τϓοτͷอଘɺੳɺΞϥʔτ •
ωοτϫʔΫύέοτͷղੳ • &MBTUJDTFBSDI -PHTUBTI ,JCBOB #FBUT • ։ൃখɺத • ϊʔυ • υΩϡϝϯτɺΣϒαΠτͷݕࡧ • ཧใΞϓϦέʔγϣϯαʔϏεͷόο ΫΤϯυ • &MBTUJDTFBSDI -PHTUBTI • ։ൃେ • ϊʔυ -
)BOETPOڥ • ԾϚγϯ - VMWare - VirtualBox - Amazon EC2
• ΦϖϨʔςΟϯάγεςϜ - Redhat Enterprise Linux 6 - CentOS 6.x - Amazon Linux AMI 2016.03.1 • ϝϞϦ - 4GB Ҏ্ • Support Matrix - https://www.elastic.co/support/matrix • ωοτϫʔΫ - Πϯλʔωοτଓ͕Մೳͳ͜ͱ - 9200/tcp, 5601/tcp ͷଓΛڐՄ • +BWBO3VOUJNF - Oracle Java SE 1.7 Ҏ্ - OpenJDK 1.7 Ҏ্ -
-BCϝχϡʔ &MBTUJDTFBSDIͷΠϯετʔϧ 5PQCFBU 'JMFCFBUͷΠϯετʔϧ ,JCBOBͷΠϯετʔϧ 5PQCFBU
'JMFCFBUͷσʔλͷ֬ೝ "QBDIFͷϩάͷऔΓࠐΈ MPHTUBTI $36%ͱݕࡧ ϞχλϦϯάઐ༻Ϋϥελʔͷߏ -
-BCԋशखॱ &MBTUJDTFBSDIΛ31.ͰΠϯετʔϧ .BSWFMϓϥάΠϯΛΠϯετʔϧ αʔϏεͱͯ͠ొ͠ɺىಈ͢Δ ಈ࡞֬ೝ
σʔλσΟϨΫτϦΛ֬ೝ͢Δ • Downloads > Marvel - https://www.elastic.co/downloads/marvel • Downloads > Elasticsearch - https://www.elastic.co/downloads/elasticsearch -
&MBTUJDTFBSDIΛ31.ͰΠϯετʔϧ $ sudo rpm -i https://download.elastic.co/elasticsearch/release/org/elasticsearch/ distribution/rpm/elasticsearch/2.3.2/elasticsearch-2.3.2.rpm Creating elasticsearch
group... OK Creating elasticsearch user... OK ### NOT starting on installation, please execute the following statements to configure elasticsearch service to start automatically using chkconfig $ sudo chkconfig --add elasticsearch ### You can start elasticsearch service by executing -
1MVHJOͷΠϯετʔϧ $ cd /usr/share/elasticsearch/ $ sudo bin/plugin install license
-> Installing license... $ sudo bin/plugin install marvel-agent -> Installing marvel-agent... @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: plugin requires additional permissions @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ * java.lang.RuntimePermission setFactory * javax.net.ssl.SSLPermission setHostnameVerifier See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html for descriptions of what these permissions allow and the associated risks. Continue with installation? [y/N]y • Docs > Marvel Documentation > Installing Marvel > Installing Marvel on Offline Machines - https:// www.elastic.co/guide/en/marvel/current/installing-marvel.html#offline-installation -
&MBTUJDTFBSDIͷىಈͱ֬ೝ $ sudo service elasticsearch start $ curl localhost:9200
{ "name" : "Juggernaut", "cluster_name" : "elasticsearch", "version" : { "number" : "2.3.2", "build_hash" : "b9e4a6acad4008027e4038f6abed7f7dba346f94", "build_timestamp" : "2016-04-21T16:03:47Z", "build_snapshot" : false, "lucene_version" : "5.5.0" }, "tagline" : "You Know, for Search" } $ cd /var/lib/elasticsearch $ ls elasticsearch -
-BCԋशखॱ 5PQCFBU 'JMFCFBUΛΠϯετʔϧ͢Δ /var/log/*Λऔಘ͢ΔΑ͏ʹઃఆ &MBTUJDTFBSDIʹϝτϦοΫΛૹΔΑ͏ʹઃఆ
ىಈ͢Δ • Downloads | Topbeat - https://www.elastic.co/downloads/beats/topbeat • Downloads | Filbeat - https://www.elastic.co/downloads/beats/filebeat -
5PQCFBUͷΠϯετʔϧͱىಈ $ sudo rpm -i https://download.elastic.co/beats/topbeat/topbeat-1.2.2-x86_64.rpm $ grep hosts
/etc/topbeat/topbeat.yml hosts: [“localhost:9200"] $ curl -XPUT 'http://localhost:9200/_template/topbeat' -d@/etc/topbeat/topbeat.template.json {"acknowledged":true} $ sudo service topbeat start Starting topbeat: [ OK ] -
'JMFCFBUͷΠϯετʔϧͱىಈ $ sudo rpm -i https://download.elastic.co/beats/filebeat/filebeat-1.2.2-x86_64.rpm $ less /etc/filebeat/filebeat.yml
… paths: - /var/log/*.log … hosts: ["localhost:9200"] … $ curl -XPUT 'http://localhost:9200/_template/filebeat' -d@/etc/filebeat/ filebeat.template.json {"acknowledged":true} $ sudo service filebeat start Starting filebeat: [ OK ] -
-BCԋशखॱ ,JCBOBΛΠϯετʔϧ͢Δ .BSWFM 4FOTFϓϥάΠϯΛΠϯετʔϧ͢Δ ىಈ͢Δ •
Downloads | Kibana - https://www.elastic.co/downloads/kibana • Downloads | Marvel - https://www.elastic.co/downloads/marvel • Sense Documentation » Installing Sense - https://www.elastic.co/guide/en/sense/current/installing.html -
,JCBOBͷΠϯετʔϧ $ sudo rpm -i https://download.elastic.co/kibana/kibana/kibana-4.5.0-1.x86_64.rpm $ cd /opt/kibana
$ sudo bin/kibana plugin --install elasticsearch/marvel Installing marvel … Plugin installation complete $ sudo bin/kibana plugin --install elastic/sense Installing sense … Plugin installation complete $ sudo chown -R kibana:root /opt/kibana/optimize $ sudo service kibana start kibana started • Docs > Marvel Documentation > Installing Marvel > Installing Marvel on Offline Machines - https:// www.elastic.co/guide/en/marvel/current/installing-marvel.html#offline-installation • Docs > Sense Documentation » Installing Sense > Manual Download - https://www.elastic.co/guide/en/ sense/current/installing.html#manual_download -
ΠϯσοΫεͷҰཡ $ curl localhost:9200/_cat/indices yellow open megacorp 5 1
3 0 11.9kb 11.9kb yellow open .marvel-es-1-2016.04.30 1 1 6783 0 2.9mb 2.9mb yellow open topbeat-2016.04.30 5 1 24399 0 5.4mb 5.4mb yellow open filebeat-2016.04.30 5 1 1290 0 354.8kb 354.8kb yellow open .marvel-es-data-1 1 1 3 1 8.1kb 8.1kb yellow open .kibana 1 1 4 0 32.2kb 32.2kb -
-BCԋशखॱ IUUQTHJUIVCDPNFMBTUJDFYBNQMFTUSFFNBTUFS&MBTUJD4UBDL@BQBDIFͷ खॱʹैͬͯɺࣗͷΫϥελʔʹ"QBDIFͷϩάΛऔΓࠐΈɺ %BTICPBSEΛදࣔ͢Δ ΞΫηεͷଟ͍ύε݅Λදࣔ͢Δ ՃͰνϟϨϯδ •
Kibana User Guide [4.5] » Visualize » Data Table - https://www.elastic.co/guide/en/kibana/current/data- table.html -
-PHTUBTIͷΠϯετʔϧͱϑΝΠϧͷ४උ $ sudo rpm -i https://download.elastic.co/logstash/logstash/packages/centos/ logstash-2.3.2-1.noarch.rpm $ mkdir
Apache_ElasticStack_Example $ cd Apache_ElasticStack_Example $ wget https://raw.githubusercontent.com/elastic/examples/master/ElasticStack_apache/ apache_logstash.conf $ wget https://raw.githubusercontent.com/elastic/examples/master/ElasticStack_apache/ apache_template.json $ wget https://raw.githubusercontent.com/elastic/examples/master/ElasticStack_apache/ apache_kibana.json $ wget https://raw.githubusercontent.com/elastic/examples/master/ElasticStack_apache/ -
ϩάͷऔΓࠐΈ $ cat apache_logs | /opt/logstash/bin/logstash -f apache_logstash.conf Settings:
Default pipeline workers: 2 Pipeline main started Pipeline main has been shutdown stopping pipeline {:id=>"main"} curl localhost:9200/apache_elk_example/_count?pretty { "count" : 10000, "_shards" : { "total" : 5, "successful" : 5, "failed" : 0 } } -
-PHTUBTIͷઃఆϑΝΠϧJOQVU GJMUFS input { stdin { } } filter
{ grok { match => { "message" => '%{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] "%{WORD:verb} %{DATA:reques t} HTTP/%{NUMBER:httpversion}" %{NUMBER:response:int} (?:-|%{NUMBER:bytes:int}) %{QS:referrer} %{QS:agent}’ # ϩάͷύʔε } } date { # λΠϜελϯϓͱͯ͠࠾༻͢ΔϑΟʔϧυ match => [ "timestamp", "dd/MMM/YYYY:HH:mm:ss Z" ] locale => en } geoip { source => "clientip" } # IPΞυϨε͔Βཧใͷల։ useragent { # User-agent ͷղੳ source => "agent" target => "useragent" } } -
-PHTUBTIͷઃఆϑΝΠϧPVUQVU output { stdout { codec => plain }
# ඪ४ग़ྗ elasticsearch { hosts => “http://localhost:9200” # ग़ྗઌ index => “apache_elk_example" # ࡞͢ΔΠϯσοΫε template => “./apache_template.json" # Ϛοϐϯάͷઃఆ template_name => “apache_elk_example" # ϚοϐϯάΛఆٛ͢ΔςϯϓϨʔτͷ໊শ template_overwrite => true } } -
-BCԋशखॱ 4FOTFΛͬͯemployee 1, 2, 3Λ࡞͢Δ ‒ https://www.elastic.co/guide/en/elasticsearch/guide/current/_indexing_employee_documents.html
υΩϡϝϯτͷऔಘΛߦ͏ υΩϡϝϯτΛݕࡧ͢Δ υΩϡϝϯτͷߋ৽ɺআΛߦ͏ • Elasticsearch: The Definitive Guide [2.x] » Getting Started » You Know, for Search… » Retrieving a Document - https://www.elastic.co/guide/en/elasticsearch/guide/current/_retrieving_a_document.html • Elasticsearch: The Definitive Guide [2.x] » Getting Started » You Know, for Search… » Search Lite - https://www.elastic.co/guide/en/elasticsearch/guide/current/_search_lite.html • Elasticsearch: The Definitive Guide [2.x] » Getting Started » Data In, Data Out » Updating a Whole Document - https://www.elastic.co/guide/en/elasticsearch/guide/current/update-doc.html • Elasticsearch: The Definitive Guide [2.x] » Getting Started » Data In, Data Out » Deleting a Document - https://www.elastic.co/guide/en/elasticsearch/guide/current/delete-doc.html -
υΩϡϝϯτͷ࡞ PUT /megacorp/employee/1 { "first_name" : "John", "last_name" :
"Smith", "age" : 25, "about" : "I love to go rock climbing", "interests": [ "sports", "music" ] } { "_index": "megacorp", "_type": "employee", "_id": "1", "_version": 1, "_shards": { "total": 2, "successful": 1, "failed": 0 }, "created": true } ϦΫΤετͱͯ͠ೖྗ FNQMPZFF ಉ༷ʹ࡞͢Δ Ϩεϙϯε -
υΩϡϝϯτͷऔಘ GET/megacorp/employee/1 { "_index": "megacorp", "_type": "employee", "_id": "1",
"_version": 1, "found": true, "_source": { "first_name": "John", "last_name": "Smith", "age": 25, "about": "I love to go rock climbing", "interests": [ "sports", "music" ] } } -
υΩϡϝϯτͷݕࡧ GET /megacorp/employee/_search?q=last_name:Smith { "took": 30, "timed_out": false, "_shards":
{ "total": 5, "successful": 5, "failed": 0 }, "hits": { "total": 2, "max_score": 0.30685282, "hits": [ { "_index": "megacorp", "_type": "employee", "_id": "2", "_score": 0.30685282, "_source": { "first_name": "Jane", ҎԼলུ -
υΩϡϝϯτͷߋ৽ PUT /megacorp/employee/3 { "first_name" : "Richard", "last_name" :
"Roe" } { "_index": "megacorp", "_type": "employee", "_id": "3", "_version": 2, "_shards": { "total": 2, "successful": 1, "failed": 0 }, "created": false } GET /megacorp/employee/3 { "_index": "megacorp", "_type": "employee", "_id": "3", "_version": 2, "found": true, "_source": { "first_name": "Richard", "last_name": "Roe" } } υΩϡϝϯτશʹߋ৽͞ΕΔ ෦తʹΞοϓσʔτ͢Δʹ@VQEBUF"1*Λ༻͢Δ -
σϓϩΠϝϯτϞσϧ 62 Cluster “elasticsearch” ES node marvel-agent Monitoring Cluster “es-monitor”
ES node marvel-agent Kibana marvel-ui # config/elasticsearch.yml marvel.agent.exporters: id1: type: http host: [“es-mon-1:9200”,…] -
-BCԋशखॱ • Marvel Documentation > Installing Marvel > Setting
up a Separate Monitoring Cluster: https:// www.elastic.co/guide/en/marvel/current/installing-marvel.html#monitoring-cluster • Downloads | Elasticsearch - https://www.elastic.co/downloads/elasticsearch • Downloads | Marvel - https://www.elastic.co/downloads/marvel • Downloads | Kibana - https://www.elastic.co/downloads/kibana 1. “es-monitor” Ϋϥελʔ ͱ Kibana ΛηοτΞοϓ • ৽ͨʹ Elasticsearch ͷϊʔυΛΠϯετʔϧ͢Δ • Ϋϥελʔ໊ “es-monitor” ͱ͢Δ • ϙʔτ9001Λ༻͢Δ • “es-monitor” Ϋϥελʔʹଓ͢ΔΑ͏ Kibana ΛηοτΞοϓ͢Δ 2. طଘͷΫϥελʔͷઃఆมߋ • Marvel ͷϝτϦοΫΛ “es-monitor” ΫϥελʔʹૹΔ 3. ֬ೝ -
lFTNPOJUPSzΫϥελʔͷηοτΞοϓ 64 $ cd $ mkdir es-monitor $ cd es-monitor
$ curl https://download.elastic.co/elasticsearch/release/org/elasticsearch/distribution/tar/ elasticsearch/2.3.2/elasticsearch-2.3.2.tar.gz | tar zxf - $ cd elasticsearch-2.3.2 $ bin/plugin install license $ bin/plugin install marvel-agent $ vi config/elasticsearch.yml cluster.name: es-monitor http.port: 9201 $ bin/elasticsearch -
.BSWFMϝτϦοΫૹ৴ઌͷมߋ 66 $ sudo vi /etc/elasticsearch/elasticsearch.yml marvel.agent.exporters: id1: type: http
host: [ "http://localhost:9201" ] $ sudo service elasticsearch restart