‹#› Kosho Owa, Solutions Architect, Elastic Edition July-2016 Elastic Stack Hands-on Workshop

Elastic’s Product Portfolio

3 Elastic Cloud Security X-Pack Kibana User Interface Elasticsearch Store, Index,
 & Analyze Ingest Logstash Beats + Introducing the Elastic Stack, X-Pack, and Cloud Alerting Monitoring Reporting Graph &MBTUJD4UBDL

4 Store, Index, and Analyze • Resilient; designed for scale-out • High availability; multitenancy • Structured & unstructured data Distributed & Scalable Developer Friendly Search & 
 Analytics • Schemaless • Native JSON • Client libraries • Apache Lucene • Real-time • Full-text search • Aggregations • Geospatial • Multilingual

5 Visualize and Explore • Explore and analyze patterns in data; drill down to any level • Leverage powerful analytical capabilities in Elasticsearch Discover Insights Customize & Share Window into Elastic Stack • Create bar charts, line and scatter plots, maps and histograms • Share and embed dashboards into operational workflows • Unified user interface for data visualization • Administration and management for the Elastic Stack • Pluggable architecture to create custom visualizations and applications

6 Ingest • Data collection and enrichment; 200+ plugins • Next generation data pipeline; micro-batches, process groups of events ES-Hadoop • Platform to build lightweight, data shippers • Forward host-based metrics and any data to Elasticsearch • Two-way connector to integrate with HDFS, Spark, MapReduce, etc. • Enable real-time search queries on Hadoop data

7 Security for the Elastic Stack (Shield) Security Monitoring for the Elastic Stack (Marvel) Monitoring Notifications for the Elastic Stack (Watcher) Alerting Security X-Pack Alerting Monitoring Reporting Graph Automated reporting for the Elastic Stack Reporting Real-time graph analytics for the Elastic Stack Graph A Single Extension

8 Simply Secure the Elastic Stack • Username/password protection Advanced Security When Needed • LDAP/AD integration • Role-based access control • IP filtering • Field and document level security • Encrypted communications • Audit logging • Kibana plugin for login and session management Security (Shield) External Authentication (optional)

9 Setup Alerts • Create Watches based on data • Trigger automatic notifications • Setup chained inputs Notify and Integrate • Slack, Hipchat, JIRA, Pagerduty • Email • Elastic Monitoring (Marvel) • Other Alerting (Watcher)

10 Monitor Elasticsearch • Real-time statistics and metrics for all clusters and nodes Diagnose Issues • Analyze historical or real-time data for root cause analyses Optimize Performance • Utilize in-depth analyses to improve cluster performance Monitoring (Marvel)

11 Query and Visualize Relationships • Use relevance as a guide to uncover and explore new relationships in all your data stored in Elasticsearch • Interact with Graph via a Kibana plugin or use the Graph API to integrate with your applications • Enable new use cases – behavioral analysis, fraud, cybersecurity, drug discovery, and recommendations Graph Analytics

12 Generate and share reports • Export PDF’s of dashboards and visualizations with a click • Use alerting features to email reports ‒ Time-based (weekly) ‒ Event-based (when X happens, send me a picture of the dashboard) • Export to CSV Reporting

13 The only Elasticsearch as a Service offering powered by the creators of the Elastic Stack • Always runs on the latest software • One-click to scale/upgrade with no downtime • Free Kibana and backups every 30 minutes • Dedicated, SLA-based support • Easily add X-Pack features: security (Shield), alerting (Watcher), and monitoring (Marvel) • Pricing starts at $45 a month Hosted Elasticsearch Search Analytics Logging

14 Elastic Drives Revenue & Cost Savings Value Data Complex/Diverse Meets Developer Requirements Use Cases Many users/use cases Real-Time Availability Rapid Query Execution Flexible Data Model Schemaless Horizontal Scale Sophisticated Query Language Value/Impact Short/mid/long term Location Machine/Log Files User-Activity Documents Social Revenue Growth Launch new applications, Monetize services; Personalize user experiences Cost Savings/Risk Mgmt Application Search Embedded Search Logging Security Analytics Operational Analytics Metrics Analytics Next generation architecture; Retool existing systems; manage risk and compliance

elastic.co github.com/elastic 15 Learning Resources

downloads Latest products, alpha and installation instructions https://www.elastic.co/downloads

Learn > Docs Guides and References https://www.elastic.co/guide/index.html

Learn > Blog Product releases, tutorials and user stories https://www.elastic.co/blog

Learn > Videos & Webinars How-to’s and success stories https://www.elastic.co/jp/videos

GitHub Source code, issues and pull requests https://github.com/elastic

21 Hands-on Lab

Prerequisites • A virtual machine - VMWare - VirtualBox - Amazon EC2 • Operating system - Redhat Enterprise Linux 6 - CentOS 6.x - Amazon Linux AMI 2016.03.1 • Memory assignment - 4GB or higher recommended • Support Matrix - https://www.elastic.co/support/matrix • Network - Internet connection - Allow incoming 9200/tcp and 5601/tcp traffic • Javan runtime installed - Oracle Java SE 1.7 or later - OpenJDK 1.7 or later

Lab Installing Elasticsearch Installing Topbeat, Filebeat Installing Kibana Verifying data from Topbeat, Filebeat Working with Apache access log CRUD and Search Setting up separate monitoring cluster

Installing Elasticsearch 24 Lab 1

Lab 1: Exercise Steps 1. Install Elasticsearch via RPM 2. Install Marvel plugin 3. Register and run Elasticsearch as service 4. Verify 5. Locate data directory • Downloads > Elasticsearch - https://www.elastic.co/downloads/elasticsearch • Downloads > Marvel - https://www.elastic.co/downloads/marvel

1/3: Install Elasticsearch via RPM $ sudo rpm -i https://download.elastic.co/elasticsearch/release/org/elasticsearch/ distribution/rpm/elasticsearch/2.3.2/elasticsearch-2.3.2.rpm Creating elasticsearch group... OK Creating elasticsearch user... OK ### NOT starting on installation, please execute the following statements to configure elasticsearch service to start automatically using chkconfig $ sudo chkconfig --add elasticsearch ### You can start elasticsearch service by executing

2/3: Install Plugins $ cd /usr/share/elasticsearch/ $ sudo bin/plugin install license -> Installing license... $ sudo bin/plugin install marvel-agent -> Installing marvel-agent... @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: plugin requires additional permissions @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ * java.lang.RuntimePermission setFactory * javax.net.ssl.SSLPermission setHostnameVerifier See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html for descriptions of what these permissions allow and the associated risks. Continue with installation? [y/N]y • Docs > Marvel Documentation > Installing Marvel > Installing Marvel on Offline Machines - https:// www.elastic.co/guide/en/marvel/current/installing-marvel.html#offline-installation

3/3: Run Elasticsearch as service $ sudo service elasticsearch start $ curl localhost:9200 { "name" : "Juggernaut", "cluster_name" : "elasticsearch", "version" : { "number" : "2.3.2", "build_hash" : "b9e4a6acad4008027e4038f6abed7f7dba346f94", "build_timestamp" : "2016-04-21T16:03:47Z", "build_snapshot" : false, "lucene_version" : "5.5.0" }, "tagline" : "You Know, for Search" } $ cd /var/lib/elasticsearch $ ls elasticsearch

Installing Topbeat, Filebeat 29 Lab 2

Lab 2: Exercise Steps 1. Install Topbeat, Filebeat 2. Configure Filebeat to retrieve logs under /var/log/* 3. Configure Topbeat to send metrics to Elasticsearch 4. Launch • Downloads | Topbeat - https://www.elastic.co/downloads/beats/topbeat • Downloads | Filbeat - https://www.elastic.co/downloads/beats/filebeat

1/2: Install and run Topbeat $ sudo rpm -i https://download.elastic.co/beats/topbeat/topbeat-1.2.2-x86_64.rpm $ grep hosts /etc/topbeat/topbeat.yml hosts: [“localhost:9200"] $ curl -XPUT 'http://localhost:9200/_template/topbeat' -d@/etc/topbeat/topbeat.template.json {"acknowledged":true} $ sudo service topbeat start Starting topbeat: [ OK ]

2/2: Install and run Filebeat $ sudo rpm -i https://download.elastic.co/beats/filebeat/filebeat-1.2.2-x86_64.rpm $ less /etc/filebeat/filebeat.yml … paths: - /var/log/*.log … hosts: ["localhost:9200"] … $ curl -XPUT 'http://localhost:9200/_template/filebeat' -d@/etc/filebeat/ filebeat.template.json {"acknowledged":true} $ sudo service filebeat start Starting filebeat: [ OK ]

Installing Kibana 33 Lab 3

Lab 3: Exercise Steps 1. Install Kibana 2. Install Marvel, Sense plugin 3. Launch • Downloads | Kibana - https://www.elastic.co/downloads/kibana • Downloads | Marvel - https://www.elastic.co/downloads/marvel • Sense Documentation » Installing Sense - https://www.elastic.co/guide/en/sense/current/installing.html

1/2: Install Kibana $ sudo rpm -i https://download.elastic.co/kibana/kibana/kibana-4.5.0-1.x86_64.rpm $ cd /opt/kibana $ sudo bin/kibana plugin --install elasticsearch/marvel Installing marvel … Plugin installation complete $ sudo bin/kibana plugin --install elastic/sense Installing sense … Plugin installation complete $ sudo chown -R kibana:root /opt/kibana/optimize $ sudo service kibana start kibana started

2/2: Verify Kibana is running Click on Plug-in chooser 4FMFDU,JCBOB

Verifying data from Topbeat, Filebeat 37 Lab 4

Lab 4: Exercise Steps 1. Open Kibana 2. Add Index “filebeat-*”,” topbeat-*” from Settings tab 3. Verify Filebeat and Topbeat documents from Discover tab

1/5: Add Filebeat index pattern Filebeat index name Specify the date part as wildcard character Click on “Create” button

2/5: Verify logs sent by FIlebeat

3/5: Add Topbeat index pattern Topbeat index name Specify the date part as wildcard character Click on “Create” button

4/5: Verify metrics sent by Topbeat Specify time window Zoom-in with click’n drag

5/5: List indices GET _cat/indices yellow open megacorp 5 1 3 0 11.9kb 11.9kb yellow open .marvel-es-1-2016.04.30 1 1 6783 0 2.9mb 2.9mb yellow open topbeat-2016.04.30 5 1 24399 0 5.4mb 5.4mb yellow open filebeat-2016.04.30 5 1 1290 0 354.8kb 354.8kb yellow open .marvel-es-data-1 1 1 3 1 8.1kb 8.1kb yellow open .kibana 1 1 4 0 32.2kb 32.2kb

Working with Apache access logs 44 Lab 5

Lab 5: Exercise Steps 1. Import Apache access logs to your cluster by following https://github.com/ elastic/examples/tree/master/ElasticStack_apache 2. Open imported Dashboard 3. Show the top 10 frequently accessed path as an optional challenge • Kibana User Guide [4.5] » Visualize » Data Table - https://www.elastic.co/guide/en/kibana/current/data- table.html

1/8: Install Logstash and download necessary files $ sudo rpm -i https://download.elastic.co/logstash/logstash/packages/centos/ logstash-2.3.2-1.noarch.rpm $ mkdir Apache_ELK_Example $ cd Apache_ELK_Example $ wget https://raw.githubusercontent.com/elastic/examples/master/ElasticStack_apache/ apache_logstash.conf $ wget https://raw.githubusercontent.com/elastic/examples/master/ElasticStack_apache/ apache_template.json $ wget https://raw.githubusercontent.com/elastic/examples/master/ElasticStack_apache/ apache_kibana.json $ wget https://raw.githubusercontent.com/elastic/examples/master/ElasticStack_apache/

2/8: Import logs $ cat apache_logs | /opt/logstash/bin/logstash -f apache_logstash.conf Settings: Default pipeline workers: 2 Pipeline main started Pipeline main has been shutdown stopping pipeline {:id=>"main"} GET /apache_elk_example/_count?pretty { "count" : 10000, "_shards" : { "total" : 5, "successful" : 5, "failed" : 0 } }

3/8: Configure Logstash - input, filter input { stdin { } } filter { grok { match => { "message" => '%{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] "%{WORD:verb} %{DATA:reques t} HTTP/%{NUMBER:httpversion}" %{NUMBER:response:int} (?:-|%{NUMBER:bytes:int}) %{QS:referrer} %{QS:agent}’ # Extract to fields } } date { # Specify timestamp match => [ "timestamp", "dd/MMM/YYYY:HH:mm:ss Z" ] locale => en } geoip { source => "clientip" } # Retrieve GIS from IP address useragent { # Analyze user-agent source => "agent" target => "useragent" } }

4/8: Configure Logstash - output output { stdout { codec => plain } # Standard output elasticsearch { hosts => “http://localhost:9200” # Elasticsearch node index => “apache_elk_example" # Index to be ingested template => “./apache_template.json" # Index settings and field type mappings template_name => “apache_elk_example" # Name of the template to be saved template_overwrite => true } }

5/8: Create index pattern Specify the index pattern Click on “Create”

6/8: Import dashboard object Click on “Settings” Click on “Objects” Select apache_kibana.json

7/8: Open Dashboard Click on “Dashboard” Open saved dashboard Select

8/8: Top 10 access path Go to Visualize > Data table, select apache_elk_example from From a new search Select “Terms” Select “request.raw” Show top 10

CRUD and Search 54 Lab 6

Lab 6: Exercise Steps 1. Use Sense to create employee 1, 2, 3 https://www.elastic.co/guide/en/elasticsearch/guide/current/_indexing_employee_documents.html 2. Read documents 3. Search document 4. Update and delete documents • Elasticsearch: The Definitive Guide [2.x] » Getting Started » You Know, for Search… » Retrieving a Document - https://www.elastic.co/guide/en/elasticsearch/guide/current/_retrieving_a_document.html • Elasticsearch: The Definitive Guide [2.x] » Getting Started » You Know, for Search… » Search Lite - https://www.elastic.co/guide/en/elasticsearch/guide/current/_search_lite.html • Elasticsearch: The Definitive Guide [2.x] » Getting Started » Data In, Data Out » Updating a Whole Document - https://www.elastic.co/guide/en/elasticsearch/guide/current/update-doc.html • Elasticsearch: The Definitive Guide [2.x] » Getting Started » Data In, Data Out » Deleting a Document - https://www.elastic.co/guide/en/elasticsearch/guide/current/delete-doc.html

1/5: Sense Select “Sense” from plugin chooser Type in the request then, Command + Return Verify the response

2/5: Create documents PUT /megacorp/employee/1 { "first_name" : "John", "last_name" : "Smith", "age" : 25, "about" : "I love to go rock climbing", "interests": [ "sports", "music" ] } { "_index": "megacorp", "_type": "employee", "_id": "1", "_version": 1, "_shards": { "total": 2, "successful": 1, "failed": 0 }, "created": true } Request Create employee/2, 3 Response

3/5: Read Document GET/megacorp/employee/1 { "_index": "megacorp", "_type": "employee", "_id": "1", "_version": 1, "found": true, "_source": { "first_name": "John", "last_name": "Smith", "age": 25, "about": "I love to go rock climbing", "interests": [ "sports", "music" ] } }

4/5: Search Documents GET /megacorp/employee/_search?q=last_name:Smith { "took": 30, "timed_out": false, "_shards": { "total": 5, "successful": 5, "failed": 0 }, "hits": { "total": 2, "max_score": 0.30685282, "hits": [ { "_index": "megacorp", "_type": "employee", "_id": "2", "_score": 0.30685282, "_source": { "first_name": "Jane",

5/5: Update Documents PUT /megacorp/employee/3 { "first_name" : "Richard", "last_name" : "Roe" } { "_index": "megacorp", "_type": "employee", "_id": "3", "_version": 2, "_shards": { "total": 2, "successful": 1, "failed": 0 }, "created": false } GET /megacorp/employee/3 { "_index": "megacorp", "_type": "employee", "_id": "3", "_version": 2, "found": true, "_source": { "first_name": "Richard", "last_name": "Roe" } } Entire document is updated Use _update API for partial update

Setting up separate monitoring cluster 61 Lab 7

Deployment Model 62 Cluster “elasticsearch” ES node marvel-agent Monitoring Cluster “es-monitor” ES node marvel-agent Kibana marvel-ui # config/elasticsearch.yml marvel.agent.exporters: id1: type: http host: [“es-mon-1:9200”,…]

Lab 7: Exercise Steps • Marvel Documentation > Installing Marvel > Setting up a Separate Monitoring Cluster: https:// www.elastic.co/guide/en/marvel/current/installing-marvel.html#monitoring-cluster • Downloads | Elasticsearch - https://www.elastic.co/downloads/elasticsearch • Downloads | Marvel - https://www.elastic.co/downloads/marvel • Downloads | Kibana - https://www.elastic.co/downloads/kibana 1. Set up “elasticsearch” cluster ‒ Install single or multiple node elasticsearch cluster with cluster name “elasticsearch” ‒ Configure exporting marvel metrics to “es-monitor” cluster 2. Set up “es-monitor” cluster and Kibana ‒ Install another Elasticsearch cluster with cluster name “es-monitor” ‒ Install Kibana instance which connects to “es-monitor” cluster 3. Verify ‒ Open Kibana with a web browser and goto Marvel app, make sure two clusters appear on the Clusters screen

1/4: Setup “es-monitor” Cluster 64 $ cd $ mkdir es-monitor $ cd es-monitor $ curl https://download.elastic.co/elasticsearch/release/org/elasticsearch/distribution/tar/ elasticsearch/2.3.2/elasticsearch-2.3.2.tar.gz | tar zxf - $ cd elasticsearch-2.3.2 $ bin/plugin install license $ bin/plugin install marvel-agent $ vi config/elasticsearch.yml cluster.name: es-monitor http.port: 9201 $ bin/elasticsearch

2/4: Configure Kibana 65 $ sudo vi /opt/kibana/config/kibana.yml elasticsearch.url: "http://localhost:9201" $ sudo service kibana restart

3/4 Send metrics to “es-monitor” cluster 66 $ sudo vi /etc/elasticsearch/elasticsearch.yml marvel.agent.exporters: id1: type: http host: [ "http://localhost:9201" ] $ sudo service elasticsearch restart

4/4: Verify Your Clusters Make sure two clusters appear Select Marvel