Getting to Kibana 4 ELK Stack Upgrading Tips

This is the presentation I gave at Elastic User Group LA Meetup on April, 22 2015.

Traversing the migration path from Kibana 3 to Kibana 4 can be tricky, challenging and a little frustrating, but without a doubt it's worth it. The goal of this presentation will be to review the challenges, provide tips and walk through a migration from a Kibana 3 environment to Kibana 4.

Transcript

1. GETTING TO KIBANA 4 ELK STACK UPGRADING TIPS AND MORE RICH HORACE ELASTIC USER GROUP LA ORGANIZER 4/22/2015
2. ABOUT ME Lead DevOps at Nativo, Inc •  a native advertising startup in L A • We’re Hiring!!!! 10 years in online advertising space and early-stage startups Working with log aggregation systems since 2008. •  Starting off with Splunk •  then Logstash in 2012 •  Splunk again in 2013 •  back the the ELK Stack in 2014.
3. BACKGROUND On my 2013 hiatus away a few things happened •  Elasticsearch.com was formed •  Bringing the ELK Stack under one roof •  Elasticsearch v1.0 Release •  Major feature: Aggregations •  Major challenge: Migration from v .90 to v1.0 •  Logstash •  Solved manual mapping with Elasticsearch •  More plugins (Input – Filters – Output) •  Kibana 3 •  What still missing? •  Providing sample data from Webinar and Demos
4. AGENDA Walkthrough Goals Overview ELK Stack Elastic’s Demo using NYC Traffic Incidents Review Environment Upgrade Environment Summary of Tips
5. WALKTHROUGH GOALS Start with complete ELK Stack created by Elastic •  NYC Traffic Incidents Demo •  Test environment with sample data •  Elasticsearch v1.3.5 •  Logstash v 1.4.2 •  Kibana 3 End with complete ELK Stack •  Elasticsearch v1.5.1 •  Logstash v 1.4.1 •  Kibana 3 and v 4.0.2 running in parallel
6. ELASTICSEARCH - OVERVIEW Store, search, analyze JSON Document-oriented search engine Scheme Free / Schema-Less Built on top of Apache Lucene API centric & RESTful Aggregations: Powerful Analytics
7. LOGSTASH - OVERVIEW Managing event and logs Collect, parse and enrich data Modular: 160+ input, filters and outputs JRuby Like an ETL Swiss Army Knife
8. KIBANA - OVERVIEW Kibana 4 •  Game Changer – Bring ELK Stack to Next Level •  Total Rewrite •  Create Intuitive Flow to Explore Data •  D3.js •  Node.js •  Aggregation •  Dashboards can have multi-indexes
9. NYC TRAFFIC INCIDENTS DEMO Created by Elastic Self Contained ELK Environment •  Vagrant with Virtual Box •  Elasticsearch Snapshots My Tweak to simplify •  Shell Scripts instead of Puppet •  Apache instead of Nginx
10. SPIN UP VAGRANT BOX Review Configuration: •  Environment •  Installs openjdk-7 and Apache •  Elasticsearch •  Install using deb package •  Install Elasticsearch Head plugin •  Set Snapshot and Restore •  Kibana 3 •  Unzip and move to /var/www/html •  Logstash •  Install using deb package
11. CHECK INSTALLATION Elasticsearch •  Status: http://localhost:9200/ •  Head Plugin: http://localhost:9200/_plugin/head/ •  Play with the Cat: http://localhost:9200/_cat/ Kibana 3 •  Kibana Intro Dashboard: http://localhost:8080/kibana Logstash •  Installed and available •  But not used is this demo
12. REVIEW SAMPLE DATA NYC Accidents v.2 Dashboard: http://localhost:8080/kibana/#/dashboard/elasticsearch/ NYC%20Accidents%20v.2
13. TIME TO UPGRADE Follow upgrade Instructions in •  /vagrant/provisioning/files/upgrades/kibana4-upgrade-steps.txt 1.  vagrant ssh 2.  Install Kibana 4 3.  Rolling Upgrade to Elasticsearch 1.5 •  (Following process as if multi-node cluster) 4.  Check Kibana 3 5.  Update Elasticsearch.yml 6.  Check Kibana 3
14. KIBANA 4 SETTINGS TAB •  http://localhost:5601/ •  Create Configure an index pattern •  Index name or pattern: •  nyc_visionzero •  Time-field name •  @timestamp •  Follow Kibana 4 Visualization Creation Steps •  /vagrant/provisioning/files/upgrades/kibana4- visualization-creation-steps
15. KIBANA 4 VISUALIZE TAB – EXAMPLE 1 •  First, gather info from Kibana 3 People Killed Row •  Fields: •  number_of_persons_killed •  number_of_pedestrians_killed •  number_of_motorist_killed •  number_of_cyclist_killed
16. KIBANA 4 VISUALIZE TAB – EXAMPLE 1 •  Create New Visual: METRIC •  New Search •  Metric-Aggregation: Sum •  Metric-Field: number_of_persons_killed •  Repeat for each number killed: •  number_of_pedestrians_killed •  number_of_motorist_killed •  number_of_cyclist_killed
17. KIBANA 4 VISUALIZE TAB – EXAMPLE 1
18. KIBANA 4 VISUALIZE TAB – EXAMPLE 2 •  First, gather info from Kibana 3 Reasons for Death •  Fields: •  contributing_factor_vehicle
19. KIBANA 4 VISUALIZE TAB – EXAMPLE 2 •  Create New Visual: Pie Chart •  New Search •  Add Aggregation •  metrics-Slice Size: Count •  bucket-Split Slices-Aggregation: Terms •  bucket-Split Slices-Field: contributing_factor_vehicle •  bucket-Split Slices-Order: Top Size: 5 •  bucket-Split Slices-Order by : metric: Count
20. KIBANA 4 VISUALIZE TAB – EXAMPLE 2
21. KIBANA 4 VISUALIZE TAB – EXAMPLE 2 Advanced-Exclude: Unspecified
22. KIBANA 4 VISUALIZE •  Build the remaining visuals •  Determine why People Injured numbers are not matching up with Kibana 3 and Kibana 4 •  Learn how to Longitude and Latitude need to be converted for Kibana 4
23. LOGSTASH TIPS •  Be careful conf files in /etc/logstash/conf.d •  File Input Plugin •  sincedb_path => "/opt/logstash/.sincedb-myfile" •  Redis Output Plugin for Redundancy •  Add Multiple Hosts (host => [‘host1’, ‘host2’]) •  Shuffle Hosts (shuffle_hosts => true) •  Elasticsearch Output Plugin •  Use HTTP (protocol => ‘http’)
24. ELASTICSEARCH TIPS Elasticsearch.yml •  bootstrap.mlockall: true •  discovery.zen.minimum_master_nodes: 2 •  discovery.zen.ping.multicast.enabled: false If in AWS use AWS Autodiscovery: •  cloud.aws.access_key: XXXXXXXXXXXXXXXXXXXX •  cloud.aws.secret_key: XXXXXXXXXXXXXXXXXXXX •  cloud.aws.region: us-xxxx-# •  discovery.type: ec2 •  discovery.ec2.groups: "my_es_cluster"
25. KIBANA TIPS Play with Kibana 4 Run Kibana 3 and Kibana 4 in parallel HAVE FUN!!!!
26. RESOURCES FOR NYC DATA https://www.elastic.co/blog/byodemos-new-york-city- traffic-incidents/ https://www.elastic.co/blog/intro-to-aggregations/ https://www.elastic.co/blog/intro-to-aggregations-pt-2-sub- aggregations https://github.com/elastic/demo
27. KIBANA 4 RESOURCES https://www.elastic.co/blog/kibana-4-video-tutorials-part-1 https://www.elastic.co/blog/kibana-4-video-tutorials-part-2 https://www.elastic.co/blog/kibana-4-video-tutorials-part-3 https://www.elastic.co/blog/elasticon-sessions-are-online https://www.elastic.co/blog/kurrently-kibana-2015-03-26
28. ADD’L KIBANA 4 RESOURCES Most comprehensive blog post that I’ve found. https://www.timroes.de/2015/02/07/kibana-4-tutorial- part-1-introduction https://www.timroes.de/2015/02/07/kibana-4-tutorial- part-2-discover/ https://www.timroes.de/2015/02/07/kibana-4-tutorial- part-3-visualize/ https://www.timroes.de/2015/02/07/kibana-4-tutorial- part-4-dashboard/
29. SUMMARY Started with Self-Contain ELK Stack with Kibana 3 Upgraded ELK Stack to support Kibana 4 Completed Steps for Rolling Elasticsearch Upgrade Added Requirements for Kibana 3 on for ES v1.5.x Configured Kibana 4 Started Exploring data!!!! Review Additional Resource
30. PRESENTATION MARTIALS Github: https://github.com/richhorace/ELKStack-Getting-to-Kibana-4 Speaker Deck: https://speakerdeck.com/richhorace/getting-to-kibana-4-elk-stack- upgrading-tips
31. NATIVO OPEN POSITIONS Publisher Account Coordinator Account Manager Python Engineer Big Data Engineer Content Strategy Intern Marketing Intern https://ciedigitallabs.bamboohr.com/jobs/ jmeza@ciedigital.com