$30 off During Our Annual Pro Sale. View Details »

Kibana, Timelion, Graph Meetup

Elasticsearch Inc
January 18, 2016
740

Kibana, Timelion, Graph Meetup

This was the PDF used in Minneapolis on January 12th, 2016

Elasticsearch Inc

January 18, 2016
Tweet

Transcript

  1. Timelion/Graph Meetup Nathan Zamecnik

  2. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written

    permission is strictly prohibited 2 Agenda • Quick Kibana 4.2 overview • A look at Timelion • Explore the Graph Plugin
  3. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written

    permission is strictly prohibited 3 Kibana • Democratize your data • Create "Visualizations" and "Dashboards" • Slide and dice log data using Elasticsearch Aggregations • Plugins to extend functionality • Timelion • Graph UI Plugin • Marvel • Sense • Community…
  4. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written

    permission is strictly prohibited 4 Timelion • Kibana 4.2 plugin • "Do more with time series data" • Easy query language based on "chaining" functions together • Simple functions such as add and subtract as well as moving averages, cumulative sums and derivatives • Custom styling • Connects to outside data sources as well as Elasticsearch in the same query • Worldbank's Data API • More to be added - community too!
  5. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written

    permission is strictly prohibited 5 Timeline Example
  6. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written

    permission is strictly prohibited 6 Graph Plugin • Why Graph? • Find connections between terms • Suggestion engines • Related terms • Connect user queries with products & departments • Domain specific research • Why Elasticsearch graphs are different • Generated "on-the-fly" • Avoids "super nodes" • Use sampling and diversity settings to generate personalized (and useful) graphs
  7. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written

    permission is strictly prohibited 7 Graph Terms • Graph: The actual data structure returned by your query • Vertex: Represent terms in your indices • As your data set new cities will become part of your graph • Edge: The connection between vertices • These connections are generated on-the-fly and can change as your data changes • Significant Terms Aggregation • Graph API relies on this aggregation to generate edges between "strongly connected" vertices. • Avoids super nodes
  8. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written

    permission is strictly prohibited 8 Example { "vertex_fields": ["artists.raw"], "query": { "terms": { "artists.raw": [ "faith no more" ] } } } { "edges": [ { "source": "artists.raw:mr. bungle", "target": "artists.raw:peeping tom", "weight": 1.4966674877093182 }, { "source": "artists.raw:mr. bungle", "target": "artists.raw:faith no more", "weight": 6.446279632618163 }, ... ], "vertices": [ { "id": "artists.raw:peeping tom", "field": "artists.raw", "hopDepth": 1, "weight": 0.3542307820922224, "term": "peeping tom" }, { "id": "artists.raw:mr. bungle", "field": "artists.raw", "hopDepth": 1, "weight": 0.445934308316037, "term": "mr. bungle" }, ... ] } Request Response
  9. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written

    permission is strictly prohibited 9 Result Graphically Displayed
  10. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written

    permission is strictly prohibited 10 Faster & Better Results From Sampling • "Sampler Aggregation" allows results to be calculated over a particular subset - in this case users from around the world, not just the USA { "vertex_fields": ["artists.raw"], "options": { "sampleSize": 1000, "diversity": { "field": "country.raw" "maxDocsPerValue": 200 } } "query": { "terms": { "artists.raw": [ "faith no more", "peeping tom", "the beatles" ] } } }
  11. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written

    permission is strictly prohibited 11 FAQ • When? • Scheduled for 2.2 release - soon • Will Kibana have a UI plugin? • Yes, still working on something • How is this different from graph DB's? • Relevance: Graph DBs have no relevance ranking algos so always get tangled up in the super-popular nodes in networks (Twitter=Bieber, Wikipedia=US, Music=coldplay...). In search-world we are very familiar with Zipf's law and super-popular terms like "the" so know how to apply ranking. • Aggregate views: We don't need to return millions of individual docs as nodes. We can use a summary representation of the connection between 2 bank accounts as a single link and use aggs to summarize potentially millions of transactions as properties of that single link at high speed using date histogram, max, sum aggs etc