Timelion/Graph Meetup Nathan Zamecnik

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 2 Agenda • Quick Kibana 4.2 overview • A look at Timelion • Explore the Graph Plugin

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 3 Kibana • Democratize your data • Create "Visualizations" and "Dashboards" • Slide and dice log data using Elasticsearch Aggregations • Plugins to extend functionality • Timelion • Graph UI Plugin • Marvel • Sense • Community…

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 4 Timelion • Kibana 4.2 plugin • "Do more with time series data" • Easy query language based on "chaining" functions together • Simple functions such as add and subtract as well as moving averages, cumulative sums and derivatives • Custom styling • Connects to outside data sources as well as Elasticsearch in the same query • Worldbank's Data API • More to be added - community too!

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 5 Timeline Example

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 6 Graph Plugin • Why Graph? • Find connections between terms • Suggestion engines • Related terms • Connect user queries with products & departments • Domain specific research • Why Elasticsearch graphs are different • Generated "on-the-fly" • Avoids "super nodes" • Use sampling and diversity settings to generate personalized (and useful) graphs

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 7 Graph Terms • Graph: The actual data structure returned by your query • Vertex: Represent terms in your indices • As your data set new cities will become part of your graph • Edge: The connection between vertices • These connections are generated on-the-fly and can change as your data changes • Significant Terms Aggregation • Graph API relies on this aggregation to generate edges between "strongly connected" vertices. • Avoids super nodes

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 8 Example { "vertex_fields": ["artists.raw"], "query": { "terms": { "artists.raw": [ "faith no more" ] } } } { "edges": [ { "source": "artists.raw:mr. bungle", "target": "artists.raw:peeping tom", "weight": 1.4966674877093182 }, { "source": "artists.raw:mr. bungle", "target": "artists.raw:faith no more", "weight": 6.446279632618163 }, ... ], "vertices": [ { "id": "artists.raw:peeping tom", "field": "artists.raw", "hopDepth": 1, "weight": 0.3542307820922224, "term": "peeping tom" }, { "id": "artists.raw:mr. bungle", "field": "artists.raw", "hopDepth": 1, "weight": 0.445934308316037, "term": "mr. bungle" }, ... ] } Request Response

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 9 Result Graphically Displayed

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 10 Faster & Better Results From Sampling • "Sampler Aggregation" allows results to be calculated over a particular subset - in this case users from around the world, not just the USA { "vertex_fields": ["artists.raw"], "options": { "sampleSize": 1000, "diversity": { "field": "country.raw" "maxDocsPerValue": 200 } } "query": { "terms": { "artists.raw": [ "faith no more", "peeping tom", "the beatles" ] } } }

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 11 FAQ • When? • Scheduled for 2.2 release - soon • Will Kibana have a UI plugin? • Yes, still working on something • How is this different from graph DB's? • Relevance: Graph DBs have no relevance ranking algos so always get tangled up in the super-popular nodes in networks (Twitter=Bieber, Wikipedia=US, Music=coldplay...). In search-world we are very familiar with Zipf's law and super-popular terms like "the" so know how to apply ranking. • Aggregate views: We don't need to return millions of individual docs as nodes. We can use a summary representation of the connection between 2 bank accounts as a single link and use aggs to summarize potentially millions of transactions as properties of that single link at high speed using date histogram, max, sum aggs etc