$30 off During Our Annual Pro Sale. View Details »

Kibana, Timelion, Graph Meetup

Elasticsearch Inc
January 18, 2016
750

Kibana, Timelion, Graph Meetup

This was the PDF used in Minneapolis on January 12th, 2016

Elasticsearch Inc

January 18, 2016
Tweet

Transcript

  1. Timelion/Graph
    Meetup
    Nathan Zamecnik

    View Slide

  2. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    2
    Agenda
    • Quick Kibana 4.2 overview
    • A look at Timelion
    • Explore the Graph Plugin

    View Slide

  3. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    3
    Kibana
    • Democratize your data
    • Create "Visualizations" and "Dashboards"
    • Slide and dice log data using Elasticsearch Aggregations
    • Plugins to extend functionality
    • Timelion
    • Graph UI Plugin
    • Marvel
    • Sense
    • Community…

    View Slide

  4. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    4
    Timelion
    • Kibana 4.2 plugin
    • "Do more with time series data"
    • Easy query language based on "chaining" functions
    together
    • Simple functions such as add and subtract as well as
    moving averages, cumulative sums and derivatives
    • Custom styling
    • Connects to outside data sources as well as Elasticsearch
    in the same query
    • Worldbank's Data API
    • More to be added - community too!

    View Slide

  5. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    5
    Timeline Example

    View Slide

  6. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    6
    Graph Plugin
    • Why Graph?
    • Find connections between terms
    • Suggestion engines
    • Related terms
    • Connect user queries with products & departments
    • Domain specific research
    • Why Elasticsearch graphs are different
    • Generated "on-the-fly"
    • Avoids "super nodes"
    • Use sampling and diversity settings to generate
    personalized (and useful) graphs

    View Slide

  7. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    7
    Graph Terms
    • Graph: The actual data structure returned by your query
    • Vertex: Represent terms in your indices
    • As your data set new cities will become part of your
    graph
    • Edge: The connection between vertices
    • These connections are generated on-the-fly and can
    change as your data changes
    • Significant Terms Aggregation
    • Graph API relies on this aggregation to generate edges
    between "strongly connected" vertices.
    • Avoids super nodes

    View Slide

  8. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    8
    Example
    {
    "vertex_fields": ["artists.raw"],
    "query": {
    "terms": {
    "artists.raw": [
    "faith no more"
    ]
    }
    }
    }
    {
    "edges": [
    {
    "source": "artists.raw:mr. bungle",
    "target": "artists.raw:peeping tom",
    "weight": 1.4966674877093182
    },
    {
    "source": "artists.raw:mr. bungle",
    "target": "artists.raw:faith no more",
    "weight": 6.446279632618163
    }, ...
    ],
    "vertices": [
    {
    "id": "artists.raw:peeping tom",
    "field": "artists.raw",
    "hopDepth": 1,
    "weight": 0.3542307820922224,
    "term": "peeping tom"
    },
    {
    "id": "artists.raw:mr. bungle",
    "field": "artists.raw",
    "hopDepth": 1,
    "weight": 0.445934308316037,
    "term": "mr. bungle"
    }, ...
    ]
    }
    Request Response

    View Slide

  9. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    9
    Result Graphically Displayed

    View Slide

  10. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    10
    Faster & Better Results From Sampling
    • "Sampler Aggregation" allows results to be calculated
    over a particular subset - in this case users from around
    the world, not just the USA
    {
    "vertex_fields": ["artists.raw"],
    "options": {
    "sampleSize": 1000,
    "diversity": {
    "field": "country.raw"
    "maxDocsPerValue": 200
    }
    }
    "query": {
    "terms": {
    "artists.raw": [
    "faith no more", "peeping tom", "the beatles"
    ]
    }
    }
    }

    View Slide

  11. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    11
    FAQ
    • When?
    • Scheduled for 2.2 release - soon
    • Will Kibana have a UI plugin?
    • Yes, still working on something
    • How is this different from graph DB's?
    • Relevance: Graph DBs have no relevance ranking algos so always get
    tangled up in the super-popular nodes in networks (Twitter=Bieber,
    Wikipedia=US, Music=coldplay...). In search-world we are very familiar with
    Zipf's law and super-popular terms like "the" so know how to apply ranking.
    • Aggregate views: We don't need to return millions of individual docs as
    nodes. We can use a summary representation of the connection between 2
    bank accounts as a single link and use aggs to summarize potentially millions
    of transactions as properties of that single link at high speed using date
    histogram, max, sum aggs etc

    View Slide