Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Why we built Elasticsearch

Elasticsearch Inc
December 06, 2013
1.7k

Why we built Elasticsearch

In this talk Shay Banon outlines the reason why we built Elasticsearch, and how and why it’s being used by companies like Foursquare, GitHub, Facebook, SoundCloud and many more.

Elasticsearch Inc

December 06, 2013
Tweet

Transcript

  1. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    Data Zen
    or why we built elasticsearch
    Shay Banon - @kimchy

    View full-size slide

  2. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    just 140 characters?
    Tweet

    View full-size slide

  3. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    just 140 characters?
    Tweet

    View full-size slide

  4. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    just a message?
    Log
    I’m broken. Please show this to someone who can fix can fix

    View full-size slide

  5. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    just a message?
    Log
    I’m broken. Please show this to someone who can fix can fix
    timestamp
    code location
    hostname
    ip address
    process
    parameter values
    request id
    who committed this?!!!

    View full-size slide

  6. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    just a number?
    Metric
    URL: https://download.elasticsearch.org/
    elasticsearch/elasticsearch-0.90.1.zip

    View full-size slide

  7. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    just a number?
    Metric
    URL: https://download.elasticsearch.org/
    elasticsearch/elasticsearch-0.90.1.zip
    timestamp
    remote ip
    geo location
    host name
    package
    format product

    View full-size slide

  8. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    just code?
    Code

    View full-size slide

  9. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    just a lat/lon?
    Location

    View full-size slide

  10. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    just a lat/lon?
    Location
    venue
    rating
    reviews
    name
    city
    country
    neighborhood

    View full-size slide

  11. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    and you shall be answered
    Ask
    show me the tweets that mention obama

    View full-size slide

  12. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    and you shall be answered
    Ask
    show me the tweets that mention obama
    unstructured

    View full-size slide

  13. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    and you shall be answered
    Ask
    show me the tweets that mention obama
    unstructured
    in ohio

    View full-size slide

  14. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    and you shall be answered
    Ask
    show me the tweets that mention obama
    unstructured
    in ohio
    structure

    View full-size slide

  15. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    and you shall be answered
    Ask
    show me the tweets that mention obama
    unstructured
    in ohio
    structure
    in the past month

    View full-size slide

  16. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    and you shall be answered
    Ask
    show me the tweets that mention obama
    unstructured
    in ohio
    structure
    in the past month
    moar structure

    View full-size slide

  17. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    and you shall be answered
    Ask
    show me the tweets that mention obama
    in ohio
    in the past month
    total: 255010294

    View full-size slide

  18. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    and you shall be answered
    Ask
    show me the tweets that mention obama
    in ohio
    in the past month
    broken by day
    analytics

    View full-size slide

  19. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    and you shall be answered
    Ask Anything
    show me the tweets that mention romney
    in ohio
    in the past month
    broken by day
    analytics

    View full-size slide

  20. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    and you shall be answered
    Ask Anything
    show me the tweets that mention romney
    in california
    in the past month
    broken by day
    analytics

    View full-size slide

  21. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    and you shall be answered
    Ask Anything
    show me the tweets that mention romney
    in california
    in the past year
    broken by day
    analytics

    View full-size slide

  22. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    and you shall be answered
    Ask Anything
    show me the tweets that mention romney
    in california
    in the past year
    broken by month
    analytics

    View full-size slide

  23. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    and you shall be answered
    Ask Anything
    with as little data munging as possible!

    View full-size slide

  24. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    Data Triangulation
    text
    meta data analytics

    View full-size slide

  25. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    Data Triangulation
    unstructured
    structure aggregation

    View full-size slide

  26. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    realtime is the only time
    Fresh!

    View full-size slide

  27. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    what is realtime?
    Fresh!
    how quickly can we get results?

    View full-size slide

  28. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    what is realtime?
    Fresh!
    how quickly can we get results?
    took: 70ms

    View full-size slide

  29. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    what is realtime?
    Fresh!
    how quickly can we see new data?

    View full-size slide

  30. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    what is realtime?
    Fresh!
    how quickly can we see new data?
    milliseconds!

    View full-size slide

  31. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    what is realtime?
    Fresh!
    how big is the data?

    View full-size slide

  32. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    what is realtime?
    Fresh!
    how big is the data?
    irrelevant
    (but make sure enough HW)

    View full-size slide

  33. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    collocation
    Data Fight Club

    View full-size slide

  34. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    collocation
    Data Fight Club
    the first rule of distributed systems
    collocation

    View full-size slide

  35. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    collocation
    Data Fight Club
    the second rule of distributed systems
    collocation

    View full-size slide

  36. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    collocation
    Data Fight Club
    in order to achieve data triangulation
    a system should provide all of them

    View full-size slide

  37. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    Final Words
    elasticsearch!

    View full-size slide