Data Streams on ElasticSearch

Midwest.io '15 talk on #datastreams #realtimeAPIs #ElasticSearch

Transcript

1. Data Streams on ElasticSearch - @siddharthlatest
3. We have a problem… Databases have been around for over half a century. Indexes, Caches, Distributed, Materialized Views, Stored Procedures, Aggregations
4. We have a problem… Databases have been around for over half a century. Indexes, Caches, Distributed, Materialized Views, Stored Procedures, Aggregations Not Unix Philosophy!
5. NoSQL: Webscale Movement - 2000s Static Webpages - 1990
6. NoSQL: Webscale Movement - 2000s Static Webpages - 1990 #RealtimeWeb - Today
7. #blackfriday #SxSW2015 #worldcup #realtimeweb
9. Topics: 1. What is the point of “data streams”? 2. ElasticSearch as the Query Layer 3. Data Streams in Action 4. Seriously, where would I use this 5. Scale 6. The Next 5 Years
14. Visualizing Percolation X
17. Topics: 1. What is the point of “data streams”? 2. ElasticSearch as the Query Layer 3. Data Streams in Action 4. Seriously, where would I use this 5. Scale 6. The Next 5 Years
18. Appbase.io: Data Streams + ElasticSearch
19. create a continuous query interface
20. Tweet with hashtag -> “#midwestio” https://github.com/siddharthlatest/twitter-stream-midwestio
21. Tweet with hashtag -> “#midwestio” client.stream('statuses/filter', {track: 'midwestio', language: 'en'}, function(stream) { stream.on('data', function(tweet) { appbaseRef.index({ "type": "onlymidwestio", "body": tweet }); }); }); Indexes incoming streams from twitter.
22. Live Tweets Browser
23. var response = appbase.searchStream({ type: "onlymidwestio", body: {query: {match_all: {}}} }); response.on('data', function(res) { var div = document.getElementById('tweetshero'); div.innerHTML = res._source.text + "<br><br>" + div.innerHTML; }); Live MidwestIO Tweets - http://midwest.appbase.io
24. appbaseio.github.io/meetupblast
25. Topics: 1. What is the point of “data streams”? 2. ElasticSearch as the Query Layer 3. Data Streams in Action 4. Seriously, where would I use this 5. Scale 6. The Next 5 Years
27. Game Leaderboards, Price Surge Newsfeeds, Geoquerying Systems
28. Topics: 1. What is the point of “data streams”? 2. ElasticSearch as the Query Layer 3. Data Streams in Action 4. Seriously, where would I use this 5. Scale 6. The Next 5 Years
29. Game Leaderboards, Price Surge
30. Game Leaderboards, Price Surge
31. Scaling websockets, http-streams with Pushpin https://github.com/fanout/pushpin Benchmarked to handle 100k simultaneous clients, with ~500ms avg. latency.
32. Topics: 1. What is the point of “data streams”? 2. ElasticSearch as the Query Layer 3. Data Streams in Action 4. Seriously, where would I use this 5. Scale 6. The Next 5 Years
33. Next 5 Years Native Stream support in all major languages.
34. Next 5 Years Native Stream support in all major languages. Data Streams as an interface for every DB.
35. Next 5 Years Native Stream support in all major languages. Data Streams as an interface for every DB. Data Streams for processing all computations, la Dataflow model.