Data Streams with Elasticsearch

Datalayer: 2017 talk - datalayer.com

Why do we need data streams,
Current approaches,
Proposing an alternative: Let's build a streaming DB,
How to build one yourself,
Previewing Streams: A realtime layer for Elasticsearch,
Benchmarks and upcoming features.

Transcript

1. Data Streams with Elasticsearch - Siddharth Kothari @siddharthlatest appbase.io
2. Founder and CEO, appbase.io GET /programming/stacktrace 
 [“Games”, “C”, “Python”, “Java (sigh)”, “AI”, “JS”, “Databases”]
3. appbase.io
4. Over 4,000 stars on github github.com/appbaseio
5. Results as Streams
7. Results as Streams
8. #slack Facebook Twitch.TV WhatsApp Periscope Postmates Who Uses Streams?
9. 1. Streams and Firehoses from #IoT 2. Monitoring Systems 3. Analytics 4. E-commerce: Search, Price Monitoring 5. Fraud Detection (Cyber Security, Payments) Use Cases
10. Typical Approach
11. Typical Approach: Limitations • Middleware logic that connects DB with 
 realtime protocol like WS. Edge Cases
 • Cannot handle complex realtime scenarios.
 • Scaling of WS connections.
12. Samza Spark Flink Storm Kafka Streams Beam Approach to Complex Realtime Scenarios
13. Streaming Framework DB Layer Middleware + Realtime Protocol Scaling RP Edge Cases Monitoring A complex System Approach to Complex Realtime Scenarios
14. • Take the best parts of a DB system
 • Bake in realtime protocol
 • Make the middleware layer optional Alternative: A Streaming Database System
15. Distributed Full-text Search based on Lucene Can scale to many nodes and highly available Analytics, Document Oriented, Open Source Elasticsearch
16. aka Search in Reverse 1. Indexing a Query
 2. Matches when new documents are added
 3. Distributed design since v1.0.0 Percolation in Elasticsearch
17. M1 M2 Mn JSON Index Of Queries Index Of Queries Search Filters Aggregations
18. An Example Percolation Snippet Registering a Query
19. Match a Document Response
20. • Queries as subscriptions (HTTP Streaming / Websockets)
 • Publish matches to subscribers • Works as is with the ES API Baking Realtime
21. • Beyond Percolation, keep the doc store model of ES. • Every document is a topic (channel). • Every query to the DB also maps to a topic (channel). Data Streams: Topology
22. https://store.docker.com/images/appbaseio $ docker pull store/appbaseio/streams Live Demo: appbaseio streams
23. Built on top of Nginx. Live Demo: streams
24. • Performant, because Nginx! • Can work anywhere, because Docker! • All the good parts of existing data layers. Streams: Benefits
25. • A nifty Time To Live (ttl) feature. • Optionally, stream without storing. • Interval and frequency based queries. Streams: Features
26. Index > Store > Stream > Act AWS Lambda Another REST API Send an E-mail Design Pattern
27. Median Throughput / sec 0 32500 65000 97500 130000 m3.2xlarge nodes 3-nodes 6-nodes 9-nodes Elasticsearch Elasticsearch with Streams Benchmarks: Indexing Overhead <1% indexing overhead Dataset: 8 Million Location Docs
28. Median Throughput / sec 0 100000 200000 300000 400000 m3.2xlarge nodes 3-nodes 6-nodes 9-nodes Elasticsearch Elasticsearch with Streams <1% indexing overhead Dataset: 8 Million Location Points Benchmarks: Indexing Overhead
29. Benchmarks: More Broadcasts at <1s latency for 10,000 clients. ~20% CPU overhead at indexing time. Number of streams nodes can scale.
30. Streams: Upcoming Features Time Windowed Aggregations Webhooks
31. Data Streams with Elasticsearch - Siddharth Kothari @siddharthlatest appbase.io