Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Building a streaming database service
Search
Siddharth Kothari
May 21, 2015
Technology
1
220
Building a streaming database service
Streaming Queries with ElasticSearch
Siddharth Kothari
May 21, 2015
Tweet
Share
More Decks by Siddharth Kothari
See All by Siddharth Kothari
Data Streams with Elasticsearch
siddharthlatest
0
130
Appbase.io - Customer Deck
siddharthlatest
0
140
Appbase.io - Investor Deck
siddharthlatest
0
330
Meteor Talk: Building Mitter
siddharthlatest
1
99
Data Streams on ElasticSearch
siddharthlatest
0
160
Appbase - A realtime events-based document store
siddharthlatest
1
800
Scalability Best Practices
siddharthlatest
0
180
Other Decks in Technology
See All in Technology
Tech-Verse 2025 Keynote
lycorptech_jp
PRO
0
1.8k
Flutter向けPDFビューア、pdfrxのpdfium WASM対応について
espresso3389
0
130
SmartNewsにおける 1000+ノード規模 K8s基盤 でのコスト最適化 – Spot・Gravitonの大規模導入への挑戦
vsanna2
0
120
OSSのSNSツール「Misskey」をさわってみよう(右下ワイプで私のOSCの20年を振り返ります) / 20250705-osc2025-do
akkiesoft
0
140
Beyond Kaniko: Navigating Unprivileged Container Image Creation
f30
0
130
ビギナーであり続ける/beginning
ikuodanaka
3
720
ビズリーチが挑む メトリクスを活用した技術的負債の解消 / dev-productivity-con2025
visional_engineering_and_design
3
6.6k
Delegating the chores of authenticating users to Keycloak
ahus1
0
140
自律的なスケーリング手法FASTにおけるVPoEとしてのアカウンタビリティ / dev-productivity-con-2025
yoshikiiida
1
15k
高速なプロダクト開発を実現、創業期から掲げるエンタープライズアーキテクチャ
kawauso
2
8.2k
American airlines ®️ USA Contact Numbers: Complete 2025 Support Guide
airhelpsupport
0
350
Backlog ユーザー棚卸しRTA、多分これが一番早いと思います
__allllllllez__
1
140
Featured
See All Featured
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
stephaniewalter
281
13k
[RailsConf 2023] Rails as a piece of cake
palkan
55
5.6k
A Tale of Four Properties
chriscoyier
160
23k
The Web Performance Landscape in 2024 [PerfNow 2024]
tammyeverts
8
680
Build The Right Thing And Hit Your Dates
maggiecrowley
36
2.8k
Building a Modern Day E-commerce SEO Strategy
aleyda
42
7.4k
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
eileencodes
35
2.4k
Helping Users Find Their Own Way: Creating Modern Search Experiences
danielanewman
29
2.7k
Bash Introduction
62gerente
614
210k
How to Ace a Technical Interview
jacobian
277
23k
The MySQL Ecosystem @ GitHub 2015
samlambert
251
13k
Connecting the Dots Between Site Speed, User Experience & Your Business [WebExpo 2025]
tammyeverts
5
270
Transcript
Streaming @siddharthlatest
The title could also be: * Streaming Queries with ElasticSearch
* Streaming DBs will take over the world * What the heck is a streaming database?
Siddarth, sith in making Co-founder and CEO, appbase.io Give me
a shout at @siddharthlatest GET /programming/stacktrace [“Games”, “C”, “Python”, “Java (sigh) ”, “AI”, “JS”, “Databases”]
Streaming, like videos?! You see …
Not Video Streaming via Ben Rubin, CEO @ Meerkat
• live
Not just synchronous and 1:Many, but also: CRUD with structured
data
Topics: 1. What is a streaming database? 2. The use-cases
3. ElasticSearch as the query layer 4. Streaming Topology 5. How does it scale? 6. The future
Results as Streams
Results as Streams
1. Streams and Firehoses from #IoT 2. Monitoring Systems 3.
Analytics 4. E-commerce: Search, Price Monitoring Use-cases
Topics: 1. What is a streaming database? 2. The use-cases
3. ElasticSearch as the query layer 4. Streaming Topology 5. How does it scale? 6. The future
Elasticsearch Distributed Full-text Search based on Lucene Can scale to
many nodes and highly available Analytics, Document Oriented, Open Source
ES: Percolation aka Search in Reverse 1. Indexing a Query
2. Matches when new documents are added 3. Distributed design since v1.0.0
Not just synchronous and 1:Many, but also: CRUD with structured
data
An Example Snippet Registering a Query
Match a Document Response
Topics: 1. What is a streaming database? 2. The use-cases
3. ElasticSearch as the query layer 4. Streaming Topology 5. How does it scale? 6. The future
Streaming Topology • Queries are subscriptions (HTTP Streaming / Websockets)
• Publish matches to subscribers. • Works as is with the ES API.
Streaming Topology
Streaming Topology • Beyond Percolation, keep the document store model
of ES. • Every document is a topic, which can have references. • When a doc is created, updated, or deleted; notify all the docs that refer to it.
Streaming Workflows • Every document has an ES Path exposed
by REST. Endpoint Worker Push back to stream • Topology like Apache Storm, but you can notify the workers using the entire ElasticSearch API.
Topics: 1. What is a streaming database? 2. The use-cases
3. ElasticSearch as the query layer 4. Streaming Topology 5. How does it scale? 6. The future
How does it distribute, scale? • Underlying Store: ES is
highly available, can scale to many nodes. • Were able to ingest 100,000 documents per second on 20 C4.2x large nodes (AWS). • Eventually consistent, with a very small t. • Distributed Streaming Topology is a work in progress.
Looking forward “The web has moved to #realtime, why shouldn’t
the Backend Infrastructure too?” “DBs are moving to having RESTful APIs, percolators, streaming interfaces are the next steps”.
Bold claim “In three years, every modern DB will have
a streaming interface.”
@siddharthlatest