Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Building a streaming database service
Search
Siddharth Kothari
May 21, 2015
Technology
1
220
Building a streaming database service
Streaming Queries with ElasticSearch
Siddharth Kothari
May 21, 2015
Tweet
Share
More Decks by Siddharth Kothari
See All by Siddharth Kothari
Data Streams with Elasticsearch
siddharthlatest
0
130
Appbase.io - Customer Deck
siddharthlatest
0
130
Appbase.io - Investor Deck
siddharthlatest
0
330
Meteor Talk: Building Mitter
siddharthlatest
1
99
Data Streams on ElasticSearch
siddharthlatest
0
160
Appbase - A realtime events-based document store
siddharthlatest
1
800
Scalability Best Practices
siddharthlatest
0
180
Other Decks in Technology
See All in Technology
Абьюзим random_bytes(). Фёдор Кулаков, разработчик Lamoda Tech
lamodatech
0
350
セキュリティの民主化は何故必要なのか_AWS WAF 運用の 10 の苦悩から学ぶ
yoh
1
180
How Community Opened Global Doors
hiroramos4
PRO
1
120
Fabric + Databricks 2025.6 の最新情報ピックアップ
ryomaru0825
1
140
本が全く読めなかった過去の自分へ
genshun9
0
560
あなたの声を届けよう! 女性エンジニア登壇の意義とアウトプット実践ガイド #wttjp / Call for Your Voice
kondoyuko
4
470
rubygem開発で鍛える設計力
joker1007
2
220
Tech-Verse 2025 Keynote
lycorptech_jp
PRO
0
220
監視のこれまでとこれから/sakura monitoring seminar 2025
fujiwara3
11
3.9k
M3 Expressiveの思想に迫る
chnotchy
0
110
Snowflake Summit 2025全体振り返り / Snowflake Summit 2025 Overall Review
mtpooh
2
400
第9回情シス転職ミートアップ_テックタッチ株式会社
forester3003
0
250
Featured
See All Featured
The Myth of the Modular Monolith - Day 2 Keynote - Rails World 2024
eileencodes
26
2.9k
jQuery: Nuts, Bolts and Bling
dougneiner
63
7.8k
Building an army of robots
kneath
306
45k
Agile that works and the tools we love
rasmusluckow
329
21k
Gamification - CAS2011
davidbonilla
81
5.3k
Visualizing Your Data: Incorporating Mongo into Loggly Infrastructure
mongodb
46
9.6k
How to Think Like a Performance Engineer
csswizardry
24
1.7k
Helping Users Find Their Own Way: Creating Modern Search Experiences
danielanewman
29
2.7k
ピンチをチャンスに:未来をつくるプロダクトロードマップ #pmconf2020
aki_iinuma
124
52k
Building a Modern Day E-commerce SEO Strategy
aleyda
42
7.3k
Bootstrapping a Software Product
garrettdimon
PRO
307
110k
Raft: Consensus for Rubyists
vanstee
140
7k
Transcript
Streaming @siddharthlatest
The title could also be: * Streaming Queries with ElasticSearch
* Streaming DBs will take over the world * What the heck is a streaming database?
Siddarth, sith in making Co-founder and CEO, appbase.io Give me
a shout at @siddharthlatest GET /programming/stacktrace [“Games”, “C”, “Python”, “Java (sigh) ”, “AI”, “JS”, “Databases”]
Streaming, like videos?! You see …
Not Video Streaming via Ben Rubin, CEO @ Meerkat
• live
Not just synchronous and 1:Many, but also: CRUD with structured
data
Topics: 1. What is a streaming database? 2. The use-cases
3. ElasticSearch as the query layer 4. Streaming Topology 5. How does it scale? 6. The future
Results as Streams
Results as Streams
1. Streams and Firehoses from #IoT 2. Monitoring Systems 3.
Analytics 4. E-commerce: Search, Price Monitoring Use-cases
Topics: 1. What is a streaming database? 2. The use-cases
3. ElasticSearch as the query layer 4. Streaming Topology 5. How does it scale? 6. The future
Elasticsearch Distributed Full-text Search based on Lucene Can scale to
many nodes and highly available Analytics, Document Oriented, Open Source
ES: Percolation aka Search in Reverse 1. Indexing a Query
2. Matches when new documents are added 3. Distributed design since v1.0.0
Not just synchronous and 1:Many, but also: CRUD with structured
data
An Example Snippet Registering a Query
Match a Document Response
Topics: 1. What is a streaming database? 2. The use-cases
3. ElasticSearch as the query layer 4. Streaming Topology 5. How does it scale? 6. The future
Streaming Topology • Queries are subscriptions (HTTP Streaming / Websockets)
• Publish matches to subscribers. • Works as is with the ES API.
Streaming Topology
Streaming Topology • Beyond Percolation, keep the document store model
of ES. • Every document is a topic, which can have references. • When a doc is created, updated, or deleted; notify all the docs that refer to it.
Streaming Workflows • Every document has an ES Path exposed
by REST. Endpoint Worker Push back to stream • Topology like Apache Storm, but you can notify the workers using the entire ElasticSearch API.
Topics: 1. What is a streaming database? 2. The use-cases
3. ElasticSearch as the query layer 4. Streaming Topology 5. How does it scale? 6. The future
How does it distribute, scale? • Underlying Store: ES is
highly available, can scale to many nodes. • Were able to ingest 100,000 documents per second on 20 C4.2x large nodes (AWS). • Eventually consistent, with a very small t. • Distributed Streaming Topology is a work in progress.
Looking forward “The web has moved to #realtime, why shouldn’t
the Backend Infrastructure too?” “DBs are moving to having RESTful APIs, percolators, streaming interfaces are the next steps”.
Bold claim “In three years, every modern DB will have
a streaming interface.”
@siddharthlatest