Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
[NYJavaSig] Riding The Distributed Streams
Search
Viktor Gamov
February 03, 2017
Technology
1
200
[NYJavaSig] Riding The Distributed Streams
Presentation on Hazelcast and Distributed Streams.
Presented on NYJavaSig
Viktor Gamov
February 03, 2017
Tweet
Share
More Decks by Viktor Gamov
See All by Viktor Gamov
Processing Streaming Data with KSQL
vikgamov
4
380
[VirtualJUG] Apache Kafka — A Streaming Data Platform
vikgamov
3
380
[SF JUG] Apache Kafka — A Streaming Data Platform
vikgamov
4
83
[OracleCode NYC-2018] Apache Kafka A Streaming Data Platform
vikgamov
1
170
[OracleCode NYC-2018] Rethinking Stream Processing with KStreams and KSQL
vikgamov
2
230
[JBreak-2018] Это кто там твитить про #jbreak?
vikgamov
0
210
[DevNexus-2018] Apache Kafka A Streaming Data Platform
vikgamov
2
290
[DataSciCon] Divide, Distribute and Conquer: Stream v. Batch
vikgamov
0
110
[Philly JUG] Divide, Distribute and Conquer: Stream v. Batch
vikgamov
0
480
Other Decks in Technology
See All in Technology
あなたの知らない Linuxカーネル脆弱性の世界
recruitengineers
PRO
3
150
だいたい分かった気になる 『SREの知識地図』 / introduction-to-sre-knowledge-map-book
katsuhisa91
PRO
3
1.3k
初めてのDatabricks Apps開発
taka_aki
1
240
もう外には出ない。より快適なフルリモート環境を目指して
mottyzzz
13
9.2k
Building a cloud native business on open source
lizrice
0
170
AI-Readyを目指した非構造化データのメダリオンアーキテクチャ
r_miura
1
280
Azureコストと向き合った、4年半のリアル / Four and a half years of dealing with Azure costs
aeonpeople
1
250
From Natural Language to K8s Operations: The MCP Architecture and Practice of kubectl-ai
appleboy
0
110
ViteとTypeScriptのProject Referencesで 大規模モノレポのUIカタログのリリースサイクルを高速化する
shuta13
2
150
AIエージェント入門 〜基礎からMCP・A2Aまで〜
shukob
1
160
現場データから見える、開発生産性の変化コード生成AI導入・運用のリアル〜 / Changes in Development Productivity and Operational Challenges Following the Introduction of Code Generation AI
nttcom
1
450
生成AI時代のPythonセキュリティとガバナンス
abenben
0
110
Featured
See All Featured
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
31
2.6k
The Language of Interfaces
destraynor
162
25k
Building Better People: How to give real-time feedback that sticks.
wjessup
369
20k
Practical Tips for Bootstrapping Information Extraction Pipelines
honnibal
PRO
23
1.5k
Bootstrapping a Software Product
garrettdimon
PRO
307
110k
The Cult of Friendly URLs
andyhume
79
6.6k
Connecting the Dots Between Site Speed, User Experience & Your Business [WebExpo 2025]
tammyeverts
10
610
Context Engineering - Making Every Token Count
addyosmani
8
300
How to Think Like a Performance Engineer
csswizardry
27
2.1k
Balancing Empowerment & Direction
lara
5
700
Fireside Chat
paigeccino
40
3.7k
Unsuck your backbone
ammeep
671
58k
Transcript
None
> whoami • Solutions Architect @Hazelcast • Hang out with
awesome people • @gamussa in internetz Please, follow me in Twitter I’m very interesting ©
Agenda • Refreshing knowledge on Java 8 Streams • Distribute
and Conquer • Distributed Data • Distributed Streams • How we did all this
Java 8 Streams
Java 8 Streams… • An abstraction represents a sequence of
elements • Is not a data structure • Convey elements from a source through a pipeline of operations • Operation doesn’t modify a source
Why I should care about Stream API? • You’re Java
developer
What does regular Java developer think about Scala? advanced
Why I should care about Stream API? • You’re Java
developer • Many Java developers know Java • It’s all about data processing
java.util.stream operations • map(), flatMap(), filter() • reduce(), collect() •
sorted()
None
None
None
Problem • One does not simply put all Big Data
in one machine
Problem • Data doesn’t fit just one machine
Problem • One does not simply put all Big Data
in one machine • Data is too important to have it only one machine
None
CACHES
Replication on Sharding? http://book.mixu.net/distsys/single-page.html
Solution • Use Distributed Map aka IMap
What’s Hazelcast IMDG? • In-memory Data Grid • Apache v2
Licensed • Distributed • Caches (IMap, JCache) • Java Collections (IList, ISet, IQueue) • Messaging (Topic, RingBuffer) • Computation (ExecutorService, M-R)
None
None
None
Green Primary Green Backup Green Shard
None
Problem • Lambda serialization 26
27
Solution • serializable version of the interfaces • Introducing DistributedStream
28
29
None
31 Jet Streams
None
What’s Hazelcast Jet? • General purpose distributed data processing framework
• Based on Direct Acyclic Graph to model data flow • Built on top of Hazelcast IMDG • Comparable to Apache Spark or Apache Flink 33
None
DAG 35
Job Execution 36
None
Future (It’s bright!) • Memory module for processing big data
• Higher level streaming and batching APIs • Reactive Streams • Distributed Classloading • Integrations (HDFS/Yarn/Mesos)
Your fuel, our Jet Engine • Public release – Feb
7th. • Developer Preview today - yay! • http://hazelcast.org/jet-signup • Send me a note
[email protected]
• Follow @hazelcast and @gamussa (duh!!) • Your questions #hazelcast #hazelcastjet
Conclusion • Java Stream API provides very white range of
data processing tools • War And Piece – is a Big (a lot of data) Book! • Now we’re pretty sure that Andrew and Pierre are the main characters
None