Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
[NYJavaSig] Riding The Distributed Streams
Search
Viktor Gamov
February 03, 2017
Technology
1
190
[NYJavaSig] Riding The Distributed Streams
Presentation on Hazelcast and Distributed Streams.
Presented on NYJavaSig
Viktor Gamov
February 03, 2017
Tweet
Share
More Decks by Viktor Gamov
See All by Viktor Gamov
Processing Streaming Data with KSQL
vikgamov
4
340
[VirtualJUG] Apache Kafka — A Streaming Data Platform
vikgamov
3
340
[SF JUG] Apache Kafka — A Streaming Data Platform
vikgamov
4
73
[OracleCode NYC-2018] Apache Kafka A Streaming Data Platform
vikgamov
1
150
[OracleCode NYC-2018] Rethinking Stream Processing with KStreams and KSQL
vikgamov
2
210
[JBreak-2018] Это кто там твитить про #jbreak?
vikgamov
0
180
[DevNexus-2018] Apache Kafka A Streaming Data Platform
vikgamov
2
240
[DataSciCon] Divide, Distribute and Conquer: Stream v. Batch
vikgamov
0
94
[Philly JUG] Divide, Distribute and Conquer: Stream v. Batch
vikgamov
0
440
Other Decks in Technology
See All in Technology
Google Cloud で始める Cloud Run 〜AWSとの比較と実例デモで解説〜
risatube
PRO
0
120
Unlearn Product Development - Unleashed Edition
lemiorhan
PRO
2
130
メンタル面でもつよつよエンジニアになる/登壇資料(井田 献一朗)
hacobu
0
140
re:Invent をおうちで楽しんでみた ~CloudWatch のオブザーバビリティ機能がスゴい!/ Enjoyed AWS re:Invent from Home and CloudWatch Observability Feature is Amazing!
yuj1osm
0
140
2024年にチャレンジしたことを振り返るぞ
mitchan
0
160
Server-Side Engineer of LINE Sukimani
lycorp_recruit_jp
0
420
AWS環境におけるランサムウェア攻撃対策の設計
nrinetcom
PRO
0
260
サイボウズフロントエンドエキスパートチームについて / FrontendExpert Team
cybozuinsideout
PRO
5
39k
大規模言語モデルとそのソフトウェア開発に向けた応用 (2024年版)
kazato
1
140
OCI技術資料 : ファイル・ストレージ 概要
ocise
3
11k
MasterMemory v3 最速確認会
yucchiy
0
220
Yahoo! ズバトクにおけるフロントエンド開発
lycorptech_jp
PRO
0
110
Featured
See All Featured
[RailsConf 2023] Rails as a piece of cake
palkan
53
5.1k
The MySQL Ecosystem @ GitHub 2015
samlambert
250
12k
Product Roadmaps are Hard
iamctodd
PRO
50
11k
Producing Creativity
orderedlist
PRO
342
39k
The World Runs on Bad Software
bkeepers
PRO
66
11k
Dealing with People You Can't Stand - Big Design 2015
cassininazir
365
25k
BBQ
matthewcrist
85
9.4k
Why You Should Never Use an ORM
jnunemaker
PRO
54
9.1k
The Cost Of JavaScript in 2023
addyosmani
46
7k
ピンチをチャンスに:未来をつくるプロダクトロードマップ #pmconf2020
aki_iinuma
111
49k
Building an army of robots
kneath
302
44k
The Art of Programming - Codeland 2020
erikaheidi
53
13k
Transcript
None
> whoami • Solutions Architect @Hazelcast • Hang out with
awesome people • @gamussa in internetz Please, follow me in Twitter I’m very interesting ©
Agenda • Refreshing knowledge on Java 8 Streams • Distribute
and Conquer • Distributed Data • Distributed Streams • How we did all this
Java 8 Streams
Java 8 Streams… • An abstraction represents a sequence of
elements • Is not a data structure • Convey elements from a source through a pipeline of operations • Operation doesn’t modify a source
Why I should care about Stream API? • You’re Java
developer
What does regular Java developer think about Scala? advanced
Why I should care about Stream API? • You’re Java
developer • Many Java developers know Java • It’s all about data processing
java.util.stream operations • map(), flatMap(), filter() • reduce(), collect() •
sorted()
None
None
None
Problem • One does not simply put all Big Data
in one machine
Problem • Data doesn’t fit just one machine
Problem • One does not simply put all Big Data
in one machine • Data is too important to have it only one machine
None
CACHES
Replication on Sharding? http://book.mixu.net/distsys/single-page.html
Solution • Use Distributed Map aka IMap
What’s Hazelcast IMDG? • In-memory Data Grid • Apache v2
Licensed • Distributed • Caches (IMap, JCache) • Java Collections (IList, ISet, IQueue) • Messaging (Topic, RingBuffer) • Computation (ExecutorService, M-R)
None
None
None
Green Primary Green Backup Green Shard
None
Problem • Lambda serialization 26
27
Solution • serializable version of the interfaces • Introducing DistributedStream
28
29
None
31 Jet Streams
None
What’s Hazelcast Jet? • General purpose distributed data processing framework
• Based on Direct Acyclic Graph to model data flow • Built on top of Hazelcast IMDG • Comparable to Apache Spark or Apache Flink 33
None
DAG 35
Job Execution 36
None
Future (It’s bright!) • Memory module for processing big data
• Higher level streaming and batching APIs • Reactive Streams • Distributed Classloading • Integrations (HDFS/Yarn/Mesos)
Your fuel, our Jet Engine • Public release – Feb
7th. • Developer Preview today - yay! • http://hazelcast.org/jet-signup • Send me a note viktor@hazelcast.com • Follow @hazelcast and @gamussa (duh!!) • Your questions #hazelcast #hazelcastjet
Conclusion • Java Stream API provides very white range of
data processing tools • War And Piece – is a Big (a lot of data) Book! • Now we’re pretty sure that Andrew and Pierre are the main characters
None