Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Serverless Data Processing

Avatar for Victor Wibisono Victor Wibisono
May 31, 2018
Technology
0
95

Serverless Data Processing

Avatar for Victor Wibisono

Victor Wibisono

May 31, 2018
Tweet

More Decks by Victor Wibisono

See All by Victor Wibisono
Predictive Maintenance Pipeline using Kafka Connect, Streams and KSQL
Avatar for Victor Wibisono devacto
1
1.9k

Other Decks in Technology

See All in Technology
Proxmox × HCP Terraformで始めるお家プライベートクラウド
Avatar for Lamaglama39 lamaglama39
1
170
Data Engineering Guide 2025 #data_summit_findy by @Kazaneya_PR / 20251106
Avatar for 風音屋 (Kazaneya) kazaneya
PRO
11
2.1k
ググるより、AIに聞こう - Don’t Google it, ask AI
Avatar for Oikon oikon48
0
780
データ組織ゼロから投資を得るまでの軌跡と未来図 〜AIの前にやるべきこと〜 / Building a Data Organization from Scratch: The Journey to Securing Investment and a Vision for the Future
Avatar for 株式会社カオナビ kaonavi
0
110
Pythonで構築する全国市町村ナレッジグラフ: GraphRAGを用いた意味的地域検索への応用
Avatar for negi111111 negi111111
8
3.3k
激動の2025年、Modern Data Stackの最新技術動向
Avatar for Sagara sagara
0
1.2k
QAエンジニアがプロダクト専任で チームの中に入ると。。。？/登壇資料（杉森 太樹）
Avatar for Hacobu hacobu
PRO
0
180
設計は最強のプロンプト - AI時代に武器にすべきスキルとは？-
Avatar for Ken'ichirou Kimura kenichirokimura
1
340
AIと共に開発する時代の組織、プロセス設計 freeeでの実践から見えてきたこと
Avatar for freee freee
3
510
Design and implementation of "Markdown to Google Slides" / phpconfuk 2025
Avatar for Ken’ichiro Oyama k1low
1
390
よくわからない人向けの IAM Identity Center とちょっとした落とし穴
Avatar for kazzpapa3 kazzpapa3
2
680
[JDDStudy #10] 社内Agent勉強会の取り組み紹介
Avatar for Kyojin.Syo yp_genzitsu
1
130

Featured

See All Featured
Building Better People: How to give real-time feedback that sticks.
Avatar for Will Jessup wjessup
370
20k
Side Projects
Avatar for Sacha Greif sachag
455
43k
Building a Modern Day 
E-commerce SEO Strategy
Avatar for Aleyda Solis aleyda
45
8k
Making Projects Easy
Avatar for Brett Harned brettharned
120
6.4k
Why Our Code Smells
Avatar for Brandon Keepers bkeepers
PRO
340
57k
Visualization
Avatar for Eitan Lees eitanlees
150
16k
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
Avatar for Stéphanie Walter stephaniewalter
285
14k
Build your cross-platform service in a week with App Engine
Avatar for Jose L jlugia
234
18k
XXLCSS - How to scale CSS and keep your sanity
Avatar for Zaharenia Atzitzikaki sugarenia
249
1.3M
Designing for humans not robots
Avatar for Tammie Lister tammielis
254
26k
Building Flexible Design Systems
Avatar for Yesenia Perez-Cruz yeseniaperezcruz
329
39k
A designer walks into a library…
Avatar for Paul Jervis Heath pauljervisheath
210
24k

Transcript

  1. Serverless Data Processing

  2. Contents • What it is and why • Architecture •

    Demo

  3. Serverless?

  4. Serverless is about focusing your efforts on what provides value

    to users.

  5. No servers?

  6. None
  7. None

  8. https://serverless.com/learn/

  9. Data processing?

  10. None
  11. None
  12. None

  13. Architecture

  14. None

  15. • Infinitely scalable • 99.9999999% data durability • Pay-per-use, no

    pre- provisioning

  16. AWS Glue • Spark-as-a-service • No cluster management • Pay-per-use,

    no pre-provisioning • Services include: data crawling, cataloging (Hive metastore)

  17. AWS Athena • Pay-per-query • No pre-provisioning • Infinitely scalable

  18. https://blog.panoply.io/an-amazonian-battle-comparing-athena-and-redshift

  19. None

  20. Learn more...

  21. https://unnik.s3.amazonaws.com/public-files/unnik-lab-guides/aws-summit-2018/datalake/unnik-aws-summit-2018-datalake-demo.html

  22. Demo