Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Serverless Data Processing

Avatar for Victor Wibisono Victor Wibisono
May 31, 2018
Technology
0
92

Serverless Data Processing

Avatar for Victor Wibisono

Victor Wibisono

May 31, 2018
Tweet

More Decks by Victor Wibisono

See All by Victor Wibisono
Predictive Maintenance Pipeline using Kafka Connect, Streams and KSQL
Avatar for Victor Wibisono devacto
1
1.9k

Other Decks in Technology

See All in Technology
[VPoE Global Summit] サービスレベル目標による信頼性への投資最適化
Avatar for sato-s satos
0
140
衛星画像超解像化によって実現する2D, 3D空間情報の即時生成と“AI as a Service”/ Real-time generation spatial data enabled_by satellite image super-resolution
Avatar for Yuji Kobayashi lehupa
0
190
Introduction to Sansan, inc / Sansan Global Development Center, Inc.
Avatar for Sansan, Inc. sansan33
PRO
0
2.8k
HR Force における DWH の併用事例 ~ サービス基盤としての BigQuery / 分析基盤としての Snowflake ~@Cross Data Platforms Meetup #2「BigQueryと愉快な仲間たち」
Avatar for Ryo Suzuki ryo_suzuki
0
240
なぜAWSを活かしきれないのか？技術と組織への処方箋
Avatar for NRI Netcom nrinetcom
PRO
5
1k
プロダクトのコードから見るGoによるデザインパターンの実践 #go_night_talk
Avatar for 弁護士ドットコム bengo4com
1
2.7k
AI Agent Dojo #2 watsonx Orchestrateフローの作成
Avatar for Akira Onishi (IBM) oniak3ibm
PRO
0
130
Introduction to Sansan Meishi Maker Development Engineer
Avatar for Sansan, Inc. sansan33
PRO
0
310
ローカルLLMとLINE Botの組み合わせ その2（EVO-X2でgpt-oss-120bを利用） / LINE DC Generative AI Meetup #7
Avatar for you（@youtoy） you
PRO
0
100
Geospatialの世界最前線を探る [2025年版]
Avatar for Yasunori Kirimoto dayjournal
1
260
Databricks AI/BI Genie の「値ディクショナリー」をAmazonの奥地（S3）まで見に行く
Avatar for camay kameitomohiro
1
310
20251014_Pythonを実務で徹底的に使いこなした話
Avatar for 森田　一平 ippei0923
0
220

Featured

See All Featured
Evolution of real-time – Irina Nazarova, EuRuKo, 2024
Avatar for Irina Nazarova irinanazarova
9
980
Context Engineering - Making Every Token Count
Avatar for Addy Osmani addyosmani
7
270
jQuery: Nuts, Bolts and Bling
Avatar for Doug Neiner dougneiner
65
7.9k
Statistics for Hackers
Avatar for Jake VanderPlas jakevdp
799
220k
Let's Do A Bunch of Simple Stuff to Make Websites Faster
Avatar for Chris Coyier chriscoyier
508
140k
The Invisible Side of Design
Avatar for Vitaly Friedman smashingmag
302
51k
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
Avatar for Dan Newman danielanewman
230
22k
Rebuilding a faster, lazier Slack
Avatar for samanthasiow samanthasiow
84
9.2k
[RailsConf 2023] Rails as a piece of cake
Avatar for Vladimir Dementyev palkan
57
5.9k
Performance Is Good for Brains [We Love Speed 2024]
Avatar for Tammy Everts tammyeverts
12
1.2k
Building Flexible Design Systems
Avatar for Yesenia Perez-Cruz yeseniaperezcruz
329
39k
GraphQLの誤解/rethinking-graphql
Avatar for sonatard sonatard
73
11k

Transcript

  1. Serverless Data Processing

  2. Contents • What it is and why • Architecture •

    Demo

  3. Serverless?

  4. Serverless is about focusing your efforts on what provides value

    to users.

  5. No servers?

  6. None
  7. None

  8. https://serverless.com/learn/

  9. Data processing?

  10. None
  11. None
  12. None

  13. Architecture

  14. None

  15. • Infinitely scalable • 99.9999999% data durability • Pay-per-use, no

    pre- provisioning

  16. AWS Glue • Spark-as-a-service • No cluster management • Pay-per-use,

    no pre-provisioning • Services include: data crawling, cataloging (Hive metastore)

  17. AWS Athena • Pay-per-query • No pre-provisioning • Infinitely scalable

  18. https://blog.panoply.io/an-amazonian-battle-comparing-athena-and-redshift

  19. None

  20. Learn more...

  21. https://unnik.s3.amazonaws.com/public-files/unnik-lab-guides/aws-summit-2018/datalake/unnik-aws-summit-2018-datalake-demo.html

  22. Demo