Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
End-to-end serverless data pipeline on AWS
Search
Alex Casalboni
October 16, 2017
Technology
1
93
End-to-end serverless data pipeline on AWS
Use case powered by Amazon Kinesis, Lambda, Athena, and QuickSight.
Alex Casalboni
October 16, 2017
Tweet
Share
More Decks by Alex Casalboni
See All by Alex Casalboni
Data-driven & cost-aware performance tuning for AWS Lambda
alexcasalboni
0
150
AWS Lambda function performance tuning - AWS re:Invent 2019
alexcasalboni
1
1k
Supercharge product development with cloud best practices - Web Summit 2019
alexcasalboni
0
85
AWS data services for machine learning - AWS Innovate Online
alexcasalboni
0
52
Advanced Serverless Architectural Patterns on AWS [Devoxx Poland]
alexcasalboni
0
210
Building a Modern Data Platform in the Cloud [AWS Dev Day @ Kyiv]
alexcasalboni
0
97
How to build a global serverless service [AWS Dev Day @ Kyiv]
alexcasalboni
0
49
Become a machine learning developer using AWS Machine Learning Services [AWS Summit @ Warsaw]
alexcasalboni
0
96
Getting started with AWS for Developers [Collision @ Toronto]
alexcasalboni
0
55
Other Decks in Technology
See All in Technology
"好き"との生活/Regularly update profile with GitHub Actions
judeeeee
0
110
疲弊しない!AWSセキュリティ統制の考え方 #devio_osakaday1
masahirokawahara
6
5.7k
Databricks:『生成AI World Cup』のご案内
databricksjapan
1
110
なぜ NOT A HOTEL が Web3 に取り組むのか - NOT A HOTEL TECH TALK
ynunokawa
0
160
Hands-on / Kaname Frusawa / Cloud Compare Users Meetup 2024 at University of Tokyo on April 17
paraworld
2
450
Why we expect the Microservices
shkitayama
2
320
Signals Unleashed: The Full Guide
rainerhahnekamp
0
340
NLP2024 参加報告LT ~RAGの生成評価と懇親戦略~ / nlp2024_attendee_presentation_LT_masuda
taro_masuda
1
190
Kubernetesでアプリの安定稼働と高頻度のアップデートを両立するためのプラクティス / Best Practices for Applications on Kubernetesto Achieve Both Frequent Updates and Stability
hhiroshell
10
2.9k
キャラクター制御のためのプロンプト術 for LINE Bot
uezo
0
490
【SORACOM UG】(2024年度版) SIMってなんだ? ~セルラー通信がつながる仕組み、解説します~
soracom
PRO
0
120
Autonomous Database Cloud 技術詳細 / adb-s_technical_detail_jp
oracle4engineer
PRO
13
35k
Featured
See All Featured
Debugging Ruby Performance
tmm1
69
11k
The Language of Interfaces
destraynor
151
23k
JavaScript: Past, Present, and Future - NDC Porto 2020
reverentgeek
39
4.3k
I Don’t Have Time: Getting Over the Fear to Launch Your Podcast
jcasabona
20
1.6k
The Pragmatic Product Professional
lauravandoore
24
5.8k
Building Adaptive Systems
keathley
29
1.8k
Infographics Made Easy
chrislema
237
18k
How GitHub (no longer) Works
holman
302
140k
Documentation Writing (for coders)
carmenintech
59
3.9k
Side Projects
sachag
451
41k
The Cost Of JavaScript in 2023
addyosmani
13
3.8k
The Straight Up "How To Draw Better" Workshop
denniskardys
227
130k
Transcript
clda.co/sls-milano-data-pipeline End-to-end serverless data pipeline on AWS Powered by Amazon
Kinesis, Lambda, Athena, and QuickSight 10/19/2017
About Me @alex_casalboni clda.co/sls-milano-data-pipeline Computer Science Background Master in Sound
& Music Engineering Sr. SoBware Engineer & Web Developer Cloud Evangelist @ Cloud Academy
clda.co/serverless-italy Call for Speakers! clda.co/sls-milano-data-pipeline
Agenda Prerequisites Use Case / Requirements Serverless Architecture Hands-on Demo
clda.co/sls-milano-data-pipeline
Prerequisites Breve introduzione dei singoli servizi clda.co/sls-milano-data-pipeline
AWS Lambda clda.co/sls-milano-data-pipeline FaaS (FuncOon as a Service) NodeJs, Python,
Java, C#, etc. Event-driven
Amazon S3 clda.co/sls-milano-data-pipeline PB-scale Object-storage 99.999999999% durability SSE Support Can
trigger Lambda (per-object)
Amazon Kinesis (Streams) clda.co/sls-milano-data-pipeline Real-Ome data ingesOon Shards scale up/down
Almost Serverless Can trigger Lambda (batch)
Amazon Athena clda.co/sls-milano-data-pipeline InteracOve SQL queries over S3 Transparent compute
provisioning “Serverless Database” Results are stored on S3 too
Amazon QuickSight clda.co/sls-milano-data-pipeline Business Intelligence (BI) Powered by SPICE (in-memory
engine) Can read from RedshiB, RDS, Aurora, Athena, S3, EMR, etc. Monthly subscripOon (not really PAYG)
Amazon IAM clda.co/sls-milano-data-pipeline IdenOty & Access Management Very granular permissions
Free service (use it!)
Amazon Cognito clda.co/sls-milano-data-pipeline Federated idenOOes Role-based temporary credenOals Data Sync
and User Management (Pools) Complex but useful (use it!)
Use Case / Requirements Descrizione del caso d’uso clda.co/sls-milano-data-pipeline
Real-Ome Fraud DetecOon clda.co/sls-milano-data-pipeline Real-0me ingesOon of credit card transacOons
Stream processing, data validaOon, and fraud detecOon Secure storage of transacOons Real-0me analysis and reporOng
AddiOonal Requirements clda.co/sls-milano-data-pipeline Many heterogeneous event producers (temporary credenOals) Elas0c
architecture (no upfront costs & easy to scale up) Extensible architecture (plug-and-play components) A lot of data (cheap storage, please!)
Serverless Architecture Come lo implemen7amo senza ges7re server? clda.co/sls-milano-data-pipeline
Architecture Diagram clda.co/sls-milano-data-pipeline
IngesOon & Processing (simple) clda.co/sls-milano-data-pipeline
IngesOon & Processing (opOmized) clda.co/sls-milano-data-pipeline
IngesOon & Processing (real-Ome analyOcs)
Serverless Data Analysis clda.co/sls-milano-data-pipeline
Hands-on Demo Sporchiamoci un po’ le mani clda.co/sls-milano-data-pipeline
clda.co/sls-milano-data-pipeline < demo >
Grazie =) Domande? 10/19/2017 clda.co/sls-milano-data-pipeline