End-to-end serverless data pipeline on AWS

Use case powered by Amazon Kinesis, Lambda, Athena, and QuickSight.

Transcript

1. clda.co/sls-milano-data-pipeline End-to-end serverless data pipeline on AWS Powered by Amazon Kinesis, Lambda, Athena, and QuickSight 10/19/2017
2. About Me @alex_casalboni clda.co/sls-milano-data-pipeline Computer Science Background Master in Sound & Music Engineering Sr. SoBware Engineer & Web Developer Cloud Evangelist @ Cloud Academy
3. clda.co/serverless-italy Call for Speakers! clda.co/sls-milano-data-pipeline
4. Agenda Prerequisites Use Case / Requirements Serverless Architecture Hands-on Demo clda.co/sls-milano-data-pipeline
5. Prerequisites Breve introduzione dei singoli servizi clda.co/sls-milano-data-pipeline
6. AWS Lambda clda.co/sls-milano-data-pipeline FaaS (FuncOon as a Service) NodeJs, Python, Java, C#, etc. Event-driven
7. Amazon S3 clda.co/sls-milano-data-pipeline PB-scale Object-storage 99.999999999% durability SSE Support Can trigger Lambda (per-object)
8. Amazon Kinesis (Streams) clda.co/sls-milano-data-pipeline Real-Ome data ingesOon Shards scale up/down Almost Serverless Can trigger Lambda (batch)
9. Amazon Athena clda.co/sls-milano-data-pipeline InteracOve SQL queries over S3 Transparent compute provisioning “Serverless Database” Results are stored on S3 too
10. Amazon QuickSight clda.co/sls-milano-data-pipeline Business Intelligence (BI) Powered by SPICE (in-memory engine) Can read from RedshiB, RDS, Aurora, Athena, S3, EMR, etc. Monthly subscripOon (not really PAYG)
11. Amazon IAM clda.co/sls-milano-data-pipeline IdenOty & Access Management Very granular permissions Free service (use it!)
12. Amazon Cognito clda.co/sls-milano-data-pipeline Federated idenOOes Role-based temporary credenOals Data Sync and User Management (Pools) Complex but useful (use it!)
13. Use Case / Requirements Descrizione del caso d’uso clda.co/sls-milano-data-pipeline
14. Real-Ome Fraud DetecOon clda.co/sls-milano-data-pipeline Real-0me ingesOon of credit card transacOons Stream processing, data validaOon, and fraud detecOon Secure storage of transacOons Real-0me analysis and reporOng
15. AddiOonal Requirements clda.co/sls-milano-data-pipeline Many heterogeneous event producers (temporary credenOals) Elas0c architecture (no upfront costs & easy to scale up) Extensible architecture (plug-and-play components) A lot of data (cheap storage, please!)
16. Serverless Architecture Come lo implemen7amo senza ges7re server? clda.co/sls-milano-data-pipeline
17. Architecture Diagram clda.co/sls-milano-data-pipeline
18. IngesOon & Processing (simple) clda.co/sls-milano-data-pipeline
19. IngesOon & Processing (opOmized) clda.co/sls-milano-data-pipeline
20. IngesOon & Processing (real-Ome analyOcs)
21. Serverless Data Analysis clda.co/sls-milano-data-pipeline
22. Hands-on Demo Sporchiamoci un po’ le mani clda.co/sls-milano-data-pipeline
23. clda.co/sls-milano-data-pipeline < demo >
24. Grazie =) Domande? 10/19/2017 clda.co/sls-milano-data-pipeline