End-to-end serverless data pipeline on AWS

Transcript

1. clda.co/sls-milano-data-pipeline End-to-end serverless data pipeline on AWS Powered by Amazon

   Kinesis, Lambda, Athena, and QuickSight 10/19/2017

2. About Me @alex_casalboni clda.co/sls-milano-data-pipeline Computer Science Background Master in Sound

   & Music Engineering Sr. SoBware Engineer & Web Developer Cloud Evangelist @ Cloud Academy

3. clda.co/serverless-italy Call for Speakers! clda.co/sls-milano-data-pipeline

4. Agenda Prerequisites Use Case / Requirements Serverless Architecture Hands-on Demo

   clda.co/sls-milano-data-pipeline

5. Prerequisites Breve introduzione dei singoli servizi clda.co/sls-milano-data-pipeline

6. AWS Lambda clda.co/sls-milano-data-pipeline FaaS (FuncOon as a Service) NodeJs, Python,

   Java, C#, etc. Event-driven

7. Amazon S3 clda.co/sls-milano-data-pipeline PB-scale Object-storage 99.999999999% durability SSE Support Can

   trigger Lambda (per-object)

8. Amazon Kinesis (Streams) clda.co/sls-milano-data-pipeline Real-Ome data ingesOon Shards scale up/down

   Almost Serverless Can trigger Lambda (batch)

9. Amazon Athena clda.co/sls-milano-data-pipeline InteracOve SQL queries over S3 Transparent compute

   provisioning “Serverless Database” Results are stored on S3 too

10. Amazon QuickSight clda.co/sls-milano-data-pipeline Business Intelligence (BI) Powered by SPICE (in-memory

    engine) Can read from RedshiB, RDS, Aurora, Athena, S3, EMR, etc. Monthly subscripOon (not really PAYG)

11. Amazon IAM clda.co/sls-milano-data-pipeline IdenOty & Access Management Very granular permissions

    Free service (use it!)

12. Amazon Cognito clda.co/sls-milano-data-pipeline Federated idenOOes Role-based temporary credenOals Data Sync

    and User Management (Pools) Complex but useful (use it!)

13. Use Case / Requirements Descrizione del caso d’uso clda.co/sls-milano-data-pipeline

14. Real-Ome Fraud DetecOon clda.co/sls-milano-data-pipeline Real-0me ingesOon of credit card transacOons

    Stream processing, data validaOon, and fraud detecOon Secure storage of transacOons Real-0me analysis and reporOng

15. AddiOonal Requirements clda.co/sls-milano-data-pipeline Many heterogeneous event producers (temporary credenOals) Elas0c

    architecture (no upfront costs & easy to scale up) Extensible architecture (plug-and-play components) A lot of data (cheap storage, please!)

16. Serverless Architecture Come lo implemen7amo senza ges7re server? clda.co/sls-milano-data-pipeline

17. Architecture Diagram clda.co/sls-milano-data-pipeline

18. IngesOon & Processing (simple) clda.co/sls-milano-data-pipeline

19. IngesOon & Processing (opOmized) clda.co/sls-milano-data-pipeline

20. IngesOon & Processing (real-Ome analyOcs)

21. Serverless Data Analysis clda.co/sls-milano-data-pipeline

22. Hands-on Demo Sporchiamoci un po’ le mani clda.co/sls-milano-data-pipeline

23. clda.co/sls-milano-data-pipeline < demo >

24. Grazie =) Domande? 10/19/2017 clda.co/sls-milano-data-pipeline