Save 37% off PRO during our Black Friday Sale! »

End-to-end serverless data pipeline on AWS

End-to-end serverless data pipeline on AWS

Use case powered by Amazon Kinesis, Lambda, Athena, and QuickSight.

36ca8ae7e19067021d0e39c0b72acc2a?s=128

Alex Casalboni

October 16, 2017
Tweet

Transcript

  1. clda.co/sls-milano-data-pipeline End-to-end serverless data pipeline on AWS Powered by Amazon

    Kinesis, Lambda, Athena, and QuickSight 10/19/2017
  2. About Me @alex_casalboni clda.co/sls-milano-data-pipeline Computer Science Background Master in Sound

    & Music Engineering Sr. SoBware Engineer & Web Developer Cloud Evangelist @ Cloud Academy
  3. clda.co/serverless-italy Call for Speakers! clda.co/sls-milano-data-pipeline

  4. Agenda Prerequisites Use Case / Requirements Serverless Architecture Hands-on Demo

    clda.co/sls-milano-data-pipeline
  5. Prerequisites Breve introduzione dei singoli servizi clda.co/sls-milano-data-pipeline

  6. AWS Lambda clda.co/sls-milano-data-pipeline FaaS (FuncOon as a Service) NodeJs, Python,

    Java, C#, etc. Event-driven
  7. Amazon S3 clda.co/sls-milano-data-pipeline PB-scale Object-storage 99.999999999% durability SSE Support Can

    trigger Lambda (per-object)
  8. Amazon Kinesis (Streams) clda.co/sls-milano-data-pipeline Real-Ome data ingesOon Shards scale up/down

    Almost Serverless Can trigger Lambda (batch)
  9. Amazon Athena clda.co/sls-milano-data-pipeline InteracOve SQL queries over S3 Transparent compute

    provisioning “Serverless Database” Results are stored on S3 too
  10. Amazon QuickSight clda.co/sls-milano-data-pipeline Business Intelligence (BI) Powered by SPICE (in-memory

    engine) Can read from RedshiB, RDS, Aurora, Athena, S3, EMR, etc. Monthly subscripOon (not really PAYG)
  11. Amazon IAM clda.co/sls-milano-data-pipeline IdenOty & Access Management Very granular permissions

    Free service (use it!)
  12. Amazon Cognito clda.co/sls-milano-data-pipeline Federated idenOOes Role-based temporary credenOals Data Sync

    and User Management (Pools) Complex but useful (use it!)
  13. Use Case / Requirements Descrizione del caso d’uso clda.co/sls-milano-data-pipeline

  14. Real-Ome Fraud DetecOon clda.co/sls-milano-data-pipeline Real-0me ingesOon of credit card transacOons

    Stream processing, data validaOon, and fraud detecOon Secure storage of transacOons Real-0me analysis and reporOng
  15. AddiOonal Requirements clda.co/sls-milano-data-pipeline Many heterogeneous event producers (temporary credenOals) Elas0c

    architecture (no upfront costs & easy to scale up) Extensible architecture (plug-and-play components) A lot of data (cheap storage, please!)
  16. Serverless Architecture Come lo implemen7amo senza ges7re server? clda.co/sls-milano-data-pipeline

  17. Architecture Diagram clda.co/sls-milano-data-pipeline

  18. IngesOon & Processing (simple) clda.co/sls-milano-data-pipeline

  19. IngesOon & Processing (opOmized) clda.co/sls-milano-data-pipeline

  20. IngesOon & Processing (real-Ome analyOcs)

  21. Serverless Data Analysis clda.co/sls-milano-data-pipeline

  22. Hands-on Demo Sporchiamoci un po’ le mani clda.co/sls-milano-data-pipeline

  23. clda.co/sls-milano-data-pipeline < demo >

  24. Grazie =) Domande? 10/19/2017 clda.co/sls-milano-data-pipeline