Save 37% off PRO during our Black Friday Sale! »

End-to-end serverless data pipeline on AWS

End-to-end serverless data pipeline on AWS

Use case powered by Amazon Kinesis, Lambda, Athena, and QuickSight.


Alex Casalboni

October 16, 2017


  1. End-to-end serverless data pipeline on AWS Powered by Amazon

    Kinesis, Lambda, Athena, and QuickSight 10/19/2017
  2. About Me @alex_casalboni Computer Science Background Master in Sound

    & Music Engineering Sr. SoBware Engineer & Web Developer Cloud Evangelist @ Cloud Academy
  3. Call for Speakers!

  4. Agenda Prerequisites Use Case / Requirements Serverless Architecture Hands-on Demo
  5. Prerequisites Breve introduzione dei singoli servizi

  6. AWS Lambda FaaS (FuncOon as a Service) NodeJs, Python,

    Java, C#, etc. Event-driven
  7. Amazon S3 PB-scale Object-storage 99.999999999% durability SSE Support Can

    trigger Lambda (per-object)
  8. Amazon Kinesis (Streams) Real-Ome data ingesOon Shards scale up/down

    Almost Serverless Can trigger Lambda (batch)
  9. Amazon Athena InteracOve SQL queries over S3 Transparent compute

    provisioning “Serverless Database” Results are stored on S3 too
  10. Amazon QuickSight Business Intelligence (BI) Powered by SPICE (in-memory

    engine) Can read from RedshiB, RDS, Aurora, Athena, S3, EMR, etc. Monthly subscripOon (not really PAYG)
  11. Amazon IAM IdenOty & Access Management Very granular permissions

    Free service (use it!)
  12. Amazon Cognito Federated idenOOes Role-based temporary credenOals Data Sync

    and User Management (Pools) Complex but useful (use it!)
  13. Use Case / Requirements Descrizione del caso d’uso

  14. Real-Ome Fraud DetecOon Real-0me ingesOon of credit card transacOons

    Stream processing, data validaOon, and fraud detecOon Secure storage of transacOons Real-0me analysis and reporOng
  15. AddiOonal Requirements Many heterogeneous event producers (temporary credenOals) Elas0c

    architecture (no upfront costs & easy to scale up) Extensible architecture (plug-and-play components) A lot of data (cheap storage, please!)
  16. Serverless Architecture Come lo implemen7amo senza ges7re server?

  17. Architecture Diagram

  18. IngesOon & Processing (simple)

  19. IngesOon & Processing (opOmized)

  20. IngesOon & Processing (real-Ome analyOcs)

  21. Serverless Data Analysis

  22. Hands-on Demo Sporchiamoci un po’ le mani

  23. < demo >

  24. Grazie =) Domande? 10/19/2017