A simple serverless data pipeline

Building and maintaining data pipelines when it’s not your full time job is a pain! So better keep things simple without the need to manage the system yourself. In this talk I’ll show a data pipeline architecture built leveraging some cloud offerings by AWS and Preset.

In this talk we’ll discuss:
- an overview of the architecture
- the data lake: AWS S3, AWS Athena
- the exploration and visualization platform: Apache Superset
- data formats and Python implementations
- vendors lock-in

Transcript

1. A simple serverless data pipeline Riccardo Magliocchetti - @rmistaken / @rmistaken@hachyderm.io Pycon(IT) 23
2. I am not a data engineer
3. Agenda for today - An overview of a simple serverless data pipeline architecture - The data lake: AWS S3, AWS Glue, AWS Athena - Apache Superset, the exploration and visualization platform - Data formats - Vendor lock-in
4. The architecture
6. The context - Small team - No data engineer - End users are business people - No real time requirements, data produced daily - Small quantity of data
7. The rest of the team - Nicola Martino - Bartolo Albanese
8. The data lake
10. AWS S3 - Cloud object storage: a persistent cache that looks like a filesystem - Cheap unless you move a lot of data outside AWS - De facto standard
11. AWS Glue - Serverless data integration service - Pay per use - Can do data transformations but you can DIY: event notifications plus lambdas - Data catalog functionality for data in S3
12. AWS Athena - Serverless query service: query Glue data in SQL - Pay per use, 5$ / TB
13. Apache Superset
14. Apache Superset - Data exploration and visualization platform - Written in Python (Flask, SQLAlchemy, Pandas) and Javascript - Plenty of databases supported - SQL Editor included for exploration - Hosted version from Preset.io
15. Apache Superset: Databases and Datasets - You connect to databases - You choose which database table to expose as a Dataset
16. Apache Superset: Chart
17. Apache Superset: Dashboard
18. Apache Superset: SQL Editor
19. Data formats
20. Data formats - Started simple with CSV: text, best debugging - Optimized size with ORC: binary, columnar, compressed - Optimized DX with Parquet: same but a bit less space efficient, bigger ecosystem
21. Vendor lock-in
22. Vendor lock-in - This solution is based on AWS because it’s already in use - Every cloud provider has a different take on data engineering, e.g. in Google Cloud Platform it would be BigQuery based - Can be replicated using open source software: - minIO as cloud storage - Apache Hive (or similar) to ingest data and get queried by Apache Superset - Apache Superset, self-hosted
23. Conclusions
24. Conclusions - You can build simple data pipelines cheaply without much operations overhead - Interesting for small teams without data specific roles or knowledge - Serverless makes sense for small scale
25. References - https://aws.amazon.com/big-data/datalakes-and-analytics/modern-data-architecture/ The AWS idea of a data architecture - https://preset.io/blog/2021-5-25-data-lake-athena/ An example of connecting Apache Superset to AWS Athena - https://superset.apache.org/docs/intro Apache Superset Documentation
26. Thanks! Riccardo Magliocchetti @rmistaken / @rmistaken@hachyderm.io