Un data lake semplice e serverless su AWS

Video: https://video.linux.it/w/6HoHhhyrnfqYnzZiM9sRx5?start=28m&stop=40m40s

Creare e mantenere data pipeline, quando non è il tuo lavoro, risulta particolarmente noioso: quindi meglio fare le cose semplici, magari facendo amministrare i sistemi agli altri.

In questo intervento vediamo un'architettura per un data lake, ossia dove conservare e processare i dati su cui vogliamo fare analisi, in modalità serverless, cioè senza dover amministrare noi server e sistemi, usando l'infrastruttura di AWS.


Riccardo Magliocchietti — Freelance software tinkerer

Transcript

1. Un data lake semplice e serverless su AWS Riccardo Magliocchetti - @rmistaken / @rmistaken@hachyderm.io Data Beers Torino
2. Agenda for today - The context and the architecture - The data lake - Data formats - Vendor lock-in
3. The architecture
4. The context - The problem: pushing business metrics out of a web application - Small team, no data engineer - No real time requirements, data produced daily - Small quantity of data
5. The team - Nicola Martino - Bartolo Albanese
7. The data lake
9. AWS S3 - Cloud object storage: a persistent cache that looks like a filesystem - Cheap unless you move a lot of data outside AWS - De facto standard
10. AWS Glue - Serverless data integration service - Pay per use - Can do data transformations but you can DIY: event notifications plus lambdas - Data catalog functionality for data in S3
11. AWS Athena - Serverless query service: query Glue data in SQL - Pay per use, 5$ / TB
12. Data formats
13. Data formats - Started simple with CSV: text, best debugging - Optimized size with ORC: binary, columnar, compressed - Optimized DX with Parquet: same but a bit less space efficient, bigger ecosystem
14. Vendor lock-in
15. Vendor lock-in - Can be replicated using open source software: - minIO as object storage - Apache Hive (or similar) to ingest data and get queried in SQL
16. Conclusions
17. Conclusions - You can build simple data pipelines cheaply without much operations overhead - Interesting for small teams without data specific roles or knowledge - Serverless makes sense for small scale
18. References - The AWS idea of a data architecture https://aws.amazon.com/big-data/datalakes-and-analytics/modern-data-architecture/
19. Thanks! Riccardo Magliocchetti @rmistaken / @rmistaken@hachyderm.io