DevOpsPorto Meetup 38: Using cutting-edge open-source technologies to build one of the biggest industrial Data Lake of the World by Allan Sene

Transcript

1. www.datasprints.com Using cutting-edge open-source technologies to build a World-class Industrial

   Data Lake

2. Hello! My name is Allan Sene Co-Founder & CTO @Data

   Sprints Co-Founder & Podcaster & Instructor @Data Hackers +10 years in Data & Software, +4 years as Data Engineer www.datasprints.com

3. Agenda www.datasprints.com The Challenge What is a Data Lake? Awesome

   cutting-edge data tools Putting everything together do build a IDL Results and Next Steps Q&A

4. THE CHALLENGE

5. • Multinational Steel Industry, with plants worldwide • They need

   to give to global managers the capability of track the production line • Data columns have BLOBs, Arrays and complex data types • Migrating from on-prem to cloud • Data sets with 70 Gb (compressed) and 8 million lines, more than 1600 columns, very complex queries (+200 joins), 15 minutes delay • Maximum Query Response time of 30s • Very limited cloud budget - USD 15.000,00/year DSPT Webinar + DevOps Porto: Cutting-edge open-source to build Data Lakes www.datasprints.com The Challenge

6. DSPT Webinar + DevOps Porto: Cutting-edge open-source to build Data

   Lakes www.datasprints.com Stakeholder's name, can you guess it?

7. DSPT Webinar + DevOps Porto: Cutting-edge open-source to build Data

   Lakes www.datasprints.com Milagres (Miracles)

8. What is a Data Lake?

9. “Is a Data Repository that holds a huge quantity of

   data on raw state, structured or not. The schema is only defined when is necessary for consumption" (Anne Buff - Best Practices Leader at SAS) DSPT Webinar + DevOps Porto: Cutting-edge open-source to build Data Lakes www.datasprints.com

10. DSPT Webinar + DevOps Porto: Cutting-edge open-source to build Data

    Lakes www.datasprints.com So, Data Lake is...

11. DSPT Webinar + DevOps Porto: Cutting-edge open-source to build Data

    Lakes www.datasprints.com Not really...

12. DSPT Webinar + DevOps Porto: Cutting-edge open-source to build Data

    Lakes www.datasprints.com • Security & Audit • Catalog & Access • Data Pipelines & Orchestration Basically, a Platform that ensure safe and efficient data consumption A Data Lake must have

13. Awesome cutting-edge data tools

14. “ • Data Lake Engine • PMCs of Apache Arrow

    • User-Friendly • Open-source • Does have an Enterprise Version Dremio

15. “ • SQL & API • Wiki • Connections: ◦

    S3, HDFS, AzureFS ◦ Mongo, Elastic, ◦ Postgres, MySQL • Supports JSON, CSV, Parquet, Avro… • Reflections Dremio

16. DSPT Webinar + DevOps Porto: Cutting-edge open-source to build Data

    Lakes www.datasprints.com Dremio's UI

17. “ • Dbt = "Data Build Tool" • Built by

    Fisthtown Analytics • Data Pipeline Orquestration • "Airflow for Data Analysts" • Open-source • SaaS Version dbt

18. “ • ETL over a MPP • Code Versioning •

    Data Lineage & Docs • Data Validation • Seed Data dbt

19. DSPT Webinar + DevOps Porto: Cutting-edge open-source to build Data

    Lakes www.datasprints.com dbt's UI

20. Putting everything together do build a DL

21. DSPT Webinar + DevOps Porto: Cutting-edge open-source to build Data

    Lakes www.datasprints.com Bringing all together • Pipelines & Orchestration => dbt • Catalog & Access => Dremio • Processing => Spark • Storage => Amazon S3

22. DSPT Webinar + DevOps Porto: Cutting-edge open-source to build Data

    Lakes www.datasprints.com Bringing all together JDBC Micro-batch Data Repartitioned Data Repartition & Schema Evolution Optimization Coalesced Data Data Port Data Explorer

23. “

24. Results and Next Steps

25. DSPT Webinar + DevOps Porto: Cutting-edge open-source to build Data

    Lakes www.datasprints.com Results, until now • Platform built on 6 months by a team of 4 engineers full-time • <add plots here>

26. DSPT Webinar + DevOps Porto: Cutting-edge open-source to build Data

    Lakes www.datasprints.com Next steps • Integrating with very common Analytics tools (Power BI, Tableau…), deployed worldwide • Data Science Models consuming data through the Data Bus • Data Quality Monitoring

27. www.datahackers.com.br

28. Questions? www.datasprints.com

29. www.datasprints.com allan@datasprints.com Thanks! https:/ /www.linkedin.com/in/allansene/