Modern Data Warehouse using Azure - ADLS & ADF

Slidedeck of the part of building Modern Data Warehouse using Azure. We focus on Petabyte scale Azure Data lake Gen 2 (ADLS) and the Hybrid Cloud integration Azure Data Factory in-depth in this session.

The recording of the session is available on Youtube
https://www.youtube.com/watch?v=d6X6XlCpowo&WT.mc_id=DP-MVP-5003170

Transcript

1. Nilesh Gule @nileshgule | www.HandsOnArchitect.com Modern Data Warehouse Using Azure
2. $whoami { “name” : “Nilesh Gule”, “website” : “https://www.HandsOnArchitect.com", “github” : “https://github.com/NileshGule" “twitter” : “@nileshgule”, “linkedin” : “https://www.linkedin.com/in/nileshgule”, “email” : “nileshgule@gmail.com", “likes” : “Technical Evangelism, Cricket”, “co-organizer” : “Azure Singapore UG” }
5. Credits: James Serra
7. Data Lake https://dzone.com/articles/data-lake-governance-best-practices
8. Azure Data Lake Storage
9. Azure Data Factory
10. Azure Data Factory – copy data
11. Azure Data Factory – Mapping Data Flow Credits: Harun Legoz for football match analysis example
12. Azure Data Factory - Monitoring
13. Summary - Azure Data Lake Storage • Petabyte scale storage • Hierarchical namespace • Hadoop compatible access with ABFS driver Main features • Use Service Principles • Use Security Groups over individual users • Enable Gen 2 firewall with Azure services access ADLS best practices
14. Summary - Azure Data Factory • Cloud ETL service • Scale-out serverless data integration & data transformation • Code-free UI • Monitoring & Management Main features • Linked Services • Datasets • Pipelines • Triggers Main components • Auto Resolver • Self Hosted • SSIS Integration Runtimes
15. ADLS Gen 2 Storage Account Azure Data Factory Azure Data Factory Mapping Data Flows Data Lake Governance Best Practices Azure Services Supporting Managed Identities
16. References – MS Learn https://docs.microsoft.com/en-us/learn/paths/data-processing-with-azure-adls/
17. Thank you very much Code with Passion and Strive for Excellence https://www.slideshare.net/nileshgule/presentations https://speakerdeck.com/nileshgule/
18. Nilesh Gule ARCHITECT | MICROSOFT MVP “Code with Passion and Strive for Excellence” nileshgule @nileshgule Nilesh Gule NileshGule www.handsonarchitect.com
19. Q&A