Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Modern Data Warehouse using Azure - ADLS & ADF

Modern Data Warehouse using Azure - ADLS & ADF

Slidedeck of the part of building Modern Data Warehouse using Azure. We focus on Petabyte scale Azure Data lake Gen 2 (ADLS) and the Hybrid Cloud integration Azure Data Factory in-depth in this session.

The recording of the session is available on Youtube
https://www.youtube.com/watch?v=d6X6XlCpowo&WT.mc_id=DP-MVP-5003170

9e33a1d43a88f23f6c545c1e0f07f4b5?s=128

Nilesh Gule

August 18, 2020
Tweet

More Decks by Nilesh Gule

Other Decks in Technology

Transcript

  1. Nilesh Gule @nileshgule | www.HandsOnArchitect.com Modern Data Warehouse Using Azure

  2. $whoami { “name” : “Nilesh Gule”, “website” : “https://www.HandsOnArchitect.com", “github”

    : “https://github.com/NileshGule" “twitter” : “@nileshgule”, “linkedin” : “https://www.linkedin.com/in/nileshgule”, “email” : “nileshgule@gmail.com", “likes” : “Technical Evangelism, Cricket”, “co-organizer” : “Azure Singapore UG” }
  3. None
  4. None
  5. Credits: James Serra

  6. None
  7. Data Lake https://dzone.com/articles/data-lake-governance-best-practices

  8. Azure Data Lake Storage

  9. Azure Data Factory

  10. Azure Data Factory – copy data

  11. Azure Data Factory – Mapping Data Flow Credits: Harun Legoz

    for football match analysis example
  12. Azure Data Factory - Monitoring

  13. Summary - Azure Data Lake Storage • Petabyte scale storage

    • Hierarchical namespace • Hadoop compatible access with ABFS driver Main features • Use Service Principles • Use Security Groups over individual users • Enable Gen 2 firewall with Azure services access ADLS best practices
  14. Summary - Azure Data Factory • Cloud ETL service •

    Scale-out serverless data integration & data transformation • Code-free UI • Monitoring & Management Main features • Linked Services • Datasets • Pipelines • Triggers Main components • Auto Resolver • Self Hosted • SSIS Integration Runtimes
  15. ADLS Gen 2 Storage Account Azure Data Factory Azure Data

    Factory Mapping Data Flows Data Lake Governance Best Practices Azure Services Supporting Managed Identities
  16. References – MS Learn https://docs.microsoft.com/en-us/learn/paths/data-processing-with-azure-adls/

  17. Thank you very much Code with Passion and Strive for

    Excellence https://www.slideshare.net/nileshgule/presentations https://speakerdeck.com/nileshgule/
  18. Nilesh Gule ARCHITECT | MICROSOFT MVP “Code with Passion and

    Strive for Excellence” nileshgule @nileshgule Nilesh Gule NileshGule www.handsonarchitect.com
  19. Q&A