Microsoft Fabric and Azure Databricks - Better together

Nesta palestra iremos explorar as características e capacidades do Azure Databricks e Fabric — uma plataforma analítica fácil e colaborativa, e uma solução unificada para análise de IA. 🤖
Vou mostrar como o Azure Databricks e o Fabric se integram através do OneLake, possibilitando experiências analíticas e de IA enriquecidas.🤯
Preparar para discutir arquiteturas e o posicionamento do produto para o Fabric e o Databricks.

Transcript

1. Apoio Balneário Camboriú 01 JUNHO, 2024 Realização
2. Sidney Cirqueira Microsoft Fabric e Azure Databricks Better together Nesta palestra iremos explorar as características e capacidades do Azure Databricks e Fabric — uma plataforma analítica fácil e colaborativa, e uma solução unificada para análise de IA. Vou mostrar como o Azure Databricks e o Fabric se integram através do OneLake, possibilitando experiências analíticas e de IA enriquecidas. Preparar para discutir arquiteturas e o posicionamento do produto para o Fabric e o Databricks.
3. About Me • + 10 years working with Data & Analytics • Sr. Cloud Solution Architect (Mission Critical) – Azure Data • MBA in Business Analytics & Big Data – FGV • Speaker in the MS Technical Community • YouTube: Sidney Cirqueira • Twitter: @sidneyocirqueira • Instagram: @sidneycirqueiradataandai • LinkedIn: /sidneyoliveiracirqueira
4. Microsoft Fabric does it all—in a unified solution An end-to-end analytics platform that brings together all the data and analytics tools that organizations need to go from the data lake to the business user
5. Unified data analytics platform for accelerating innovation across data science, data engineering, and business analytics Original creators of popular data and machine learning open source projects Global company with 5,000 customers and 450+ partners What is Databricks?
6. Lakehouse Architecture • A data lake for all data with an open, transactional curated layer • A foundational compute service to support all primary data lake use cases • Easy integrations with other tools and services to enable any additional or future customer use cases Key Capabilities:
7. Medallion Architecture
8. Data Engineering Real-time & streaming analytics Data Warehouse Data Science Lakehouse Data Integration 150+ Data Sources Migration Federation Virtualization / Mirroring AWS S3 Google Cloud Storage Azure CosmosDB Dataverse Amazon RedShift Google BigQuery Data Sources Azure SQL Azure DataLake Storage Snowflake Power BI Datasets Microsoft Excel Copilot Azure AI Studio Mosaic AI Azure Machine Learning The best of both worlds
9. Gold Layer Meets functional consumption requirements Ingest Copy data using Data Factory Bronze Layer Typically, raw (technical) and unprocessed data Process Silver Layer Typically, cleaned, filtered, light modifications Process Synapse Engineering Operational systems Blob landing zone OneLake Tables Tables Tables Orchestrate using Data Factory Microsoft Fabric (Workspace) Landing area Temporary storage location Synapse Warehouse Microsoft Fabric - Lakehouse Architecture Data Sources
10. ADLS Gold Ready for consumption Ingest Copy data using Data Factory ADLS Bronze Typically, raw and different file formats Process ADLS Silver Typically, cleansed and standardized file formats Process Azure Databricks Azure Databricks Operational systems Blob landing zone Import mode for PBI datasets Orchestrate using Azure Data Factory Direct Query for fast updates Databricks SQL Ad-hoc and slow reporting Lakehouse OneLake Process Process Different PowerBI integration patterns Data Sources Azure Databricks – Current Lakehouse Architecture
11. ADLS Gold Ready for consumption Ingest Copy data using Data Factory ADLS Bronze Typically, raw and different file formats Process ADLS Silver Typically, cleansed and standardized file formats Process Azure Databricks Azure Databricks Operational systems Blob landing zone Import mode for PBI datasets Orchestrate using Azure Data Factory Direct Query for fast updates Databricks SQL Ad-hoc and slow reporting Lakehouse OneLake Tables Direct Lake (Fabric) Process Process Data Sources Azure Databricks – Lakehouse Architecture
12. Lakehouse Architecture – ADLS as Gold layer Ingest Copy data using Data Factory ADLS as Bronze Typically, raw and different file formats Process ADLS as Silver Typically, cleansed and standardized file formats Process Serve Provide shortcuts Azure Databricks Azure Databricks Operational systems Blob landing zone ADLS as Gold Ready for consumption Orchestrate using Azure Data Factory Microsoft Fabric (SaaS) Process Process Integration with Shortcuts Data Sources
13. Lakehouse architecture – OneLake as Gold layer Ingest Copy data using Data Factory ADLS as Bronze Typically, raw and different file formats Process ADLS as Silver Typically, cleansed and standardized file formats Process Operational systems Blob landing zone Microsoft Fabric (SaaS) Synapse Data Science Power BI Azure Databricks Azure Databricks OneLake as Gold Ready for consumption Tables Orchestrate using Azure Data Factory Process Process Data Sources
14. Store all Delta tables in OneLake Lakehouse / Warehouse Gold Ready for consumption Ingest Copy data using Data Factory Lakehouse Bronze Typically, raw and different file formats Lakehouse Silver Typically, cleansed and standardized file formats Serve Provide shortcuts Azure Databricks Azure Databricks Operational systems Blob landing zone OneLake Tables Tables Tables Microsoft Fabric (SaaS) Data science Reporting Orchestrate using Azure Data Factory Process Process Data Sources
15. Store all Delta tables in OneLake Lakehouse / Warehouse Gold Ready for consumption Ingest Copy data using Data Factory Lakehouse Bronze Typically, raw and different file formats Lakehouse Silver Typically, cleansed and standardized file formats Serve Provide shortcuts Azure Databricks Azure Databricks Operational systems Blob landing zone OneLake Tables Tables Tables Microsoft Fabric (SaaS) Data science Reporting Orchestrate using Data Factory in Microsoft Fabric Process Process Data Sources
16. Demo - End-To-End Fabric with Databricks Lakehouse architecture Ingest Copy data using Data Factory Bronze raw file format Process Silver Process Lakehouse in OneLake Azure Databricks Azure Databricks Gold Orchestrate using Data Factory in Microsoft Fabric Microsoft Fabric (SaaS) Process Process Direct Lake Mode Azure SQL DB Source Shortcuts Z-ORD V-ORD
17. Online Resources – Databricks + Fabric Don't be beguiled by Microsoft Fabric Shortcuts (yet) | Databricks Blog (archive.org) Integrate OneLake with Azure Databricks - Microsoft Fabric | Microsoft Learn Integrate Databricks Unity Catalog with OneLake - Microsoft Fabric | Microsoft Learn Azure Databricks personal access token authentication - Azure Databricks | Microsoft Learn Azure Databricks activity - Microsoft Fabric | Microsoft Learn • Don't be beguiled by Microsoft Fabric Shortcuts (yet) | Databricks Blog (archive.org) • Integrate OneLake with Azure Databricks - Microsoft Fabric | Microsoft Learn • Integrate Databricks Unity Catalog with OneLake - Microsoft Fabric | Microsoft Learn • Azure Databricks personal access token authentication - Azure Databricks | Microsoft Learn • Azure Databricks activity - Microsoft Fabric | Microsoft Learn
18. Thanks for all & below are my contacts YouTube: Sidney Cirqueira Twitter: @sidneycirqueira
19. Realização Apoio Balneário Camboriú 01 JUNHO, 2024 Agradecimentos