Snowflake - Big Data Expo

Transcript

1. © 2022 Snowflake Inc. All Rights Reserved 1 One Size Does Not Fit All: Make The Right Data Mesh For You Prepared for presentation at Big Data Expo NL, 2022 Dr. Jennifer Belissent | Principal Data Strategist | Snowflake Fernando Raposo | Senior Data Engineer | Findhotel
2. © 2022 Snowflake Inc. All Rights Reserved BEWARE OF DATA SILOS Finance Web/ Log Data Product Usage Marketing OLTP Databases IoT Enterprise Applications Sales Suppliers Operations Third Party Agencies
3. © 2020 Snowflake Inc. All Rights Reserved AND DATA BOTTLENECKS Organization Product Finance Marketing Data Project Data Project Data Project Data Team
4. © 2020 Snowflake Inc. All Rights Reserved DECENTRALIZED DATA TEAMS?! Organization Finance Product Data Project Data Project Data Project Data Team Data Team Data Team Marketing
5. © 2021 Snowflake Inc. All Rights Reserved A Data Mesh Enables Distributed Self-Service With Federated Governance
6. © 2021 Snowflake Inc. All Rights Reserved … “a decentralized socio-technical approach to managing and accessing analytical data at scale.” It is distributed, not centralized or monolithic It’s about architecture and organizational principles. It’s about data management for analytics, not for operational systems. It aims to improve organizational scalability. DATA MESH IS…
7. © 2021 Snowflake Inc. All Rights Reserved DATA MESH: FOUR PRINCIPLES Distribute responsibility for data pipelines and data quality to people with domain knowledge. Serve data as-a-product using a common self-service IT infrastructure platform to ensure consistency. Domain-Centric Ownership & Architecture Data as-a-Product Self-Serve Data Platform Federated Governance • Data products reflect use cases, and are easily discoverable • Data products are well-documented, easy to obtain and use • Data products define quality requirements • Domains are accountable to “customers” • Domain-agnostic, common tool set • Easy to use and low maintenance to support • Easy to deploy repeatable patterns for common requirements: cleansing, transformation, automation, storage, security, governance, sharing • Data governance is coordinated global to ensure interoperability across domains • Access and use policies are coordinated • Governance is applied within domains • Data pipelines owned by teams with domain knowledge • Domains own cleansing, refinement, historization, pre-aggregation, etc. • Domains responsible for data governance • Domains deliver data products to “customers”
8. © 2021 Snowflake Inc. All Rights Reserved The 4 principles support each other Domain-Centric Ownership & Architecture Data-as-a-Product Self-Service Data Platform Federated Governance FOUR INTERCONNECTED PRINCIPLES Source: Zhamak Dehghani Reduce cost and complexity of building and maintaining data products Empower domain teams Enforce global standards Ensure interoperability Reduce domain isolation Ensure product quality Connect data products Prevent data silos Ensure data use Data-as-a-Product
9. THE DATA MESH ARCHITECTURE Domain: Customer Domain: Sales Domain: Products Domain: Marketing Domain: Customer 360 Inventory of shared data products Snowflake Managed Account The Data Mesh Architecture Consumers Data Sources Interoperability Standards, Federated Governance, 3rd Party Tools Data Domains: • Can be functional or topical (e.g. customer, item/sku, etc.) • Can consume and share data or functions • Control access policies, data masking, etc. for downstream consumers • Can share external tables, i.e. provide access to data outside of Snowflake • Can provide reader accounts for non-Snowflake consumers • Must deliver products, and comply with federated governance • Data Marketplace / Catalog: • Connects providers to consumers • Inventory of available assets • No central storage of shared data • Providers retain full control over shared assets (data, functions) • Consumers access live provider data, no copies or ETL required Data Consumers: • Discovers and uses data or services or applications directly (no copying or moving) Snowflake Data Marketplace or 3rd-party catalog 3rd party marketing agency Reseller Sales Analysts Churn & Retention Business optimization Finance & Controlling
10. © 2021 Snowflake Inc. All Rights Reserved 10 Centers of Excellence coordinate data and analytics processes and projects, with a central hub and distributed spokes. Function Function Region LOB LOB CoE Hub At the hub of a CoE, data insights leaders drive strategy, establish enterprise-wide data governance and deliver services including talent recruitment, training, vendor management and data and analytic services. Gray areas reflect shared responsibilities depending on spoke-level capabilities. The spokes of a CoE lie within the lines of business, functional teams or geographical units. They define use cases, oversee data execution, track performance and measure outcomes LOB THE DATA ORGANIZATION
11. © 2022 Snowflake Inc. All Rights Reserved Let’s hear about how Findhotel leverages its data mesh.
12. Get every traveler the best accommodation deal, worldwide Our mission Confidential ⓒ FindHotel 2022. All rights reserved
13. About Us ⊲ Launched in 2010, we have scaled profitability within the online travel (marketplace) industry, offering price comparison and direct booking options ⊲ Headquartered in Amsterdam, with a distributed team of 150+ experienced professionals active across three continents ⊲ We work with the one of the largest & most competitive inventory in the world, using it on a CPA basis. This ensures the highest liquidity of supply in the market, better prices for consumers and a win-win-win formula for publishers-booking engines-consumers. ⊲ We are a trusted partner to all of the major (travel) marketing networks, like Google, TripAdvisor, Bing, etc. Our Major Suppliers Confidential ⓒ FindHotel 2022. All rights reserved
14. Our Data Mesh so far… ⊲ Ad hoc evangelism ⊲ Framework to assess the data maturity of each team ⊲ Tools that automate the deployment of data infrastructure ⊲ Central repository for metadata ⊲ Domain mapping framework (simplified) ⊲ Split the monolith into business domains and delegated ownership Confidential ⓒ FindHotel 2022. All rights reserved
15. Learnings and next steps ⊲ Clear communication of the strategy and direction ⊲ Domain mapping should be agreed upon before any deployment ⊲ Focus on Governance very early ⊲ Assemble a Data Council Confidential ⓒ FindHotel 2022. All rights reserved
16. 1. Assess your organizations needs and current frustrations. This is often called a listening tour. 2. Determine skills levels and gaps across teams. This will suggest which domains will be more autonomous first, and how to support others. 3. Start with a domain and use case, then iterate to expand domain ownership and self-service infrastructure access. KEY TAKEAWAYS HOW TO GET STARTED, TODAY!
17. © 2022 Snowflake Inc. All Rights Reserved THANK YOU
18. Reference Material Capgemini on "Why Snowflake is a good match for implementing Data Mesh" https://www.capgemini.com/no-no/2021/05/why-snowflake-is-a-good-match-for-implementing-data-mesh/ Data Mesh at Siemens Healthineers https://resources.snowflake.com/case-study/snowflake-enables-siemens-healthineers-it-to-optimize-business-operations-around-the-gl obe The Data Mesh Journey at Roche https://www.snowflake.com/blog/data-mesh-perspectives-a-qa-with-roche-diagnostics/ https://f.hubspotusercontent30.net/hubfs/5870630/Roche%20and%20DataOps.live%20case%20study.pdf Building a Data Mesh with Snowflake at Iterable https://www.snowflake.com/blog/building-iterables-data-mesh-using-snowflake-three-components-of-an-innovative-data-management-strategy/ Data Mesh at Flexport: Driving Buy-in and Social/Org Challenges https://www.youtube.com/watch?v=-POiudR2_R0 https://docs.google.com/presentation/d/e/2PACX-1vQpZUaVDrBh1ZnMic-2BP1f_N-Zu3QKpaACagA1lrOkXlGJ5s_cMKmx342N2FpB3FAryKMXW46BNDDG/pub#sli de=id.gf1a3223b00_0_30 Empower Data Teams with a Data Mesh Built on Snowflake https://www.snowflake.com/guides/data-mesh-self-service-data https://www.snowflake.com/blog/empower-data-teams-with-a-data-mesh-built-on-snowflake/