Domain Driven Design & Data Mesh

Data mesh is a decentralized sociotechnical approach for data management. Focusing on the technology, we tend to neglect the socio aspect of moving to new decentralized approaches. If we want to invest in data mesh architecture work, we must also invest in the people to make our data mesh more sustainable. How do we make people feel responsible for the data they expose and use? How do we make data management meet business objectives instead of a bureaucratic burden.

Decentralized approaches, e.g. like microservices, have been around for a while now. And we notice similar patterns of companies moving to a microservices architecture happening again when moving towards data mesh. Join this webinar to learn from these experiences and prevent obstacles for the future.

Transcript

1. Domain Driven Design & Data Mesh Webinar 15/03/2022 Kenny Baas-Schwegler Peter Kromhout Niels Zeilemaker
2. Photo by Şahin Sezer Dinçer on Unsplash
3. “Most organizations don’t have their Data Management in order, and have found themselves in a data mess because of it”
4. Businesses miss out on opportunities
5. It is a waste of resources
6. You break the law 746 225 90 60 60 50 35 27,8 22 20,4 17 12,3 10,4 9 8,5 8,15 7 6 5 4,5 0 100 200 300 400 500 600 700 800 Amazon WhatsApp Google Ireland Facebook Google LLC Google H&M TIM British Airways Marriott Wind Vodafone Italia Notebooksbilliger Austrian Post Eni Vodafone Spain Google Caixabank BBVA bank Fastweb In millions € 20 biggest GDPR fines till date (900 in total)
7. Agenda • Why do we do what we do? • From centralised to decentralised, we've been here before.. • What do the data mesh principles have to do with data management? • How do we get to a domain-oriented ownership? • The engineering way of solving this problem • Why a Data Mesh is a decentralized sociotechnical approach
8. Why do we do what we do? * * = Data- engineers, analysts, scientists, architects, leaders ..
9. How do we try to achieve this goal of Data Driven Organizations? Volume, Velocity & Variety Tooling & technology Growth in number of data professionals
10. “The percentage of firms identifying themselves as being data-driven has declined in each of the past 3 years — from 37.1% in 2017 to 32.4% in 2018 to 31.0% this year” (HBR - Companies Are Failing in Their Efforts to Become Data-Driven - 2019) “Today, most firms use data in some fashion for decision making, but many can’t claim to be fully “data-driven.” In fact, only 24% said they had created a data-driven organization.” (NVP - Big Data and AI Executive Survey - 2021)
11. From centralised to decentralised approach to data
12. From centralised to decentralised, we've been here before..
14. How do we split a Monolithic application? https://martinfowler.com/articles/microservices.html
15. Big ball of Mud When we have no correct boundaries of functionality, meaning a software systems lack an explicit architecture around these functionalities. Changing one functionality impacts the other functionalities.
16. Leads to
17. From 2003 - Get a shared understanding of the problems our domain experts have.
20. Instead of one canonical language, create multiple bounded languages
21. A model is a simplified representation of a thing or phenomenon that intentionally emphasizes certain aspects while ignoring others. Abstraction with a specific use in mind. — Rebecca Wirfs-Brock
22. Stakeholders Team developing software Others involved designing the software
23. What do the data mesh principles have to do with data management?
24. One of the opportunities with Data Mesh is to address the Data Management problem organizations face (O’Reilly – Data Mesh – Zhamak Dehgani - 2022) Data Mesh principles Very much needed & Huge risk for chaos, inefficiency and misalignment Engineering way of solving the data management issue
25. “Data mesh, at its core, is founded in decentralization and distribution of data responsibility to people who are closest to the data” “Data product ownership: long-term ownership of responsibilities to create, model, maintain, evolve, and share data as a product to meet the needs of data users.” (O’Reilly – Data Mesh – Zhamak Dehgani - 2022) (O’Reilly – Data Mesh – Zhamak Dehgani - 2022) Data Owner Data Users
26. (O’Reilly – Data Mesh – Zhamak Dehgani - 2022) One long-standing challenge of existing analytical data architectures is the high friction and cost of using data: discovering, understanding, trusting, exploring, and ultimately consuming quality data. Data becomes a 1st class citizen instead of an after thought Data product characteristics: (O’Reilly – Data Mesh – Zhamak Dehgani - 2022)
27. How do we get to a domain-oriented ownership?
28. Domain • A sphere of knowledge, influence, or activity. The subject area to which the user applies a program is the domain of the software. • an area of interest or an area over which a person has control • It is all about grouping concepts. Photo by Randy Fath on Unsplash
29. Photo by Michal Balog on Unsplash Photo by Raul Gonzalez Escobar on Unsplash
34. What influences our grouping of the domains? • Customers • Resources • People • Team cognitive load • Cohesion of change • Change rate • Culture • Business strategy and there value streams • Software architecture • Organisation structure • Knowledge & Practice • ….. • Aka Contextual!
35. Collaborative Modelling
37. The engineering way of solving this problem
38. A refreshed look at data governance is the missing and final piece to make data mesh work. (O’Reilly – Data Mesh – Zhamak Dehgani - 2022) … rely on automated mechanisms built by the platform and embedded into the distributed architecture ... (O’Reilly – Data Mesh – Zhamak Dehgani - 2022) Examples of governance that can be automated: But How?!
39. Data product Catalogue Data product monitoring Experience plane Data Product plane Output port Output port Bounded context Metadata store Bounded context Metadata: - Descriptions - Classifications - SLA’s - Quality - Sample data - Endpoint - Owner - Usage - Access The 'Experience plane'
40. Data product Catalogue Data product monitoring Metadata store Experience plane Data Product plane Output port Output port Bounded context Bounded context Bounded context Output port The 'Experience plane'
41. Why a Data Mesh is a decentralized sociotechnical approach
44. End2End Live Conference: https://www.youtube.com/watch?v=VxPVWjEVc6Q
45. Wrap up Data Mesh is an investment in technology and in the people! Use collaborative modelling to design domain-oriented ownership Use an engineering approach to solve the governance problem Getting your data management in order is not limited to Data Mesh
46. Contact us Peter Kromhout – pkromhout@xebia.com Kenny Baas-Schwegler – kbaas@xebia.com – twitter: @kenny_baas Niels Zeilemaker – nielszeilemaker@godatadriven.com