The REWE Data Platform - A step towards a data mesh

A data mesh in the wild – in this talk we will talk openly about our journey of setting up a data mesh at REWE International. We will cover the good, the bad and the ugly, starting from the reason why we decided on a data mesh, the initial attempts and challenges and
the current state and architecture. We will end with an overview of our future plans for the REWE data platform.

Transcript

2. 2 HAPPY 10TH EDITION OF THE DATA ENGINEERING MEETUP
3. THE REWE DATA PLATFORM 1. Vision 2. The billion-euro cake 3. Building blocks of a data mesh 4.Challenges 5. What we got right – and the impact it had 6. A platform is never finished
4. 4 REWE International AG is part of the REWE Group - one of Europe's leading trading and travel and tourism groups. REWE Interna+onal AG 9,21 bn Euro Total External Sales 2021 92 Thousand Employees 2021 2 Thousand Apprentices 2021 2,65 Thousand Markets 2021 The REWE International AG in figures
5. 5 The IT department with around 480 employees, creates customized software and hardware solutions for all domestic and international divisions of the group. Focus IT: Pleased to meet you!
6. 6 Jobs at REWE IT – Data Platform Interested to join us? https://rewe-group.jobs/jobs/it-handelinternational Contact details: Mail: F.Pribahsnik@rewe-group.at
7. 7 WE ARE Follow us on LinkedIn:
8. THE VISION 1
9. 9 • Self-service producers and consumers • Data catalogue • Centralized compliance THE VISION
10. BILLION-EURO CAKE 2
11. 11 • Obvious data error • Requires collaboration between different departments • Sensitive data (prices) • Difficult to propagate the fix through all the systems WHY A CAKE
12. BUILDING BLOCKS OF A DATA MESH 3
13. 13 MEANING • Data is scattered across teams • Meaining is unclear or different outside the context of the team
14. 14 QUALITY • Quality gates do no exists, data is not checked • Data graveyards • By-product of the operational processes
15. 15 SECURITY • Some data is sensitive • Usage of some data must be tracked and reported • The people that produce the data are not always aware of all the rules
16. 16 AVAILABILITY • Data is difficult to find • It takes weeks to get access to data • It takes weeks to give access to data
18. 18 DOMAIN OWNERSHIP • Article masterdata domain is responsible for all article data • The team makes the data available to all
19. 19 DATA PRODUCTS • The team is responsible for the data quality • Internal customer satisfaction is tracked • Tests catch errors
20. 20 FEDERATED GOVERNANCE • A governence team decides on rules and access control • Rules are enforced by the system
21. 21 SELF-SERVICE DATA PLATFORM • Data is easy to find • Data is easy to publish and consume • The platform enforces the rules
22. ORGANISATIONAL AND TECHNICAL CHALLENGES 4
23. 23 TECHNICAL CHALLENGES • Ecosystem is not mature, tools don’t always work • Massive amounts of data
24. 24 ORGANIZATIONAL CHALLENGES • Adoption of a product- based mindset • Golden path • Freedom vs. control
25. WHAT WE GOT RIGHT – AND THE IMPACT IT HAD 5
26. 26 ARCHITECTURE
27. 27 ARCHITECTURE – DEEP DIVE Sources Storage and Processing Data Integration article curated Semantics Layer (Processing) product Observability/monitoring GCS ✔schema ✔quality data contract Data platform run orders raw Level 1 Level n Level 1 Level n central /shared HaFi specific Cloud composer central orchestration Meta data quality check to be published in the data catalog • Table/Data description • 70% column description Data catalog
28. A PLATFORM IS NEVER FINISHED 6
29. 29 • Multiplatform • Easy onboarding • Data contracts • MLOps and AI • Streaming data THE WAY FORWARD
30. 30 WANT TO KEEP THE DISCUSSION GOING? MESSAGE ME ON LINKEDIN!