The REWE Data Platform - Speaker Deck

Slide 1

Slide 1 text

No content

Slide 2

Slide 2 text

2 HAPPY 10TH EDITION OF THE DATA ENGINEERING MEETUP

Slide 3

Slide 3 text

THE REWE DATA PLATFORM 1. Vision 2. The billion-euro cake 3. Building blocks of a data mesh 4.Challenges 5. What we got right – and the impact it had 6. A platform is never ﬁnished

Slide 4

Slide 4 text

4 REWE International AG is part of the REWE Group - one of Europe's leading trading and travel and tourism groups. REWE Interna+onal AG 9,21 bn Euro Total External Sales 2021 92 Thousand Employees 2021 2 Thousand Apprentices 2021 2,65 Thousand Markets 2021 The REWE International AG in figures

Slide 5

Slide 5 text

5 The IT department with around 480 employees, creates customized software and hardware solutions for all domestic and international divisions of the group. Focus IT: Pleased to meet you!

Slide 6

Slide 6 text

6 Jobs at REWE IT – Data Platform Interested to join us? https://rewe-group.jobs/jobs/it-handelinternational Contact details: Mail: [email protected]

Slide 7

Slide 7 text

7 WE ARE Follow us on LinkedIn:

Slide 8

Slide 8 text

THE VISION 1

Slide 9

Slide 9 text

9 • Self-service producers and consumers • Data catalogue • Centralized compliance THE VISION

Slide 10

Slide 10 text

BILLION-EURO CAKE 2

Slide 11

Slide 11 text

11 • Obvious data error • Requires collaboration between different departments • Sensitive data (prices) • Difﬁcult to propagate the ﬁx through all the systems WHY A CAKE

Slide 12

Slide 12 text

BUILDING BLOCKS OF A DATA MESH 3

Slide 13

Slide 13 text

13 MEANING • Data is scattered across teams • Meaining is unclear or different outside the context of the team

Slide 14

Slide 14 text

14 QUALITY • Quality gates do no exists, data is not checked • Data graveyards • By-product of the operational processes

Slide 15

Slide 15 text

15 SECURITY • Some data is sensitive • Usage of some data must be tracked and reported • The people that produce the data are not always aware of all the rules

Slide 16

Slide 16 text

16 AVAILABILITY • Data is difficult to find • It takes weeks to get access to data • It takes weeks to give access to data

Slide 17

Slide 17 text

Slide 18

Slide 18 text

18 DOMAIN OWNERSHIP • Article masterdata domain is responsible for all article data • The team makes the data available to all

Slide 19

Slide 19 text

19 DATA PRODUCTS • The team is responsible for the data quality • Internal customer satisfaction is tracked • Tests catch errors

Slide 20

Slide 20 text

20 FEDERATED GOVERNANCE • A governence team decides on rules and access control • Rules are enforced by the system

Slide 21

Slide 21 text

21 SELF-SERVICE DATA PLATFORM • Data is easy to ﬁnd • Data is easy to publish and consume • The platform enforces the rules

Slide 22

Slide 22 text

ORGANISATIONAL AND TECHNICAL CHALLENGES 4

Slide 23

Slide 23 text

23 TECHNICAL CHALLENGES • Ecosystem is not mature, tools don’t always work • Massive amounts of data

Slide 24

Slide 24 text

24 ORGANIZATIONAL CHALLENGES • Adoption of a product- based mindset • Golden path • Freedom vs. control

Slide 25

Slide 25 text

WHAT WE GOT RIGHT – AND THE IMPACT IT HAD 5

Slide 26

Slide 26 text

26 ARCHITECTURE

Slide 27

Slide 27 text

27 ARCHITECTURE – DEEP DIVE Sources Storage and Processing Data Integration article curated Semantics Layer (Processing) product Observability/monitoring GCS ✔schema ✔quality data contract Data platform run orders raw Level 1 Level n Level 1 Level n central /shared HaFi specific Cloud composer central orchestration Meta data quality check to be published in the data catalog • Table/Data description • 70% column description Data catalog

Slide 28

Slide 28 text

A PLATFORM IS NEVER FINISHED 6

Slide 29

Slide 29 text

29 • Multiplatform • Easy onboarding • Data contracts • MLOps and AI • Streaming data THE WAY FORWARD

Slide 30

Slide 30 text

30 WANT TO KEEP THE DISCUSSION GOING? MESSAGE ME ON LINKEDIN!