Vanderlande-Beyond_theory__Practical_lessons_from_4_years_of_data_platform_evolution.pdf

Transcript

1. Beyond theory Sander Kerstens | IT Director | Vanderlande Data Expo | 2025 September 11th | Utrecht Practical lessons from 4 years of data platform evolution
2. 2 To succeed in this environment, we need to focus on a strong and future proof data platform core Organizational Change › Cross-company collaborations, acquisitions, carve- outs and internal reorganizations ask your data platform to be scalable and open, while secure and governed. › Business is becoming more mature in the field of Data & AI, changing the role of the CDAO (department). Practice (at scale) vs. Theory › Excessive costs due to dedicated DEV-ACC-PROD environments. › Complex operations and loss of speed due to BDP chaining. › Large amount of Data Products due to ‘architectural quantum’ concept. Technology Advances › The offerings of companies like Microsoft, Databricks and Snowflake rapidly develop, driving constant changes in your data platform architecture. › Evolution of AI towards deep learning and generative/agentic AI brings new requirements. Landing Zones How do we rapidly ingest data from external or newly acquired companies/systems? AI How do we make sure AI developments within our company source data from our platform only? Governance How can we provide a smooth user experience while maintaining strict governance? Labs Where do we draw the line between an experiment and an actual analytics workload that requires more formal data productst? Data Streams How tot answer almost ‘operational’ data requests with sufficient speed? Be prepared to ‘kill your darlings’. The world of Data & AI is changing rapidly and requires us to adapt. LEARNINGS
3. Problem Statement Atomic Fragmentation BDP Layering Complex Setup Redundant Inclusions Highly granular SDPs (e.g. 10 different order types) due to ‘architectural quantum’ concept. Multiple BDPs build on top of other BDPs creating deep dependency chains. Medallion architecture per data product causes an ‘overkill’ when logic is simple. Same SDP appearing multiple times in different BDP chains caused by ‘bolted on’ functionality in the chain. Practice (at scale) vs. Theory
4. SDP Granularity - Issue DP00026 Internal Projects DP00025 Service Projects DP00015 Installation Projects DP00186 Projects DP00035 Cost Centers DP00143 Company Balance Sheet Units DP00148 Project Specific Branches DP00144 Inventory Branches DP00149 Project Warehouses JDE F0005 JDE F0006 JDE F0010 DP00137 Financial Business Units FINANCE Practice (at scale) vs. Theory
5. SDP Granularity - Solution JDE F0005 JDE F0006 JDE F0010 DP00317 Internal Projects – P02 Financial Business Units – P01 Service Projects – P03 Installation Projects – P04 Company Balance Sheet Units – P06 Projects – P05 Project Specific Branches – P07 Inventory Branches – P08 Cost Centers – P10 Project Warehouses – P09 FINANCE Practice (at scale) vs. Theory
6. BDP Chaining – Issue SDP 002 SDP 003 SDP 001 BDP 008 BDP 005 BDP 007 BDP 006 BDP 004 BDP 009 BDP 010 BDP 011 BDP 012 BDP 013 BDP 014 BDP 015 BDP 016 Practice (at scale) vs. Theory
7. BDP Chaining – Solution SDP 002 SDP 003 SDP 001 BDP 008 BDP 005 BDP 007 BDP 006 BDP 004 BDP 009 BDP 010 BDP 011 BDP 012 BDP 013 BDP 014 BDP 015 BDP 016 BDP 004 BDP 005 View View Practice (at scale) vs. Theory
8. Changing the rules of the game AT Layer DS/DT SDP BDP BDP SDP SDP SDP SDP Reduce complexity in the mesh, while keeping complexity in the nodes under control. Maximum depth of tree SDPs can only be succeeded by a maximum of two levels of BDPs. First BDP Layer Has tables and is a full fledged BDP with all logic and projections. Second BDP Layer BDPs on top the first layer of BDPs can only be a view on that data, no dedicated tables or data is stored. Practice (at scale) vs. Theory
9. Turning documents into insights with the use of AI Requires us to parse unstructured documents into structured information Files Parsing Parse Text Extract images Extract tables Extract Entities Semi-Structured Content Metadata Governance Data RAG Chat applications, content generation Automation Extracting key clauses, summarizing documents BI Sentiment analysis, Risks, Emerging trends Feeding Technology Advances
10. Data Analytics Franchise Model Data Governance Team D&A ART – Data Product Platform ML/AI Platform Operations Team Core D&A ML/AI CDAO Markets & Solutions Technology Sales Supply Life Cycle Services Finance Human Resources Information Proj. Execution S S L S S M L L M D&A ML/AI Organizational Change
11. T-shirt sizes Product Manager Analytics Business Domain Data Stewards (2) Visualization Specialist Data Engineers (2) Data Scientist Domain Team (L) Product Manager Analytics (0.5) Business Domain Data Steward (0.5) Domain Team (S) Business Domain Owner Product Manager Analytics Business Domain Data Steward Analytics Engineer Domain Team (M) Business Domain Owner Business Domain Owner Analytics Applications Business Data Products Source Data Products Business Object Models Data Governance Franchise Team S M L Landing Zones / AT Layer Organizational Change
12. Data Discovery Data Analysis Data Engineering ML & AI Enabling Functions Deliver to Service Supply Innovate to Sell Strategic Partners Customers Organizational Change
13. 13 To succeed in this environment, we need to focus on a strong and future proof data platform core Organizational Change › Design your processes with scalability, security and governance in mind. › Accept your central organization changes over time. Encourage developers to move towards the business and replace them with profiles that fit the next phase of your maturing business. Practice (at scale) vs. Theory › Know when to hold on to theory and when to let go. › Not every standard or best practice, should be your standard or is the best practice in your environment. › Be very much open to changing the rules of the game halfway. Technology Advances Be prepared to ‘kill your darlings’. The world of Data & AI is changing rapidly and requires us to adapt. WRAP-UP › Choose one or multiple strong partners in the field of Data & AI and follow their roadmaps. › Prevent ‘shopping’ and the introduction of ‘niche’ vendors by offering a strong and broad toolbox. › Focus on integration with your data platform and existing governance structures. Sander Kerstens Sander.Kerstens@Vanderlande.com