Agrico

Transcript

1. 1 & Data transformations without Data Engineering? How Agrico builds dashboards in no time with Analytics Engineering
2. 2 & 12-9-2023 Data transformations without Data Engineering? How Agrico builds dashboards in no time with Analytics Engineering Today’s speakers
3. 3 & 1. Introduction Agrico 2. Introduction Digital Power 3. What do Analytics Engineers do? 4. Agrico’s challenge 5. Our set-up, architecture & business context 6. What can you gain with Analytics Engineering? 7. Questions 12-9-2023 Data transformations without Data Engineering? How Agrico builds dashboards in no time with Analytics Engineering Outline What will we cover today?
4. 4 International group Active all over the world Employer of choice ~300 colleagues Research driven we actively search for the best proprietary potato varieties Cooperative Our members ( ~1700) are both owners as well as suppliers 12-9-2023 Data transformations without Data Engineering? How Agrico builds dashboards in no time with Analytics Engineering Who we are
5. 5 & 12-9-2023 Data transformations without Data Engineering? How Agrico builds dashboards in no time with Analytics Engineering The challenges in today's world ….and how we cope with them Growing demand Growing conditions Technology Operational effort to optimize yields and add acreage R&D driven efforts to introduce new varieties ICT driven efforts to become data driven
6. 6 & 12-9-2023 Data transformations without Data Engineering? How Agrico builds dashboards in no time with Analytics Engineering Our goal is to seek "good growth" ….but what does that mean? In volume In quality As an organization For our growers / members For the environment Good growth
7. 7 Team Foundation & Subsidary Founding Locations 12-9-2023 Data transformations without Data Engineering? How Agrico builds dashboards in no time with Analytics Engineering About Digital Power Amsterdam Den Bosch London 145 colleagues Digital Power B.V. founded in 2016
8. 8 & Analytics Engineer 12-9-2023 Data transformations without Data Engineering? How Agrico builds dashboards in no time with Analytics Engineering What do Analytics Engineers do? Build data models Optimize transformations Manage data & documentation Technology Business Data sources Build infrastructure Optimize performance Manage data platform Data Engineer Data outputs Build reports Optimize insights Manage data products Data Analyst
9. 9 & 12-9-2023 Data transformations without Data Engineering? How Agrico builds dashboards in no time with Analytics Engineering The typical ERP1 conditions do not apply Our business and the processes applied are very different.... 1. ERP = Enterprise Resource Planning # The typical ERP system The situation at Agrico 1 Continuous process Season based 2 Article / product structure fixed Same physical product, different articles 3 Volumes / stocks known Volumes / stocks estimated 4 Prices known Prices determined at year end 5 Sales = delivery Volumes invoiced differ from volumes delivered ("Tarra")
10. 10 & 12-9-2023 Data transformations without Data Engineering? How Agrico builds dashboards in no time with Analytics Engineering Agrico’s challenge Agrico process & specific data D365 entities Data transformation s Information requirements Generic ERP data Customized ERP data
11. 11 & 12-9-2023 Data transformations without Data Engineering? How Agrico builds dashboards in no time with Analytics Engineering Our set-up, architecture & business context How do we address the challenge?
12. 12 12-9-2023 Data transformations without Data Engineering? How Agrico builds dashboards in no time with Analytics Engineering Storing Processing engine Business logic & transformations Supporting tools & services Serving Costs control Monitoring Key vault Version control Business users bronze silver gold Landing zone Raw data Intermediate data models Final data models
13. 13 Stakeholder Request Report Scope & Definitions Source Data Check Mock Tables & Report Bronze Layer Document Sources & Data Quality Checks Build Silver Models & Document Transformations Build Gold Models & Document Transformations Landing Zone Stakeholder Request Report Scope & Definitions Source Data Check Mock Tables & Report Bronze Layer Document Sources & Data Quality Checks Build Silver Models & Document Transformations Build Gold Models & Document Transformations Landing Zone 12-9-2023 Data transformations without Data Engineering? How Agrico builds dashboards in no time with Analytics Engineering General Workflow
14. 14 & 12-9-2023 Data transformations without Data Engineering? How Agrico builds dashboards in no time with Analytics Engineering What can you gain with Analytics Engineering? Improved Data Quality Reproducibility & Documentation Faster Time-to-Insight Collaboration & Communication
15. 15 & Questions? You can also find us at booth 8! Analytics Engineers in their natural habitat Mark Kranenburg
16. 16 12-9-2023 Data transformations without Data Engineering? How Agrico builds dashboards in no time with Analytics Engineering Developer Iga Developer Pim feature 01 feature 02 develop main tag 1.0 tag 1.1 tag 1.2 Databricks DEV warehouse Core Cloud Databricks ACC warehouse Databricks PRD warehouse feature N Data Model Repo PowerBI DEV PowerBI ACC PowerBI PRD Developer Daan Scheduled jobs
17. 17 & 12-9-2023 Data transformations without Data Engineering? How Agrico builds dashboards in no time with Analytics Engineering What can you gain with Analytics Engineering? Improved Data Quality Reproducibility & Documentation Faster Time-to-Insight Collaboration & Communication Consistency Validation Cleansing Accountability Versioning Transparency Coherency Standards Efficiency Holistic Interdisciplinary Sound
18. 18 & 12-9-2023 Data transformations without Data Engineering? How Agrico builds dashboards in no time with Analytics Engineering Improved Data Quality Consistency Validation Cleansing Stock?
19. 19 & 12-9-2023 Data transformations without Data Engineering? How Agrico builds dashboards in no time with Analytics Engineering Faster Time-to-Insight Coherency Standards Efficiency
20. 20 & 12-9-2023 Data transformations without Data Engineering? How Agrico builds dashboards in no time with Analytics Engineering Reproducibility & Documentation Accountability Versioning Transparency
21. 21 & 12-9-2023 Data transformations without Data Engineering? How Agrico builds dashboards in no time with Analytics Engineering Collaboration & Communication Holistic Interdisciplinary Sound Data project Stakeholders Skillsets Roles Tools
22. 22 & 12-9-2023 Data transformations without Data Engineering? How Agrico builds dashboards in no time with Analytics Engineering What can you gain with Analytics Engineering? Improved Data Quality Faster Time-to-Insight Reproducibility & Documentation Collaboration & Communication
23. 23 & 12-9-2023 Data transformations without Data Engineering? How Agrico builds dashboards in no time with Analytics Engineering Questions? You can also find us at booth 8! Analytics Engineer in their natural habitat Connect with Iga Connect with Mark
24. 24 & Questions? 12-9-2023 Data transformations without Data Engineering? How Agrico builds dashboards in no time with Analytics Engineering Analytics Engineer in their natural habitat
25. 25 12-9-2023 Data transformations without Data Engineering? How Agrico builds dashboards in no time with Analytics Engineering How to edit the footer • To change the footer date and description, go to "Insert" at the top left. • Then choose: "Header and footer." After clicking, a window will open with the different functions. • To add a presentation title, enter your own text under the heading: ''Footer‘’. • After you have made your choice click on: "Apply anywhere." Now a footer will appear on every slide in the PowerPoint Presentation. • If you uncheck the footer, the footer disappears completely.
26. 26 12-9-2023 Data transformations without Data Engineering? How Agrico builds dashboards in no time with Analytics Engineering Style components
27. 27 12-9-2023 Data transformations without Data Engineering? How Agrico builds dashboards in no time with Analytics Engineering Building Blocks model
28. 28 12-9-2023 Data transformations without Data Engineering? How Agrico builds dashboards in no time with Analytics Engineering Icons Interactions Notifications Visualisations Tools & technology Shapes Other Expertise Services DA TWA DS DE CX Research Strategy Consultancy Projects Services Academy
29. 29 12-9-2023 Data transformations without Data Engineering? How Agrico builds dashboards in no time with Analytics Engineering Illusrations
30. 30 12-9-2023 Data transformations without Data Engineering? How Agrico builds dashboards in no time with Analytics Engineering Timeline 2012 Founding of Digital Power 2000 Title 2000 Title 2000 Title 2000 Title
31. 31 Data projects Develop solutions, planning & budget Data academy Specialist training and creative sessions about data Data services Guaranteed continuation of your data operations Data consultancy Advice, interim management & hands-on work 12-9-2023 Data transformations without Data Engineering? How Agrico builds dashboards in no time with Analytics Engineering What we do
32. 32 & Analytics Engineer 12-9-2023 Data transformations without Data Engineering? How Agrico builds dashboards in no time with Analytics Engineering What do Analytics Engineers do? Technology Business Data sources Data Engineer Data outputs Data Analyst
33. 33 Data Strategy Research Customer experience Technical web analytics Data engineering Data science Data analytics 12-9-2023 Data transformations without Data Engineering? How Agrico builds dashboards in no time with Analytics Engineering Expertise
34. 34 & • Climate change • Regulation (e.g. Green deal) • R&D driven efforts to introduce new varieties 12-9-2023 Data transformations without Data Engineering? How Agrico builds dashboards in no time with Analytics Engineering The challenges in today's world ….and how we cope with them • Population growth • Demographic shifts • Consumption patterns • Operational effort to optimize yields and add acreage Growing demand Growing conditions • Precision agriculture • Connectivity through the chains • Data availability • ICT driven efforts to become data driven Technology
35. 35 & 12-9-2023 Data transformations without Data Engineering? How Agrico builds dashboards in no time with Analytics Engineering Our goal is to seek "good growth" ….but what does that mean? In volume In quality As an organization For our growers / members For the environment Good growth
36. 36 Team Foundation & Subsidary Founding Locations 12-9-2023 Data transformations without Data Engineering? How Agrico builds dashboards in no time with Analytics Engineering About Digital Power Amsterdam Den Bosch London 145 colleagues Digital Power B.V. founded in 2016
37. 37 & Analytics Engineer 12-9-2023 Data transformations without Data Engineering? How Agrico builds dashboards in no time with Analytics Engineering What do Analytics Engineers do? Build data models Optimize transformations Manage data & documentation Technology Business Data sources Build infrastructure Optimize performance Manage data platform Data Engineer Data outputs Build reports Optimize insights Manage data products Data Analyst
38. 38 & 12-9-2023 Data transformations without Data Engineering? How Agrico builds dashboards in no time with Analytics Engineering The typical ERP conditions do not apply Our business and the processes applied are very different.... The typical ERP system • Continuous process • Article / product structure • Volumes / stocks known • Prices known • Sales = delivery • Procurement first, sales follows • Can only deliver what is on stock The situation at Agrico • Season based – discontinued • Article structures flexible: same physical product, different articles • Volumes / stocks estimated • Prices determined at year end • Volumes invoiced differ from volumes delivered ("Tarra") • Sales (delivery) often determines procurement • Sale is often different than what is on stock
39. 39 & 12-9-2023 Data transformations without Data Engineering? How Agrico builds dashboards in no time with Analytics Engineering Agrico’s challenge Agrico process & specific data D365 entities Data transformations Information requirements Generic ERP data Customized ERP data
40. 40 & 12-9-2023 Data transformations without Data Engineering? How Agrico builds dashboards in no time with Analytics Engineering Our set-up, architecture & business context How do we address the challenge?
41. 41 12-9-2023 Data transformations without Data Engineering? How Agrico builds dashboards in no time with Analytics Engineering Landing zone bronze silver gold Storing Processing engine Business logic & transformations Supporting tools & services Serving Costs control Monitoring Key vault Version control Business users
42. 42 12-9-2023 Data transformations without Data Engineering? How Agrico builds dashboards in no time with Analytics Engineering Stakeholder Request Report Scope & Definitions Source Data Check Mock Tables & Report Bronze Layer Document Sources & Data Quality Checks Build Silver Models & Document Transformations Build Gold Models & Document Transformations Landing Zone General Workflow
43. 43 & 12-9-2023 Data transformations without Data Engineering? How Agrico builds dashboards in no time with Analytics Engineering What can you gain with Analytics Engineering? Improved Data Quality Faster Time-to-Insight Reproducibility & Documentation Collaboration & Communication
44. 44 & Questions? You can also find us at booth 8! Analytics Engineers in their natural habitat
45. 45 12-9-2023 Data transformations without Data Engineering? How Agrico builds dashboards in no time with Analytics Engineering Developer Iga Developer Pim feature 01 feature 02 develop main tag 1.0 tag 1.1 tag 1.2 Databricks DEV warehouse Core Cloud Databricks ACC warehouse Databricks PRD warehouse feature N Data Model Repo PowerBI DEV PowerBI ACC PowerBI PRD Developer Daan Scheduled jobs
46. 46 & 12-9-2023 Data transformations without Data Engineering? How Agrico builds dashboards in no time with Analytics Engineering Today’s speakers Iga Jarosz Analytics Engineer Mark Kranenburg CFO
47. 47 & Data transformations without Data Engineering? How Agrico builds dashboards in no time with Analytics Engineering