Bol.com

Transcript

1. Big Data Expo 2023 How to accelerate your analytics team with analytics engineering Zhou Su Analytic s Enginee r zhou.su @xebia. com Loek Botman Analytic s Enginee r
2. / Logistic Services Analytics • Logistic Services: warehousing & delivery services to sellers on our platform • Analytics: business decisions backed by data
3. Back to a pre-digital era
4. But it’s hard to gather information from a huge pile of books
5. Someone needed to clean up the mess ?
6. The librarian needed to clean up the mess
7. Data warehouses our modern libraries Data engineer Data analyst ?
8. Analytics Engineers our modern librarians Data engineer Data analyst Analytics Engineer
9. ‘Transform’ bigger than you think • Data modeling • Data testing • Source freshness testing • Data timeliness • Data documentation & definitions • Implementing right to be forgotten • Code reusability • Code linting • Data accessibility • Development vs production environments • …
10. Why? Prevent duplicate definitions Increase productivity Increase discoverability Prevent data errors Prevent data downtime Not re-inventing the wheel
11. Data engineer Supplies data to the data warehouse Tech-focused Strong programming component Analytics Engineer Data Analyst Brings engineering principles to the field of analytics Business- & Tech-focused Data warehousing SQL fluency Gets insights from the data Business-focused Communication & stakeholder management skills Data visualization
12. Our journey
13. Centralized ETL All transformations in one project Testing testing testing! Testing has become part of our definition of done Documentation in GIT/BQ Documentation is part of our definition of done and is exposed CI/CD Easy to develop and automate tests before going to production Atscale cubes, BQ scripts, etc over separate locations and tools Scattered ETL Testing in scripts was tedious and mostly manual. Hardly any testing Lack of documentation Both on our data products, as well as conventions, pipelines and stack Development Even with great care; hard to control progress and roll back if necessary 01 02 03 04 01 02 03 04 Before and after Q4 2020 In Q4 2020 we started rethinking our analytics workflow
14. Build the foundation 󰠺󰠻 • Tooling • Modeling convention & standards
15. Tech stack
16. Workflow container registry
17. We have a development environment next to our production environment lsa-dev lsa DEV PRO <username> logistic_services project project dataset dataset logistic_services dataset dbt run dbt run dbt run
18. Build foundational data models Create data models for key entities as basic building blocks
19. Data warehouse architecture Source s Bases -recasted -renamed Marts -entities Commo n -enriched entity Exposu res -dashboard s -authorized views One big table -for analysis
20. Data warehouse architecture
21. Data warehouse architecture
22. Data warehouse architecture
23. Building data products Dashboards Analysis Data
24. Ensure quality • CI/CD • Tests • Monitoring
25. GitLab CI
26. Data tests
27. Source freshness tests
28. Data quality dashboard dbt-project-evaluator + custom macro
29. Increase analyst productivity • Proper local development setup • Macros to compare tables after refactors
30. Proper local development setup
31. Macros to compare tables after refactors
32. Looking back Higher productivity More collaboration Less stress More trust
33. Tips
34. Some practical tips • Start with the technical stuff • Continue with more focus on data modeling and conventions • Think about centralization vs. decentralization
35. Thanks