Big Data Expo 2023 How to accelerate your analytics team with analytics engineering Zhou Su Analytic s Enginee r zhou.su @xebia. com Loek Botman Analytic s Enginee r

/ Logistic Services Analytics ● Logistic Services: warehousing & delivery services to sellers on our platform ● Analytics: business decisions backed by data

Back to a pre-digital era

But it’s hard to gather information from a huge pile of books

Someone needed to clean up the mess ?

The librarian needed to clean up the mess

Data warehouses our modern libraries Data engineer Data analyst ?

Analytics Engineers our modern librarians Data engineer Data analyst Analytics Engineer

‘Transform’ bigger than you think ● Data modeling ● Data testing ● Source freshness testing ● Data timeliness ● Data documentation & definitions ● Implementing right to be forgotten ● Code reusability ● Code linting ● Data accessibility ● Development vs production environments ● …

Why? Prevent duplicate definitions Increase productivity Increase discoverability Prevent data errors Prevent data downtime Not re-inventing the wheel

Data engineer Supplies data to the data warehouse Tech-focused Strong programming component Analytics Engineer Data Analyst Brings engineering principles to the field of analytics Business- & Tech-focused Data warehousing SQL fluency Gets insights from the data Business-focused Communication & stakeholder management skills Data visualization

Our journey

Centralized ETL All transformations in one project Testing testing testing! Testing has become part of our definition of done Documentation in GIT/BQ Documentation is part of our definition of done and is exposed CI/CD Easy to develop and automate tests before going to production Atscale cubes, BQ scripts, etc over separate locations and tools Scattered ETL Testing in scripts was tedious and mostly manual. Hardly any testing Lack of documentation Both on our data products, as well as conventions, pipelines and stack Development Even with great care; hard to control progress and roll back if necessary 01 02 03 04 01 02 03 04 Before and after Q4 2020 In Q4 2020 we started rethinking our analytics workflow

Build the foundation 󰠺󰠻 ● Tooling ● Modeling convention & standards

Tech stack

Workflow container registry

We have a development environment next to our production environment lsa-dev lsa DEV PRO logistic_services project project dataset dataset logistic_services dataset dbt run dbt run dbt run

Build foundational data models Create data models for key entities as basic building blocks

Data warehouse architecture Source s Bases -recasted -renamed Marts -entities Commo n -enriched entity Exposu res -dashboard s -authorized views One big table -for analysis

Data warehouse architecture

Data warehouse architecture

Data warehouse architecture

Building data products Dashboards Analysis Data

Ensure quality ● CI/CD ● Tests ● Monitoring

GitLab CI

Data tests

Source freshness tests

Data quality dashboard dbt-project-evaluator + custom macro

Increase analyst productivity ● Proper local development setup ● Macros to compare tables after refactors

Proper local development setup

Macros to compare tables after refactors

Looking back Higher productivity More collaboration Less stress More trust

Tips

Some practical tips ● Start with the technical stuff ● Continue with more focus on data modeling and conventions ● Think about centralization vs. decentralization

Thanks