How we set-up a Compliant Hybrid Cloud platform at the Belgian National Employment Office

Transcript

1. DATA EXPO UTRECHT Setting up a Compliant Hybrid Data Platform How NEO Belgium turned strict PII compliance into a fast, self-service data platform. Martijn Bauters Chief Data Officer — NEO Belgium
2. INTRODUCTION Data Career Journey Master in Business Engineering: Data Science Master in Statistics: Data Science Academic foundation in data science Data Scientist @ Accenture Developed ML Models in various industries Head of Data Intelligence @ Easyfairs Group Build out an award-winning data platform Head of Data & Digital @ Cerba Research Group Developed and rolled out group wide data platform strategies Founder, Head of Delivery @ Agoya Strategical Data Advisor @ D'ieteren Chief Data Officer @ NEO Belgium Build out the delivery unit of Agoya Advised on operational model Accountable for the data department 2
3. CONTEXT What is NEO Belgium? A public social security institution administering unemployment insurance and career-break / time-credit systems for the whole country. 2.040 919 16 1.57M employees in core business employees at headquarters regional offices unemployment cases handled 96% of unemployment cases and 94% of first-line calls are resolved correctly — the platform has to protect that bar, not slow it down. 3
4. “ Data at the heart of every key decision. Every fraud flag, every benefit payout, every audit — starts with data someone can trust. 4
5. THE PROBLEM Where we started 5
6. THE PROBLEM Where we started 5
7. THE CONSTRAINT One rule to rule them all “No PII in the cloud at RVA” — every design choice had to satisfy the IVC before it could ship. Public cloud providers must not have access to data and key management. ✓ Confidential Computing must be certified. ✓ Encryption of data and secrets. ✓ Authentication mechanisms must be in place. ✓ Authorization mechanisms must be in place. ✓ Data must be deleted properly. ✓ The technology used must be monitored. ✓ A service level agreement (SLA) must be in place. ✓ For disputes, the competent court must be located in Belgium or Europe. ✓ 7
8. THE PLATFORM An encapsulated cloud platform 8
9. THE PLATFORM • • • • • If data needs to move to the cloud, we need to make sure that it does not contain any sensitive information. An on-prem pseudonymization step helps removing this sensitive information. Open Metadata is used to validate if data is labeled as PII sensitive. On-prem hop is used to ingest the data to our MSSQL Server and do the transformations & pseudonymizations Data quality is assured during this process. All metadata is monitored using Open Metadata. 9
10. • • Moving data from A to B is done by Airbyte, our data ingestion tool. This process is monitored by Open Metadata to achieve end to end data lineage, as well as governing the tool. 10
11. • Data arrives in the bronze layer in Databricks and undergoes multiple transformations during its lifetime in the data lakehouse. Data is split up into PII catalogs and Non-PII catalogs, based on the label in Open Metadata. • Medallion architecture is implemented: bronze (raw), silver (cleansed) and gold (modelled) layer. Transformations between each layer is done using dbt. • Platinum layer added for business domains. This allows business domains to have full control over their own data transformations, reports and AI/ML models without impacting other teams. SQL, DBT and Python can be used to create the Platinum layer. • Soda is responsible for data quality between each layer in the data lakehouse. • All metadata is managed using Open Metadata. 11
12. • Power BI Reports can be created based on the final data • Data Lakehouse: gold layer and platinum layer • Data Warehouse: presentation area • Pseudonymized PII catalogs can only be used by Power BI Server - to depseudonymize the data whilst garanteeing data integrity and safety. • Non-PII catalogs can be used by Power BI Service to create reports • Presentation area can be used by both Power BI Server & Service, depending on the type of information (PII or not). This PII label can be gathered from the Open Metadata tool. • All metadata is managed by Open Metadata 12
13. THE SOLUTION — OUR MEDALLION ARCHITECTURE MDP Architecture Simplified Layer Overview 01 02 ➜ 03 ➜ On-Prem Hop Bronze 04 ➜ 05 ➜ Silver Gold Platinum • Pre-cloud prep • Raw data • Clean & validate • Curate • Domain products • Pseudonymise & filter • Partition & archive • Quality applied • Analytics-ready • Reports & AI/ML 9
14. CASE STUDY — THUNDERSTROM USE CASE Predicting fraudulent temporary unemployment Description: detecting fraudulent companies in applications for temporary unemployment by cross-validating KMI meteorological data. Rapporten Metadata Management • Identificatie PII data • Pseudonymisation procedure ↕ Operationele data → Hop → Cloud Data Platform List of fraudulent companies (KBO numbers with a risk score) for statistics AI & ML Calculate fraud risk for a temporary unemployment application • PII data (SSN, addresses, …) • Pseudonymisation SSN (01.01.01-111.11 → • Pseudonymised data • Non-PII data (zip code, date of az1qs234n143265) • Anonymised data request, …) • Only keep the zip code (Wetstraat 16, 1000 • Non-PII data Extra input operationele systemen Brussel → 1000) • Anonymisation: remove day & month of birthdate (01/01/1950 → 1950) Push the list of fraudulent companies into WAVE (operational system) for follow-up by the inspectors 11
15. ADOPTION — ORIGINAL DIAGRAM How do we get the organization on board? Introduction of Forward Deployed Engineering Organisation IDS Data Mining Statistics … HRA Modern Data Platform Governance 13
16. ADOPTION — FORWARD DEPLOYED ENGINEERING How do we get the organization on board? Introduction of Forward Deployed Engineering Forward Deployed Engineering Organisation Our engineers sit inside the business team — not in a separate data team handing over specs. IDS Data Mining Statistics … HRA 1 Embedded, not external We work from inside IDS, KDD, SBS and HRA — in their systems, their meetings, their context. 2 Ships product, not slides Every sprint moves working software forward, not another architecture deck. 3 Modern Data Platform Stays until it sticks We don't leave when it technically works — we leave when the team owns it independently. Governance 13
17. GOVERNANCE Clear roles, clear ownership Six roles, one ladder — access grows as trust and skill grow. 6 5 4 3 2 1 PBI Reports User PBI Self-Service User Platinum Layer User Data Modeling Contributor Data Modeling Owner PBI Reports User — Consumes shared reports & dashboards • PBI Self-Service User — Builds reports from curated data • Platinum Layer User — Develops advanced products & models data, under supervision • Data Modeling Owner — Owns models across all layers • MDP Contributor — Oversees the entire platform • MDP Contributor Data Modeling Contributor — Transforms 14
18. REPLACEABILITY — FULL COMPARISON Built for Replaceability A reflection made on non-Azure components — every layer still works if Azure, or the cloud, disappears. Layer Current Ideal No Azure No Cloud Second Best Data Ingestion Airbyte Airbyte Airbyte Airbyte dlthub Data Orchestration Dagster Hybrid Dagster Hybrid Dagster Hybrid Dagster OSS Airflow Data Transformation Dbt Core Dbt Core Dbt Core Dbt Core Dbt Core Data Visualization PBI Report Server + Service PBI Report Server + Service PBI Report Server PBI Report Server Qlik On-Prem Hop MSSQL Server Clickhouse MSSQL Server MSSQL Server Postgres 16
19. KEY TAKEAWAYS Three things worth stealing 1 Make compliance “boring” Bake pseudonymisation and PII-labelling into the pipeline, not into a review meeting. 2 Build for scalability, design for replaceability Every component has a fallback — no single vendor can stall the platform. 3 Give every user a clear next step A visible ladder of roles turns platform access into a growth path, not a gate. 17
20. Thank you. Questions, war stories, or your own “boring compliance” story — let's talk. Martijn Bauters Chief Data Officer, NEO Belgium 18