Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Dataops - The WAT, The Pain, The How

Dataops - The WAT, The Pain, The How

A high-level introduction to DataOps and how to deploy or nail it within real organisations.

Xavier Bruhiere

August 22, 2019

More Decks by Xavier Bruhiere

Other Decks in Programming


  1. Hey hi, I’m Xavier Bruhiere VP Data engineering @ Lazada

    And I AM HIRING Data and full stack engineers Shameless Plug
  2. “DevOps is the combination of cultural philosophies, practices, and tools

    that increases an organization’s ability to deliver applications and services at high velocity: evolving and improving products at a faster pace than organizations using traditional software development and infrastructure management processes.” - AWS:*What is DevOps?* DevOps
  3. “A collaborative data manager practice, really focused on improving communication,

    integration, and automation of data flow between managers and consumers of data within an organization,” - Gartner DataOps
  4. Orchestrate, monitor Cycle Velocity Sensitive information DataOps Long lasting Slow

    changing Stateful People Massive Same goals Data != Software Analytics iterations ML / Operations
  5. - Time to delivery and Reliability - Fast iterations -

    Safe iterations - Correctness - Cost down - Insight value Goal
  6. - Reproducibility? - Local sandbox of a warehouse? Staging? -

    Mocking source data - Complex, stateful DAGs But… About tooling About testing
  7. - QA Testing with non-tech-savy people - Gap between who

    request and own the domain, and who performs - Various backgrounds and expectations But… About security About people - Access management at the column level - GDPR - Data leakage - Anonymization
  8. 01. Ignore the buzz, capture automations to escape chaos Takeaway

    02. Listen to people 03. Hunt relentlessly bottlenecks and risks 05. Have faith 02-bis. No heroism – self-organizing team