What Makes or Breaks a Data Engineer?

Some call them research engineers, some machine learning engineers, some BI engineers and some data engineers. Still cloud migrations, Hacker News frontpage darlings and evil APIs can break them. How to become a good one and what makes a good investment to learn in the Dépêche Mode of data technology?

Transcript

1. What Makes or Breaks
   is a Data Engineer?
   Daniel Molnar
   Pipeline Data Engineering Academy
   PAKCon, 2021
   

   View Slide

2. Quick Agenda
   » whoami
   » No such thing?
   » Maybe still?
   » What breaks them?
   » What makes them?
   » Hope, anyone?
   

   View Slide

3. whoami
   » I co-funded the first data engineering bootcamp ...
   » ... during Covid
   » 12 years of building data teams
   @ Shopify, Microsoft, Wunderlist, Zalando
   » 23 years building startups
   B2C, e-commerce, productivity, music, e-learning
   » I built web before Java (1995)
   

   View Slide

4. Data Engineer?
   Some call them:
   » BI developers,
   » research engineers,
   » machine learning engineers,
   » dataops,
   » analytics engineers (aka pissed off data analysts),
   » data engineers.
   

   View Slide

5. "Is there even such a role
   like data engineer???"
   — ex-DS, ex-PO at now-DAX, 2018
   

   View Slide

6. "Not one of today's data engineers
   grew up as a kid imagining
   becoming one."
   (Peter Fabian, co-founder, MD)
   

   View Slide

7. A data engineer
   solves data-related
   engineering problems
   in a maintainable way.
   Also talks a lot.
   

   View Slide

8. The Pyramid
   

   View Slide

9. The DE Pyramid
   

   View Slide

10. The DE Pyramid
    

    View Slide

11. State of the Union at Company X
    » (C*O/VP): "I have a Data Scientist. :)"
    » me: "What does your Data Scientist do?"
    » (C*O/VP): "Bad Data Engineering. At least 80% of the time."
    » me: "You need a Data Engineer. You don't need a Data Scientist."
    » (C*O/VP): "But it's impossible to hire a Data Engineer."
    » me: "
    !
    "
    

    View Slide

12. Hiring DEs?
    !"
    » I dare you to change your title on LinkedIn to
    data engineer
    for a week.
    » Be nice with the headhunters. It's the hard part.
    » Most people getting hired as a data engineer know exactly that
    much about DE as you.
    

    View Slide

13. What breaks them?
    » Trying to hire other DEs
    » Marketing
    » Dépêche Mode
    » (Cloud migrations)
    

    View Slide

14. Marketing
    

    View Slide

15. When we were young (2016)
    

    View Slide

16. View Slide

17. View Slide

18. Funding, 2021 Q1-Q3, million$
    » Platform: Databricks 2600, Dataiku 400, Datarobot 300
    » BI: Grafana 220, Preset 36, Streamlit 35, Metabase 30
    » DQ: Monte Carlo 85, Great Expectations 21
    » DWH: neo4J 325, Cockroach 160, Dremio 135, Firebolt 127,
    Startburst 100, Clickhouse 50, Timescale 40
    » ETL: dbt 150, Matillion 100, Prefect 43, Airbyte 31, Snowplow 10,
    Meltano 4
    

    View Slide

19. Hype = bullshit
    » data lakehouse,
    » reverse ETL,
    » data mesh.
    » Sound like sex positions to me.
    

    View Slide

20. Reflect on the Zeitgeist
    Wired Tired
    dbt most Apache
    Great Expectations especially Hadoop
    Prefect Airflow (shit, but popular)
    Presto Spark, Databricks (aged badly pretty fast)
    Superset/Preset.io chart.io (shot dead by Atlassian)
    Clickhouse Snowflake
    

    View Slide

21. What makes
    a data engineer?
    

    View Slide

22. Why to become one?
    » Plumbers are system relevant workers
    » There is a need for them
    » DE open positions >> DS open positions
    » DE salaries >> DS salaries
    » Market justifies added value by salary
    

    View Slide

23. Who would become a DE?
    » data scientists (Covid reality check),
    » business analysts (more moat),
    » any engineer.
    

    View Slide

24. What to learn?
    "One is never over-dressed or under-
    dressed with a Little Black Dress."
    — Karl Lagerfeld
    

    View Slide

25. What to learn?
    "If you know how to make a Little Black
    Dress, then you can do fast fashion."
    — me.
    

    View Slide

26. Where? Fast? Deep?
    » self-driven, tool oriented band aids (cloud providers, OS-as-a-
    marketing-tool - Cloudera, Databricks, Udemy),
    » self-driven bytesize Sudoku (DataQuest, DataCamp),
    » lonely places (Coursera, Udacity),
    » bootcamps, few,
    » few universities.
    

    View Slide

27. The past is the future,
    choose boring
    » UNIX Shell (1971),
    » SQL (1974),
    » Python (1991),
    » Kubernetes YAML hell (2014)?
    "The longer a technology lives, the longer it can be expected to live."
    — Nassim N. Taleb (way of Mandelbrot, aka Lindy effect)
    

    View Slide

28. #HOPE
    

    View Slide

29. How to become
    (a good) one?
    

    View Slide

30. Knowing this, ...
    

    View Slide

31. ... this, ...
    

    View Slide

32. ... and this.
    

    View Slide

33. Thank you!
    @soobrosa
    Read Morozov: "The Meme Hustler" and thanks fly for the to visuals: @mrogati, @xkcd,
    @DorsaAmir, @luismisanchez, Coco Chanel, Depeche Mode, Christopher Bolard, Tomasz Dudek,
    James Mickens, @bfaludi, @FirstMarkCap.
    

    View Slide