What Makes or Breaks a Data Engineer?

Some call them research engineers, some machine learning engineers, some BI engineers and some data engineers. Still cloud migrations, Hacker News frontpage darlings and evil APIs can break them. How to become a good one and what makes a good investment to learn in the Dépêche Mode of data technology?

Transcript

1. What Makes or Breaks is a Data Engineer? Daniel Molnar Pipeline Data Engineering Academy PAKCon, 2021
2. Quick Agenda » whoami » No such thing? » Maybe still? » What breaks them? » What makes them? » Hope, anyone?
3. whoami » I co-funded the first data engineering bootcamp ... » ... during Covid » 12 years of building data teams @ Shopify, Microsoft, Wunderlist, Zalando » 23 years building startups B2C, e-commerce, productivity, music, e-learning » I built web before Java (1995)
4. Data Engineer? Some call them: » BI developers, » research engineers, » machine learning engineers, » dataops, » analytics engineers (aka pissed off data analysts), » data engineers.
5. "Is there even such a role like data engineer???" — ex-DS, ex-PO at now-DAX, 2018
6. "Not one of today's data engineers grew up as a kid imagining becoming one." (Peter Fabian, co-founder, MD)
7. A data engineer solves data-related engineering problems in a maintainable way. Also talks a lot.
8. The Pyramid
9. The DE Pyramid
10. The DE Pyramid
11. State of the Union at Company X » (C*O/VP): "I have a Data Scientist. :)" » me: "What does your Data Scientist do?" » (C*O/VP): "Bad Data Engineering. At least 80% of the time." » me: "You need a Data Engineer. You don't need a Data Scientist." » (C*O/VP): "But it's impossible to hire a Data Engineer." » me: " ! "
12. Hiring DEs? !" » I dare you to change your title on LinkedIn to data engineer for a week. » Be nice with the headhunters. It's the hard part. » Most people getting hired as a data engineer know exactly that much about DE as you.
13. What breaks them? » Trying to hire other DEs » Marketing » Dépêche Mode » (Cloud migrations)
14. Marketing
15. When we were young (2016)
18. Funding, 2021 Q1-Q3, million$ » Platform: Databricks 2600, Dataiku 400, Datarobot 300 » BI: Grafana 220, Preset 36, Streamlit 35, Metabase 30 » DQ: Monte Carlo 85, Great Expectations 21 » DWH: neo4J 325, Cockroach 160, Dremio 135, Firebolt 127, Startburst 100, Clickhouse 50, Timescale 40 » ETL: dbt 150, Matillion 100, Prefect 43, Airbyte 31, Snowplow 10, Meltano 4
19. Hype = bullshit » data lakehouse, » reverse ETL, » data mesh. » Sound like sex positions to me.
20. Reflect on the Zeitgeist Wired Tired dbt most Apache Great Expectations especially Hadoop Prefect Airflow (shit, but popular) Presto Spark, Databricks (aged badly pretty fast) Superset/Preset.io chart.io (shot dead by Atlassian) Clickhouse Snowflake
21. What makes a data engineer?
22. Why to become one? » Plumbers are system relevant workers » There is a need for them » DE open positions >> DS open positions » DE salaries >> DS salaries » Market justifies added value by salary
23. Who would become a DE? » data scientists (Covid reality check), » business analysts (more moat), » any engineer.
24. What to learn? "One is never over-dressed or under- dressed with a Little Black Dress." — Karl Lagerfeld
25. What to learn? "If you know how to make a Little Black Dress, then you can do fast fashion." — me.
26. Where? Fast? Deep? » self-driven, tool oriented band aids (cloud providers, OS-as-a- marketing-tool - Cloudera, Databricks, Udemy), » self-driven bytesize Sudoku (DataQuest, DataCamp), » lonely places (Coursera, Udacity), » bootcamps, few, » few universities.
27. The past is the future, choose boring » UNIX Shell (1971), » SQL (1974), » Python (1991), » Kubernetes YAML hell (2014)? "The longer a technology lives, the longer it can be expected to live." — Nassim N. Taleb (way of Mandelbrot, aka Lindy effect)
28. #HOPE
29. How to become (a good) one?
30. Knowing this, ...
31. ... this, ...
32. ... and this.
33. Thank you! @soobrosa Read Morozov: "The Meme Hustler" and thanks fly for the to visuals: @mrogati, @xkcd, @DorsaAmir, @luismisanchez, Coco Chanel, Depeche Mode, Christopher Bolard, Tomasz Dudek, James Mickens, @bfaludi, @FirstMarkCap.