Upgrade to Pro — share decks privately, control downloads, hide ads and more …

What Makes or Breaks a Data Engineer?

October 18, 2021

What Makes or Breaks a Data Engineer?

Some call them research engineers, some machine learning engineers, some BI engineers and some data engineers. Still cloud migrations, Hacker News frontpage darlings and evil APIs can break them. How to become a good one and what makes a good investment to learn in the Dépêche Mode of data technology?


October 18, 2021

More Decks by soobrosa

Other Decks in Technology


  1. What Makes or Breaks is a Data Engineer? Daniel Molnar

    Pipeline Data Engineering Academy PAKCon, 2021
  2. Quick Agenda » whoami » No such thing? » Maybe

    still? » What breaks them? » What makes them? » Hope, anyone?
  3. whoami » I co-funded the first data engineering bootcamp ...

    » ... during Covid » 12 years of building data teams @ Shopify, Microsoft, Wunderlist, Zalando » 23 years building startups B2C, e-commerce, productivity, music, e-learning » I built web before Java (1995)
  4. Data Engineer? Some call them: » BI developers, » research

    engineers, » machine learning engineers, » dataops, » analytics engineers (aka pissed off data analysts), » data engineers.
  5. "Not one of today's data engineers grew up as a

    kid imagining becoming one." (Peter Fabian, co-founder, MD)
  6. State of the Union at Company X » (C*O/VP): "I

    have a Data Scientist. :)" » me: "What does your Data Scientist do?" » (C*O/VP): "Bad Data Engineering. At least 80% of the time." » me: "You need a Data Engineer. You don't need a Data Scientist." » (C*O/VP): "But it's impossible to hire a Data Engineer." » me: " ! "
  7. Hiring DEs? !" » I dare you to change your

    title on LinkedIn to data engineer for a week. » Be nice with the headhunters. It's the hard part. » Most people getting hired as a data engineer know exactly that much about DE as you.
  8. What breaks them? » Trying to hire other DEs »

    Marketing » Dépêche Mode » (Cloud migrations)
  9. Funding, 2021 Q1-Q3, million$ » Platform: Databricks 2600, Dataiku 400,

    Datarobot 300 » BI: Grafana 220, Preset 36, Streamlit 35, Metabase 30 » DQ: Monte Carlo 85, Great Expectations 21 » DWH: neo4J 325, Cockroach 160, Dremio 135, Firebolt 127, Startburst 100, Clickhouse 50, Timescale 40 » ETL: dbt 150, Matillion 100, Prefect 43, Airbyte 31, Snowplow 10, Meltano 4
  10. Hype = bullshit » data lakehouse, » reverse ETL, »

    data mesh. » Sound like sex positions to me.
  11. Reflect on the Zeitgeist Wired Tired dbt most Apache Great

    Expectations especially Hadoop Prefect Airflow (shit, but popular) Presto Spark, Databricks (aged badly pretty fast) Superset/Preset.io chart.io (shot dead by Atlassian) Clickhouse Snowflake
  12. Why to become one? » Plumbers are system relevant workers

    » There is a need for them » DE open positions >> DS open positions » DE salaries >> DS salaries » Market justifies added value by salary
  13. Who would become a DE? » data scientists (Covid reality

    check), » business analysts (more moat), » any engineer.
  14. What to learn? "One is never over-dressed or under- dressed

    with a Little Black Dress." — Karl Lagerfeld
  15. What to learn? "If you know how to make a

    Little Black Dress, then you can do fast fashion." — me.
  16. Where? Fast? Deep? » self-driven, tool oriented band aids (cloud

    providers, OS-as-a- marketing-tool - Cloudera, Databricks, Udemy), » self-driven bytesize Sudoku (DataQuest, DataCamp), » lonely places (Coursera, Udacity), » bootcamps, few, » few universities.
  17. The past is the future, choose boring » UNIX Shell

    (1971), » SQL (1974), » Python (1991), » Kubernetes YAML hell (2014)? "The longer a technology lives, the longer it can be expected to live." — Nassim N. Taleb (way of Mandelbrot, aka Lindy effect)
  18. Thank you! @soobrosa Read Morozov: "The Meme Hustler" and thanks

    fly for the to visuals: @mrogati, @xkcd, @DorsaAmir, @luismisanchez, Coco Chanel, Depeche Mode, Christopher Bolard, Tomasz Dudek, James Mickens, @bfaludi, @FirstMarkCap.