Why you should become big data engineer now

"В останні декілька років, разом з професією data scientist, зросла популярність професії big data engineer. Більшість людей в сфері ІТ вважають, що big data engineer працює з великими об'ємами даних, але чи це дійсно так?
Що таке big data, в чому різниця big data і data science, що входить в обов'язки big data engineer, типові задачі і цікаві кейси, шляхи входу в професію - все це і багато іншого ви дізнаєтесь в цій доповіді."
Presented at Impulse Fest (Kyiv, Ukraine)

Transcript

1. Why you should become big data engineer now
2. Roksolana Diachuk • Big Data Developer at Captify • Women Who Code Kyiv Data Engineering Lead • Speaker and traveller • Big data and Functional programming fan
3. BIG DATA WHAT IS
4. 5 VS
5. Big data Structured Unstructured Semi-structured
7. Main tasks • Building data pipelines • Data Storage • Data Processing • Infrastructure
9. Building data pipelines
10. Data storage
11. Data processing
12. Infrastructure
13. BIG DATA DATA SCIENCE
14. • Data processing and cleaning • Data pipelines • Testing and maintenance • Storing data • Data Pre-Processing • Data Analysis • Building ML models
15. Data Scientist Data Analyst Data Engineer Data Communication Math, Stats, Algorithms Software Engineering
16. General ML workflow
17. RDBMS ML model Metrics
18. RDBMS ML model Metrics BIG DATA DATA SCIENCE
19. Back to big data engineering…
20. Challenges • Nature of the data • Instant data processing • Historical data processing • Storage of large volumes of data
21. Pros • Innovative sphere • Complicated tasks • Little competition • Diverse tech stack Cons • Tech stack might be too diverse • High entry threshold • Invisible work
22. Background
23. Software engineer
24. Databases developer
25. Data analysis
26. How to become big data engineer
27. Big data degree Internship Switch from other IT professions
28. My career
29. 2014 2017 First years in NAU Freelance as Java developer at Upwork Jan Sep May French Spring School in KPI Coursera Big Data Specialization Aug
30. 2017 Java courses in EPAM Oct Dec DataRoot university Job offer from Ciklum Graph databases research Jan
31. Graph databases research
32. 2019 Jun Job offer from Captify Big data developer at Captify Jul Project in banking Feb 2020 Kubernetes research Oct
33. Kubernetes research
34. 2019 Jun Job offer from Captify Big data developer at Captify Jul Project in banking Feb 2020 Kubernetes research Oct
35. Project in banking
36. 2019 Jun Job offer from Captify Big data developer at Captify Jul Project in banking Feb 2020 Kubernetes research Oct
37. Why you should become big data engineer • Edge of innovation • Diverse projects/products • Challenging tasks • Popularity rise
38. My article about big data https://dou.ua/lenta/articles/what-is-big- data-engineering/
39. Coursera Big data specialisation https://www.coursera.org/specializations/big-data
40. dead_flowers22 roksolana-d roksolanadiachuk roksolanad My contact info
41. Thank you for attention