How to master the basics of Data Engineering

Data Engineers need to be good at data exploration, create data pipelines, have appropriate monitoring & productivity tools, automate & innovate, communicate well with stakeholders, learn the latest tools and methods associated with their trade, and be motivated.

Full article on Medium: https://medium.com/@SoftDevLife/how-to-master-the-basics-of-data-engineering-7a16244dd148

Transcript

2. Like in every draft, you have to prove your own potential within your college days, boot camps, and interviews. But that is just the first step of the journey. The rest, you have to learn them as well within the time-frame of your career. PREFACE: EDUCATION
3. Although practical degrees can get you ahead of the curve, theoretical degrees provide foundations that come in handy when problems are more complicated or too big to manage. Do get your hands dirty, but also learn a tip or two from those who have mastered their craft.
4. CHAPTER 1: DATA EXPLORATION Remember, it is better for your crew to wait a little bit longer for arriving at the treasure to avoid getting your boat shipwrecked.
5. CHAPTER 2: DATA PIPELINES Creating a data pipeline requires you to learn different types of data operations that each takes time to master. Setting the proper data models, writing reusable general scripts and taking advantage of what each database offers helps in making data pipelines more efficient.
6. CHAPTER 3: MONITORING TOOLS Arthur IV is said to be protected by the gods. But if we look under the surface, the ship is bundled with several monitoring tools. Having all your source code organized in a version control system, documenting everything from a top-down and bottom-up view, being proactive by logging all your activity and debugging whenever an alert shows up will make your ship bulletproof.
7. CHAPTER 4: PRODUCTIVITY TOOLS Make sure your productivity toolbox is fully stacked. Using interpreted languages and doing audit tests can take you miles away from the rest.
8. CHAPTER 5: AUTOMATION & INNOVATION Developer tools are great for building stuff that cannot be abstracted away. But remember, it is foolish on reinventing the wheel. Find ways to automate mundane work in order to focus more on adding value in new ways within your organization.
9. CHAPTER 6: COMMUNICATION In order for a captain’s message can reach out to his crew, artifacts and processes must be created within his island. Having a laboratory where anybody can grab their insights with ease and flexibility, having a town hall in order to make arrangements and retrospectives, training users in a gym on all aspects of the business and IT infrastructure, and taking the strengths of the other members within your group in order to create synergy are all great signs of leadership.
10. CHAPTER 7: SKILLS To be good at something today does not mean it will still count as good tomorrow. Always learn the latest tools and methods associated with your trade and re-assess yourself with side projects in order to fill any gaps that you may have missed.
11. CHAPTER 8: MOTIVATION Remember, your target is not a specific destination, but wherever a treasure crosses the map. Do research on several journeys that you desire to explore. Always find a resolution within the obstacles along your way. When the tides become too high, don’t be afraid to fall back to your other journeys.
12. CONCLUSION Think of Data Engineering like a soccer match. Not all are attackers, neither all are defenders, even star players on the midfield may not be proportionally all good on the different type of tasks that we have just described so far.