Should you be a data engineer or a software engineer?
It depends on what you enjoy building with code. You should try both and see. Here is how I would find out.
Build a small software program that does something interesting. An example could be a python program that web scrapes the top headline on a website that you follow.
Manipulate some data in python pandas. Take a dataset that is related to one of your interests and transform (clean up) the data with pandas. Then explore the data and display some charts. (Kaggle has tons of great datasets.)
Which of those exercises did you enjoy the most?
If you chose the first one, continue with software engineering. If you liked cleaning and manipulating the data in pandas, try data engineering. And if you preferred exploring the data and displaying charts, data analytics or data science might be your route.
I was a software engineer before becoming a data engineer. To me, both are super fun and challenging. I would still be happy doing either.
This is why I moved from software engineer to data engineer.
I saw that Machine Learning and Artificial Intelligence was going to be extremely important in industry. So I studied Machine Learning and AI very intently.
I completed a Udacity Machine Learning Engineer nanodegree. (This was very difficult and I don’t recommend it unless you are sure about becoming a machine learning engineer.)
But I realized that to get a job as a Machine Learning Engineer, I might need a PhD or a masters degree.
Also, I really enjoyed manipulating data with Pandas in Python. There was something fun about transforming wild data into a format that could be used for machine learning, AI, or analysis. I guess this could be called data cleaning, and I wouldn’t want to do it all the time, but I still find manipulating data in Pandas to be an enjoyable challenge.
At that point I realized that there would be a lot of demand for the people that prepared the data and kept the machine learning pipelines going. So I focused on becoming a data engineer.
A couple more things pushed me towards data engineering.
I enjoyed building pipelines in AWS. I did a lot of pipeline building as a software engineer and found it to be challenging and rewarding.
I liked studying machine learning and AI. But I didn’t really enjoy the tediousness of tuning little parameters and then waiting a long time for the machine learning models to produce an output. As a data engineer, I can be around machine learning models, but I don’t have to do the tedious part.
Most importantly I loved solving coding puzzles. Every bug you run into is a puzzle. Every project requirement is a puzzle. Like software engineering, data engineering has plenty of puzzles to solve.
If you want to continue learning data engineering, try out Data Camp’s Data Engineering track. They also have a Python Programmer track if you want to try the software engineer track.
See which subject you enjoy most and then continue the journey.
There is a huge demand for software engineers and data engineers right now, so either one is a good choice.
