… or how I realized there isn't one single path to becoming a data engineer.

I had been lurking forums like r/dataengineering for a while now and every few months I see these two questions pop up again and again:

What is a data engineer?

and

How much code do I need to know to be a data engineer?

And every single time, it sparks vibrant yet inconclusive discussions. This is not surprising, though. It's a relatively new field that's evolving like crazy, and it's interesting to see the different roles and backgrounds that make up this diverse community.

I switched to the glamorous job of data plumbing a while back, but I used to be just a plain backend developer. So I believe I can give a bit of shape to the different varieties of data engineering roles we see. I couldn't think of different names for each of the roles since they all indeed do the one essential thing: move data from one place to another so that it makes more sense. If data engineering were a role-playing game, there would be 4 classes: The Wizards with their magical SQL queries, The Jedis who wield the force of software engineering principles, The Ninjas of no-code solutions, and last but not least, The Chameleons who can adapt to any data needs.

1. The Wizards: Traditional SQL-Driven Data Engineer

First up, we have the SQL-based traditional data engineers, the Parselmouths of the data world. They are proficient in writing extremely complex SQL queries, designing and optimizing database structures, and transforming data for reporting and analytics purposes. Often, these data engineers come from backgrounds in SQL development, data analysis, or business intelligence (BI).

Strengths

  • Proficient in data querying and modeling, enabling quick and accurate data extraction and analysis.
  • Strong understanding of databases, optimizing their structures for efficient data processing.
  • Swift responders to business data requests in data-driven organizations.

Weaknesses

  • Limited ability to handle complex ETL processes or automate data workflows without additional coding skills.
  • May face challenges in adopting modern tech stacks or integrating machine learning algorithms.

Skills Needed

To join the ranks of the wizards, you'll need a solid foundation in SQL and a deep understanding of databases:

  • Start with the Basics: Familiarize yourself with SQL syntax, data types, and simple SELECT statements.
  • Master Data Manipulation: Learn how to filter, aggregate, and join data using advanced SQL concepts like GROUP BY, JOIN, and subqueries.
  • Understand Database Concepts: Grasp the fundamentals of relational databases and their design principles, including normalization and indexing.
  • Real-world Practice: Gain practical experience in data manipulation using SQL by working on personal projects or analyzing open datasets on sites like Kaggle.

2. The Jedis: When Software Engineers Use The Force

Next, we've got the coders turned data Jedis, who use the Force aka software engineering principles to build data-intensive platforms. With a coding background, they thrive in utilizing technologies like Apache Spark or Apache Airflow to tackle complex tasks.

The fundamental difference between these engineers and the SQL-driven DEs is the ability to use application development tools like docker, logging, microservices, and APIs to implement domain-driven solutions.

Strengths

  • Proficient in coding and software engineering practices.
  • Skilled in building robust data pipelines and automating ETL processes.
  • Comfortable with building complex data architectures and services.
  • Adaptive in integrating modern tech stacks and AI/ML algorithms into data workflows.

Weaknesses

  • May lack in-depth knowledge of SQL querying and modeling techniques.
  • Might be overkill in certain environments where no/low code or SQL based is preferred.

Skills Needed

Coders looking to transition into data engineering can leverage their programming skills to become Data Jedis:

  • Programming languages: Create data pipelines using programming languages like Python or Java, integrating them with data sources and storage solutions.
  • Learn Data Engineering Tools: Familiarize yourself with popular data engineering frameworks like Pandas/Polars, Apache Airflow, and Prefect. Online tutorials and documentation can be valuable resources.
  • Embrace Big Data Technologies: Dive into distributed computing software like Apache Spark, Apache Flink, Dask, and storage technologies like Hadoop and Amazon S3, which are crucial in handling large-scale data.
  • Explore Cloud Platforms: Familiarize yourself with cloud services like AWS, Google Cloud, or Azure, as they offer robust data engineering tools and infrastructure. Also learn Docker, Kubernetes, etc to deploy your code there.

3. The Ninjas: No-Code or Low-Code Tools Instead of Katanas

These data engineers may not be hardcore coders or SQL junkies, but they can amaze you with their GUI-based tools that make data workflows a breeze. Often coming from non-technical backgrounds like finance or business, they've picked up data engineering skills through BI tools or drag-and-drop ETL platforms. Their main focus is to get the job done as soon as possible, be fast just like a ninja.

Strengths

  • Excel in using GUI-based tools and low-code platforms for building data workflows quickly and efficiently.
  • Ideal for business-oriented teams or individuals with non-technical backgrounds seeking to engage in data engineering tasks.
  • Rapid adoption and ease of use make them valuable assets for fast-paced projects.

Weaknesses

  • May lack the flexibility and customization options of traditional coding approaches.
  • Some GUI tools may not support complex data engineering tasks or scale well with growing data volumes.

Skills needed

No-code and low-code platforms empower non-technical individuals to participate in data engineering. Aspiring Wizards can follow these steps:

  • Discover No-Code Tools: Explore user-friendly platforms like Microsoft Power Automate, Zapier, or Google Cloud Data Fusion, which enable data workflows without coding.
  • Data Integration: Learn how to connect various data sources and automate data transformations using drag-and-drop interfaces.
  • Master Data Transformation: Utilize GUI tools to cleanse, enrich, and prepare data for downstream analytics and reporting.
  • Practice with Business Use Cases: Work on real-world business scenarios, creating data workflows that cater to specific business needs.

4. The Chameleon: Hybrid Data Engineers

Last but not least, we got Hybrid Data Engineers who are essentially the chameleons of the data world. As data engineering practices have evolved, a hybrid role has emerged that combines software engineering expertise with a strong focus on data processes and business needs. These data engineers understand the technical aspects of data engineering while also aligning solutions with the organization's broader goals. Just like how chameleons adapt to any environment depending on the requirement, these engineers can encompass data/business analysis, ETL pipeline development, visualization, etc.

In comparison to software developers turned data engineers, these folks take just the bare minimum required skills from software engineering and focus the rest of their attention on understanding the business.

Strengths

  • Effective translators between technical teams and stakeholders, navigating both software engineering and data management challenges.
  • Proficient in crafting data solutions that align with specific business needs and requirements.
  • Valuable problem solvers, capable of navigating both software engineering and data management challenges.

Weaknesses

  • Striking the right balance between technical best practices and business agility can be a delicate challenge.
  • May require a diverse skill set, making their learning curve potentially steeper compared to specialized roles.

Skills needed

Hybrid Data Chameleons combine technical skills with business acumen:

  • Develop Data and Business Skills: Master SQL querying, data modeling, and programming languages while also enhancing your understanding of business operations and requirements.
  • Communication and Collaboration: Hone your communication skills to effectively translate technical concepts to non-technical stakeholders and vice versa.
  • Infrastructure: Learn different IaC tools such as Docker Compose, Terraform, and Pulumi to maintain reproducible data infrastructure to help with setting up

Challenges and Opportunities in Data Engineering

From what I have seen, I believe the data engineering landscape will keep changing in the next 5 years, branching into distinct specializations within the field. Some key points to consider:

1. Legacy vs. Modern Tech Stacks

Many data engineering teams face the challenge of dealing with legacy technologies and processes that might not align with modern software engineering practices. Transitioning from traditional BI-focused platforms to modern data stacks can be a bumpy road, especially if the team lacks a strong software development background.

2. The Human Element: Embracing Change

Change is hard, and this holds true in data engineering too. Introducing new technologies, best practices, and workflows can sometimes meet resistance from team members who prefer sticking to familiar methods. Building a culture of openness to change and continuous learning is crucial for successful data engineering teams.

3. Balancing Flexibility and Best Practices

Striking a perfect balance between business pressure and software engineering principles is essential. While serving the pressing needs of the stakeholders is our foremost priority, it is important that we adopt modern software dev practices like TDD, version control, and log management. This ensures the data infrastructure is something that can be recreated from scratch again and again, making it the resilient data backbone of the organization.

Into The Looking Glass: The Future of Data Engineering

With the rise of machine learning and AI, data engineers will play an even more pivotal role in extracting meaningful insights from the vast ocean of data. With that in mind, more and more teams will adopt cloud-centric products for cost-effective solutions with better ML integration. DataOps practices and DataOps engineering teams will emerge, leading to greater collaboration between data engineers, data scientists, and other stakeholders.

As data privacy and security become paramount concerns, data engineers will embrace practices that prioritize data governance and compliance. These are pure speculations, however, the future is bright for data engineers. The more AI and ML get embedded in our data-driven decision-making process, the need for clean and reliable data pipelines becomes more important.

Data engineering is an exciting and ever-evolving domain with diverse roles and backgrounds shaping its landscape. From SQL Wizards to software Jedis and no-code Ninjas, every type of data engineer has a place in the world of data.

Thank you for reading my analysis of the data engineering role in the year 2023. Feel free to hit me up through my email address or DM me on Twitter @ratulotron.