Every modern company depends on data to drive its decisions and products. With all of that data to capture and interpret, there is a huge demand for skilled data engineers. Jeff Katz of Jigsaw Labs demystifies the data engineering career path, breaking down what it takes to pivot into data engineering, the typical responsibilities (and salaries!) you can expect at any level of your data engineer career, and how to learn data engineering. Find out how Jigsaw Labs is preparing students for today’s data engineering field with a fresh perspective on tools, techniques, and skills and an externship that gives students real world experience.
Meet the Expert: Jeff Katz, Founder and Instructor of Jigsaw Labs
In a modern organization, teams make decisions based on data. The sales team wants to target the best leads. Marketing wants to promote the best ads on the right platforms. The Operations team needs to understand how they’re gaining and losing money. All of that involves data.
But of course, someone needs to capture and analyze this data, whether it’s from inside the company’s web application, or located in a third party tool, like Salesforce, Google Ads, or Google Analytics. The first step is designing a system that can collect this data repeatedly and accurately, so that it can be used for further analysis. That’s the foundation of a data driven organization, and that’s the job of a data engineer.
It’s the data engineer’s job to collect data from various sources, change that data so it’s easier to use, and store that data in an analytical database. The data engineer needs to automate this to ensure the data is current, accurate, and standard throughout the organization. From there, the data engineer may be involved in cleaning and analyzing the data to provide to the marketing, operations, sales, or product teams.
Where does a data engineer fit into a team?
Once the software is developed to collect the data into an analytical database, how data engineers work with the broader company depends on the organization’s needs.
3 Traits of Successful Data Engineers
The typical data engineer has three main skills:
There are a couple of main types of data engineers and there are some key differences in the skills and tools used.
Analytics engineers use extract & load (EL) tools. These tools pipe data into a data warehouse automatically, after some initial setup. Once the data is in the data warehouse, analytics engineers use SQL to select, manipulate, and analyze that data.
These SQL queries can grow to be quite long, consisting of dozens of files. To organize these commands, analytics engineers use a tool called Data Build Tool (DBT). DBT is the standard way to organize these transformations. Once the queries have been performed, next is to use this data to develop dashboards in business intelligence tools, such as Looker or Tableau.
Analytics Engineers mainly rely on their knowledge of third-party services to build data pipelines, and SQL to manipulate and query that data to connect to dashboards. But as the pipelines become more complicated, an out-of-box solution may not be available. In this case, companies will need data engineers to build their pipelines and organize their data.
Data engineers use SQL and Python to extract data, and cloud computing tools like Docker and AWS to deploy their data pipelines. Because the data needs to be repeatedly updated, data orchestration tools Airflow are used to repeatedly move data from one location to the other. Data engineers may use web frameworks like Flask to make their data available from a website or API.
One specialization in data engineering is to become a Machine Learning Operations practitioner. An MLOps practitioner is used to deploy a machine learning model, continually update (train) the machine learning model with new data, and to ensure that the machine learning model makes predictions quickly (when necessary) and without consuming too many resources. Doing so requires knowledge of web frameworks like Flask, SQL and Python, and cloud computing skills like AWS and Docker.
Soft skills for Data Engineers
Data engineers are serving internal stakeholders, so it’s necessary for them to communicate and understand what the use case is and have that in mind when building the system.
For example: It could be that you’re noticing teams using different metrics for what is considered an active user. It might be better for an organization if everyone is on the same page and it might not, but it’s the data engineer’s job to think about that. It’s important to have these perspectives and translate them into a data product to serve teams with the appropriate data.
Data engineering isn’t so much a linear career ladder as it is a jungle gym. Once you have learned the fundamentals of data engineering, you can choose a specialization, climb the ladder, or switch into other tech careers fairly easily. Another nice thing about this field is that you don’t need a certain amount of years of experience to progress. The technologies are new and if you learn them, there’s nothing stopping you!
A “typical” data engineering career path could look like…
Entry-Level vs Senior-Level Data Engineering Roles
The differences between junior and senior-level data engineers are similar to the differences between junior and senior back end engineers.
Data Engineer Salaries
Data engineers at the entry-level typically make between $80K-$100K. Mid-level data engineers can make $115K-140K. Senior-level data engineers make anywhere from $170K-$200K.
What types of jobs are Jigsaw Labs students landing after graduation?
Graduates of Jigsaw Labs have gone on to become back end engineers, data engineers, Python engineers, and even machine learning engineers. They have landed roles at cryptocurrency companies, fintech companies like AmEx, and medical organizations like Memorial Sloan Kettering. A few of our students incorporated what they learned at Jigsaw Labs into their business and product development roles. Jigsaw Labs graduates have starting salaries of around $115K a year.
In a data engineering interview, you can absolutely expect questions on SQL. You can also expect a soft question about the types of data pipelines you’ve built. Finally, they’ll ask about Python. Python questions arise slightly less often, but strong data engineers are strong Python engineers so you should not be surprised when it’s assessed.
If you want to ace an interview, you’ll need to know SQL and be able to pass the technical assessments. If you want to wow your potential employers, show them that you’ve built an end-to-end data pipeline or written code that uses object oriented programming and Airflow that’s deployed on a cloud service like AWS. In the interview, you have to be able to discuss your reasoning behind your builds and be able to organize your thoughts. It’s important to be someone people want to work with and to be enthusiastic.
Which careers transfer to data engineering well?
If you automated something in your last job – for example, maybe you wrote an Excel Macro because you found yourself doing the same process over and over – take that as a sign that you could switch over to data engineering.
For example: One of our Jigsaw Labs students was a salesperson and his interest in data engineering started when he wrote a LinkedIn scraper to identify leads. In my former career, I was a lawyer and I wrote something that pulled information from various legal contracts (like the contract date, title, and signatories) so it could place this information into Excel for me. Things like that are great because they show you know how to think through a problem and deliver an efficient solution.
Former marketers can find a good fit in data engineering because there are so many use cases for marketing. Every company has issues with selling their products, and if you have experience working in marketing then you will be attuned to the marketing team's needs as a data engineer.
At Jigsaw Labs, we teach the full analytics engineering and data engineering stack and we are thorough about each skillset involved.
When we teach SQL, we don't stop at relational queries; we also teach window functions, CTEs, and subqueries – and how to organize these queries in a large codebase with DBT. Students learn the tradeoffs of using different databases like Snowflake and Postgres. Students also learn modern tools like Fivetran and Meltano to organize the data.
Then it’s back end engineering with Flask, Python, Object Oriented Programming, and testing. After that, we teach cloud computing with Docker and AWS, and orchestration with Airflow.
We are constantly iterating on our curriculum. So far we've thrown out a third of our curriculum every time we’ve launched a new cohort. We want to make sure we’re teaching exactly what employers are looking for now. Because our curriculum is closely aligned with the job market, our graduates are in high demand from employers.
What can students expect to learn in the internship at Jigsaw Labs?
Halfway through the program, six hours of class time per week is devoted to internships. Some students work up to 20 hours a week in the internship, and do so with top companies. Our previous externships have included:
Is there an ideal student for Jigsaw Labs’ Data Engineering Bootcamp?
In admissions, we are looking to see that a student is a good culture fit for the bootcamp and that we are confident they will land a job after graduation. Part of that is ensuring students respond well to feedback, collaborate with others, and that they’re curious about the subject matter. Halfway through the program, we offer an internship program, so we need to be able to count on students to be on time to meetings with partnering companies, and be engaged and respectful throughout the partnership.
Do students need to be able to code to apply to Jigsaw Labs?
To prepare for the bootcamp, we recommend that applicants complete our free training, which includes 10 readings and 4 labs. Once an applicant understands that material, they will have the technical skills to apply for Jigsaw Labs. From there, we’ll have a technical interview, which allows us to make a good prediction about your skills coming out of the program.
How does Jigsaw Labs support its students in their career journeys?
Like other bootcamps, we source our students interviews for various positions we think our graduates would be a good fit for. But after students graduate, we’re not done with meeting as a class. We have twice weekly meetings in the evening to help students continue with their internships, prepare for technical interviews, and apply for jobs. This is important because students have many responsibilities when they graduate and often feel pulled in multiple directions. Meeting biweekly means students have instructors to help them balance their time, learn additional skill sets, and prepare and give feedback on interviews. All graduates are undergoing this same process, so it makes sense that they do so together so they can share their learnings.
For anyone starting in data engineering or looking to level up their career, we recommend:
All the coding bootcamp news you may have missed in June!
Top 4 Tips for Launching a Cybersecurity Career after Bootcamp!
Find out why bootcamp instructor Lindsey loves teaching data science at Flatiron School!