I had a great moment today. A moment where I felt like I conquered the beast known as “ETL.” For those of you who don’t know (lucky you!) ETL stands for Extract, Transform, Load. Like it suggests: it is the term used to move data around and taking elements from “raw” forms and making them usable data. And it is has been a beast in my life because I feel like it gets used for “gotcha” moments in the data/IT/technical world.
Let me side-track and say that I have a background in physics and mathematics AKA problem solving. I didn’t go to school for computer science and I have no formal training in the proper terminology of data management, ETL, data warehousing. I appreciate those who did, because they have the benefit of being taught the history and terminology that drives a lot of what goes on. Unfortunately, I never did that.
I was driven to a job in data because of my ability to problem solve and communicate. Math and physics are very algorithmic, they require you to assess your surroundings, identify the problem, and work on a solution. I started out in operations, moved to quality of operations, then process improvement of operations, and finally into what I would consider the “IT world.” Each layer that I dove into required more and more data access and to get closer and closer. You can imagine in my quality role, I received flat files from existing front end systems to do analytical work. Once I got to process improvement, I actually interacting with reporting associates who would pull custom SQL queries for me out of our transaction databases. Today I work in that same type of “reporting” environment.
What’s kind of funny is – now that I’m here I realize it’s all the same. If you’re good at understanding canned flat files and deriving value (typically analytically, but also just in general), you’ll be just as good as you get closer to the data.
So today I feel like I conquered ETL. Once in an interview someone asked me how good at ETL I was. I had to say that I didn’t have much ETL experience, and I think that’s what cost me the job. I wish I would have known that an ETL tool like SSIS leverages a GUI that hauntingly resembles Microsoft Visio. I wish I would have known that. Because you know what I am really good at? Visualizing processes, making process maps, streamlining processes, communicating and understanding what processes are happening.
What makes me feel like I conquered ETL? Today I had a meeting where I used my communication skills to bridge the gap between two groups of people knee deep in the data weeds. I moved past the barrier of never having developed an SSIS package, making a stored procedure, or developing in SSRS to solve the problem we were facing. Does knowing SQL make it easier? Slightly – only in the sense that I can find the fields and tables the data comes from. But you want to know what the “problem” we were facing was? It was “are we deriving the right fields to answer the business ask?’ Huh. SSIS doesn’t know the answer to that.
So if you’re out there and want to get more into data, and someone says to you “rate your ETL skills.” Push back and let them know that you are an amazing communicator, that you take time and effort to fundamentally understand data (from both the system capture end and how business users interpret it). And today I give my ETL skills a 10, because I bridged the gap between everyone while leveraging knowledge I’ve picked up along the way and not being intimidated by ETL.
Okay – my “yay” moment is done. Those of you that do ETL, I thank you. There are a lot of important components that go into making data systems that contain the information people need to do their jobs and for businesses to succeed. By no means is my aim to discredit you. I love your ERDs.