Me vs. ETL

I had a great moment today.  A moment where I felt like I conquered the beast known as “ETL.”  For those of you who don’t know (lucky you!) ETL stands for Extract, Transform, Load.  Like it suggests: it is the term used to move data around and taking elements from “raw” forms and making them usable data.  And it is has been a beast in my life because I feel like it gets used for “gotcha” moments in the data/IT/technical world.

Let me side-track and say that I have a background in physics and mathematics AKA problem solving.  I didn’t go to school for computer science and I have no formal training in the proper terminology of data management, ETL, data warehousing.  I appreciate those who did, because they have the benefit of being taught the history and terminology that drives a lot of what goes on.  Unfortunately, I never did that.

I was driven to a job in data because of my ability to problem solve and communicate.  Math and physics are very algorithmic, they require you to assess your surroundings, identify the problem, and work on a solution.  I started out in operations, moved to quality of operations, then process improvement of operations, and finally into what I would consider the “IT world.”  Each layer that I dove into required more and more data access and to get closer and closer.  You can imagine in my quality role, I received flat files from existing front end systems to do analytical work.  Once I got to process improvement, I actually interacting with reporting associates who would pull custom SQL queries for me out of our transaction databases.  Today I work in that same type of “reporting” environment.

What’s kind of funny is – now that I’m here I realize it’s all the same.  If you’re good at understanding canned flat files and deriving value (typically analytically, but also just in general), you’ll be just as good as you get closer to the data.

So today I feel like I conquered ETL.  Once in an interview someone asked me how good at ETL I was.  I had to say that I didn’t have much ETL experience, and I think that’s what cost me the job.  I wish I would have known that an ETL tool like SSIS leverages a GUI that hauntingly resembles Microsoft Visio.  I wish I would have known that.  Because you know what I am really good at?  Visualizing processes, making process maps, streamlining processes, communicating and understanding what processes are happening.

What makes me feel like I conquered ETL?  Today I had a meeting where I used my communication skills to bridge the gap between two groups of people knee deep in the data weeds.  I moved past the barrier of never having developed an SSIS package, making a stored procedure, or developing in SSRS to solve the problem we were facing.  Does knowing SQL make it easier?  Slightly – only in the sense that I can find the fields and tables the data comes from.  But you want to know what the “problem” we were facing was?  It was “are we deriving the right fields to answer the business ask?’  Huh.  SSIS doesn’t know the answer to that.

So if you’re out there and want to get more into data, and someone says to you “rate your ETL skills.”  Push back and let them know that you are an amazing communicator, that you take time and effort to fundamentally understand data (from both the system capture end and how business users interpret it).  And today I give my ETL skills a 10, because I bridged the gap between everyone while leveraging knowledge I’ve picked up along the way and not being intimidated by ETL.

Okay – my “yay” moment is done.  Those of you that do ETL, I thank you.  There are a lot of important components that go into making data systems that contain the information people need to do their jobs and for businesses to succeed.  By no means is my aim to discredit you.  I love your ERDs.

 

 

Dot Plots

Today I was reading Stephen Few’s Information Dashboard Design aloud while Josh was doing some fall PC clean up and was on the chapter “An Ideal Library of Graphs.” As Stephen describes it there are several charts or graphs that should make their way onto dashboards and he goes into detail on the reason behind each and how to properly apply them.

We stopped to discuss the dot plot, because it is one that both of us under utilize.  From my perspective there’s a lot of under utilized space below each dot.  As we were exploring it deeper, I also felt uncomfortable with dots above a certain threshold, because at that point the actual data point was lost.  We decided to open up Tableau and start playing around.  In my mind I was thinking that using a Gantt bar as the point would better represent the data.  (Mind you, Stephen says that you use dot plots when you may not go to zero, otherwise stick to bar graphs).  I thought Gantt bars presented the perfect “happy medium” between the dot plot and the bar chart without causing the end data reader to incorrectly visualize length.

Below are the four different representations we came up.  All things equal, I think I might be most partial to the ‘Plus Plot.’  I think the general shape tends to draw your eye directly to the center point, and even if they were larger you’d still know where the center was.  I can see how this might get cluttered quickly, or if more encoding were done things could go awry.

I also found that I liked the Gantt bar, but needed to shorten the bar significantly to eliminate mentally drawing drop lines and visualizing as bars.

Getting started in data

Continuing on the path I started the other day, here’s another blog post that I first wrote internally back in March of this year.

The question: How do you think you can encourage more women in data?

I think the toughest part of getting started in data is understanding the field and feeling confident working within it. It took me some time to realize that working with data is a mindset. It isn’t a functional area reserved for someone who has amazing coding or technical skills. It is for people who are detail oriented and want to understand the “why” behind things.

Finding the reason or purpose behind causes is one of my favorite components of working in data. Within this field, you get the opportunity to do that everyday. I am also extremely passionate about taking the data and creating a visual story with it.

If you’re interested in data or want to know if it is the right field for you, ask yourself if you like knowing the “why.” I’d also be curious if you are driven to mastery. Within the data field there are daily opportunities to learn and apply that newly found knowledge to your job. If that’s something you like doing, data and data analysis is the right place for you.

I’ve been in several analyst roles and they have meant different things. The technical skill set and applications have always varied widely, but there have been a few universal constants. Those constants are extensions of your personality – being a great problem solver, a critical thinker, and someone who can both reach solid conclusions AND (more importantly) communicate them. These few skills are what make people great analysts. The tools they use are just there to help them broadcast their findings.

If you’re on the fence, I would also encourage you to go find some data available in your daily life. There’s literally mountains of data out there. Find something that you’re interested in knowing more about and analyze it. It is a great exercise to start thinking like a data analyst and I guarantee you that you’ll find something new and interesting.

Once you’ve analyzed one data set, find something totally different. See what commonalities in the structuring of the data you can find. After having received hundreds of different data sets, I can tell you there are several commonalities. The more you practice with any set of data, the better you’re going to get at overall data analysis.

My last comment would be to remember there’s never one right answer in data analysis. It is a creative and interpretive field. Your interpretations should be based on the truth of the data, but realize once you’re in the analysis driver’s seat it’s up to you to showcase what’s important.

Quick and easy – parameters to aggregate dates


(Now with video – video uses different data as an example)

One of my favorite uses of parameters is to dynamically change date aggregations without the pesky drill symbol.  Sometimes I want to just see week or month, quarters tend to be pretty worthless.  Especially if I’m doing something discrete, I really don’t like the way Tableau breaks apart the data view.

The creation of the parameter leaves me with a very clean user experience and ensures users don’t negatively interact with the viz.

Here’s the setup – first create a parameter of data type string and type in your allowed values.  Mine is going to have week and month:

parameter1

(note I changed the casing on the words later on for look/feel)

Next create custom dates based on your primary date field.  Mine is called “Start”:

dates

Right click on the field, go to Create -> Custom date…

For the sake of this viz, I made both a Months and Weeks (both are date value).  At one point I had days (shown as “Exact”, but I abandoned it)

Last step is create the calculated field.  Right click on the parameter at the bottom, create calculated field:

dates2

Place it on your designated shelf.  You may have to do some wrangling on the field to get it to display right.  For the viz I ended up with, the field is set to “Exact Date” and “Discrete.”

Check out the final experience below:

Data Analysis – First Steps

Earlier in the year (March), I participated in a panel discussion for a data+women series at the Phoenix TUG.  As prep I was given a list of questions and asked to pick a few I felt comfortable answering.

To facilitate the sharing of my experiences, I wrote a blog post on some of the more intriguing questions.  What comes next is a blog post I originally published within my company’s social media site on 3/23/16.

What is your first step of investigation when you are given a dataset?

So before investigating a data set, I think it is critical to understand what the data output contains. I think it is important to meet with the business owners or subject matter experts that are responsible for creating, maintaining, or using the data. This helps to ensure you understand what they are looking for and what the different fields mean.

Assuming the pre-work has been done with business owners, I always approach a new dataset the same way. There’s three things I do every time.

The first is to find a categorical dimension that seems to be at the highest level. Typically I’m looking for a category that has less than 20 members (10 to 15 being ideal). This data can set a great foundation to understanding the overall population of records. Once I’ve found that categorical dimension, I immediately create a Pareto chart. Recall that a Pareto chart is a bar chart where the bar volumes are ranked from greatest to smallest. Additionally it has a line chart on the secondary axis displaying the cumulative percentage. The goal of the tool is to see immediately where most (or a majority) of problems are. And it usually follows the 80/20 rule logic.

The next thing I do is plot records over time, usually by month, sometimes by week. This is great for understanding any seasonality and for finding immediate trends. Often systems are changing and new processes are implemented. By plotting records over time, you can immediately take any funny data points back to your business owner and ask for clarity.

The more clarity you have around your dataset, the more successful you’re going to be at understanding the information you can retrieve from it.

The last thing I do is take that categorical dimension and plot each one against time. This really is the combination of the first two to a large extent. From this you’ll be able to further pinpoint any trends that are category specific and find even more potential seasonality or process changes.

 

By creating the three visuals I mentioned above, I usually feel pretty confident about the “vitals” of the dataset. I get a great sense of being able to communicate in the same language and accurately reference volume. I’ve found that this is a great first step when working with almost any set of data. It makes an excellent conversation starter with business owners and also proves itself as a useful communication tool when bringing new team members into a project.

That’s my initial analysis workflow, what’s yours?

#IronViz Entry – Mobile Design

Part of being involved in the Tableau community means publicly publishing visualizations to learn and grow.  It’s also a great way to find inspiration.

As I’ve pushed myself to be more active within the local Phoenix Tableau community and social (Twitter) community, I knew it was time to “step up” and make an Iron Viz.

My design aesthetic tends to be minimal, slightly formal, and geared toward (in my mind) elegance.  I like to make dashboards and visualizations that highlight the data, but don’t jump to many conclusions.  I’m very conclusion agnostic, so leading people too far down a path doesn’t always seem right.

All that being said – I wanted to make an Iron Viz entry, but it needed to be simple.  The deadline is September 18 (today as I’m writing this).  So I wanted to develop something relatively straightforward that got to the heart of mobile design.

The data inspiration for the viz actually came from an animated bar chart .gif showing the “Top 10 Economies” growth from 1969 to 2030 that I saw on Twitter.  I thought it would make a nice bump chart or slope chart, and the conclusion of the data was already compelling.

First the data gathering process – relatively simple on this one.  The .gif referenced the source, a quick Google search led me to the results.  I’m going to loosely promise to publish the excel file at some point.

Next up was diving right in.  I recently made a “micro viz” in Tableau 10, designing it exclusively for a very tiny space.  I actually didn’t use the device designer for this one, instead opting to develop the whole thing with my intended sizing.  With the sizing set, development was similar to what I’ve done in the past in v9.3 (version of Tableau I use at my job).

Transitioning to device specific design was different than I thought.  Since I knew the final product (in my mind) would need to have more emphasis placed on the mobile view.  It is after all a mobile design challenge!

Like I mentioned above, I had a pretty good idea of what I wanted the final viz to look like.  I knew there needed to be a bump chart and I was going to call attention to China and India.  What I didn’t realize is that the device designer is really geared toward creating a “master view” and then augmenting that master for the device.  This makes sense to me as I rethink the way the feature was presented.

What this meant for the creative process?  I wasn’t able to make visualizations (sheets) and quickly drop them on the mobile layout.  For each new “potential viz” I had to first drop it on the overall dashboard and then bring it on to the mobile specific dash.  It made the whole process kind of clunky.

I also struggled a bit with getting to formatting features quickly.  I can’t double click on titles to adjust font sizes in device preview, gotta go back to the default layout.  I’ll have to adjust my thought process next time and really think about starting from the default view and optimizing a mobile version.

I probably cheated the viz out of more depth that could have been added if I had truly started with the default dashboard and then made a mobile design.  It is fair to say that the default dashboard has real estate for more data insight and data depth.  I am still very pleased with the overall final results.

 

Blog post ideas

Ann

  • How Apple Watch has changed my life
  • How tracking calories has changed my life
  • Why is my life always changing (har har)
  • New books
  • Walk through improvements on micro viz
  • Revamp D+W posts
  • Tableau – continuous fields from discrete values for heat maps

Josh

  • Starting a home automation system
  • Hobby photography
  • Stretch project  – IFTTT our data
  • FLIR

Low carb brownies

One of the biggest struggles of a low carbohydrate lifestyle is sweets/desserts.  At least in my mind, sweet is synonymous with carbohydrates.  The easiest way to remedy this is typically by using artificial sweeteners.  But those sweeteners alone don’t usually satisfy food that is traditionally associated with dessert.

Enter eggs into the equation.  Eggs are surprisingly versatile and can actually mimic the structure of things we would typically classify as “bready” or carbs.

So here it is – arguably my favorite recipe of all time.  I eat these for breakfast and feel absolutely no guilt (because they’re essentially cooked eggs mixed with cocoa powder).

Ingredient List

  • Unsalted butter (160 grams)
  • 6 large eggs
  • Cream cheese (120 grams)
  • Cacao powder (60 grams) – can also use cocoa powder
  • Vanilla extract (2 tsp)
  • Baking powder (1/2 tsp)
  • Salt (1/2 tsp)
  • Liquid sucralose (1/2 tsp, measured out as 60 drops) – this is equivalent to Splenda

Oven Prep: 350 degrees F

General Instructions

  1. Melt butter and cream cheese in microwave, 1 minute on soften setting.
  2. Mix eggs, sweetener, butter, and cream cheese in blender.
  3. Add in all powder ingredients and mix thoroughly.  (When doing this, I say that I’m “buzzing” the mix.  Mix very well.)
  4. Line baking dish with parchment paper.  I use an 11 x 13 glass Pyrex dish.
  5. Bake at 350 degrees F for 30 minutes.
  6. Once done, remove from baking dish and cool on cooling rack.
  7. Reuse baking dish as storage vessel.

The resulting dish is amazing.  In my mind it is exactly like brownies.  The mixture of eggs with the leavening of the baking powder perfectly mimics the cake-y, slightly heavy texture you’d expect with a brownie.  And the cacao powder (and likely the butter) add a level of decadence that ensures you won’t over indulge.

The home automation mindset

Home automation is something that I have been following for several years.  I think we all have seen automation on TV.  Bill Gates’ house with music and pictures that follow you around in the 90s.  For me, it is about getting to Star Trek level of voice control with automation that does what I want with minimal interaction.  Getting to that ideal state is one of the most thought intensive projects I have taken on.  To use a meme, one does not simply automate a house. It requires a mindset.

The first thing to keep in mind is that the home automation landscape is like any other technology ecosystem but even more complex.  The one thing that makes it more complex than gaming, or photography is that home automation is so open.  I know that I can ignore news on PlayStation because I don’t have that console and I know I won’t ever be able to use it.  I also know that gaming has a limited scope that won’t impact much outside of gaming.  With automation, anything can be made compatible with almost anything and it affects my entire life.  For example, I’m thinking about getting a wi-fi souve machine.  How well does that work with SmartThings or any of the other platforms I have?  If my fridge broke tomorrow, I would have to consider automation in my replacement decision.

This openness is great because it means that I can do anything I put my mind, and time, to.  Literally, I could write an app that uses the accelerometer in my iPhone to change light colors.  I could then paint with all the colors that Hue can produce, while I dance around the living room.  I don’t have the time to do that, but that is really all that is stopping me.  Thankfully, others have used their time productively to produce tools that are more useful. But, of course, that means searching and testing the work of other hobbyists.

With the help of talented coders around the world, I can get better and more effective automation.  But that brings up the next thing to keep in mind: what do I even want to happen?  When I come home, what lights should be on?  Should it be the same lights during the day and during the night?  When does “night” start inside a house?  Essentially, I am always trying to determine what I usually do with lights and fans and temperature, and thinking about how I can communicate that to the automation systems.

All of that is the basis for the mindset needed to do home automation.  It is always on my mind.  If I’m out, I am thinking about what the house should be doing.  Should I communicate my distance to home?  What would I want the house to do with that?  When I am home, I think about what could be better.  What lights are working? Should the house be adjusting to my actions? And what would those adjustments look like? While thinking about this stuff, I also have to keep in mind how much time any solution would take, is the problem already solved, and if it is worth it.

The worst part of all of this is that the biggest hurdle faced in home automation is getting past the automation vendors themselves.  They want to take this complex and involved mind set and make it one button.  That would be fine, but nobody is that good at it yet, and the least good are the people that make the products.

So why even bother? Because of the way it feels when it works.  The feeling of a giant complex machine that culminates in a fan turning off. It’s basically magic, with magical tools, and me as the magician.