Very often in market research, we know the answer to our client’s burning question…or at least we think we do.
In a very real-world example (and one that has been all-consuming for me…hence my ‘radio silence’ on Twitter), our local school district is trying to find a solution to a very overcrowded elementary school that houses three distinct programs. Given the option of moving all, or part of one program (Program “A”) brought a flurry of emails, petitions, and you-name-it from angry parents advocating to keep Program “A” intact.
The school district, to their credit, offered to send a survey to parents to gather more feedback. But, with such an overwhelming response to keep Program A together, the district wondered whether they even needed to send out a survey.
This is where I remind folks that even if you think you know the answer…you need to ask the question.
Not just for the obvious reason to ensure all points of view are accounted for, but also, asking “the question” gives you an excellent chance to engage with your constituents. They will welcome the opportunity to tell you what they are thinking–because they feel you are listening…and don’t we all want to listen to our audience/customers/clients?
In the case of the school district, they did send a short survey–three questions, short and sweet.
So, even if you really, truly do know the answer–please take a bit of time and ask the question.
As I’ve mentioned in previous posts, I will happily take any survey, especially ones that directly affect me or my family…although I don’t always enjoy the experience.
The one pictured above is one from our local school district’s ‘strategic plan’…yep, full of those dreaded open-ended responses. Lots of fill-in-the-blank vague questions about improvement opportunities, academic excellence and other edu-speak buzzwords.
I hate surveys like these…open-ended questions such as the ones shown here are as enjoyable as trying on bathing suits in the dead of winter when I’m still digesting all of those holiday meals.
Why do I mock these questions like others ridicule pie charts? Because they are useless.
A bold statement, I know, but just think about this:
- Someone has to read each and every response
- That person (or someone else) has to organize the responses into manageable categories
- Then, only then, can the responses be tabulated into something that remotely resembles a list
- And finally, one can draw conclusions and plan a path forward
Oh, did I mention there are upwards of 40,000 students in our district? You do the math of how much time this is going to take to analyze and report…boggling, eh?
Best practice tip: have a few focus groups to ask these kinds of questions and develop a short response set based on the focus group feedback. This will get rid of almost all of steps 1 through 4.
Don’t forget to allow for an “Other” response because you can’t always capture every possible response, but I bet they could have come up with at least 80%–more than enough to offset the costs of the focus groups.
And, you’ll find most folks will be happy to fill in an “Other” open-end, including me.
My mud-filled dug up driveway
If any of you follow me on twitter and have seen my hashtag #failuretoplan, you might have an inkling of what this post is about…
We have had one of the wettest, coldest and generally miserable winters (and springs!) on record. I think Seattle has had fewer than 5 days above 60 degrees since something like October. So, when the sun is out we embrace it like a long lost friend, soaking up copious amounts of Vitamin D and generally acting like goofy children.
Sunny days are also perfect for scooter riding. Rainy days are not. So, my little scooter has been wintering in our garage hooked up to a trickle battery charger waiting for weather more conducive to riding than sliding. And, lo and behold, we have had a few such wonderful days lately…oh joy!
But, we are also in the midst of our last large (read: expensive) home improvement project. Without going into a lot of detail, we are replacing the retaining walls that border our driveway…one of those projects where the end result is very unsatisfying visually, but structurally necessary.
Thus my #failuretoplan begins…driveway gets torn up, walls get taken down and suddenly, no access to the garage. Argh. If I had planned ahead, my scooter would be happily parked on the street and I would have gotten at least 4 days of riding in.
The same thing is true for most industries, including market research. Planning ahead can make all the difference in a project that comes in on-time and on-budget. Taking that little bit of extra time in the beginning can make all the difference in questionnaire/discussion guide design, fielding contingencies, and analysis. I’ve found projects that have that extra time built in tend to be the ones that are most successful–or at least run into the least amount of ‘bumps in the road’…or torn up driveways.
And the proverbial scooter isn’t left in the garage!
The poor pie chart has been much maligned by data visualization guru Edward Tufte and more recently in this Voice of Vovici blog post.
Really, what has that poor pie chart done that makes it so abhorrent to folks? Yes, overuse and misuse are the main culprits, but that can be said of many charting options.
The beauty of a pie chart is that each ‘slice’ is instantly recognizable as a proportion of a 100% total. The same cannot be said for the bar chart…without the scale below, one would have to rely on grade school arithmetic to quickly add up the sections. Is the pie chart glamorous? Nope, but it is quick and easy for readers to understand the proportions you are reporting about.
I admit that I was nice and put the actual data labels (aka: percentages) in both charts, but just envision them without and see how easier the pie chart is to read…
Now where the pie chart fails miserably is when comparing more than one data set. In the example below we have three products that respondents are either currently offering, planning to offer, or have no plans to offer. It’s much easier to line up three bar charts (ensuring the 100% scale is there!) than try to compare the three pies.
So, think about whether your reader needs a quick visual of a single data set…and maybe the lowly pie chart is just the ticket.
Sometimes I wonder if I only write blog posts when I see egregious errors by other research firms…maybe it’s because it spurs me to action in my otherwise ordinary workdays.
Many of us researchers take surveys to keep up with trends and new ideas as well as learning what not to do when designing and fielding our own surveys.
My latest foray into survey-taking was a rather lengthy one about a local grocery store. Earlier in the survey I was asked where I buy groceries and I indicated not only did I shop at the stores listed, but I also happen to shop at a local chain…in fact I do a significant portion of my shopping at this local store.
But, when it came to allocating my proportion of grocery shopping across the stores, the survey designer failed to include my little local one…even though I had indicated I shopped there in an earlier question. This is what I saw when I tried to allocate my grocery shopping dollars:
As you can see, making matters worse, the survey folks expected answers to this question to total to 100%…kind of hard when you don’t have enough choices! Ugh.
I was very tempted to abandon the survey at this point as I’m sure other folks will who potentially find themselves in my dilemma.
So, tip for today, if you allow an “Other” choice earlier in a survey, make sure you account for that in any subsequent questions that use the same list of choices.
We all like having lots of choices…