Friday, March 11, 2011


I generally respect a scientific analysis much more than random hunches.  Normally that is on the basis of two things:  Proper data collection and proper data analysis.  Tonight I got a phone call from a phone survey company and it illustrated the extreme weakness of the data collection for anyone who relies on such surveys.  The interviewer asked me all kinds of questions like:

"How would you go about looking for government services if you needed assistance for an aging relative?"

While I tried to answer "I would go to Google and type 'Government Canada Old Age Service'" they didn't seem especially interested in that answer.  The interviewer wanted yet more data about which other ways I would look for answers... as if there was anything useful I could do after fully investigating the result of my Google search.

"Why have you not done more to Green your home in the past year?"

Well, it is complicated.  First, I live in a condo.  I can't just start tearing out the walls to put in more insulation, I have no control over the type of heating used and since I already made real efforts to maximize my recycling and reusing I don't know how I *could* do a lot more.  Since the interviewer never asked what sort of home I lived in the information they got is pretty near useless and they don't have the option to enter more random data to make my answers make sense.  Some people probably answer "It costs too much" or whatever but I can't see how they make the answers usefully correspond to the actual reasons.

"What is your household yearly income?"

Do you want income as reported to the government counting Wendy's stipend as zero (which it is for tax purposes), or how much money we had coming in?  Do we subtract tuition, do we count appreciation of investments or the increase in value of the condo over the year?  I gave a random answer but honestly there is no reason to think people are going to respond using the same sorts of rules I did.  Never mind the fact that people regularly misreport their income depending on who they are talking to, we don't even know what the correct answer is.  I can't even tell if I was lying or not!

"The Canadian government is against the UN resolution making access to water and sanitation a basic human right.  Do you agree with this stance?"

How the heck should I know?  What exactly does this resolution say?  It is clear that Canada tries to make sure all its citizens have access to water and sanitation so there must be something more to this but I can't exactly take an hour off to read up on the complexities of this situation.  Does adopting this resolution mean that countries with cruddy infrastructure are committing human rights violations?  Does it mean I can sue the government if they don't supply me with proper sanitation for my shack in the wilderness?  I have no idea!  I guess I answer something in the middle of the spectrum of answers allowed and move on.

The trouble with all these things is that my answers are uninformed (about the UN!), wildly inconsistent with other reports (income reporting), not useful (Greening my condo), or incomplete (finding government services).  The interviewer also did not speak English as a first language so I had all kinds of difficulty interpreting options for answers and I probably ended up giving flat out incorrect answers due to challenges with translation.  Now this data is going to be used to make government decisions, supply data to environmental organizations and develop models of living conditions based on location.  All of that is just going to be incorrect, and that is with a cooperative subject!  There are plenty of people who end up just taking the first option for their answer instead of getting the right one and probably a few who give wildly incorrect responses for their own amusement.  Any time you see studies being quoted that talk about surveys done interviewing 2,000 people with particular confidence intervals you have to wonder if the data is any good at all; are we really just guessing at these things?

Of course I don't have a better solution.  Presumably phone data gives you some interesting information, primarily what sorts of things patient, bored, polite people say when they want to get off the phone.  It seems sort of like democracy... pretty wretched from nearly any perspective but everything else is even worse.


  1. true!
    and then there's people like me who just end the conversation as soon as I realize it's a survey ... altering the results again.
    interesting observations!

  2. That's a part of the problem. I answer them occasionally because I am interested to see what sorts of questions people are paying them to ask but it isn't at all consistent, and consistency is key to getting good data.

  3. I think you have a misperception on how effective statistics is, even with such bias. I've been to a number of talks on statistics and I feel reasonably confident that they can adjust for such difficulties. Whether I think that they'll pay to do so... is another story.

  4. I'm pretty sure there are enough sources of bias in this particular one that accounting for them all isn't going to work.

    I have no doubt at all that stuff like the involuntary stats can survey worked wonders. But when you're outsourcing the questions to people who have no ability or desire to comprehend the responses?

  5. I answered a survey recently that I was very concerned about. It was basically asking the same question over and over with different phrasings, some more confusing than others. It dealt with political issues. I was pretty sure the eighth time I answered a question about how I felt about Canadian health care (and how I thought it compared to the US) that the person who requested the poll was going to use whichever question was answered the way they wanted to justify their position.

    Also, the water as a human rights question might be "push polling". That is, you probably didn't know about that issue before they asked - I know I didn't know about it until I read it here. Maybe one of us will go look that up and find out we are really upset with the government's position. Then we'll tell our friends and family, etc. Even more likely there are lots of people out there who got that question and were outraged without even finding out more about it. It's not impossible that the people who paid for the survey don't actually care about the answer so much as they do about asking the question.