I generally respect a scientific analysis much more than random hunches. Normally that is on the basis of two things: Proper data collection and proper data analysis. Tonight I got a phone call from a phone survey company and it illustrated the extreme weakness of the data collection for anyone who relies on such surveys. The interviewer asked me all kinds of questions like:
"How would you go about looking for government services if you needed assistance for an aging relative?"
While I tried to answer "I would go to Google and type 'Government Canada Old Age Service'" they didn't seem especially interested in that answer. The interviewer wanted yet more data about which other ways I would look for answers... as if there was anything useful I could do after fully investigating the result of my Google search.
"Why have you not done more to Green your home in the past year?"
Well, it is complicated. First, I live in a condo. I can't just start tearing out the walls to put in more insulation, I have no control over the type of heating used and since I already made real efforts to maximize my recycling and reusing I don't know how I *could* do a lot more. Since the interviewer never asked what sort of home I lived in the information they got is pretty near useless and they don't have the option to enter more random data to make my answers make sense. Some people probably answer "It costs too much" or whatever but I can't see how they make the answers usefully correspond to the actual reasons.
"What is your household yearly income?"
Do you want income as reported to the government counting Wendy's stipend as zero (which it is for tax purposes), or how much money we had coming in? Do we subtract tuition, do we count appreciation of investments or the increase in value of the condo over the year? I gave a random answer but honestly there is no reason to think people are going to respond using the same sorts of rules I did. Never mind the fact that people regularly misreport their income depending on who they are talking to, we don't even know what the correct answer is. I can't even tell if I was lying or not!
"The Canadian government is against the UN resolution making access to water and sanitation a basic human right. Do you agree with this stance?"
How the heck should I know? What exactly does this resolution say? It is clear that Canada tries to make sure all its citizens have access to water and sanitation so there must be something more to this but I can't exactly take an hour off to read up on the complexities of this situation. Does adopting this resolution mean that countries with cruddy infrastructure are committing human rights violations? Does it mean I can sue the government if they don't supply me with proper sanitation for my shack in the wilderness? I have no idea! I guess I answer something in the middle of the spectrum of answers allowed and move on.
The trouble with all these things is that my answers are uninformed (about the UN!), wildly inconsistent with other reports (income reporting), not useful (Greening my condo), or incomplete (finding government services). The interviewer also did not speak English as a first language so I had all kinds of difficulty interpreting options for answers and I probably ended up giving flat out incorrect answers due to challenges with translation. Now this data is going to be used to make government decisions, supply data to environmental organizations and develop models of living conditions based on location. All of that is just going to be incorrect, and that is with a cooperative subject! There are plenty of people who end up just taking the first option for their answer instead of getting the right one and probably a few who give wildly incorrect responses for their own amusement. Any time you see studies being quoted that talk about surveys done interviewing 2,000 people with particular confidence intervals you have to wonder if the data is any good at all; are we really just guessing at these things?
Of course I don't have a better solution. Presumably phone data gives you some interesting information, primarily what sorts of things patient, bored, polite people say when they want to get off the phone. It seems sort of like democracy... pretty wretched from nearly any perspective but everything else is even worse.