Asking the wrong questions

[tigerfort]

Why is it that so many surveys appear to be designed to not collect information? I don't even mean the ones that are obviously designed to lead to a specific set of answers so that the news business/political party/think-tank/NGO/whatever who commissioned it can say "see, everyone agrees with us". I mean the ones where the questions don't even make any sense.

For example, I'm on the local (county-level) "Residents' panel", which means I get a questionnaire every three or four months asking about whatever the council (and other bodies, like the NHS trusts) happens to feel they should be showing an interest in. Often, the questions in these are sensibly constructed ("We've been told that our budget will be 10% lower next year; which two of the following non-essential council services would you protect from cuts and why?" Not a nice question to have to answer, but a reasonable and solidly constructed one.)

The latest one, however, includes the following two marvels:

If you were to use sexual health services which of these would be most important to you? (Please rate your preference, where 1 is your preferred option and 3 is your least preferred option)

• Location - where the service is
  
• Access - how easy it is to get there
  
• Services provided - what the service is able to help you with
  

So you want me to choose whether it's better for the clinic to be useless to someone because it doesn't provide the services they need or because they can't get to it? If not, what are you trying to ask?

Which is followed by:

If someone had concerns about their sexually-related health, which one of these do you think they would do first?
[With a long list of possible answers, ranging from 'do nothing' through 'look at a website' to 'visit their GP' and 'go to a private specialist clinic'.]

The bold is theirs; the italics are mine. The question asks me to speculate about the likely actions of a non-specific 'someone', and the order in which they would take them. I can't see that totally uninformed guesses of this sort are of any value whatsoever as a basis for their decision-making. If the question asked what I would do first, it would be marginally more useful - the answer is no longer a guess, but the sample is probably non-representative (consisting as it does exclusively of people who have signed up to fill in random surveys from the county council. Yes, they attempt to match the demographics of the region age- and gender- wise, but a lot of groups are inherently excluded because they don't want (or aren't easily able) to deal with questionnaires). Asking what I think 'someone' should do is a bit better too, since that's presumably the advice I would give them if they asked... (and is likely to be the same as what I would do in their position).

Perhaps more importantly, though, the only correct answer is 'it depends'. Presumably they mean non-emergencies (go to A&E isn't an option), but even so... and of course the first thing is probably not the most important thing anyway, since I can't actually see a medic the instant I notice something's wrong.

Comments

[naath.livejournal.com]

A narrow-range of services might suit most people most of the time, a hard-to-get-to location is not necessarily impossible to get to. And I think the question is a legitimate one - would you prefer that they limit the services available (presumably forcing you to go a lot further afield to get other services, or wait longer or something), or that they centralise the services to a perhaps less-convenient location?

[kargicq.livejournal.com]

Yeah, I agree, sorry. I thought it was fairly clear what they were trying to get at - clinics at lots of small sites which don't offer all possible services, or one big clinic in the city centre.

Re the second, I guess they are trying to get at "what would you do" without offending people who go "Hmph! I would never have an STD, what are you implying?!"

Neuromancer

[tigerfort.livejournal.com]

On the assumption (for the second) that most people will assume "everyone else is like me" rather than being all logical about it the way I was? Possibly, and I suppose it does avoid the problem of using 'should' (which is certainly not ideal). But it still invites problems, I think.

See also my answer to Naath.

[tigerfort.livejournal.com]

I should possibly have added a paragraph about this, but I was busy ranting :)

The answer here is "yes, but..."; the question doesn't provide enough information to express a preference on that front. If it said "We have enough money to either keep all the existing [number of] facilities open, but only provide [list of services] at them, or else to provide [longer list of services] at [a smaller number of] facilities. Which option would be better?" that would be fine. But the question as asked is on the same level as "which is preferable, chocolate or comedy?"

[monkeyhands.livejournal.com]

I used to get very cross at badly-written surveys until I got the job of editing/rewriting a survey for a client. I realised how chuffing hard it is to get it right; although I made it a lot clearer and cut the damn thing down from 6 pages to 2, it still wasn't much cop.

I think creating a good survey is an art, and it requires training, but it's mostly done by people who don't realise this.

[tigerfort.livejournal.com]

Oh, absolutely (to all of that). Surveys are a difficult art to get right, especially if you actually want to get people's thoughts without imposing a filter of your own biases. I think partly those two questions rankled so much because the surveys in question are actually mostly pretty good examples and the contrast was... notable. (In particular, given that policies are allegedly at least amended on the basis of things that come up in the surveys, the presence of questions on a potentially contentious issue so badly phrased that any answer could be interpreted as supporting any policy rang instinctive warning bells, quite possibly wrongly.)

[monkeyhands.livejournal.com]

I think you were right to have warning bells ringing! I was sort of rambling a bit about my own experience, but I agree with absolutely everything you said in the post. And TBH I *still* get annoyed at badly-written surveys!