WHOAful survey?

In the last week, many Placitas residents received a phone call asking their opinion about Placitas free-roaming horses. The survey is sponsored by the local Wild Horse Observers Association (WHOA) and is being fielded by Southwest Planning, a New Mexico marketing firm that conducts public opinion polls.

In a call last week to Southwest Planning, we learned the survey sample draws from 900 Placitas households with telephone landlines. The firm hopes to complete at least 200 surveys, and the results will be statistically weighted to reflect the demographics of the Placitas population. The pollster will calculate both response rates and confidence intervals for each survey item. The poll started on Tuesday, August 19 and was scheduled to end on Sunday, August 24. However, as of August 25, it was still being fielded.

Unfortunately, any survey is only as valid as its design. To be accurate, a survey has to have what researchers call content or face validity. That means survey items must reasonably address all the relevant issues – not just a cherry-picked few – and provide an appropriate range of response options. In addition, a valid survey’s questions are worded in an unbiased, neutral manner and don’t “push” the respondent towards a particular response.phone

Uh, could you repeat the question?

We understand the polling firm has gotten an earful from Placitans upset or confused about the survey questions. The short poll asks the respondent if they live in Placitas, how long they have lived here, their age, and how familiar they are with the free-roaming horse issue. So far so good. Another question asks if they prefer the use of contraceptives and cattle guards or round-up and removal as a means to manage the horses (of course, the survey doesn’t let on that contraceptives and cattle guards are inadequate methods of management).

However, the most interesting item is a long, convoluted question that conflates development of a loop road/highway with free-roaming horses. The gist being, do you support a loop road that would bring 20,000 cars a day, more real estate development, more gravel mining, and other negative consequences, and which, by the way, would also impact free-roaming horses? It’s a long question, and by the end your head is spinning.

If you answer “Yes”, you are indicating you support a loop road and its various potential adverse impacts – development, traffic, mining, etc. It would be surprising if even one person responds “Yes”. The vast majority of respondents will answer “No,” they don’t support a loop road. And by slipping in the mention of free-roaming horses at the end of the question, WHOA undoubtedly hopes to be able to claim that nearly 100% of Placitas residents support keeping free-roaming horses on the land, when in fact, the question is about a loop road, not horses.

Any researcher who designs surveys can tell you that because of poor design, key components of the survey aren’t valid and won’t be able to accurately represent public opinion. We’re puzzled that WHOA apparently thinks no one would notice this.