In a recent survey of my household, 100% of people and 50% of cats agreed they love Travel Rants too.

This is a pet peeve of mine too, because not only are statistics so easily manipulated anyway, but when the surveys are carried out by people with a clearly vested interest … makes my blood boil!

(PS 50% of the cat population was asleep at the time of the survey)

Quote Amanda “This is a pet peeve of mine too, because not only are statistics so easily manipulated anyway, but when the surveys are carried out by people with a clearly vested interest … makes my blood boil! ”

The Same could be said of Price Comparison sites as well Darren!!

Sorry to ride to the defense of market research here but…a sample size of 1001 gives you a confidence interval of about 3% at a 95% confidence level in a population of 56 million.

In other words, the sample size is not at fault. But without knowing the other options or the wording of the survey, it’s difficult to tell how leading (or otherwise) the question was.

Hi Darren - I’m not unsympathetic to your take on this, but want to clarify a few things, since we reported this on Travel Weekly and I invited comments on the blog:

- The survey said 30% of consumers find holidays more expensive than they expected online, not that holidays are more expensive online than off.

- You are right to be sceptical about surveys, but wrong about sample sizes. A sample of 1,000 can be statistically meaningful if the poll is conducted properly - Gallup uses samples of around that size to represent the US electorate, which is far bigger than the UK holiday market. The important questions are how a sample is selected and what questions it is asked.

- @Amanda: the survey was commissioned by Advantage, not carried out by it.

- Finally, the price finding doesn’t strike me as that unusual - on several occasions I’ve gone online expecting a good deal, and not found one even after exhaustive searching on both big travel retailers and price comparison sites.

Also, a recent poll of me found that 100% of people wish it wasn’t Monday and want a cup of tea.

Darren, I’m all for making research more transparent but think that it’s probably unrealistic to have full supporting methodology and demographic breakdowns contained in the press release.

In this context, it’s worth looking at the MRS guidelines (rules B55 onwards). Put simply, decent market research companies are obliged to ensure that their clients accurately report the findings in press releases.

I think its right to be skeptical and ask hard questions but not to be automatically cynical .

Interesting, though - Stephen is clearly right that a full methodology is unrealistic, but as Darren says he is the target reader, and between them the researcher/PR company/media outlet have produced something he doesn’t trust.

So something is wrong. Laypeople don’t have the MRS guidelines in mind, and I do think there’s an argument for PRs always including some very top-line stuff (even if it’s just to assert that the sample was random) in ‘notes to editors’ - which also means journalists have to pick up on it and put it somewhere in the story.

If Advantage had just put a bullet point stating ‘random sample of 1,000 people from across the UK’ at the end of the release we wouldn’t be having this discussion!

Lively stuff as always Darren

Darren, I don’t think we’re disagreeing - just a difference of emphasis! The view of the MRS seems to be that if research is made public in any way, then the public should be able to catch sight of the data behind it, even if this is via a link to data rather than appearing on a press release.

Before I sign off on this one, I’ll just add a little insight about where the respondents for a survey like this come from based on my own experience.

I’m going to guess that it was an online survey fed to a panel of volunteers. The reason I say it is an online survey is that its cheaper to do it that way (but with some exceptions that I’ll come back to) and the reason that I think it has been served to a panel is the number of responses (it’s near enough to a nice round number to suggest that the survey was closed down automatically once a quota had been reached - phone surveys are little more hit and miss when it comes to final numbers).

I would say that there are indeed issues of under and over representation of certain demographics in online panels - but probably not as much as you would think and any divergence between the ‘internet population’ and the ‘real population’ will decrease over time.

The other thing to add about online surveys using panels is that the panels can still be VERY expensive and picking your customers to get the result you want really only adds a lot of cost onto the process. Basically, for each additional filter you put on your sample, the cost goes up. For some researchers this is good because it enables them to research a very targeted market (eg homeowning women in SE England under 30) but for others it would be an unnecessary extra cost. This isn’t to say that there are not shysters out there with dodgy data – but in my opinion it’s an expensive form of lying if that is your game!

did you know 72% of statistics are invented?

I’m always very sceptical about these things.

How do we know they haven’t taken a sample of 2000 people and picked out 1000 of them, which helps to support their views/business/expectations?

Similarly, there may be a number of questions/answers which they haven’t publicised which do not support them, so they don’t get mentioned.

It’s all swings and roundabouts.

If surveys are too long, people get bored and start to fill out any old answer just to finish, if they’re too short you may not get a full collection of data, if the questions are phrased incorrectly you will end up with complete incorrect data.

If you offer an incentive to complete a survey, which a lot of people do these days - chances are (or at least most of my friends do this) it will be completed without a care in the world for the answers - they just want their chance to win that amazing holiday or luxury spa treatment

“How do we know they haven’t taken a sample of 2000 people and picked out 1000 of them, which helps to support their views/business/expectations?”

Sorry to be boring and repetitive but this goes back to the good old MRS guidelines. A decent market research company should either be an MRS member or follow MRS guidelines. If the market research company’s client lies or distorts the research findings (and excluding the ‘bad’ responses from a question is distorting the research) then the market research company stands to be fined, named and shamed for letting this happen. Therefore, we have an incentive not to allow bad practices like the one you describe above from happening.

Darren mentioned yesterday that he hadn’t heard of the MRS and there is no reason why he should have done. But for those of us in this business, they are a pretty good incentive to prevent your client cheating and lying. Put simply, a dodgy press release can destroy your livelihood.

By all means be skeptical but don’t assume that we all have the morals of a politician!

I had a similar run-in with a PR agency recently. It’s not just about the stats (though I agree with almost all the points Darren raises - sample sizes might be OK, but how are the questions phrased?) It’s also about the way the stats are manipulated to tell a particular story. (One that usually favours the client.) In my recent spat it was the claim that it was ‘quite common’ to collide with a kangaroo when driving in New South Wales in Australia.

The stat quoted was that of the 9000 collisions involving animals in NSW last year, 7000 were with kangaroos. This number is meaningless if you don’t know the total number of collisions - which wasn’t provided. All this shows is that IF you hit an animal, it’s highly likely it will be a kangaroo. But the PR was trying to come up with a witty headline… and presentation is everything these days.

I got two responses in the comments on my blog from the agency, neither of them addressing my issue… just side stepping it and waffling on about what a great job they were doing for their client. It’s about time PR agencies woke up and realised that the people they pump their remorseless, poorly researched press releases at read them and make judgements based on them… about them, their client and the industry they are part of.

http://www.travelblather.com/2008/06/writing-a-press-release-quote-your-stats-in-context-please.html

Brainwashing the consumer… and also attempting to brainwash the journalist (who writes for the consumer). PR is quietly trying to erode journalistic integrity… In a world where journos are paid less and less for their work the temptation to take a press release at face value and publish its content without proper investigation and verification is increasingly tempting…
Yes… great debate!