Are surveys worth any more than the paper they're written in or who commissions them?

Yesterday Populus published a survey in which it 'proved' that 61% of the population thinks that schools should be free to set the pay of individual teachers based on performance. 

I bet Mr Gove was relieved to hear that since this new rule - where pay rates are set locally, not nationally and are based on performance rather than 'time served' as is currently the case - comes into force in September.

In response the NUT quoted its own survey, undertaken by YouGov in March of this year, which 'proved' that only 25% of parents believe that schools should set their own pay system; that only 19% of parents think academies and free schools are taking education in the right direction and that, three years in, only 8% of parents think that this coalition Government is having a positive impact on education.

Now I don't now about you, but I'm finding something a bit jarring about these two bits of 'proof'.

The NUT survey polled 2,008 people, the Populus survey 1,723 both were covering essentially the same subject and both have been: 'weighted and are representative of all parents (aged 18 and over) in England by region and family type.' (NUT) and:  'the results have been weighted to be representative of all British adults.' (Populus).

I have asked Populus and the NUT for a copy of the questions (in order) that were asked. The NUT ones can (sort of) be seen in the Word attachment to the press release which can be found here  (incidentally thank you to the NUT for providing me with this press release and document in response to my tweet of yesterday). The Populus announcement - you can drill down to data which again sort of provides the questions- can be found here.

Firstly this 'weighting' that both Populus and YouGov talk about. What does that mean? Presumably if you control the weighting you can control the results/conclusion of the whole survey? If so, that makes the thing pretty meaningless as a piece of 'news'. But then who am I kidding, if you want to prove anything by statistics and survey, obviously you can: 

Before we go any further, do yourself a favour and have a quick (2 mins) watch of this. It's brilliant and will save me some time in explanation.

OK so I think most of us realise that this is how surveys work: They're not really there to expand the wealth of human knowledge or to find out the facts or the truth, but are there to help one side or the other to prove or promote its argument. And that you can effectively prove anything you like with a survey.

What amazes me is that news outlets continue to run stories on these things, especially (of course) if the results support their own political stance, and that they actually work in PR terms - they must do otherwise they wouldn't be done so frequently and the survey companies wouldn't be earning so much money.

It's a bit like the old argument that violence on television has no effect whatsoever on people's behaviour (I'm not Mary Whitehouse), whereas the £billions spent on advertising on TV every year does. Erm, just me?

Anyway it (the survey) is a well-known PR trick: The creation of a story that you can control, which purports to have a level of 'credibility' (even if us cynics know that to be untrue, there are obviously enough blythely trusting souls out there for it to be worth the effort) and they also have the added value for the publication of 'proving' what its readers want to be 'proved'. It's win win then.

Except it's bullshit.

I'm not immune, I don't think any of us are: Oh joyous day when that scientific survey found proof that drinking a glass of red wine a day was actually good for you. A glass of red a day would make you live longer. Think what two glasses might achieve? Some of us might live forever.

Presumably this wonderful 'scientific' news only related to Red wine because it was funded by the Shiraz Society and the Sauvignon Blanc Supporters Club refused to pay up? (obviously made up names). This huff Post thing (left) is amusing. It's always had the opposite effect on me to be honest. ;)

From an objective standpoint, looking at these surveys, presented as credible and highly scientific, can be quite funny - as the old saying goes, follow the money. And I love the way that a survey of ten people, seven of whom are, either because of the way the question has been framed or the narrowness of the group itself, in favour of, say,  Scottish independence, can then be extrapolated to the headline '70% of Scots want to leave the union.' It's just silly, but also quite dangerous in the wrong hands. 

It seems to me that a large part of the population is being duped, in this way, into believing shit that just isn't true. Opinions are being formed on the basis of falsehoods. Yes we all know that media outlets are biased in one way or another: one tends to believe as fact those items which support our view of the world and dismiss those that don't.

But where do you go to in order to find the truth these days?

Thanks for reading.