The Guardian reports Professor Dylan Williams as saying that exam results are unreliable:
“People who manage and produce tests have a responsibility to be honest about the margins of error and report them. By pretending exam results are completely reliable, we have encouraged people to rely more on them.”
This is not really news to anyone in education, but may shock students and parents who, I’m sure, would like to think that we always get it right. And we should be striving to improve the system, because, well, it should be as fair as we can possibly make it.
But wait a minute …
“By pretending exam results are completely reliable …”
Who’s pretending? The Institute of Education? Schools?
Well, maybe. No-one likes the appearance of being unfair. But I think there’s another party here, too: the media.
The message of this news story is that the system is not perfect. But of course, no real, living, breathing system will ever be perfect! There will always be exceptions, and in any system involving measurement, there will be a margin of error — except that this is hardly ever reported in the UK media. Here, on a good day, you get sampling information.
It’s a different story in North America: there, it’s routine to find statistical data, such as polls measuring political approval ratings or voting intentions, accompanied by information — often quite detailed — about the margin of error. And they take it pretty seriously, too.
I’m not saying this is all the media’s fault – we could do way more to ensure statistical literacy while people are still in school. But maybe reporters should try including information about margin of error anyway. I’m thinking that even a vague awareness among the general public that there is some uncertainty about the results of any statistical exercise would be better than unthinking acceptance of whatever numbers emerge. What’s the worst that can happen?
(PS – I’m reminded that polls are the worst way of measuring public opinion and public behaviour — except for all the others ;o)