Being clear about uncertainty

The Guardian reports Professor Dylan Williams as saying that exam results are unreliable:

“People who manage and produce tests have a responsibility to be honest about the margins of error and report them. By pretending exam results are completely reliable, we have encouraged people to rely more on them.”

This is not really news to anyone in education, but may shock students and parents who, I’m sure, would like to think that we always get it right. And we should be striving to improve the system, because, well, it should be as fair as we can possibly make it.

But wait a minute …

“By pretending exam results are completely reliable …”

Who’s pretending? The Institute of Education? Schools?

Well, maybe. No-one likes the appearance of being unfair. But I think there’s another party here, too: the media.

The message of this news story is that the system is not perfect. But of course, no real, living, breathing system will ever be perfect! There will always be exceptions, and in any system involving measurement, there will be a margin of error — except that this is hardly ever reported in the UK media. Here, on a good day, you get sampling information.

It’s a different story in North America: there, it’s routine to find statistical data, such as polls measuring political approval ratings or voting intentions, accompanied by information — often quite detailed — about the margin of error. And they take it pretty seriously, too.

When I read these stories in the New York Times or Globe And Mail it makes me feel like we’re statistically illiterate in this country.

I’m not saying this is all the media’s fault – we could do way more to ensure statistical literacy while people are still in school. But maybe reporters should try including information about margin of error anyway. I’m thinking that even a vague awareness among the general public that there is some uncertainty about the results of any statistical exercise would be better than unthinking acceptance of whatever numbers emerge. What’s the worst that can happen?

(PS – I’m reminded that polls are the worst way of measuring public opinion and public behaviour — except for all the others ;o)

Leave a comment

Filed under other people's stuff

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s