Statistics are great unless they measure the wrong things: Don Pittis

If prices are rising by about two per cent, as inflation data is likely to show this week, why did one of my newspaper subscriptions just go up by 17 per cent?

And if wages are rising at about four per cent, as recent jobs data has shown, why are some provincial governments insisting that wage increases be held below one per cent?

As house prices go through the roof, the fact that the price of the biggest purchase Canadians make in their lives is not included in our inflation statistics makes it easy to see why many young people have expressed doubts about the accuracy of those figures.

It is a struggle that Statistics Canada faces every day as it tries to sketch out with numbers an authentic picture of the reality Canadians experience. But Oxford fellow and bestselling author of Age of Discovery Chris Kutarna says the task is far more complicated than many statisticians like to admit.

Kutarna worries that Statistics Canada's plan to plunge into the ocean of "Big Data" so beloved of retailers and credit card companies — described last week by chief statistician Anil Arora — will inevitably create bias in the results simply because we are measuring the wrong things. 

"One of the terrifying and most fundamental sources of risk is that we only consider what we're now measuring as real," said Kutarna, on the phone from London, England.

For example, long-standing data sets built on debt, spending, prices and gross domestic product simply close the door on values such as family, respect, happiness and species extinction.

"There is far more that is real and not being measured than there is that is real and we are measuring it," said Kutarna.

One practical example from his book is the failure of modern statistics to measure the value of the online encyclopedia Wikipedia which, despite providing value to billions, adds less to GDP figures than the old Encyclopedia Britannica which reached far fewer people.

In a recent speech, Stephen Poloz, governor of the Bank of Canada, described an economy changing so fast that our statistical models fail to grasp it.

Poloz paraphrased the Solow Paradox, the observation by economist Robert Solow that computers had led to an increase in productivity everywhere but in the statistics. Poloz suggested GDP is being understated by as much as two per cent.

One example he offered was the way so many companies are distributing computer services to the cloud, turning whole computer divisions into a budget line item.

"How does StatCan deal with that?" asked Poloz.

Last week, Arora boasted to a gathering of the Empire Club that Statistics Canada was respected everywhere as a global leader, but he acknowledges it is constantly struggling to keep up with changing technology and the shifting understanding of how the world works.

"Look, that's what statistics is, right? To take what are evolving concepts, nebulous concepts, things that haven't even taken a lot of shape and then quickly try to turn them into numerics," said Arora in an interview.

The statistics chief calls it a "team sport" where governments and individuals need to decide which of the millions and millions of things that could possibly be measured should be addressed by the some 5,000 employees at Statistics Canada. Their job, he says, is to bring scientific rigour to the process, so that the numbers are as accurate as possible.

"This is always going to be a journey," said Arora, adding that finding and incorporating into our figures what have so far been labelled intangibles may be a never-ending task. 

Part of that journey that those employees are now undertaking is the attempt to mine the immense bodies of information embedded in Big Data, those traces of activity we leave behind when we do almost anything on the internet from buying to searching. Not only are they readily available for quick analysis but they reduce the employee hours required in traditional surveys. 

"Alternate sources of data are increasing exponentially and we have the technologies and the mechanisms to convert them to public good with high quality statistics," said Arora in his speech.

When it comes to the inadequacies of GDP, a big part of the problem has less to do with Statistics Canada than how we continue to use familiar indicators that may be out of date.

Arora says the University of Waterloo's Canadian Index of Wellbeing includes 200 indicators — from crime and safety to sustainable growth — most of which come from Statistics Canada data. But it's GDP that gets the attention.

Statisticians are always groping to find the data sets that matter. But even in areas we think we know and understand, statistics are merely an indicator — an estimate, of reality. Things we don't understand are, by definition, even harder to measure.

"We live in this culture where what is real is what we measure," said Kutarna, "That the things we measure are reality."

In which case, a certain amount of healthy skepticism, whether about this week's inflation numbers, about GDP, productivity or the many other financial statistics that are often offered as solid immutable facts, may well be in order.  

Follow Don on Twitter @don_pittis