One of the better known surveys of business confidence is NAB Business Confidence index which is part of their monthly business survey. I have a few gripes about how it is used and reported and an interesting observation about how you should respond if you are ever surveyed.

 

Fluctuations in this index are deemed newsworthy. For instance, when it changed from -30% to -20% the jump was noted with some skepticism as it was against the general trend of gloom and doom. The jump was followed by a “reversal” back to -32% in January.

I was not able to find out how the index is calculated on the NAB website, though I could no doubt find out with enough digging. I assume it is a difference between the proportion of bulls and bears, a negative number indicating that pessimists outnumber optimists. This is similar to supporters minus detractors results in politics and the Net Promoter Score in marketing. It appears that the survey is given to about 350 small to large sized companies in the non-farm business sector.

It is a pity that margin of error is never reported for these surveys, a practice which has happily become standard for most other surveys. With 350 firms the standard error would be around 5%. I could not find if the same companies are surveyed each time or if a new batch are selected each time. This is important in assessing the reliability of changes in the index. But if they are surveying different firms each time then the standard error of the difference is about 7%. So the 11% jump and 12% reversal were not beyond the bounds of chance.

My more general gripe is that not reporting margins of error in survey figures leads to an excess of spurious explanation. I like to present my MBA students with a 13 week chart of surveyed brand recognition and ask them to spend 5 minutes coming up with an explaination of the features. They are given background of the company and specific dates and economic events to compare the chart against. They are very creative and insightful and see subtle patterns that I completely miss. They are also disappointed when I tell them that the data are randomly generated from a stationary process. There is nothing to explain.

So that is my first gripe. Why do the NAB get away with publishing this index without saying how it is calculated or what is the margin of error is? It is just not there in their standard reports. In fact you will find that the word “error” only appears ocne in the legal disclaimer on the final page. Apparently they do not make errors and are not liable for them anyway. I think they should be liable – for not acknowledging unavoidable sampling error.

My second gripe is that I am not sure what business confidence really measures. It seems to be bad science. When I respond to a market survey I am being asked how much I value a product feature, or how much I spent last month. When I respond to a business confidence survey I am being asked to judge how the economy will react to recent developments. I am essentially being asked to predict how other people will feel about the events I am being asked about. And their reactions will be determined by how they think me and my peers will feel. It is like a snake swallowing its own tail.

There is a real signal there somewhere, generated by objective economic data. These are surely worth publishing. But the business confidence index is measuring something that is so full of feedback that I wonder if it means anything at all. It is rational for me to become a bear if when see the NAB index fall because I know the other players will be reasoning the same.

Following this line of thought beings me to an interesting conclusion. Business confidence surveys are different beasts to market surveys. They seek to inform the same people they measure – to inform the market about everyone else’s beliefs. However, when business confidence is low there is no public good in publicising the fact. In fact, it is positively damaging to the economy. So do I expect NAB to suppress the results? Should the NAB index be banned in bad times?

Of course not. But it does seem to me that it is rational for respondents of business surveys to completely lie on the survey. They should respond that they are supremely confident about the future. It cannot hurt them to boost the confidence of their customers and competitors alike. And saying you think the economy is just about to turn the corner does not prevent you from moving all your money into cash when nobody is looking.

And this leads to a final nasty thought. If people are already doing this then the true level of business confidence is even lower than the headline figure of -32%.