By Rodney Dickens
To properly review the accuracy of the consensus economic forecasts we need to look at what the economic forecasters were predicting versus what happen over the forecast period.
NZIER gets predictions from the economic forecasters for the current and subsequent March years.
We don’t believe economic forecasters can provide much insight beyond the year ahead, so our review focuses on what the economic forecasters were predicting roughly a year ago and what transpired.
To be useful from a business risk management perspective the economic forecasters need to be able to give advance warnings of upturn and downturns over the subsequent year or so.
The black/thick line in the chart shows rolling annual average GDP growth based on the quarterly GDP data (i.e. the percentage growth in the volume of economic activity or GDP for the last four quarters compared to the same four quarters a year earlier).
Last month Stats NZ reported that annual GDP growth for the 2010/11 March year was 1.5%.
When NZIER surveyed the economic forecasters a little over a year ago in June 2010 the average prediction for GDP growth for the 2010/11 March quarter was 3.2%. The lowest prediction by any of the economic forecasters was 2.5% and the highest prediction for GDP growth in the 2010/11 March year was 3.9%.
The economic forecasters get some marks for getting the direction right (i.e. that there would be positive rather than negative economic growth), but they were on average significantly too positive about economic growth (i.e. the outcome was slightly under half of what they were predicting and was outside the range of what all the forecasters were predicting).
The major earthquakes in Canterbury explain part of the deviation and economic forecasters can’t be expected to predict earthquakes, but their overly optimistic predictions were more because they were overly optimistic about underlying economic growth and especially about residential building prospects, as discussed below. Also see below for the quantified scoring.
Reviewing the Consensus Forecasts for residential building activity
The black/thick line in the chart shows the rolling annual level of residential building activity, which includes new buildings and alterations and additions to old buildings. It is measured in dollar terms, but the level of activity is measured in constant price terms (i.e. based on prices in the 1995/96 March year), which means the chart effectively shows the volume of residential building activity.
On average the economic forecasters were in June 2010 predicting $5.8b of residential building activity in the 2010/11 March year (in 1995/96 price terms) when the outcome was 14% lower at $5b.
The most optimistic ofthe forecasters was predicting $6.2b and the most pessimistic was predicting $5.3b. In this instance the economic forecasters didn’t even get the direction right so none of them were much use in providing useful insights for firms and individuals impacted by cycles in residential building (see below for the quantified scoring).
The underperformance of residential building activity compared to what the economic forecasters were predicting had little to do with the negative impact of the Canterbury earthquakes, with it being mainly due to the economic forecasters not understanding what was going on in the housing market and especially not realising the major negative impact of the property tax changes announced in the May 2010 Budget.
The major relevance of the property tax changes to residential building activity and overall economic growth went almost unnoticed by the economic forecasters.
Being overly optimistic about the outlook for residential building played a significant part in the economic forecasters being too optimistic about economic growth prospects. The pivotal role housing market activity and house prices play in economic cycles was covered in Chapter Two of the How the Economy Works booklet (see http://sra.co.nz/pdf/PivotalHousing.pdf).
If the economic forecasters can’t get the direction of residential building activity right they have little chance of providing advance warnings of economic upturns and downturns or of predicting the outlook for interest rates (see below).
By contrast, we provided clients with advanced warnings of the downturn in housing market activity in 2010 and we correctly warned that economic growth would be significantly weaker than predicted by the RBNZ and that interest rates would not increase anywhere near as much as the economic forecasters were predicting, although economic growth turned out even weaker than we warned (see http://sra.co.nz/pdf/EconomicTrackRecord.pdf for a review of our forecasting track record in 2010).
Reviewing the Consensus Forecasts for interest rates
NZIER surveys the economic forecasters’ predictions for the 90-day bank bill yield and the 10-year government bond yield. The 90-day bank bill yield is the benchmark short-term wholesale interest rate that both reflects the market’s assessment of the near-term outlook for the OCR and plays a major part in driving the short-term interest rates faced by borrowers (e.g. the floating and short-term fixed mortgage interest rates and the base business lending rates set by banks).
The 90-day bank bill yield is the only interest rate the RBNZ forecasts, with these predictions being proxies for OCR forecasts.
The 10-year bond yield is of more relevance as a benchmark of the longer interest rates available to investors and as the main measure of the 'risk free' interest rate used in assessing the value of shares and the like.
In general the outlook for the 90-day bank bill yield is of relevance to more people, while it has implications for the outlook for the 10-year bond yield and for medium-term and longer-term interest rates more generally. From one perspective, the market’s expectation of the OCR over the next 10 years determines the current 10-year bond yield. From another perspective, the 10-year bond yield changes in response to changes in the market’s expectation about the OCR and to changes in international and especially US bond yields. But the two perspectives should add up to the same outcome for the 10-year bond yield.
The black/thick line in the chart shows the rolling annual average of the 90- day bank bill yield. The 90-day bank bill yield has largely drifted sideways over the last year at just under 3%. The average prediction of the economic forecasters survey by NZIER in June 2010 was that it would average 3.7% in the 2010/11 March year and 5.3% in the 2011/12 March year (see the blue line).
If I interpolate between these two predictions it implies that the forecasters were on average predicting that the rolling annual average of the 90-day bank bill would be around 4.4% now compared to an outcome of 2.9%. The minimum and maximum forecasts lines in the chart show that all of the forecasters surveyed by NZIER in June 2010 were predicting significant increases in the 90-day bank bill yield over the subsequent two years.
In terms of their 90-day bank bill yield predictions the economic forecasters should get low marks in this instance (see below for the quantified scoring).
The 90-day bank bill yield predictions are roughly consistent with the predictions that economic growth would be robust, which meant an element of internal consistency between the Consensus Forecasts for economic growth and the 90-day bank bill yield. But if the 90-day bank bill yield increased as much as the economic forecasters were predicting it would result in sizeable increases in the cheapest available mortgage interest rates (i.e. the floating and short-term fixed rates), which would in turn result in falling rather than surging residential building activity.
Every past cyclical increase in mortgage interest rates has resulted in a cyclical fall in residential building activity 2-3 quarters later.
What we learn from reviewing the Consensus Forecasts is that the economic forecasters in at least this instance produced forecasts that are internally inconsistent, which is supposed to be a no-no for economic forecasters.
Dr Rod Deane, best known now for his role at Telecom, was the Chief Economist at the RBNZ when I was recruited in 1980. He was the founder of mathematical, economic forecasting models in New Zealand. He taught us novice forecasters that a key role of an economic forecasting model is to ensure internal consistency of the forecasts. Unfortunately, the modern economic forecasters seem to have forgotten this cardinal rule, which will greatly inhibit their ability to provide useful insights.
The black/thick line in the adjacent chart shows the rolling annual average of the 10-year government bond yield. The 10-year bond yield has fallen significantly over the last year, which partly reflects the market adjusting downwards expectations for the OCR and partly reflects falling US bond yields.
Being overly hawkish about the outlook for the 90-day bank bill yield will have played a significant part in the economic forecasters on average predicting a rising 10-year bond yield in June 2010, just in time for it to fall.
Again, the Consensus Forecasts didn’t get the direction right, although one of the forecasters surveyed in June 2010 got very close to accurately predicting the annual average level of the 10-year bond yield in the 2010/11 March year (see the red line following the black line up to March 2011, but thereafter deviating).
Reviewing the Consensus Forecasts for the exchange rate
NZIER survey what the economic forecasters are predicting for the trade-weighted value of the NZD (i.e. the trade-weighted index or TWI). This measure of the exchange rate is relevant to economists and the RBNZ in assessing the impact of the NZD on the economy.
The NZD TWI is based on the value of the NZD against the USD (0.3023), the Euro (0.2797), the AUD (0.2126), the Yen (0.1425) and the Pound Sterling (0.0629). It is 50:50 weighted according to:
· each currency area's share of NZ's merchandise trade (exports plus imports), normalised to total 100 percent; and
· each currency area's share of the combined nominal GDP of the five currency areas.
The weights (in parenthesis above), are updated annually, most recently on 16 December 2010.
In client reports the economic forecasters focus more on the cross rates that are relevant to exporters and importers (e.g. NZD/USD, NZD/AUD etc), but reflecting NZIER’s focus on the economy it surveys the TWI forecasts rather than NZD/USD forecasts.
The black/thick line in the adjacent chart shows the rolling annual average of the NZD TWI. Up until around March this year the TWI increased roughly in line with the June 2010 Consensus Forecasts, although it has since headed a bit higher than the economic forecasters were on average predicting in June 2010.
On this front they appear to deserve some praise. But having reviewed the Consensus Forecasts for the TWI over a number of years I can reveal that the coincidence of the predictions with the outcome was more an accident than good judgement.
The Consensus Forecasts generally predict that the TWI will revert to around the historical average level, so in the rare event this happens the Consensus Forecasts will be reasonably accurate.
But it is more often the case that the TWI ends up spiking significantly above or below the average level.
The other observation about the TWI predictions is that the economic forecasters have almost covered all bases (i.e. there is a wide spread between the maximum and minimum predictions). This means that one of them has a reasonable chance of getting it right, although you are unlikely to know which one in advance, while it is important to note that the forecaster predicting the highest/lowest TWI in one March year may not be the same forecaster who is predicting the highest/lowest TWI in the subsequent March year.
Scoring the accuracy of the Consensus Forecasts
We have adopted a 0-10 scoring system for quantifying the accuracy/usefulness of the Consensus Forecasts. The score is 10 if the Consensus Forecasts were close to the mark, meaning they would have been of great value to end-users (i.e. to business managers and individuals impacted by the economic indicator in question).
A score of zero is awarded if the Consensus Forecasts were so far from the mark that they would have been of little or any use to end-users.
Scores between zero and 10 are awarded based on how far the outcomes were from the Consensus Forecasts.
The scores for the June 2010 Consensus Forecasts are shown in the adjacent chart. Based on the forecasting scoring table we are using, the Consensus Forecasts score 4/10 for economic growth, residential building activity and the 90-day bank bill yield (i.e. well below pass GO and pick up $200 scores). The Consensus Forecasts score 5/10 for the 10-year government bond yield and a respectable 8/10 for the NZD TWI, but this outcome might be more by chance than by good judgement, as discussed above.
Based on the June 2010 Consensus Forecasts the forecasters would be kept back a year.
Some general observations about the forecasting track record
In general the Consensus Forecasts predict that most economic indicators will revert to around average or trend levels. In the 2010 June survey NZIER found that the economic forecasters were on average predicting just over 3% GDP growth for both the 2010/11 and 2012/13 March years.
When it comes to predicting economic growth the forecasters have a strong predisposition to predict around 3% growth, which reflects a somewhat optimistic assessment of the historical average rate of annual GDP growth.
If you look at the Consensus Forecasts for residential building activity, the 90-day bank bill yield and the 10- year bond yield (the respective blue lines in the charts above) you will find that they also tend to 'mean revert'.
This means that on average over the long-term the economic forecasters will be reasonably close to the mark, but in any one year economic growth, residential building activity, interest rates and the exchange rate are more likely to end up some distance from the historical averages. This implies that the economic forecasters are unlikely to provide many useful insights for business risk management purposes.
However, this is jumping the gun based on my knowledge of the past track record of the economic forecasters. If my observations are correct (i.e. the economic forecasters often get the direction of a number of key economic indicators wrong and have a tendency to cling to historical averages) it will be revealed as we review the track record of the economic forecasters in the future.
How the accuracy of the Consensus Forecasts are scored
We are using a 0-10 scoring system for quantifying the accuracy/usefulness of the Consensus Forecasts.
The score is 10 if the deviation between the outcome and the Consensus Forecasts is equal to or less than plus or minus 0.25% for GDP growth, +/- $0.25b for residential building activity, +/- 0.25% for the 90-day bank bill yield, +/- 0.25% for the 10-year government bond yield and +/- 1.5 for the NZD TWI.
If the outcomes deviate from the Consensus Forecasts by more than the following amounts we assess that the forecasts are of little or no help and a score of zero awarded: +/- 3% deviation for economic growth; +/- $1.25b for residential building activity; +/- 2.5% for the 90-day bank bill yield; +/- 1.5% for the 10-year bond
yield; and +/- 10 for the NZD TWI.
The numbers used for each of the five economic indictors reflect how much each indicator has varied in the past, which has implications for how close the Consensus Forecasts need to be to the outcomes to be of value.
The table below shows how the score falls from 10 to 0 based on increasing deviations between the outcomes and the Consensus Forecasts (e.g. if the difference between the outcome for economic growth and the Consensus Forecast is greater than +/- 0.25% but less than +/- 0.525% the score is 9; if the deviation is greater than +/- 0.525% but less than +/- 0.8% the score is 8, etc).
*Rodney Dickens is the managing director and chief research officer of Strategic Risk Analysis Limited.