Fresh perspectives on unobservable variables: Data decomposition of the Kalman smoother AN 2013/09

Freshperspectivesonunobservable
variables:Datadecompositionofthe
Kalmansmoother
AN2013/09
NicholasSander
December2013
ReserveBankofNewZealandAnalyticalNoteseries
ISSN2230­5505
ReserveBankofNewZealand
POBox2498
Wellington
NEWZEALAND
www.rbnz.govt.nz
TheAnalyticalNoteseriesencompassesarangeoftypesof
backgroundpaperspreparedbyReserveBankstaff.Unlessotherwise
stated,viewsexpressedarethoseoftheauthors,anddonot
necessarilyrepresenttheviewsoftheReserveBank.
Reserve Bank of New Zealand Analytical Note Series
2
Non-technical summary
Macroeconomic analysis, including that undertaken by the Reserve Bank, makes extensive
use of economic concepts for which no observed data exist. The output gap is one well-known
example. The gap must be estimated using various models and indicator variables. Statistical estimates of core inflation, such as the sectoral factor model (Kirker, 2010), are another
example.
In working with estimates of unobserved variables, it can be difficult to articulate the contribution of each piece of observed data to the estimate of an unobservable variable. The ”data
decomposition” tool outlined in this note does exactly that.
The tool is applied to two types of unobserved variables.
The first is the sectoral factor model estimate of core inflation. That model takes 96 component
series of the CPI and uses statistical techniques to identify the co-movement among those
series (separate tradable and non-tradable factors). The data decomposition tool enables us
to identify which component series have had the largest influence, on average through time
or in any particular quarter, on the estimate of core inflation. It turns out that over the 20
year history of the series three components (with a cumulative weight of 1.04 percent of the
CPI) contributed around 30 percent of the variability in core inflation. Tests suggest that the
methodology is robust to plausible measurement errors in these, and other, individual price
series.
The second application is to a small structural model of the New Zealand economy. In modern structural models, variables are assumed to be driven by ”structural shocks” (such as a
”monetary policy shock”) which are also not directly observable. We use the data decomposition technique to illustrate the impact on the estimated shocks of adding an additional quarter
of actual data. We can’t directly observe the behaviour of economic agents, so the data decomposition describes the process macroeconomists go through to infer from new data which
shocks must have occurred in the economy.
Reserve Bank of New Zealand Analytical Note Series
3
1 Introduction
A common method for analyzing the state of the economy is to estimate unobserved component models. These models describe the behaviour of observable data in terms of underlying
economic concepts that are unobservable. Although some of these models impose very strict
structures that give a unique value for an unobservable variable, most impose a lighter structure where many possible values of the unobservable variable are consistent with the observed
data.1 The unobserved components in the models are estimated using filtering techniques
such as Kalman smoothing or particle filtering.2
One key advantage of the filtering techniques is that they can be applied to a wide range of
models without requiring any adjustment to the filtering algorithm. However, the generalized
nature of these filters can obscure how observable data are processed for a specific model. In
cases where these filters are given a single observable variable, the unobservable estimates
are typically determined by decomposing the observable data series by frequency (i.e. cycles
in the data). Because of this, most applications of filtering with one observable variable involve
estimating low-frequency trends and high-frequency cycles in macroeconomic data.3 However,
for models with multiple observable variables, it is not immediately clear how the observable
data is processed to produce estimates of unobservable variables.
Understanding how these filters infer from observable data the likely values of unobservable
variables could yield useful insights into unobservable estimates and model behaviour. This
note develops a tool – the ‘data decomposition’ – of the type discussed in Andrle (2013a and
2013b) that is designed to separate the contributions of the various observable variables to the
Kalman smoother’s estimates of unobservable variables.4 Because of the general nature of the
Kalman smoother algorithm, this decomposition can be applied to any linear model that can be
written in state space form.
1
The HP filter introduced in Hodrick and Prescott (1997) is a commonly used example of a model where the trend is
unique given the data (and smoothing parameter). DSGE models with unobservable marginal costs (such as Smets
and Wouters (2004)) and dynamic factor models as in Cristadoro et al. (2005)) are two examples of models without
unique unobservables.
2
Simon (2006) and Durbin and Koopman (2012) both provide comprehensive coverage of both of these techniques.
3
Some examples are: Beveridge and Nelson (1981),Harvey and Jaeger (1993), King and Rebelo (1993),Cogley and
Nason (1995),Harvey (1985), Clark (1987) and Watson (1986)
4
The concept of a data decomposition was first proposed by Andrle (2013a) and applied in Andrle (2013b) to decompose various output gaps into observable data. While these papers present methods to infer the contributions of
observable data by exploiting the linearity of the model, this article uses the weights derived in Koopman and Harvey
(2003) for the Kalman smoother. Using explicit weights has the computational advantage of requiring the Kalman
smoother be run only once and provides additional information regarding how these weights change throughout the
sample.
Reserve Bank of New Zealand Analytical Note Series
4
Through two examples of policy-relevant models, this note demonstrates that the data decomposition can provide useful insights to policymakers by analyzing unobserved variables and
model-based forecasts.
This article is structured as follows: section 2 discusses state space models and the Kalman
smoothing procedure – an understanding of these techniques is useful in understanding intuitively what the data decomposition does, section 3 applies the data decomposition to the two
unobservable components models mentioned above and section 4 concludes. An algorithm
detailing how to compute the outputs required for the data decomposition is provided in the
appendix.
2 Unobservable Components Models and the Kalman Smoother
Estimating unobservable variables is an important process in macroeconomic modelling. One
of the more common tools used to estimate unobserved variables in these models is the Kalman
smoother. The Kalman smoother has the advantage of being relatively simple to implement
while being applicable to a large class of models. The disadvantage of this flexibility is that the
precise manner in which the Kalman smoother estimates unobservable variables for specific
models can be lost. The data decomposition tool can be used to address this. This section
provides an overview of how the Kalman smoother works (with technical details provided in
Appendix A). Understanding, in non-technical terms, how the Kalman smoother works can
provide some insights into how the data decomposition works and how its output should be
interpreted.
2.1 State Space
Unobserved components models are commonly written in a ‘State Space Form’. This structure
essentially distinguishes between the fundamental concepts that describe the evolution of a
system and the data that are observed by the econometrician or statistician. The mathematical
notation for the state space of a linear and Gaussian5 unobserved components model is given
below:
5
Non-linear models with non Gaussian errors can also be written in state space form, but they are not discussed
here.
Reserve Bank of New Zealand Analytical Note Series
5
Xt = AXt−1 + Rt
(1)
Yt = BXt + et
(2)
t ∼ N (0, Q)
et ∼ N (0, H)
The notation X, Y , and e each refer to groups of variables and the remaining terms A, B, R,
Q and H represent groups of parameters that link the variables together.6
Equation 1 describes the evolution of the system and is often referred to as the transition
equation. It describes how a group of n possibly unobservable variables (denoted by X) evolve
between discrete time periods (from t − 1 to t).7 The variables X are driven by unobserved
inputs . Because the inputs are unobservable and their evolution unmodelled we describe
them as being random in nature and arising from a normal distribution.8 It is common to refer
to these inputs as ‘shocks’ or ‘structural shocks’. The evolution of the variables X over time
is determined by the parameter matrix A. The response of variables X to various structural
shocks is governed by the parameter matrix R.
Equation 2 describes how the true drivers of the system X generate the observable data Y and
is called the measurement equation.9 The state space form allows for there to be measurement
errors e that reflect the imperfect measurement of the data Y . These measurement errors are
also assumed to originate from a normal distribution. The observable data Y is linked to the
6
Xt , Yt , t and et are of size n × 1, m × 1, n × 1 and ne × 1 respectively. In addition, A, B, R, Q and H are of size
n × n, m × n, n × n , n × n and m × m respectively. Note that for the Kalman smoother to estimate properly, n
must be greater than or equal to the number of observable variables m
7
Many models contain structures that impose dependence between variables spanning multiple periods - rather than
just one as indicated in the above equation. These models can be incorporated into the above structure by redefining
the lag of a variable as an additional
in X.
following
variable
Consider
for example the system: dt= ρ1 dt−1
+ρ2 dt−2 .
dt
ρ1 ρ2
dt−1
dt
ρ1 ρ2
This can be rewritten as:
=
. Defining Xt =
and A =
gives the
dt−1
1
0
dt−2
dt−1
1
0
following state space system: Xt = AXt−1 . This can be done for any multiple lag model.
8
The assumption of normality is not necessary to execute the Kalman smoother algorithm, however, the estimates
from the Kalman smoother will not be optimal without Gaussian errors.
9
This structure should not be viewed as forbidding observable data representing concepts of economic relevance. If
there is data available that represents an underlying driver of economic fluctuations such as government spending,
then the state space form can be re-written as follows:
R
t
Xt
A Xt−1
=
+
gt−1
0 . . . 1 g,t
gt
0
Yt
B
Xt
et
=
+
Gt
0 ... 1
gt
0
In this model, G, g and g each represent the same concept: government spending. By including the variable g
government spending represents an unmodelled input for which there is data.
Reserve Bank of New Zealand Analytical Note Series
6
unobservable data X by the parameter matrix B. Each observable data series is allowed to be
related to multiple variables from X, or just one if desired.
The parameter matrices Q and H represent the variance parameters for the stochastic inputs
and the measurement errors e respectively. Stochastic inputs occurring in the same time
period are allowed to be correlated with each other – as are measurement errors – however,
the measurement errors are assumed to be uncorrelated with the stochastic inputs.
As an example of how to write a model in state space form, consider a simplified version of
Harvey (1985) that calculates the output gap:
yt = tt + ct + eyt
tt = tt−1 + tt
ct = 0.6ct−1 + ct
eyt ∼ N (0, σe )
tt ∼ N (0, σt )
ct ∼ N (0, σc )
This is a model where there are three components: GDP (yt ), which is observed with a measurement error eyt ; ‘potential’ or ‘trend’ GDP (tt ), an unobservable variable; and the output gap
(ct ) – the unobservable object of interest. There exist some structural shocks tt and ct which
act as drivers of potential GDP and the output gap; and (as mentioned above) are assumed to
be random normally distributed variables.
In this model movements in potential GDP are defined as causing movements in GDP that
are permanent absent any shock tt . Movements in the output gap by contrast are defined
as causing movements in GDP that eventually decay to 0 even in the absence of any shock
ct . The rate at which the output gap decays to zero is set to 0.6 per quarter for convenience.
In other words, if the output gap were 1% of GDP this quarter, next quarter – assuming no
new shocks – it would be 0.6% of GDP. The measurement error is defined as a movement in
GDP that completely vanishes in the next period (again, absent any subsequent shock). It is
these differences in variable behaviour – rather than any relationship to other variables such as
inflation – that the Kalman smoother will exploit to estimate each unobservable variable.
Reserve Bank of New Zealand Analytical Note Series
7
In state space form this model becomes:
  

  
tt
1 0
tt−1
t
 =

 +  t
ct
0 0.6 ct−1
ct
| {z } | {z } | {z } | {z }
Xt
A
Xt−1
 
h
i tt
yt = 1 1   + eyt
|{z}
|{z}
| {z } ct
Yt
et
| {z }
B
(Transition Equation)
t
(Measurement Equation)
Xt
2.2 The Kalman Smoother
The Kalman smoother is a tool that can estimate the unobservable variables in any linear state
space model. In this way, the Kalman smoother provides economists with estimates of some
of the economic concepts and variables they are most interested in. Because the algorithm for
the Kalman smoother can be applied to any model written in state space form, it is not clear
how the algorithm produces estimates of unobservable variables for a particular model. The
data decomposition is used to understand the output from this estimation procedure in a modelspecific context. However, an understanding of the Kalman smoother procedures (independent
from any model) can be useful in interpreting the output from the data decomposition.
In the model of potential GDP and the output gap in the previous subsection, there are infinitely many combinations of the unobservable GDP trend, the output gap and measurement
errors that could explain the evolution of observable GDP data. Because of this, the Kalman
smoother is not able to estimate potential GDP or the output gap with absolute certainty. Under
the assumption of normally distributed errors (and a correctly specified model), the Kalman
smoother does however, produce the ‘optimal’ estimate of the statistical distribution of these
unobservables given the GDP data supplied. The mean of this (normal) distribution represents
the statistically most likely estimate of potential GDP and the output gap.
To estimate potential GDP and the output gap over the entire sample efficiently, the Kalman
smoother uses all the available data to inform the estimate at each point in time – including
data occurring both before and after that particular point in time (if available).
To illustrate this point, consider using a sample of GDP data from 2000Q1 to 2013Q1 to estimate
the value of potential GDP and the output gap as at 2006Q1. The model structure implies that
a rise in potential GDP for 2005Q4 will likely be followed by a similar rise in potential GDP for
Reserve Bank of New Zealand Analytical Note Series
8
the following period. In addition, a rise in the output gap for 2005Q4 predicts that the output gap
over the near future will (absent any surprises) slowly decline back to 0. Therefore, if 2005Q4
GDP data suggests changes in the estimates of potential GDP or the output gap in that quarter,
this should also suggest changes to estimates of the values of potential GDP and the output
gap for 2006Q1 (and so on).
Similarly, data for periods following 2006Q1 are useful in determining potential GDP and the
output gap in 2006Q1. If GDP were observed to increase in 2006Q2, it is possible that this
increase was due to changes in potential GDP or the output gap in 2006Q1. Data from both
before and after 2006Q1 may hold some insights into the unobservable variables for 2006Q1.
The manner in which these insights are formed depends on the model structure and the data
provided.
The Kalman smoother exploits these insights in its estimation procedure. Unlike other estimation procedures such as ordinary least squares, Kalman smoother estimation proceeds
recursively as shown in figure 1. That is, the Kalman smoother processes the observable GDP
data one period at a time. It moves through the data both from the beginning of the sample to
the end (forward) and then, having processed the entire sample, from the end to the beginning
(backward). At each part of the estimation, the mean values of the output gap and potential
GDP are estimated as well their variances.10 Iterating forward is referred to as filtering and the
addition of iterating backwards after filtering is referred to as smoothing.
Figure 1: Kalman smoother recursions
1. Filtering
Update
Predict
time
t
t+1
t+2
t+3
Update
2. Smoothing
The Kalman filter is often described as having two phases: prediction and updating. These
operations describe the forward recursive estimation of potential GDP and the output gap at
each point in the available sample. In the prediction phase, the previous estimates of the mean
10
Because the estimates of the unobservable variables Xt are normally distributed, knowledge of the mean and
variance of the unobservable estimates is sufficient to characterize the entire distribution of the estimates.
Reserve Bank of New Zealand Analytical Note Series
9
and variance of the output gap and potential GDP are used to forecast the current mean and
variance of these unobservable variables. In constructing these forecasts, the unmodelled inputs into the system (tt and ct ) are assumed to be at their mean value of 0. Using the model
structure specified in the Transition Equation above, the prediction for potential GDP will be the
same as the previous period and the output gap prediction will be 60% of the previous period’s
estimate. In the updating phase, the implications of these forecasts on observable GDP are
calculated. If these forecasts suggest a value of GDP that is at odds with the observable data,
the forecasts of potential GDP and the output gap will be updated to better reflect the observable data. The updated estimates of the mean and variance of the unobservable variables are
referred to as filtered estimates.11
The relative weights applied in the updating phase between new data and the prediction from
the model varies between each iteration. These weights make up a matrix referred to as the
Kalman gain. The Kalman gain efficiently links the new data to the prediction by accounting for
both the uncertainty of the previous estimate of the unobservable variables and the variance
of the measurement error.12 The smaller the measurement errors and the stronger the link
between the unobservable variable and the observable data, the greater the weight placed on
the new data relative to the prediction. The smaller the variance of the previous estimate, the
more weight is placed on the previous prediction relative to the new data.
Moving an extra period through the sample introduces a new period of data. The Kalman filter
only uses this data to update estimates of unobservable variables occurring in the same time
period as the data. Previously occurring estimates of unobservable variables are not updated
using the new data until the smoothing stage. This final stage contains a single updating phase
which iterates backwards from the end of the sample back to the beginning to incorporate
information processed in later iterations of the Kalman filter into earlier estimates of potential
GDP and the output gap.
If, for example, GDP data for 2006Q1 rose sharply and remained at similar levels for some time,
the most likely explanation (according to the model being applied) is that this was due to an
increase in potential GDP in 2006Q1. This is because potential GDP is characterized in the
model by its strong persistence and this persistence is able to explain the observed GDP data
11
The variance of the unobservables varies over the data set because of the different amounts of information available
in forming the estimates of the unobservable variables. As more iterations of the filter take place, each subsequent
estimate of the unobservable variables is based on more information. This reduces the estimate’s uncertainty.
The variance of the unobservable variables therefore, tends to fall further into the sample (and converge toward a
positive value). This is not the case after smoothing because each estimate is informed by the entire set of available
data rather than simply all current and preceding data.
12
As mentioned in subsection 2.1 the variance of the measurement errors are assumed to be known when running
the Kalman smoother.
Reserve Bank of New Zealand Analytical Note Series
10
subsequent to 2006Q1. The Kalman smoothing phase will update the estimate of potential GDP
in 2006Q1 to reflect that GDP remained elevated in 2006Q2 and beyond. The degree to which
the filtered estimates are adjusted in response to data occurring after 2006Q1 depends on both
the relevance of the extra data to the 2006Q1 estimates and the confidence around the filtered
estimates. If, for example, there was considerable uncertainty regarding the filtered estimate of
potential GDP in 2006Q1, then accounting for the higher subsequent GDP data will likely have
a large impact on the smoothed estimate of potential GDP in 2006Q1.
In summary, the Kalman smoother efficiently uses all the observable data to estimate each
unobservable variable at each point in time over the sample. The result is an estimate of the
distribution of the unobservable variables that is as precise as statistically possible (given the
assumptions mentioned in subsection 2.1). The mean of this distribution represents the most
likely estimate of the unobservable variables that generated the observable data.13
The data decomposition used in the next section, identifies how observable data is processed to
arrive at the mean estimates of the model’s unobservable variables. In the output gap example
mentioned above, this is trivial since there is only one observable variable: GDP. However, in
larger models it can be helpful to understand what influence various observed data are having
on the unobserved variables being estimated.
2.3 What is the Data Decomposition?
The data decomposition splits the estimate of an unobservable variable into contributions from
the various observable data. Each observable data series has an effect independent from
other observable data because the Kalman smoother’s estimate of each unobservable variable is always a linear function of the observable data. Because linear functions can always
be separated into independent terms, the effect of each observable variable on the Kalman
smoother’s estimate of a particular unobservable variable can be separately identified. This
process of separating variables within a time period, and then across time as the smoother
moves between time periods (both forward and backward), determines the weights on each
observable variable used to construct the data decomposition. The technical details of this
process and how they can be implemented are included in Appendix A.2.
As mentioned above, the estimate of an unobservable variable at a particular point in time is
13
While the mean produced represents the best estimate of the unobservable variables given the available sample,
these estimates are most inaccurate near the beginning and end of the sample. This is known as the ‘end-point’
problem.
Reserve Bank of New Zealand Analytical Note Series
11
informed by observable data throughout the entire sample. Therefore, when interpreting the
data decomposition, it is important to view the contributions as summarising all data – both
before and after the quarter being considered.
3 Applications
In this section the data decomposition is applied to unobservable estimates originating from
two different models.
3.1 Sectoral Factor Model of Core Inflation
The sectoral factor model of core inflation as detailed in Kirker (2010) and Price (2013) is
a dynamic factor model. Factor models are tools used to summarize the information in large
datasets.14 Essentially, the method is to estimate unobservable ‘factors’ that represent common
variation between all the data series.
A factor representing co-movements in prices in the economy is useful to central banks in that
it is structured to ignore idiosyncratic movements in specific price indices. These idiosyncratic
movements may affect headline CPI inflation, yet likely reflect relative price movements rather
than the concept of ‘general’ price increases (core inflation) that central banks are primarily
concerned with. Dynamic factor models include additional structure detailing how the factor
evolves over time.
The sectoral factor model of core inflation uses estimates of two factors from a data set of
96 disaggregated price indices (components of New Zealand’s CPI) to model core inflation: a
factor from 56 disaggregated price indices classified as tradable and a second factor from 50
disaggregated price indices classified as non-tradable (10 price indices are classified as a mix
of prices in tradable and non-tradable economic sectors). This allows the model to differentiate between price pressures originating domestically and those originating from internationally
traded goods and services. These factors are then combined to create the summary estimate
of core inflation.15
Figure 2 shows the tradable factor from the sectoral model against the (standardized16 ) 56
14
See Forni and Lippi (2001) and Stock and Watson (2002) for details on factor models and how they summarize
economic data.
15
The published measure can be accessed here: http://www.rbnz.govt.nz/statistics/tables/m1/.
16
A standardized data series is re-scaled so that the series has a mean of 0 and a variance of 1.
Reserve Bank of New Zealand Analytical Note Series
12
Figure 2: Tradable factor
Tradable data (56 series)
5
Standardised annual percent price change
4
3
2
1
0
−1
−2
−3
−4
Tradable factor
−5
1995
1997
1999
2001
2003
2005
2007
2009
2011
2013
This figure details how the tradable factor from Kirker (2010) (shown in black) fits through all 56 disaggregated price series classified as tradable.
Source: Price (2013)
Figure 3: Core inflation estimate
5.5
5
4.5
4
%
3.5
3
2.5
2
1.5
1
0.5
1994
1996
1998
2000
2002
2004
- Annual Percent Change in NZ CPI
2006
- Core CPI
2008
2010
2012
Reserve Bank of New Zealand Analytical Note Series
13
series classified as tradable. As the graph shows, the factor tends to run through the areas
that are most densely populated with tradable price series – thus capturing the co-movement
between these series.
Although the factors identify the common movements in all the data, some data series have
smaller idiosyncratic movements than other variables. These variables are therefore more
useful in determining the best estimate of the factor. The data decomposition can identify
which data series have had the largest contribution to the factor estimates.
Figure 4: Data decomposition of core inflation
Core
3.7
3.2
2.7
2.2
1.7
1.2
1994
1996
-
1998
2000
Hairdressing
Vehicle servicing and repairs
Ready-to-eat food
Accommodation services
2002
-
2004
Veterinary services
Dental Services
Gas
Other Data
2006
2008
2010
2012
- Household appliance repairs
- Property maintenance services
- Telecommunication services
This figure shows the data decomposition from the Core CPI model in Kirker (2010). THe black line is the overall estimate of core CPI and the coloured bars represent the
contributions from the top ten most important data series (by mean absolute error). The grey bars represent all other data.
The estimate of core inflation17 against overall New Zealand CPI inflation is shown in Figure 3
and Figure 4 shows the data decomposition of core inflation. The black line in both figures represents the estimate of core inflation and the coloured bars in Figure 4 show the contributions
from the individual price indices. The decomposition is centred around the series mean of 2.2
percent. For simplicity, we show the ten data series (out of 96) that have had the largest con17
This estimate of core was produced using data up to June 2013.
Reserve Bank of New Zealand Analytical Note Series
14
tributions to the estimate of core inflation over the full period and leave the total contributions
from the other series in grey. Overall, the top ten data series have explained about 60 percent
of the variation in core CPI and the top three components (with a total CPI weighting 0f only
1.04 percent) have explained approximately 30 percent of the variation.18
Kirker (2010) noted that the sectoral factor model’s estimate of core inflation is largely explained
by the non-tradable factor. Thus it is unsurprising that the five most important price series in
determining core CPI come from non-tradable subsectors of the economy (these series are
hairdressing, veterinary services, household appliance repairs, vehicle servicing repairs and
dental services). These series are quite labour-intensive subsectors and labour market pressures are often regarded as a key element in domestic inflation. Other CPI component series
which could also represent labour intensive sectors in the economy could have been less useful
as indicators of overall labour market pressures than hairdressing or veterinary services price
data if these sectors faced fluctuations in their costs (such as raw material costs) that were
specific to their industry.
Figure 5: Effect of modified data on core CPI
Effect of Modified Data on Core
3.4
3.2
3
2.8
%
2.6
2.4
2.2
2
1.8
1.6
1.4
1994
1996
1998
2000
- Original Estimate of Core CPI
2002
2004
2006
2008
2010
2012
- Estimate of Core CPI with Modified Data
This figure details how core CPI is revised when hairdressing, veterinary services and appliance repairs data are increased by 1 standard deviation in 2003Q1. The blue
line represents the original estimate of core CPI with original data. The black line represents how core CPI is revised when the three most important data series in terms of
contributions are increased by 1 standard deviation in 2003Q1.
18
The precise contributions to core inflation have varied over considerably over time. While the data decomposition
could be used to analyze this, this is beyond the scope of this note.
Reserve Bank of New Zealand Analytical Note Series
15
Figure 6: Data decomposition of the change in core CPI due to adjusted data
Core
0.16
0.14
0.12
0.1
0.08
0.06
0.04
0.02
0
1994
1996
1998
- Hairdressing
2000
2002
- Veterinary services
2004
2006
2008
2010
2012
- Household appliance repairs
This figure details how core CPI is revised when hairdressing, veterinary services and appliance repairs data are increased by 1 standard deviation in 2003Q1. The black
line represents the total revision to core CPI when all three data series are adjusted and the coloured bars each represent the revisions to core CPI from the adjustment to
each individual data series.
That hairdressing, veterinary services and household appliance repairs price indices have explained approximately 30 percent of core inflation fluctuations appears to suggest that the estimate of core inflation may be sensitive to the precise values of these price indices. However,
this is only the case for persistent (low frequency) movements in these data series. Short-lived
(high frequency) movements in these variables are largely interpreted as idiosyncratic movements. To demonstrate this we increase by 1 standard deviation the 2003Q1 values of these
series.19 Figure 5 shows the overall change in core CPI in 2003Q1 is approximately 0.1 percentage points or 0.15 standard deviations with smaller changes for periods before and after
this date. This demonstrates that core inflation is relatively unresponsive to noisy movements
in individual data series, even when one of these data series is an ‘important’ determinant of
core.
In addition, we can use the data decomposition to identify how each of these data series con19
We picked a time period in the middle of the available sample because this results in visually more dramatic results.
The size of the revisions is similar if the data changes are made at the end of the sample. Hairdressing, veterinary
services and appliance repairs data was increased by 1.2, 2.1 and 3.3 percentage points respectively.
Reserve Bank of New Zealand Analytical Note Series
16
tributes to the revision of core CPI. This is illustrated in Figure 6.20 It turns out that veterinary
services price data – the series with only the second largest overall contribution – leads to the
largest revision in core CPI of around 0.07 standard deviations. This is because veterinary
services prices are slightly less volatile than hairdressing and appliance repairs prices so that
a larger proportion of the price movements in veterinary services is treated as core inflation.
3.2 A small structural model
Structural macroeconomic models can be useful tools for forecasting because they are able
to provide users with an economic interpretation behind the forecasts they produce. The data
decomposition can illuminate how these economic interpretations are deduced from observable
data.
The model considered in this section is a small open economy model similar to Justiniano and
Preston (2010) with a tradable and domestic producing sector. Households supply labour to
domestic firms and consume both domestic and tradable goods. Firms are subject to nominal
rigidities which result in sluggish price adjustments to changes in economic conditions and
a role for monetary policy to stabilize prices. The equations for this model are detailed in
Appendix B.
Because this is a structural model, the behaviour of economic agents such as households,
firms and the central bank is described in some detail. These details allow the user to identify
how movements in observed economic variables are determined by changes in the behaviour
of agents in various sectors of the economy. For example, some structural models are able to
take a position on whether – given the structure of the model itself – the values of economic
variables such as the exchange rate are justified by economic conditions.
Economic variables in modern structural models are assumed to be driven by a range of ‘structural shocks’. These shocks are unobserved and are modelled as stochastic in nature. When
20
An oddity made clear by the data decomposition is that core CPI – a concept defined as summarizing co-movements
between many variables – is estimated by a method (Kalman smoothing) that interprets information from each data
series independently from the information contained in other data series. The fact that the Kalman smoother does
not adjust its estimates of the factors when the data co-moves could be interpreted as suggesting that the smoother
is an inappropriate tool to use to estimate the factor. It turns out that this is not the case when the parameters in
the model are estimated on the data set. The process of estimation involves selecting parameters to maximize the
likelihood that the model can explain the observable data. This is equivalent to minimizing the random components
in the model. Given the structure of the model, minimizing the random components largely involves minimizing the
idiosyncratic movements in variables – that is, setting the parameters so that the Kalman smoother estimates factors
that maximize the co-movement between the various data series. Therefore, so long as the data being used when
smoothing is the same as the data used in estimation, using the Kalman smoother to estimate the factor should be
appropriate.
Reserve Bank of New Zealand Analytical Note Series
17
designing these models from the basic behaviours of economic actors, structural shocks are
each given unique economic interpretations that can be used to describe the subsequent economic fluctuations these models generate.
One example is the ‘monetary policy shock’ term. The structural model considered here has an
equation representing the central bank’s monetary policy rule – how the central bank is typically
estimated to behave in response to the forecasted economic outlook. This particular structural
model has a rule under which the central bank responds to its forecast of next quarter’s CPI
inflation as well as the current output gap and exchange rate. In the model, interest rates
will sometimes deviate from the level proposed by the model’s rule. In some cases, these
deviations may occur deliberately and in other cases, these deviations may occur because
the central bank’s assessment of the level of ‘inflationary pressure’ turns out to have been
incorrect. The size of these deviations is assumed to diminish gradually over several quarters.
Conceptualizing monetary policy in this manner can be useful to assess the effects of policy
deviating occasionally from a systematic rule (either deliberately or otherwise).
There are many other unobserved shocks in the model. A ‘shock decomposition’ is a common tool used to ascertain the model implied explanation of which shocks (each with their own
economic interpretation) have caused observed economic fluctuations. The shock decomposition provides information on how the various underlying economic shocks – once identified –
contribute to the model’s forecasts.
However, the shock decomposition – even when applied over history – leaves out an important
aspect of the process: how these various underlying economic shocks are themselves estimated. Before producing the model’s forecasts the Kalman smoother needs to select a combination of the possible structural shocks available within the model to explain the observed
economic fluctuations over the available sample. Because the shock decomposition takes the
structural shocks as given, the data decomposition can complement the shock decomposition
by illustrating how observable data contributed to the estimates of these structural shocks.
In the remainder of this section, we look at a single quarter in history, in this case 2002Q1. Table
1 compares the 2002Q1 data with the structural model’s forecasts for 2002Q1 constructed with
2001Q4 data for the variables used to produce the structural model’s forecasts.21 For example,
GDP and consumption growth were lower than had been forecast whereas CPI, domestic inflation and exports were above the model’s forecasts. When producing regular forecasts it is
often a useful exercise to investigate how new releases of data will affect the model’s forecasts.
21
The differences column in table 1 may not perfectly match the values calculated from other columns due to rounding.
Reserve Bank of New Zealand Analytical Note Series
18
Table 1: Data and the previous forecast for 2002Q1
Variable
GDP growth
Consumption growth
Export growth
CPI inflation
Non-tradable inflation
Wage inflation
Interest rates
Exchange rate change
Terms of trade change
Data
0.37
-0.59
1.29
0.71
1.27
0.73
5.03
4.12
4.13
Forecast made in 2001Q4
0.58
1.10
-0.69
0.66
0.91
1.14
5.19
2.28
3.54
Difference
-0.21
-1.70
1.98
0.06
0.36
-0.41
-0.16
1.84
0.59
The black lines in Figure 7 shows the effects of this new information on the subsequent forecasts produced by the model. A black line at zero indicates no revision to the forecast in
response to new data whereas a positive value indicates that the forecast was, at that point in
time, revised upwards. This figure shows that simply adding the new quarter’s data led to the
forecast of the 90-day interest rate to be revised downwards by approximately 0.5 percentage
points (50 basis points) in 2002Q4. Similarly, the forecast for GDP growth is initially revised
downwards but is predicted to return to the previous forecast.
Figure 7: Revisions to forecasts from a structural model
Interest rate
Interest Rate
0.2
GDP growth
CPI inflation
GDP Growth
0.2
0.1
CPI Inflation
0.05
0.1
0.04
0
0.03
−0.2
−0.1
0.02
−0.3
−0.2
0.01
−0.3
0
−0.4
−0.01
0
−0.1
−0.4
−0.5
−0.6
2001
2002
2003
2004
2005
−0.5
2001
2002
2003
2004
2005
−0.02
2001
2002
2003
2004
2005
The black lines in each panel represents the revision to the model’s forecast for each variable while the points each labelled with an X the new data itself.
Reserve Bank of New Zealand Analytical Note Series
19
While not visible on the figure, the model’s inflation target assumes that the central bank will adjust interest rates over time to ensure that, eventually, CPI inflation will return to target (leading
to an eventual forecast revision of 0).
Figure 8: Shock decomposition of forecast revisions
Interest rate
GDP growth
Interest Rate
CPI inflation
GDP Growth
1
0.5
CPI Inflation
2
0.3
1.5
0.2
1
0.1
0.5
0
0
0
−0.1
−0.5
−0.5
−0.2
−1
−1
−0.3
−1.5
−1.5
2001
2002
2003
2004
2005
−2
2001
2002
2003
2004
2005
−0.4
2001
2002
2003
2004
2005
- Consumption shock
- Permanent productivity shock
- Import price shock
- Exchange rate shock
- Wage markup shock
- Domestic markup Shock
- Tradable markup shock
- Export markup shock
- Export demand shock
- Monetary policy shock
- Government spending shock
This figure shows a shock decomposition of the forecast revisions caused by introducing a new quarter of data in 2002Q1. The black lines represent the overall forecast
revisions and the coloured bars represent the contribution to these black lines from individual economic shocks.
To provide an economic interpretation of these forecast revisions we can use a shock decomposition which is detailed in Figure 8. The black lines in this figure are the same forecast revisions
shown in Figure 7. The bars represent contributions to these forecasts from the new economic
shocks inferred by the introduction of 2002Q1 data. The model interprets the changes as most
likely to have arisen from a combination of a positive export demand shock (an increase in demand for New Zealand exports at any given price) and a negative consumption shock (a desire
on the part of households to, at any given interest rate, consume a larger portion of their income
in the current period). These shocks have offsetting effects on GDP growth, and hence interest
rates. The model also identifies several smaller shocks, including an exchange rate shock (indicating the exchange rate has risen by less than suggested by economic fundamentals) and a
government spending shock.
Shocks also appear to contribute to explaining periods before the new data occurs (even though
the historical data themselves are not changed). This occurs because the new 2002Q1 data
suggests a different interpretation of history in terms of this model’s economic shocks.
The data decomposition can complement the shock decomposition by illustrating which of the
new observed data are important in generating the revisions to the estimated shocks. Figure 9
shows how observable data (in bars) contributed the model’s recommended interest rate (black
line in the left panel) as well as the difference between this recommended rate and actual
interest rates – the ‘monetary policy shock’ (the black line in the right panel). Interest rate data
includes both the changes in monetary policy that are consistent with the model’s monetary
Reserve Bank of New Zealand Analytical Note Series
20
Figure 9: Data decomposition of recommended monetary policy and the estimated monetary
policy shock
Recommended
interest
rate
Recommended Interest
Rates
Monetary
policy
shock
Monetary
Policy Shock
0.4
0.1
0.2
0
0
−0.1
−0.2
−0.2
−0.4
−0.3
−0.6
2001
2002
2003
2004
- GDP growth
- Interest rates
- Non-tradable inflation
2005
−0.4
2001
- Consumption growth
- Exchange rate
- Terms of trade
2002
2003
2004
2005
- Wage inflation
- CPI inflation
- Export growth
policy equation as well as monetary policy shocks. The interest rate forecast revision in figure
7 is made up of the combined effect of the two black lines in the panels in figure 9.
All observable data are important in determining the model’s recommended interest rate because the model recommends responding to its own forecasts of future inflation (as well as the
output gap and exchange rate). Since future inflationary pressures are partly caused by current
economic conditions, new observable data can also change the model’s recommended interest
rate. In addition, because it is assumed that the central bank takes considerable time to bring
actual monetary policy back in line with the model’s rule it also changes the forecast of the size
of the monetary policy shock (deviations from this rule).
Figure 9 suggests that the bulk of the revisions to the recommended interest rate occur from
interest rate, exchange rate and terms of trade data – while being offset by the impact of consumption growth data. Actual consumption growth data was much lower than had been forecasted, however, the model interprets this downwards revision as suggesting that consumption
growth will be stronger in the future to return consumption to previous levels. This ‘bounceback’ behaviour of consumption is forecast because the model explains a large portion of the
negative consumption surprise with a temporary positive government spending shock (displacing the need for as much private consumption).22 As private consumption returns to previous
levels this will result in higher inflation. This results in consumption data contributing to upward
22
Govenment spending data are not observed by the model, yet the model structure details a government sector. The
behaviour of this sector is inferred from the available observable data. The pink bars in figure 8 detail the overall
effects of the government spending shock. These bars are influenced by a variety of factors, not just the fall in
private consumption data.
Reserve Bank of New Zealand Analytical Note Series
21
revisions to the path of recommended interest rates.
Curiously, higher terms of trade data contribute to lower recommended interest rates. While
the surprise to terms of trade data does indicate higher export prices, export volumes data is
also an observable. Since this decomposition isolates the effects of each individual data series,
the shocks introduced by new terms of trade surprise must leave export volumes unchanged
(changes in export volumes data are dealt with in another part of the decomposition). To leave
export volumes unchanged, the terms of trade data introduces both export markup (supply) and
demand shocks. The net effect of these two shocks is a reduction in resource and inflationary
pressures in the medium term which lowers the model’s recommended interest rate.23
The monetary policy shock by contrast, seems largely driven by non-tradable inflation, exchange rate and interest rate data. The higher than forecast non-tradable inflation data seems
to revise the estimates of shocks over history, revising upward the recommended interest rate
over history. As interest rate data over the same period did not change, this causes downwards revisions to the historical estimates of monetary policy shocks (monetary policy over the
historical period now looks looser, relative to the model’s rule, than had been previously realized). These revisions to the historical estimates also affect the forecast of monetary policy
shocks because the model assumes that deviations from the recommended rate take time to
be corrected.
The model appears to interpret new consumption growth and terms of trade data as suggesting
identical revisions to the forecasts of recommended and expected monetary policy – with little
expected deviations from the recommended rule. Non-tradable inflation data raise the model’s
forecasts of inflationary pressure and raise recommended interest rates more than the model’s
forecasts of actual interest rates. Exchange rate data by contrast, lower the forecast of actual
interest rates by more than its revisions to dis-inflationary pressures lowers recommended interest rates. Interest rate data both indicates a reduction in inflationary pressures as well as a
deviation from the model’s recommended rule.
The data decomposition can also be used to identify how data affects other estimated economic
shock terms in the model. To illustrate this Figure 10 shows the data decomposition of the the
tradable markup shock and the exports demand shock. The tradable markup shock can be
23
Once accounting for export volumes data, the overall impact of the higher terms of trade and export volumes data
suggests a demand curve shift. However, export volumes data does not reverse the revisions to recommended
interest rates that were inferred from terms of trade data. In addition to the export demand shock, export volumes
data has led the model to infer additional shocks that lead to a reduction in the costs of producing exports – offsetting
the inflationary pressures caused by higher export demand. These shocks are not the same as the tradable markup
shock shown in figure 10
Reserve Bank of New Zealand Analytical Note Series
22
Figure 10: Data decomposition of the tradable markup shock and the exports demand shock
Tradable
markup
shock
Tradable
Markup Shock
Exports demand
shock
Export Demand Shock
0.3
3
0.2
2.5
2
0.1
1.5
0
1
−0.1
0.5
−0.2
0
−0.3
−0.4
−0.5
2001
2002
2003
2004
- GDP growth
- Interest rates
- Non-tradable inflation
2005
−1
2001
- Consumption growth
- Exchange rate
- Terms of trade
2002
2003
2004
2005
- Wage inflation
- CPI inflation
- Export growth
thought of as representing changes in competitive pressures in the tradable sector (therefore
resulting in changes to tradable firms’ desired markups). The exports demand shock is a modelbased measure of demand for New Zealand exports.
In the model, the CPI is a weighted combination of tradable and non-tradable prices. Since tradable inflation is unobserved by the model, this variable is calculated by the Kalman smoother
as the (weighted) difference between the observed CPI and non-tradable inflation data.24 As
table 1 shows, both the CPI and non-tradable inflation data were higher than expected, however
the non-tradable inflation surprise was much larger in magnitude than the CPI inflation surprise
indicating that tradable inflation fell during that quarter. One explanation for the lower tradable
inflation is that the tradable sector has become more competitive. An alternative explanation
is that costs fell and some of the falls in costs were passed on to consumers. Costs in the
tradable sector within the model are assumed to be a weighted average of import prices (in
New Zealand dollars) and domestic prices. This cost structure is the result of assuming that
tradable firms are retailers selling imported goods who face some (domestic) transport costs to
get their goods to market.
The black line in the left panel of figure 10 shows that a reduction in the mark-ups achieved by
tradable firms is estimated by the model to be one of the explanations of the fall in tradable inflation. The view that competitive pressures have increased (leading to lower pricing pressures in
the tradable sector) seems to be based on the new non-tradable inflation data. New information
on non-tradable inflation has two channels through which to adjust the model’s estimate of the
tradable markup shock. The first channel is that higher non-tradable inflation, while holding the
24
These two data series are the only two that affect the estimate of tradable inflation.
Reserve Bank of New Zealand Analytical Note Series
23
CPI constant, reduces the model’s estimate of tradable inflation. The model then (partly) explains this reduction in its tradable inflation estimates with changes in tradable markup shocks
- tradable firms are less able to achieve the markups they desire. The second channel by
which new information on non-tradable inflation affects the estimates of the tradable markup
shock is through changes in the domestic distribution cost component in tradable production.
As non-tradable inflation reflects costs in the domestic sector in the structural model, higher
non-tradable inflation indicates higher distribution costs for tradable firms. These higher costs,
holding the estimate of tradable inflation constant, must be offset by a reduction in other costs
or by changes in the markups of tradable firms. Therefore, the higher non-tradable inflation
data led to both a reduction in the model’s estimate of tradable inflation as well as an increase
in the model’s estimate of costs in the tradable sector. Both these changes result in estimated
increases in competitive pressures in the tradable sector.
The model’s view of competitive pressures in the tradable sector is also affected by exchange
rate and CPI inflation data. The higher than expected exchange rate lowers the model’s estimate of the domestic currency price of imports, which results in lower costs in the tradable
sector. This reduction in cost pressures, holding the estimate of tradable inflation constant,
results in a reduction in the model’s estimate of competitive pressures in the tradable sector (allowing firms to raise their markups). Higher than expected CPI data directly raises the
estimate of tradable inflation (holding non-tradable inflation data constant). This is explained
(partly) by changes in competitive pressures.
Export demand was an important shock in explaining the forecast revisions produced by new
2002Q1 data (as shown in figure 8). The right hand panel of figure 10 shows that the model’s estimate of foreign demand for New Zealand exports at any given set of prices has been revised
upward in response to 2002Q1 data. The two most important series in determining whether
export demand has increased are export volumes and terms of trade. Higher than expected
exports volumes and terms of trade are together consistent with an increase in demand for
exports.25 Other data are also important for the size of the exports demand shock. For example, higher non-tradable inflation boosts exporting firms’ costs. To explain the absence of
a reduction in export volumes when costs have increased, the model increases its estimate of
export demand shocks. This demand shock is assumed to abate over the forecast – however
at a more gradual rate than the other two shocks illustrated here.
25
Terms of trade data might also reflect a reduction in imports prices, however, in this case, the positive contribution
to exports demand suggests that the terms of trade data did result in higher export prices.
Reserve Bank of New Zealand Analytical Note Series
24
4 Conclusion
Many models in economics contain unobservable variables that must be estimated. While these
estimation procedures are often opaque, the data decomposition can make these procedures
clearer by identifying how observable data affect estimates of unobservable variables.
This note showed that the data decomposition can provide useful insights into both purely
statistical and structural models. The data decomposition identified how data was used to
estimate core CPI in the sectoral factor model, and how data is used to revise the forecasts of
structural models.
The insights gained from applying the data decomposition to these models highlights that this
tool can improve modellers’ understanding of their models. Further insights can be gained when
data decompositions are used in conjunction with other tools such as shock decompositions.
The flexibility of the data decomposition allows it to be readily applied to any linear model.
Algorithms to implement the data decomposition along with the Kalman smoother are included
in the Appendix.
Reserve Bank of New Zealand Analytical Note Series
25
References
Andrle, M. (2013a). Understanding DSGE filters in forecasting and policy analysis. IMF Working
Paper WP/13/98, IMF.
Andrle, M. (2013b). What is in your output gap? Unified framework & decomposition into
observables. IMF Working Paper WP/13/105, IMF.
Beveridge, S. and C. R. Nelson (1981). A new approach to decomposition of economic time
series into permanent and transitory components with particular attention to measurement
of the ‘business cycle’. Journal of Monetary Economics 7 (2), 151–174.
Clark, P. K. (1987). The cyclical component of U.S. economic activity. The Quarterly Journal of
Economics 102(4), 797–814.
Cogley, T. and J. M. Nason (1995). Effects of the Hodrick-Prescott filter on trend and difference stationary time series implications for business cycle research. Journal of Economic
Dynamics and Control 19(1-2), 253–278.
Cristadoro, R., M. Forni, L. Reichlin, and G. Veronese (2005, June). A core inflation indicator
for the Euro area. Journal of Money, Credit and Banking 37 (3), 539–60.
Durbin, J. and S. Koopman (2012). Time Series Analysis by State Space Methods (2 ed.).
Oxford: Oxford University Press.
Forni, M. and M. Lippi (2001). The generalized dynamic factor model: Representation theory.
Econometric Theory 17 (06), 1113–1141.
Harvey, A. C. (1985). Trends and cycles in macroeconomic time series. Journal of Business &
Economic Statistics 3(3), 216–27.
Harvey, A. C. and A. Jaeger (1993). Detrending, stylized facts and the business cycle. Journal
of Applied Econometrics 8(3), 231–47.
Hodrick, R. J. and E. C. Prescott (1997). Postwar U.S. business cycles: An empirical investigation. Journal of Money, Credit and Banking 29(1), 1–16.
Justiniano, A. and B. Preston (2010). Can structural small open-economy models account for
the influence of foreign disturbances? Journal of International Economics 81(1), 61–74.
Reserve Bank of New Zealand Analytical Note Series
26
King, R. G. and S. T. Rebelo (1993). Low frequency filtering and real business cycles. Journal
of Economic Dynamics and Control 17 (1-2), 207–231.
Kirker, M. (2010). What drives core inflation? A dynamic factor model analysis of tradable and
nontradable prices. Reserve Bank of New Zealand Discussion Paper Series DP2010/13,
Reserve Bank of New Zealand.
Koopman, S. J. and A. Harvey (2003). Computing observation weights for signal extraction and
filtering. Journal of Economic Dynamics and Control 27 (7), 1317–1333.
Price, G. (2013). Some revisions to the sectoral factor model of core inflation. Reserve Bank
of New Zealand Analytical Note Series AN 2013/06, Reserve Bank of New Zealand.
Simon, D. (2006). Time Series Analysis by State Space Methods. New Jersey: John Wiley &
Sons.
Smets, F. and R. Wouters (2004). Comparing shocks and frictions in US and euro area business
cycles: a Bayesian DSGE approach. Working Paper Series 0391, European Central Bank.
Stock, J. H. and M. W. Watson (2002). Macroeconomic forecasting using diffusion indexes.
Journal of Business & Economic Statistics 20(2), 147–62.
Watson, M. W. (1986). Univariate detrending methods with stochastic trends. Journal of Monetary Economics 18(1), 49–75.
Reserve Bank of New Zealand Analytical Note Series
27
A An Algorithm for Implementing the Data Decomposition
A.1 What is the Data Decomposition?
In this appendix it will be shown that for any linear model, the estimate of the unobservable
variables X̂ produced from the Kalman smoother can be written in the following form as a
function of Yt 26 :

X̂j,t =
T
X
h
ωlj Yl0 = ω1j ω2j . . .
l=1

Y10
i 
 . 
ωTj  .. 
 
YT0
j ∈ {1, . . . , n}
(3)
where ωlj represents a row vector of weights linking the m observable data series at time l to
the j th unobservable variable at time t.
The key insight is that each observation of data for each series has an independent effect on
the estimate of each unobservable variable. This independence of the contributions simplifies
the analysis required to decompose the unobservable estimates into the various observed data
– any movement in an unobservable variable that can be associated with a particular data
series is not associated with any movements in other data series. The weights ωlj derived for
this algorithm were originally derived in Koopman and Harvey (2003).
The data decomposition algorithm presented here is recursive and so can be implemented
alongside the Kalman smoother algorithm. To that end we discuss the forward (filtering) and
backwards recursion parts separately in the subsections A.2 and A.3.
A.2 Data Decomposition of the Kalman Filter
The algorithm presented here uses the Kalman filter equations from Durbin and Koopman
(2012, pp. 85, 110-112). These equations perform the prediction and updating phases dis26
Recall that there are n unobservable X variables and m observable data series Y .
Reserve Bank of New Zealand Analytical Note Series
28
cussed above in a single step.27 Using the notation introduced in subsection 2.1 the Kalman
filter equations are:
Ft = Wt BPt B 0 Wt0 + Wt HWt0
(4)
Kt = APt B 0 Wt0 Ft−1
(5)
f
Xt+1
= AXtf + Kt (Wt Yt − Wt BXtf )
(6)
f
Pt+1
= APtf (A − Kt Wt B)0 + RQR0
(7)
where Xtf and Ptf represent the filtered mean and variance of Xt and Kt is the Kalman gain
matrix mentioned above in subsection 2.2. The matrix Wt is used to control for missing observations. If all data is available at time t then Wt is set to the identity matrix.28 If an observation
for a particular observable data series is unavailable, then the row corresponding to that data
series is removed from this identity matrix.
Note that Equation 6 shows that the estimate of Xtf is a linear function of the estimate of
f
and the data pertaining to period t, Yt .
the unobservables in the preceding time period Xt−1
Therefore, the Kalman filter interprets each observation for each variable Yi,t i ∈ {1, . . . , m}
independently from all other observations and variables. It is this linearity that allows us to
attribute movements in an unobservable variable to a particular observable variable (at a particular time). Thus we can re-write the Equation 6 so that X f depends only on the data Y as
27
The steps can be separated into the following prediction and updating steps
Prediction:
Ft
=
Wt BPt B 0 Wt0 + Wt HWt0
f
Xt+1
=
AXt|t
f
Pt+1
=
APt|t A0 + RQR0
Kt0
=
Pt B 0 Wt0 Ft−1
Xt+1|t+1
=
f
0
f
Xt+1
+ Kt+1
(Wt+1 Yt+1 − Wt+1 BXt+1
)
Pt+1|t+1
=
f
f
0
−1
Pt+1
− Pt+1
B 0 Wt+1
Ft+1
Wt+1 BPt+1
Updating
where Xt+1|t+1 and Pt+1|t+1 are the respective updated means and variances of the unobservables using data up
to time t + 1. Note that Kt0 is different from that in equation 5.
28
The size of the identity matrix is given by the total number of observable data series.
Reserve Bank of New Zealand Analytical Note Series
29
follows:
f
Xt+1
= (A − Kt Wt B) Xtf + Kt Wt Yt
f
f
⇔ Xt+1
= (A − Kt Wt B) (A − Kt−1 Wt−1 B) Xt−1
+ (A − Kt Wt B) Kt−1 Wt−1 Yt−1 + Kt Wt Yt
" l−1
#
t−1
X Y
f
⇔ Xt+1
=
(A − Kt−s Wt−s B) Kt−l Wt−l Yt−l
l=0
s=0

⇔
f
Xt+1
=
t
X
h
ωl Yl = ω1t+1 ω2t+1 0 · · ·
ωtt+1 0 · · ·
l=1

Y1
i 
 . 
0  ..  = Dt+1,1:T vec(Y )
 
YT
(8)
where products that have an upper index number that is below the lower index number should
Q−1
j
equal one (i.e.
0 X = 1 for any X). In addition, ωl is a n × m matrix of weights on each
Yi,t i ∈ {1, . . . , m}. These weights contain all the information we will need to implement the
data decomposition of the Kalman filter. We will derive the data decomposition for the Kalman
smoother in the next subsection. To obtain the information necessary to apply the data decomposition we introduce the matrix D above which is nT × mT where n is the number of
unobservable variables in X, m is the number of unobservable variables in Y and T is the
number of observations available in the dataset. This matrix is written so that:

D1,1:T






D
2,1:T 
f

vec(X ) =  . 
 vec(Y ) = Dvec(Y )
 .. 


DT,1:T
(9)
where X f and Y are n × T and m × T matrices containing all the filter estimates of X and all
the observable data Y . Equation 9 is written so that the rows of D represent the weights on the
f
various Yi,t i ∈ {1, . . . , m} that led to the estimate Xj,t
j ∈ {1, . . . , n}. D can be split into T 2 n x
m blocks as shown:

D1,1
D1,2
···


 D2,1 D2,2 · · ·
D := 
 ..
..
..
 .
.
.

DT,1 DT,2 · · ·
The recursive algorithm for obtaining D is as follows:
D1,T



D2,T 

.. 
. 

DT,T
(10)
Reserve Bank of New Zealand Analytical Note Series
A.2.1
30
Algorithm to Produce the Data Decomposition of the Kalman filter
1. Set D to be a nT × mT matrix of zeros.
2. Pick starting points with which to initialize the Kalman filter. Usually one uses the unconditional mean and variance of the unobservable variables.
3. Loop from the beginning of the sample to the end where i represents each point in the
loop. For each i execute the following:
(a) perform the Kalman filter equations 4 to 7 and store Ki and Wi .
(b) set Di,i = Ki Wi
(c) For each i, create a new loop from i to the beginning of the sample where j represents each point in this second loop. For each j execute the following:
i. set Di,j = (A − Ki Wi B) Di−1,j
Note that this algorithm explicitly details the recursive operations needed to produce the weights.
This is to allow this algorithm to be implemented within a Kalman filter and smoother procedure.
This greatly simplifies the memory requirements of implementing the data decomposition. The
above decomposition of the filtered estimates X f will be needed for the data decomposition of
the Kalman smoother in the next section.
A.3 Data Decomposition of the Kalman Smoother
The algorithm here uses the Kalman smoother equations of Durbin and Koopman (2012, pp.
91-94)29 which are:
rt−1 = B 0 Wt0 Ft−1 (Wt Yt − Wt BXtf ) + (A − Kt Wt B)rt
X̂t = Xtf + Ptf rt−1
(12)
êt = H(Ft−1 (Wt Yt − Wt BXtf ) − Kt0 rt )
(13)
ˆt = QR0 rt
(14)
Nt−1 = B 0 Wt0 Ft−1 Wt B + (A − Kt Wt B)0 Nt (A − Kt Wt B)
Vt = Ptf − Ptf Nt−1 Ptf
29
(11)
(15)
(16)
The Kalman smoother equations can be written in several forms. This article chooses the form detailed in Durbin
and Koopman (2012) as it is applicable to the largest class of models.
Reserve Bank of New Zealand Analytical Note Series
31
where X̂t represents the smoothed estimates of the unobservable variables Xt , ˆt the estimate
of the structural shocks t , êt the estimate of the measurement shocks et and Vt is the covariance matrix of X̂t . The smoother begins one period before the end of the sample T − 1 and
proceeds to the beginning of the sample. Starting values for X̂T , rT and NT are XT |T , 0 and
030 respectively.31
Note that in equation 12 the smoothed estimate of the unobservable is a linear function of the
filtered estimate of the unobservables Xtf and an adjustment term rt . From the above section
we already know to decompose X f , so that is required find the weights for X̂t is to calculate
the decomposition of rt−1 . This can be found as follows:
rt−1 = B 0 Wt0 Ft−1 vt + (A − Kt Wt B)rt
vt := (Wt Yt − Wt BXtf )
−1
0
⇔ rt−1 = B 0 Wt0 Ft−1 vt + (A − Kt Wt B)B 0 Wt+1
Ft+1
vt+1 + (A − Kt Wt B)(A − Kt+1 Wt+1 B)rt+1
"
#
T
l−1
X
Y
⇔ rt−1 =
(A − Ks Ws B) B 0 Wl0 Fl−1 vl
l=t−1
s=0
In addition, note that:
vt = (Wt Yt − Wt BXtf ) = (Wt Yt − Wt BDt,1:T vec(Y ))
⇔ vt = (Wt (et ⊗ Im ) − Wt BDt,1:T ) vec(Y )
where Im is an m x m identity matrix and et is an indicator vector of length T which is 1 at time
t and 0 otherwise. Therefore rt−1 is given by:
rt−1 =
T
X
l=t−1
" l−1
Y
#
(A − Ks Ws B) B 0 Wl0 Fl−1 (Wl (el ⊗ Im ) − Wl BDl,1:T ) vec(Y ) = Dr vec(Y )
s=0
From this we can write X̂t as:
r
s
X̂t = Dt,1:T vec(Y ) + Pt Dt,1:T
vec(Y ) = Dt,1:T
vec(Y )
(17)
Again, note that products where the upper index number is below the lower index number
Q
should equal one (i.e. −1
0 X = 1 for any X).
30
31
0 represents a matrix of zeros of size n x n.
This is the updated estimate of the unobservables in period T which is given by: XT |T = XTf +PT B 0 WT0 FT−1 (Wt YT −
WT BXTf )
Reserve Bank of New Zealand Analytical Note Series
32
The following algorithm shows how to use this to get the weights of the smoothed estimates of
X̂t .
A.3.1
Algorithm to Produce the Data Decomposition of the Kalman smoother
1. Let Ds be the nT x mT matrix of weights for the smoothed estimates X̂1:T and Dr be a
nT x mT matrix for the weights on rt .For the remainder of the algorithm consider Ds and
Dr to be partitioned into T 2 n x m blocks as follows:

s
D1,1
s
D1,2
···

 s
s
 D2,1 D2,2
···
s
D := 
 ..
..
..
 .
.
.

s
s
DT,1
DT,2
···
s
D1,T


s 

D2,T

.. 
. 

s
DT,T

r
D0,0
r
D0,2
···

 r
r
 D2,0
D1,2
···
r
D := 
 ..
..
..
 .
.
.

r
DTr −1,1 DT,2
···
r
D0,T
r
D1,T
..
.
DTr −1,T








where the subscripts in the Dr partitions are labelled as such to match the time subscripts
of rt .
s
= (DT,1:T − PT B 0 WT0 FT−1 WT BDT,1:T )
2. Set DT,1:T
s
s
3. Set DT,T
= DT,T
+ PT B 0 WT0 FT−1 WT
4. Set DTr −1,1:T = −B 0 WT FT−1 WT BDT,1:T
5. Set DTr −1,T = DTr −1,T + B 0 WT0 FT−1 WT
6. Loop from one period prior to the end of the sample (T − 1) back to the beginning of the
sample. Let i be represent each point in this loop. For each i execute the following:
(a) It is recommended for computational efficiency that the smoother equations be executed along with the data decomposition. However, these equations need not be
executed before the remainder of this algorithm. To compute the smoothed estimates of X, execute the eautions 11 through to 15.32
r
r
(b) Set Di−1,1:T
= −B 0 Wi Fi−1 Wi BDi,1:T + (AKi Wi B)Di−1,1:T
r
r
(c) Set Di−1,i
= Di−1,i
+ B 0 Wi0 Fi−1 Wi
s
r
(d) Set Di,1:T
= Di,1:T + Ptf Di−1,1:T
7. To produce weight matrices for the structural shocks or measurement errors, simply declare additional matrices D and De of the appropriate size and evaluate the following
32
The extra two equations (15 and 16) calculate the variance of the unobservable estimates X̂t and are not necessary
to calculate the smoothed mean. Execute these equations only if the variance is desired as output.
Reserve Bank of New Zealand Analytical Note Series
33
also:
e
r
(a) Di,1:T
= −HFi−1 Wi BDi,1:T − Ki Di,1:T
e = D e + HF −1 W
(b) Di,i
i
i,i
i
r
(c) Di,1:T
= QR0 Di−1,1:T
Note that for models that are initialized with a non-zero mean, these recursions will have to be
repeated with nT x 1 matrices for X, r and X̂ to store the contributions of the initial condition
to the overall estimates.
Reserve Bank of New Zealand Analytical Note Series
34
B Model Equations of the Structural Model
Euler Equation
c
λ̂t = λ̂t+1 + ∆ω̂t+1
+ R̂th − π̂t+1
Consumption Equation
h
σ
λ̂t = − ωn
1− 1+σ
n1+σn
n
1
1−χc
(ĉt − χc ĉt−1 ) − ω n n1+σn n̂t
i
Marginal Rate of Substitution
mrs
ˆ t = σn n̂t
Wage Philips curve
π̂tw =
γW
1+βγW
w +
π̂t−1
β
w
(1+βγW ) π̂t+1
+
1
ψW (1+βγW )(λw −1)
mrs
ˆ t − ŵth + λ̂w,t
Imports Demand
m̂t = −v p̂T,t + ĉt + (1 − τ )(1 − ηT )(p̂N,t − p̂M,t )
Domestic Production
ŷt =
y
y+FN
y (ŵt
+ αn̂t )
Domestic Phillips Curve
π̂tN =
γPN
1+βγPN
N +
π̂t−1
β
1+βγPN
N +
π̂t+1
1
(λN −1)ψPN
(1+βγPN )
m̂cN,t + λ̂N
t
Domestic Marginal Cost
mcN,t = ŵth +
1−α y
α y+FN ŷt
− α1 ω̂ty − p̂N,t
Imports Phillips Curve
π̂T,t =
γPT
1+βγPT
π̂X,t =
γPX
1+β π̂X,t−1
π̂T,t−1 +
β
1+βγPT
π̂T,t+1 +
1
(1+βγPT )ψPT (λT −1)
p̂∗m,t − q̂t − p̂T,t + λ̂T,t
Exports Phillips Curve
+
β
1+β π̂X,t+1
+
1
(1+β)ψPX (λX −1)
p̂N,t + q̂t − p̂X,t + λ̂X,t
Exports Demand
ĉX,t = −vf p̂X,t + ŷt∗
Balance of Payments
by,f
=
t
πR∗ y,f
π ∗ bt−1
+
p X cX
Q y
+
G
Y ĝt
(p̂X,t + ĉX,t − p̂M,t − m̂t )
Resource Constraint
ŷt =
C
Y ĉt
+
I
Y ît
+
X
Y x̂t
−
M
Y m̂t
UIP
s
∗ ) = q̂ − q̂
∗
∗
r̂t − Et π̂t+1 − (r̂t∗ − Et πt+1
t
t+1 − ψuip r̂t−1 − π̂t − (r̂t−1 − π̂t ) + q̂t − q̂t−1 + ωt
Retail Interest Rates
r̂th = r̂t − φB b̂t + ωtR
Central Bank Policy Rule
r̂t = ρr r̂t−1 + (1 − ρr ) [φπ π̂t + φy ŷt + φQ (q̂t + q̂t−1 − π̂t + π̂t∗ ) + ωtr ]
All real variables are log-deviations from steady state. Quantity variables are in per-capita terms.
r = nominal interest rate, i = investment, π N = non-tradable inflation,
π w = nominal wage inflation, π T = Tradable Inflation, w = real wage,
q = Real Exchange Rate, pN = Relative Non-tradable prices, pT = Relative Tradable Prices, by,f = foreign debt,
c = consumption, y = GDP, m = imports, g = government, mrs = Marginal Rate of Substitution, cX = exports.
gp = Total Factor Productivity, PX = Export Prices,
z = Preference Shifter, tf
PM = Wholesale Import Prices, y ∗ = Foreign GDP,
τ C = GST, τ N = Labour Taxes,n = Labour,
^
W
tf p = World total factor productivity, ω y = temporary productivity changes,
λi = Markup Shocks, ω i = Other shocks,
Et preceding variables represent expectations taken at time t