Consolidation of Predictions of Seasonal Climate by Several Atmospheric General Circulation

Consolidation of Predictions of
Seasonal Climate by Several
Atmospheric General Circulation
Models at IRI
Anthony Barnston, Lisa Goddard, Simon Mason
and Andrew Robertson
International Research Institute
for Climate and Society (IRI)
IRI DYNAMICAL CLIMATE FORECAST SYSTEM
2-tiered
OCEAN
ATMOSPHERE
PERSISTED
GLOBAL
SST
ANOMALY
GLOBAL
ATMOSPHERIC
MODELS
ECPC(Scripps)
10
24
24
Persisted
SST
Ensembles
3 Mo. lead
ECHAM4.5(MPI)
POST
PROCESSING
FORECAST SST
TROP. PACIFIC:
THREE scenarios
(multi-models, dynamical
and statistical)
TROP. ATL, INDIAN
(ONE statistical)
EXTRATROPICAL
(damped persistence)
CCM3.6(NCAR)
12
NCEP(MRF9)
NSIPP(NASA)
COLA2
GFDL
Forecast
SST
24
30 Ensembles
3/6 Mo. lead
24
12
30
30
MULTIMODEL
ENSEMBLING
-Bayesian
-Caninical
variate
IRI DYNAMICAL CLIMATE FORECAST SYSTEM
2-tiered
OCEAN
PERSISTED
GLOBAL
SST
ANOMALY
ATMOSPHERE
GLOBAL
ATMOSPHERIC
MODELS
ECPC(Scripps)
ECHAM4.5(MPI)
FORECAST SST
TROP. PACIFIC: THREE scenarios:
1) CFS prediction 2) LDEO prediction
3) Constructed Analog prediction
TROP. ATL, and INDIAN oceans
CCA, or slowly damped persistence
EXTRATROPICAL
damped persistence
CCM3.6(NCAR)
NCEP(MRF9)
NSIPP(NASA)
COLA2
GFDL
Six GCM Precip. Forecasts, JAS 2000
RPSS Skill of Individual Models: JAS 1950-97
Goals
To combine the probability forecasts of several
models, with relative weights based on the
past performance of the individual models
To assign appropriate forecast probability
distribution: e.g. damp overconfident forecasts
toward climatology
Probabilities and
Uncertainty
Climatological
Probabilities
GCM
Probabilities
Pkt (x)  Pk (x)
m
Pkt (y)  kt
m
Above Normal
1/3

1/3
Near-Normal
Below Normal

x1

x2
y2
6/24
8/24
10/24
1/3

y1

k = tercile number
t = forecast time
m = no. ens members
Tercile boundaries are
identified for the models’
own climatology, by
aggregating all years and
ensemble members. This
corrects overall bias.
Bayesian Model
Combination
Combine climatology forecast (“prior”) and an
AGCM forecast, with its evidence of
historical skill, to produce weighted
(“posterior”) forecast probabilities, by
maximizing the historical likelihood score.
Aim to maximize the
likelihood score
N
L( w)  log  E (Qkt )
t 1
k=tercile category
t=year number
The multi-year product of the probabilities
that were hindcast for the category
that was observed.
(Could maximize other scores, such as RPSS)
Prescribed, observed SST used to force AGCMs.
Such simulations used in absence of ones using
truly forecasted SST for at least half of AGCMs.
1. Calibration of each model, individually, against climatology
Pjkt 
wc (.333)  w j Pjkt
wc  w j
k=tercile category (1,2, or 3)
t=year number
j=model number (1 to 7)
w=weight for climo (c) or for model j
Optimize
likelihood
score
PMMkt= weighted linear
comb of Pjkt over all j,
normalized by Σ(wj)
2. Calibration of the weighted model combination against climatol
P
final
kt
w *c (.333)  wMM PMMkt

w *c  wMM
Optimize
likelihood
score
where wMM uses wj proportional to results of the first step above
Algorithm used to maximize the designated score:
Feasible Sequential Quadratic Programming (FSQP)
“Nonmonotone line search for minimax problems”
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
M:
TOTAL NUMBER OF CONSTRAINTS.
ME :
NUMBER OF EQUALITY CONSTRAINTS.
MMAX : ROW DIMENSION OF A. MMAX MUST BE AT LEAST ONE AND GREATER
THAN M.
N:
NUMBER OF VARIABLES.
NMAX : ROW DIMENSION OF C. NMAX MUST BE GREATER OR EQUAL TO N.
MNN :
MUST BE EQUAL TO M + N + N.
C(NMAX,NMAX): OBJECTIVE FUNCTION MATRIX WHICH SHOULD BE SYMMETRIC AND
POSITIVE DEFINITE. IF IWAR(1) = 0, C IS SUPPOSED TO BE THE
CHOLESKEY-FACTOR OF ANOTHER MATRIX, I.E. C IS UPPER
TRIANGULAR.
D(NMAX) : CONTAINS THE CONSTANT VECTOR OF THE OBJECTIVE FUNCTION.
A(MMAX,NMAX): CONTAINS THE DATA MATRIX OF THE LINEAR CONSTRAINTS.
B(MMAX) : CONTAINS THE CONSTANT DATA OF THE LINEAR CONSTRAINTS.
XL(N),XU(N): CONTAIN THE LOWER AND UPPER BOUNDS FOR THE VARIABLES.
X(N) : ON RETURN, X CONTAINS THE OPTIMAL SOLUTION VECTOR.
U(MNN) : ON RETURN, U CONTAINS THE LAGRANGE MULTIPLIERS. THE FIRST
M POSITIONS ARE RESERVED FOR THE MULTIPLIERS OF THE M
LINEAR CONSTRAINTS AND THE SUBSEQUENT ONES FOR THE
MULTIPLIERS OF THE LOWER AND UPPER BOUNDS. ON SUCCESSFUL
TERMINATION, ALL VALUES OF U WITH RESPECT TO INEQUALITIES
AND BOUNDS SHOULD BE GREATER OR EQUAL TO ZERO.
Circumventing the effects of
sampling variability
• Sampling variability appears to be an
issue: noisy weight distribution with large
number of zero weights and some unity
weights
• Bootstrap the optimization, omitting
contiguous 6-year blocks of the 48-yr time
series
–
–
–
yields 43 samples of 42 years
shows the sampling variability of the likelihood over subsets of years
We average the weights across the samples
Example
Six GCMs’ Jul-Aug-Sep precipitation simulations
Training period: 1950–97
Ensembles of between 9 and 24 members
Model Weights – initially, by individual model
Climatological Weights – Multi-model
Model Weights – after second (damping) step
Model Weights – step 2, and Averaged over Subsamples
For more spatially smooth results, the
weighting of each grid point is averaged
with that of its 8 neighbors, using
binomial weighting.
X X X
X X X
X X X
Climatological Weights
Combination Forecasts of July-Sept Precipitation
After first stage only
After second (damping) stage
After sampling subperiods
After spatial smoothing
Reliability
JAS Precip., 30S-30N
Below-Normal
Observed relative
Freq.
Observed relative
Freq.
Above-Normal
Forecast probability
Forecast probability
(3-model)
from Goddard et al. 2003
Bayesian
Pooled
Individual
AGCM
RPSS
Precip.
from Roberson et al. (2004):
Mon. Wea. Rev.,
132, 2732-2744
RPSS
2-m
Temp.
from Roberson et al. (2004):
Mon. Wea. Rev.,
132, 2732-2744
Conclusions - Bayesian
• The “climatological” (equal-odds) forecast
provides a useful prior for combining
multiple ensemble forecasts
• Sampling problems become severe when
attempting to combine many models from
a short training period (“noisy weights”)
• A two-stage process combines the models
together according to a pre-assessment of
each against climatology
• Smoothing of the weights across data subsamples and spatially appears beneficial
IRI’s forecasts use also a second consolidation
scheme, whose result is averaged with
the result of the Bayesian scheme.
1. Bayesian scheme
2. Canonical Variate scheme
Canonical Variate Analysis (CVA)
A number of statistical techniques involve
calculating linear combinations (weighted sums) of
variables. The weights are defined to achieve
specific objectives:
• PCA – weighted sums maximize variance
• CCA – weighted sums maximize correlation
• CVA – weighted sums maximize discrimination
Canonical Variate Analysis
Let X be a set of centered explanatory variables, with
variance-covariance matrix Sxx. Let Y be a set of (noncentered) indicator variables that define the group
membership of a set of observations, with cross-products
matrix Syy. In our case, membership is with respect to tercile.
Solve the eigenproblem:
S
1
xx

S xy S yy1 Sxy  r 2I a  0 ,
which is identical to the CCA problem, but with Y as the set of
indicator variables rather than continuous values on the
dependent variables.
The loadings, a, maximize ratio of between-group variance to
total variance, represented by the canonical correlation, r. A
discrimination is maximized w.r.t. tercile group membership.
Canonical Variate Analysis
The canonical variates
are defined to
maximize the ratio of
the between-category
(separation between
the crosses) to the
within-category
(separation of dots
from like-colored
crosses) variance.
Conclusion
IRI presently using a 2-tiered prediction system.
It is interested in using fully coupled systems also,
and is looking into incorporating those.
Within its 2-tiered system it uses 4 SST prediction
scenarios, and combines the predictions of 7 AGCMs.
The merging of 7 predictions into a single one uses
two multi-model ensemble systems: Bayesian and
canonical variate. These give somewhat differing
solutions, and are presently given equal weight.