Verification of CFS forecasts Daily

Verification of Daily CFS forecasts
Huug van den Dool & Suranjana Saha
• CFS was designed as ‘seasonal’ system
• Hindcasts 1981-2005, 15 ‘members’ per month
• Here we look at CFS(T62L64) as NWP (never
mind the delayed ocean analysis)
• 25 years of forecasts (4500 forecasts out to 9
months) by a ‘constant’ T62L64 model !!!!!.
Climatology of daily scores
•
•
No ensemble average, no time average
Compare to CDAS & CDC’s MRF
: Cathy Thiaw (reruns)
Acknowledgement
and on CDAS info Bob Kistler/Fanglin Yang/Pete Caplan
Day 5 AC-scores, using the
harmonically smoothed model
and observed climatologies
(which are more ‘competitive’ than the old-old-old climo used on Pete Caplan’s page
Variables:
Extra-tropical Z500 (NH, SH)
PSI200 and CHI200
Annual Mean CFS da
Z500 - NH 1981-2005
Anomaly Correlation
80
70
60
180 forecasts per dot
50
40
30
1981
1983
1985
1987
1989
1991
1993
1995
1997
1999
2001
2003
2005
YEAR
Z500: 72.3 (4500 cases, grand mean)
Congratulations with a
constant system!
From Pete Caplan’s EMC website:
0.7
Grand Mean NH over 1984-2005: 70.5
(CDAS1)
((CDAS statistics has
several problems))
0.7
Annual Mean CFS da
Z500 - NH 1981-2005 (CD
Anomaly Correlation
80
70
60
50
40
30
1981
1983
1985
1987
1989
1991
1993
1995
1997
1999
2001
2003
2005
YEAR
CDAS
Annual Mean CFS da
Z500 - NH 1981-2005
Anomaly Correlation
80
70
60
50
40
30
1981
1983
1985
1987
1989
1991
1993
1995
1997
1999
2001
2003
2005
YEAR
CFS
Next Two Slides: Excursion to SH
Z500
Annual Mean CFS da
Z500 - SH 1981-2005
Anomaly Correlation
80
70
60
50
40
Scores ‘volatile’
in SH (still 180
per dot)
SH (CFS)
30
1981
1983
1985
1987
1989
1991
1993
1995
1997
1999
2001
2003
2005
YEAR
Z500: 62.9 (4500 cases, grand mean)
Scores go UP and UP!
“Congratulations” to
whom ???
Annual Mean CFS d
Z500 - SH 1981-2005 (CD
Anomaly Correlation
80
70
60
50
40
SH (CDAS)
30
1981
1983
1985
1987
1989
1991
1993
1995
1997
1999
2001
2003
2005
YEAR
Doing the best we can:
Comparing CFS to CDAS day 5 Z500 scores)
NH
SH
CFS 1981-2005
T62L64, coupled,
R2
0.723
0.629
CDAS 1984-2005
T62L28,
R1
0.705
0.623
Very flawed
Warning: Number of forecasts per month differ. Climos differ!!! and change in
’96 for CDAS
Prelim Conclusion
• CFS is (slightly) better than CDAS.
• NH much better than SH (typical for pre2000 technology) – this will change
completely in next CFS
• Trend-issues and non-constancy of
system (will get worse)
Next Two Slides: Excursion to
TROPICS
Annual Mean CFS da
PSI200 - TROPICS 1981-2
80
Anomaly Correlation
ENSO?
70
60
180 forecasts per dot
50
40
30
1981
1983
1985
1987
1989
1991
1993
1995
1997
1999
2001
2003
2005
YEAR
PSI200: 63.0 (4500 cases grand mean)
No increase???
Annual Mean CFS da
CHI200 - TROPICS 1981-2
80
Anomaly Correlation
ENSO?
70
180 forecasts per dot
60
50
40
30
1981
1983
1985
1987
1989
1991
1993
1995
1997
1999
2001
2003
2005
YEAR
CHI200: 45.9 (4500 cases grand mean)
This may be lowish for MJO type prediction.
Definitely no increase over
time. Perhaps a decrease!!!
How about Bias Correction???
One of the claimed usages of
hindcasts
Daily CFS Scores
Based on
375
forecasts.
Anomaly Correlation
Z500 1981-2005 NH July
100
Indisputable but
very small
improvement.
80
60
Is Z500
incorrigeable?
40
20
0
0
2
4
raw
6
8 10 12
forecast day
14
bias corrected
The gain due to bias correction in a few selected
months. Day 5 scores Z500 1981-2005
Raw
Bias Corrected
NH DEC
73.0
74.5 (+1.5)
NH JUL
65.8
68.3 (+2.5)
SH FEB
61.8
63.5 (+1.7)
SH AUG
62.2
63.7 (+1.5)
Is Z500 incorrigeable? Largely
Yes, because the systematic
error is small.
Wait till you see CHI200
Daily CFS Scores
Anomaly Correlation
CHI200 1981-2005 TR December
100
Chi200 improves
tremendously from
cleaning up the bias,
especially early on.
80
60
40
20
0
0
2
4
6
8 10 12
forecast day
14
375 forecasts
Can we do
MJO
forecasts ??
raw
bias corrected
Daily CFS Scores
Anomaly Correlation
PSI200 1981-2005 TR February
100
80
60
40
20
0
0
PSI200 in Tropics
does not improve
very much from
bias removal. !!!!
2
4
raw
6
8 10 12
forecast day
14
bias corrected
Distribution of
Skill in space is
important
and largely unexplored
and unexplaind
Skill as a function of (EOF)
mode is interesting
CFS Skill in Daily Z500 pred
Anomaly Correlation
100
80
60
40
20
0
-20
10
8
6
4
2
0
-2
0 10 20 30 40 50 60 70 80 90100
EOF mode #
skill_day-5 EV
EV by Mode (%)
February 1981-2005 NH
100
80
60
40
20
0
-20
February 1981-2005 NH
10
8
6
4
2
0
-2
0 10 20 30 40 50 60 70 80 90100
EOF mode #
skill_day-5
EV
Regular AC
persistence
persistence's regular AC
EV by Mode (%)
Anomaly Correlation
CFS Skill in Daily Z500 pr
Now: OUT TO 270 DAYS !!
100
80
Die-off curve, Z500,
NH, 1981-2005, 4500
forecasts, all seasons
aggregated
60
40
20
0
0 15 30 45 60 75 90105120135150165180195210225240255270
100
1.5
Detail: Month 2 to Month 8
80
1
Detail, first month.
60
NH
40
0.5
0
20
0
-0.5
0
5
10
15
20
25
30
30 45 60 75 90 105120135150165180195210225240255270
100
80
Die-off curve, Z500,
SH, 1981-2005, 4500
forecasts, all seasons
aggregated
60
40
20
0
0 15 30 45 60 75 90105120135150165180195210225240255270
100
2
80
1.5
60
SH
40
1
0.5
20
0
0
0
5
10
15
20
25
30
30 45 60 75 90 105120135150165180195210225240255270
FOCUS
•
•
•
•
•
Week 3
Week 4
= days 15-21 and 22-28
Physical basis of wk3/wk4
Ocean interaction is as much a liability as
a promise at this point in time.
DAY 15
DAY 28
CFS Pers
CFS Pers
NH
15.2 4.2
6.1
TR
36.2 29.3
29.6 27.5
SH
9.6
2.0
6.2
3.9
3.4
Anomaly correlation of CFS and ‘Persistence’ Z500 prediction in Feb 1981-2005
signal
______________________
ratio
NOISE
Improving signal to noise ratio: (often cosmetic) by reducing
noise by applying some operator and, hopefully, not hurting
the signal by this operator
1) Take a time mean (7 days)
2) Ensemble mean (not done here)
3) EOF filters (or better (maximally predictable modes))
Effect of a 7 day mean………
Day 15
Wk3 Wk4
Day 28
NH
15.2 15.7 9.2
TR
36.2 46.7 42.9 29.6
SH
9.6
10.6 5.2
6.1
2.0
Anomaly correlation of CFS Z500 prediction, daily as well as weekly, in Feb 19812005
Effect of EOF truncation……
Wk3
Wk3 WK3 WK4
EOF1 EOF1-10
WK4 WK4
EOF1 EOF1-10
15.7
22.0
23.9
18.7
9.2
12.0
So, overall, we went from daily scores (15.2….6.1) to
weekly scores (15.7 and 9.2) to filtered weekly scores
(22.0 and 23.9 at best)
15.7
9.2
Conclusions
• CFS is (slightly) better than CDAS (Z500).
• Over time (1981-2005), the CFS “system”
appears quite constant with various qualifiers
• If we need an as-constant-as possible in-house
system, look no further (better than CDAS)
• Bias correction: small +ve impact on Z500 and
Ψ200 (1-2pnts)
• Bias correction: large +ve impact on CHI-200
(15 pnts)
• CFS loses 5-20% in terms of SD and eDOF in
the first few days, then, admirably, stays nearly
constant out to 270 days
Conclusions
• (Very) modest skill in wk3 and wk4
• Even with 2 of 3 signal to noise
improvements in place AC is only 0.200.25. (No ensemble averagehere)
• Waiting for the next CFS and CFSRR
(higher Res, consistent IC)
Conclusions:
• The day 1 – 3 forecasts appear to be too
damped, and damp faster than a
regression would. Increasing anomaly
amplitude as a postprocessor (undoing the
sd decrease) actually improves the rms
error early on. A curiosity?
• Probably: initial conditions are damped as
well.
THE REST IS EXTRA
CFS daily scores 1981-200
78
40
76
38
74
36
72
34
70
32
68
30
66
28
64
26
9 10 11 12 13
0
1
2
3
4
5
6
7
8
AC-NH at day 5 degrees of freedom
Does the eDOF variation explain the AC variation????
CFS Skill in Daily Z500 pred
Anomaly Correlation
100
80
60
40
20
0
-20
10
8
6
4
2
0
-2
0 10 20 30 40 50 60 70 80 90100
EOF mode #
skill_day-5 EV
EV by Mode (%)
February 1981-2005 NH
skill_day-10
CFS Skill in Daily Z500 pred
100
80
60
40
20
0
-20
10
8
6
4
2
0
-2
0 10 20 30 40 50 60 70 80 90100
EOF mode #
skill_day-5
EV
skill_day15
skill_day28
skill_day-10
EV by Mode (%)
Anomaly Correlation
February 1981-2005 NH
Anomaly Correlation
Daily CFS Scores
Z500 1981-2005 NH May13/15
hurricane
100
80
60
40
20
0
It is not just
resolution….
Member 5
0
2
4
6
8 10 12
forecast day
14
Only 25 forecasts
T382('hurricane')
CFST62
May13/15 CFS day
Z500 - NH 1981-2005
T382 used
GDAS
Anomaly Correlation
90
80
70
60
50
1981
1983
1985
1987
1989
1991
1993
1995
1997
1999
2001
2003
2005
YEAR
CFS@T382
CFS@T62
RMSE growth CFS
NOSEC
RMSE (gpm)
160
140
120
100
80
60
40
20
0
Savijarvi Eq(5
CFS-jan
0 1 2 3 4 5 6 7 8 910111213141516
lead
Initial error=6.1 gpm ; Systematic error growth s=13.5 gpm/day ; Small error amplification a=0.181/day ;
e-infinity=157.7gpm ; NOSEC
Savijarvi’s equation 5:
RMSE growth CFS
SEC
RMSE (gpm)
160
140
120
100
80
60
40
20
0
Savijarvi Eq(5
CFS-jan
0 1 2 3 4 5 6 7 8 910111213141516
lead
Initial error=4.7 gpm ; Systematic error growth s=13.1 gpm/day ; Small error amplification a=0.181/day ;
e-infinity=155.7gpm ; SEC
RMSE growth CFS
SECSD
RMSE (gpm)
160
140
120
100
80
60
40
20
0
Savijarvi Eq(5
CFS-jan
0 1 2 3 4 5 6 7 8 910111213141516
lead
Initial error=4.7 gpm ; Systematic error growth s=12.6 gpm/day ; Small error amplification a=0.200/day ;
e-infinity=158.3gpm ; SECSD
Example:
Systematic errors
for mid-January at
day 5.
RMSE growth CFS NH 'jan
75
3
65
2
55
1
45
0
35
-1
25
-2
15
-3
0
raw
1
2
sec
3
secsd
4
sec-secsd
Climatological Annual Cycle of day-5
scores
Climo Day 5 Scores byNH:
Month
Best month is Feb (76.4),
Z500 - NH 1981-2005
Worst month is July (67.7). Near
sinusoidal variation.
Range=8.7pnts
Anomaly Correlation
80
70
NH
60
1
2
3
4
5
6
7
8
Month
9
10 11 12
Climo Day 5 Scores by M
Z500 - SH 1981-2005
SH: Best month is Aug (64.9),
Worst month is March (59.0).
Typically 62-64, except Feb-Apr.
Range=5.9pnts
Anomaly Correlation
70
60
SH
50
1
2
3
4
5
6
7
8
Month
Forecasts verifying in month shown
Curious: March is best/worst in NH/SH. Aug is best/worst in SH/NH.
9
10 11 12
375 forecasts per
month
Climo Day 5 Scores by Month
Z500 - NH 1981-2005
Anomaly Correlation
85
The NH has an annual cycle in
skill, every year.
75
(Each color curve is a different year.)
65
55
1
2
3
4
5
6
7
8
Month
9
10 11 12
Climo Day 5 Scores by
Z500 - SH 1981-2005
Anomaly Correlation
75
65
The SH does NOT have a clear
annual variation in skill each year.
55
45
1
2
3
4
5
6
7
8
Month
15 forecasts per month
9
10 11 12
Climo Day 5 Scores by Month
Climo Day 5 Scores by M
Z500 - NH 1981-2005
Anomaly Correlation
70
Anomaly Correlation
80
Z500 - SH 1981-2005
70
60
60
50
1
2
3
4
5
6
7
8
Month
9
10 11 12
1
2
3
4
5
6
7
8
Month
9
10 11 12
Climo Day 5 Scores by Month
Climo Day 5 Scores by M
PSI200 - TROPICS 1981-2005
Anomaly Correlation
55
Anomaly Correlation
75
CHI200 - TROPICS 1981-20
65
45
55
35
1
2
3
4
5
6
7
8
Month
9
10 11 12
1
2
3
4
5
6
7
8
Month
9
forecasts verifying in month shown
375 forecast per dot(month)
Large annual variation in
the tropics! But volatile.
10 11 12
Day 5 scores
• AC (obviously; already shown),
But also:
• SD
• eDOF
CFS daily scores 1981-200
standard deviation (gpm)
120
110
CFS misses 5-20 %
of variability
100
90
80
70
60
d
j
f m a m j
j a s o n d
j
sd (day 5)sd (OBS)
Loss of variance does not increase beyond 10-15 days and is never more than 20%
CFS daily scores 1981-200
effective degrees of freedom
44
42
40
38
36
34
32
30
28
26
CFS misses several
degrees of freedom
d
j
f m a m j
j
a s o n d
j
deg of freedom (OBS)degrees of freedom
Loss of dof does not increase beyond 10-15 days and is never more than shown above
CFS daily scores 1981-2005 Z500 NH
standard deviation (gpm)
120
110
100
90
80
70
60
d
j
f m a m j
j a s o n d
Compare the SD reduction
in the
j
sd (day 5)sd (OBS)
NH
CFS daily scores 1981-2005 Z500 SH
To that in the
standard deviation (gpm)
120
SH
110
100
90
80
70
60
d
j
f m a m j
j a s o n d
sd (day 5)sd (OBS)
j
CFS daily scores 1981-2005 Z500 NH
effective degrees of freedom
44
42
40
38
36
34
32
30
28
26
d
j
f m a m j
j
a s o n d
j
Compare the DOF
reduction in the
deg of freedom (OBS)degrees of freedom (day 5)
NH
CFS daily scores 1981-2005 Z500 SH
To that in the
effective degrees of freedom
36
SH
34
32
30
28
26
d
j
f m a m j
j
a s o n d
j
deg of freedom (OBS)degrees of freedom (day 5)