Modernization of Statistical f Information systems initiatives

Modernization of Statistical
Information
f
systems
Global initiatives
Eric Hermouet, Statistics Division, ESCAP
1
Presentation




Why modernization?
What is modernization?
How: Global initiatives and collaboration mechanisms
Main outputs of those mechanisms
1
Modernization
 Modernization of statistical information systems
 Industrialization of statistics
 Modernization of statistical production and services
The data deluge
The internet had
1800 exabytes of
data in 2011
Exabyte=1018
Or 1 million terabytes
y
2
The data deluge
50 000 exabytes by 2020
Even if only 0.1% of
these data is useful, that
leave millions of
terabytes for possible
statistical purposes
New data providers
 Google
– Real-time price indices
– Public data explorer
– First point of reference for the “data generation”
 Facebook
 Telecom companies
phones in Asia and the Pacific
– 4 billion mobile p
 How can statistical offices access and use those data sources?
3
Changing user expectations
 Expectation of data available at a faster rate
– Real-time
 Ability to customize datasets
– Linking coherently datasets across domains
– With a high degree of detail
 Data presentation addressing different target groups
– Governments
– General public
IT progress
 IT developments offers new solutions
– In processing huge amounts of data
– In accessing new sources of data
 In a more efficient way
– In terms of speed
– In terms of financial costs
4
So what is modernization?




Common generic processes
Common tools
Common methodologies
Recognizing that all statistics are produced in a similar way
– No special domains
y to adapt
 Increased flexibility
– To access new data sources
– To generate new statistical products
5
A shared production environment
From
To
Custom design
Use of information bank and
modular design
Built solutions
Assembled solutions
Design for a collection mode
Source independent design
“designed-in” quality
Working with existing data and
varying level of quality
Survey cycles with clear start and
end points
Continuous approach of ongoing collection, processing,
and release
A shared production environment
From
To
Direct data collection
supplemented with data from
administrative source
Tapping into existing data,
data
using direct data collection to
link sources and bridge gaps
Individually crafted data
structure
Management of data
Use of agreed standard
approaches
Management of data,
metadata, and paradata
Smaller field workforce with
specialized interviewing skills
A large field workforce
6
A shared production environment
From
To
Spending effort and resources
on collection and processing
More emphasis on specifying
needs, design, and analysis
Understanding user needs and Understanding data
designing collection
characteristics and negotiating
instruments
solutions that bridges gaps
between existing data and
user requirements
What is modernization and why is it
needed?
Questions so far ?
14
7
Global initiatives & collaboration mechanisms
15
Global initiatives & collaboration mechanisms
 High-Level Group for the Modernization of Statistical
Production and Services (HLG)
 Formerly HLG-BAS
 Objectives
– To promote common standards, models, tools and
methods to support the modernization of official statistics;
– To drive new developments in the production,
organization and
d products
d
off official
ff
l statistics, ensuring
effective coordination and information sharing within
official statistics, and with relevant external bodies;
8
HLG










Gosse van der Veen (Netherlands) - Chairman
Brian Pink (Australia)
Eduardo Sojo Garza-Aldape (Mexico)
Enrico Giovannini (Italy)
Mr Park Hyungsoo (Republic of Korea)
Irena Križman (Slovenia)
Katherine Wallman ((United States))
Walter Radermacher (Eurostat)
Martine Durand (OECD)
Lidia Bratanova (UNECE)
HLG reference documents
 Strategic vision
– http://www1.unece.org/stat/platform/display/hlgbas/S
http://www1 unece org/stat/platform/display/hlgbas/S
trategic+Vision
 Strategy to implement the vision of the HLG
– http://www1.unece.org/stat/platform/display/hlgbas/
HLG+Strategy
9
Technical groups - MSIS
 Management of Statistical Information Systems (MSIS)
– Objectives:
j
• To provide, through regular meetings and other means such as the
MSIS Wiki, a forum for exchange of experiences and good practices
among information systems managers from national and
international statistical organizations.
• To contribute to the coordination of activities of different national
and international organizations in the area of statistical information
y
systems.
• To facilitate and encourage implementation of international
standards and recommendations in the field of statistical
computing among national and international statistical
organizations.
Technical groups - MSIS
 Annual meetings jointly organized by UNECE, Eurostat,
OECD. And ESCAP for this 2013 MSIS
– MSIS meetings consider issues related to information
technology governance and management, system
architecture, accessibility and usability.
– First meeting in 2000
– Secretariat: UNECE
 Open
O
to all
ll UN countries and
d internationall organizations
 Reports to the Conference of European Statisticians
10
Technical Group - METIS
 Steering Group on Statistical Metadata – METIS
– Objectives:
• Promote
P
t th
the iimplementation
l
t ti off metadata
t d t systems
t
b
by d
developing
l i
advocacy targeting the senior management level and subjectmatter staff of NSOs;
• Oversee the maintenance of the Common Metadata Framework
(CMF) directing it towards a practical guide serving national
statistical offices;
• Facilitate collection, discussion and dissemination of best practices
in the field of statistical metadata
– Established in 1990
– Work sessions every 2-3 years
– Workshops in-between
– Organized with Eurostat / OECD
– Open to all UN member countries and international organizations
Technical Groups - SAB
 Sharing Advisory Board
– Objectives
• To promote harmonization of business and information
systems architectures;
• To support collaboration for the development of
statistical software
• To provide guidelines and tools to assess new statistical
software
f
tools
l and
d components
• To assist in the improvement of the technical statistical
infrastructure of countries both within and outside the
UNECE region as required.
11
Technical groups - others
– Statistical Data editing
– Statistical Data collection
 Not directly overseen by HLG
– Statistical Network
– SDMX Expert Group
– Statistical Open Standards Group
gg
group
p on q
quality
y in statistics
– Working
– ….
Main outputs
 Emerging “statistical industry” standards
– GSBPM
– GSIM
– SDMX
– DDI
12
GSBPM
 Generic Statistical Business Process Model
– To define and describe statistical processes in a coherent
way
– To standardize process terminology
– To compare and benchmark processes within and between
organisations
– To identify synergies between processes
– To inform decisions on systems architectures and
organisation of resources
GSBPM
 Applicability
– All activities undertaken by producers of official statistics
which result in data outputs
– National and international statistical organizations
– Independent of data source, can be used for:
• Surveys / censuses
• Administrative sources / register-based
g
statistics
• Mixed sources
13
GSBPM
 Not a linear model
 Sub-processes do not have to be followed in a strict order
 It is a matrix, through which there are many possible paths,
including iterative loops within and between phases
 Some iterations of a regular process may skip certain subprocesses
14
GSBPM
 Examples of use
– Harmonizing statistical computing systems
– Facilitating sharing of statistical software
– Framework for process quality management
– Structure for storage of documents
– Measuring operational costs
15
GSIM
 GSIM contains objects which
specify information about the
real world – 'information objects‘
– Examples include data and
metadata (such as
classifications) as well as the
rules and parameters needed
for production processes to
run (for example, data
editing rules).
rules)
 GSIM identifies around 150
information objects, which are
grouped into four top-level
groups
GSIM
 Generic Statistical Information Model
 To describe data and metadata objects and flows within the
statistical business process
 Implementation of GSIM, in combination with GSBPM, allow
– Creating an environment to prepare for reuse and sharing
of methods, components and processes;
– Implementing rule based process control, thus minimizing
h
human
intervention in the
h production
d
process;
– Economies of scale through development of common tools
by the community of statistical organizations.
16
SDMX and DDi
 Standard Data and Metadata eXchange
– UNSD/DFID project in Cambodia
Cambodia, Laos
Laos, Thailand
Thailand, Viet
Nam
 Data Documentation Initiatives
– World Bank Microdata Management Toolkit
SDMX and DDI
17
What’s next?
 Launch of “Plug and Play” project – February 2013
– To create a common statistical production architecture
– To create a standardized architecture for statistical
production solutions, including processes, information and
systems, coherent with the Generic Statistical Business
Process Model (GSBPM) and the Generic Statistical
Information Model (GSIM),
– To enable and advance the sharing of production processes
or components,
t th
thus reducing
d i costs.
t
– To provide the basis for a central inventory or repository
with life cycle management of sharable production
processes and components.
18