A Closer Look at Results from the Program for International

A Closer Look at Results
from the Program for International
Assessment of Adult
Competencies (PIAAC)
Colorado Adult Education and Family Literacy
Directors’ Conference
July 17, 2014
Dr. Sondra Stein
AIR/NCES PIAAC Outreach Project
What is PIAAC?
2
About
PIAAC
PIAAC is an international large-scale assessment
administered in 2011-12 in 23 countries
It assessed 16 - to 65-year-olds, non-institutionalized, residing in each
country, irrespective of nationality, citizenship, or language status
Laptop computer In the U.S., 80% took the computer tests
or paper-andand 15% took the paper-and-pencil tests.
pencil:
Assessment
subjects:
Literacy
Numeracy
Problem Solving in Technology-Rich
Environments (digital problem solving)
Assessment
was conducted
only in English
in the U.S.:
The background survey was conducted in English
or Spanish. About 4% could not complete the BQ
because of language difficulties or learning or
mental disabilities, and 1% could not complete it
for other reasons.
3
Participating Countries
2012
Australia
Austria
Belgium
Canada
Cyprus
Czech Republic
Denmark
Estonia
Finland
France
Germany
Ireland
Italy
Japan
Korea, Rep of
Netherlands
Norway
Poland
Slovak Republic
Spain
Sweden
United Kingdom
United States
2015
Chile
Greece
Indonesia
Israel
Lithuania
New Zealand
Singapore
Slovenia
Turkey
4
The Direct Assessment focuses on four domains:
 Literacy: both paper & pencil and computer versions
 Numeracy: both paper & pencil and computer versions
 Problem solving in technology-rich environments: only on
computer
 Reading components: only paper & pencil
 All countries were required to administer literacy and numeracy
assessments
 The U.S. assessed all four domains
6
PIAAC Background Questionnaire
Focused on identifying:
 Skills that are critical to functioning successfully in today’s society,
 How participants acquire those skills, and
 How those skills are distributed.
Areas of BQ include:
 Education and training, present and past,
 Work experience,
 Skills used for work and outside of work,
 Literacy, numeracy and ICT skill use at work and at home,
 Personal traits, and background information.
7
Module on Skill Use
Cognitive skills
reading, writing, mathematics, and use of
information and communication technologies
Interaction and social skills
collaboration and co-operation, work
and time planning, communication
and negotiation, and customer
contact
Learning skills
Physical skills
use of gross and
fine motor skills
coaching, formal/informal
learning and updating
professional skills
8
PIAAC provides a rich source of data that tells
us:
 What skills adults actually have and can use rather
than just the number of years of education they have
completed or the degrees they have.
 How adults acquire those skills, and what factors are
related to skill acquisition and decline.
 What the level and distribution of skills is within and
across various subgroups within the population.
 Where we can focus our efforts in order to raise the
skills of adults with the greatest needs.
9
Overview of U.S. Results
PIAAC results tell a story about the systemic nature of the
skills deficit among U.S. adults. PIAAC raises the question:
What are we going to do to make sure that
 The U.S. has the workforce it needs to succeed in the
global economy?
 U.S. citizens have the skills necessary to support a
thriving democracy?
How did we do compared to other countries?
Literacy
Japan
Finland
Netherlands
Australia
Sweden
Norway
Estonia
Flanders-Belgium
Czech Rep.
Slovak Rep.
Canada
Korea, Rep. of
U.K.
Denmark
Germany
United States
Austria
Cyprus
Poland
Ireland
France
Spain
Italy
Numeracy
Japan
Finland
Flanders-Belgium
Netherlands
Sweden
Norway
Denmark
Slovak Rep.
Czech Rep.
Austria
Estonia
Germany
Australia
Canada
Cyprus
Korea, Rep. of
U.K.
Poland
Ireland
France
United States
Italy
Spain
PS-TRE
Japan
Finland
Australia
Sweden
Norway
Netherlands
Austria
Denmark
Czech Rep.
Korea, Rep. of
Germany
Canada
Slovak Rep.
Flanders-Belgium
U.K.
Estonia
United States
Ireland
Poland
Italy
Spain
Cyprus
France
• The U.S. ranked lower
than most other
countries in all three
domains.
• The U.S. ranked better
in Literacy than in
Numeracy or Problem
Solving in technologyrich environments.
11
Literacy
Japan
Finland
Netherlands
Australia
Sweden
Norway
Estonia
Flanders-Belgium
Czech Rep.
Slovak Rep.
Canada
Korea, Rep. of
U.K.
Denmark
Germany
United States
Austria
The U.S. average literacy score (270) was
lower than the international average (273).
• Scores on literacy ranged from
296 (Japan) to 250 (Italy)
• U.S. scores were:
• Lower than in 12 countries
• Not significantly different than
in 5 countries
• Higher than in 5 countries
Cyprus
Poland
Ireland
France
Spain
Italy
12
These descriptions of the PIAAC Proficiency
Levels for Literacy define what adults can do at
each level.
Level 4
Below
Level 1
(0-175)
Locate
single piece
of
information
in familiar
texts.
Level 1
(176-225)
Read
relatively
short
digital,
print or
mixed
texts to
locate
single text.
Level 2
(226-275)
Make matches
between text
and
information
that may
require low
level paraphrasing and
drawing lowlevel
inferences.
Level 3
(276-325)
(326-375)
Perform
multiple-step
Identify,
interpret, or operations to
integrate,
evaluate
one or more interpret, or
synthesize
pieces of
information information
from
and often
complex
require
texts, and
varying
may require
levels of
complex
inference.
inferences.
Level 5
(376-500)
Integrate
information
across
multiple,
dense texts;
construct
syntheses,
ideas or
points of
view; or
evaluate
evidence
based
arguments.
13
The U.S. average is low because a higher proportion of
U.S. adults are at the bottom levels (level 1 and below
level 1) of literacy.
14
In literacy, U.S. gaps larger by educational attainment and
skill level of job, but similar to international average by
income and employment status
17
U.S. gaps in literacy scores larger than international
average by parental education and nativity status
18
Low-skilled adults in the U.S.
19
Literacy skill descriptions at the lower levels
0
Below Level 1 175
Can read brief texts on
familiar topics to locate a
single piece of specific
information. Only basic
vocabulary knowledge is
required, and the reader
is not required to
understand the structure
of sentences or
paragraphs.
Level 1
225
Can read relatively short
digital or print continuous,
non-continuous, or mixed
texts to locate a single
piece of information.
Knowledge and skill in
recognizing basic
vocabulary determining the
meaning of sentences, and
reading paragraphs of text
is expected.
Level 2
275
Can integrate two or
more pieces of
information based on
criteria, compare and
contrast or reason
about information and
make low-level
inferences. Navigate
within digital texts to
access and identify
information.
20
How many low-skilled adults in the U.S.?
33% at level 2 times 206 M adults 16-65 = ~68 million
14% at level 1
~29 million
4% below level 1
~8 million
4% literacy-related non-response
~8 million
21
Who are the low-skilled adults in the U.S. in literacy?
Educational attainment
Percentage of population
45
40
35
30
25
20
15
10
5
0
Grad or prof.
degree (10%)
Bachelor’s
degree (16%)
Below Level 1
Associate’s
degree (9%)
Level 1
High school
credential (50%)
Below high
school (15%)
Level 2
22
Who are the low-skilled adults in the U.S. in literacy?
Age
Percentage of population
40
35
30
25
20
15
10
5
0
16-24 (18%)
25-34 (20%)
Below Level 1
35-44 (20%)
Level 1
45-54 (22%)
55-65 (19%)
Level 2
23
United States
55-65
45-54
35-44
25-34
16-24
263*
266
273*
275*
272*
PIAAC
international
average
Only oldest U.S. adults outperformed the
international average in literacy
55-65
45-54
35-44
25-34
16-24
255
268
279
284
279
0
50
100
150
200
250
*p < .05. U.S. average score is significantly different from PIAAC international average.
300
Who are the low-skilled adults in the U.S. in literacy?
Race/ethnicity
Percentage of subpopulation
45
40
35
30
25
20
15
10
5
0
White, non-Hispanic Black, non-Hispanic
(65%)
(13%)
Below Level 1
Hispanic (14%)
Level 1
Other (8%)
Level 2
25
Who are the low-skilled adults in the U.S. in literacy?
Immigration status
Percentage of subpopulation
40
35
30
25
20
15
10
5
0
Born in U.S. (85%)
Below Level 1
Not Born in U.S. (15%)
Level 1
Level 2
26
Who are the low-skilled adults in the U.S. in literacy?
First language
Percent of subpopulation
40
35
30
25
20
15
10
5
0
English as first language (85%)
Below Level 1
Other as first language (15%)
Level 1
Level 2
27
Stubborn Skills Gap in America’s Workforce
“…the skill level of the American labor force is not merely
slipping in comparison to that of its peers around the world,
it has fallen dangerously behind.”
--Eduardo Porter,
New York Times, October 8, 2013
Mean task input in percentiles of 1960 distribution
Since 1970, there has been a shift in the U.S. economy
away from routine and manual tasks and towards more
analytic and interpersonal tasks in occupations
70.0
65.0
60.0
55.0
Routine manual
Non-routine manual
Routine cognitive
50.0
Non-routine analytic
Non-routine interpersonal
45.0
40.0
35.0
1960
1970
1980
1990
2000
2006
2009
Source: Autor, D. H. and B.M. Price (2013), "The Changing Task Composition of the US Labor Market:
An Update of Autor, Levy, and Murnane (2003)", MIT Mimeograph, June.
Percentage of population
Who are the low-skilled adults in the U.S. in literacy?
Employment status
45
40
35
30
25
20
15
10
5
0
Below Level 1
Level 1
Level 2
30
Who are the low-skilled adults in the U.S. in literacy?
Occupation
50
Percent of subpopulation
45
40
35
30
25
20
15
10
5
0
Elementary
occupations
(8%)
Machine
operators
(6%)
Crafts and
Service
Clerks (8%)
trades
workers/sales
workers (9%)
(21%)
Below Level 1
Level 1
Technicians Legislators, Professionals
and
officials, and
(21%)
associates
managers
(16%)
(10%)
Level 2
31
Percentage of subpopulation
Who are the low-skilled adults in the U.S. in literacy?
Industry
50
45
40
35
30
25
20
15
10
5
0
Below Level 1
Level 1
Level 2
32
Raising skills through education and training can make a difference since
participation rates in adult education and training are higher in the U.S. than
in most countries. This is true for adults at all skill levels although, as
elsewhere, low-skilled adults are less likely to participate.
When we think about the future of the U.S. workforce, these results raise
even greater concern: while U.S. 55-65 year olds scored higher in
literacy than their peers in many countries, our young people are not
moving ahead as fast as those in Germany, France, Finland and Korea.
For more information on PIAAC visit us at
www.piaacgateway.com
35
PIAAC Outreach Toolkits
These Outreach Toolkits include all documents and power point
modules you will need to share information about PIAAC with others.
*Simply download the pieces you want to use!*
Toolkit
Documents
•
PIAAC
Overview
•
Key Results
•
National
Supplement
•
Education &
Skills Online
(ESO)
•
Videos
•
Infographics
Toolkit Slides for Your Presentation
Instructions: Start with “What is PIAAC” and “Results Overview”.
Then add other modules to build a presentation that suits your
audience.
What is
Key U.S. Issues
Results
Gateway
PIAAC
• Low-skilled Workers
Overview
& Other
•The Future Workforce
Resources
•Health Status & Skills
These can be
Sample
downloaded
•The Impact of Parent
PIAAC
with a focus on
Education
National
Tasks
literacy,
Supplement
(Each of these can be
numeracy, or
digital problemsolving
downloaded with a focus
on literacy, numeracy or
digital problem-solving)
ESO
What you can find on the PIAAC Gateway
www.piaacgateway.com
Links to everything you want to know about PIAAC,
including:
•
•
•
•
•
•
The latest PIAAC reports and presentations
A calendar of PIAAC-related events
Infographics, brochures, and videos to share
Data tools and training resources
Press coverage from the U.S. and around the world
Links to release events, reports, and presentations
37
For more information about PIAAC
Visit our website at https://piaacgateway.com
Or contact us at [email protected]
Sign up for our regular newsletter, the PIAAC Buzz at
https://piaacgateway.com
Extra Slides
The PIAAC Assessment was delivered to a
nationally representative sample of households
in every country.
• In the U.S. the household sample was selected through a 4-stage
stratified area sample:
– Counties (Primary Sampling Units)
– Blocks
– Housing units with households
– Eligible persons within households
• Resulted in 5,010 respondents
• A U.S. supplement will add 3,600 more adults that represent key
populations (young adults 16-34 yrs, older adults 66-74 yrs,
unemployed adults,16-65 yrs)
• A representative prison sample will include 1,200 inmates, 16-74
yrs, in state, federal and private prisons.
40
PIAAC collects data through it’s background
questionnaire and module on skill use as well as
through direct assessment of skills.
Background
questionnaire
Module on
skill use
Direct
assessment
of key
informationprocessing
skills
41
These descriptions of the PIAAC Proficiency
Levels for Numeracy define what adults
Level 5
can do at each level.
(376-500)
Perform
analysis,
Understand complex
Level 1
& work with reasoning,
Perform 2 or
(176-225)
mathematical statistics and
more
patterns,
chance;
calculations,
Perform oneproportions, spatial
simple
step tasks:
relationships;
measurement; basic
count; sort;
statistics
and
spatial
arithmetic
representation; expressed in communicaoperations;
ting wellunderstanding estimation; and verbal or
reasoned
simple percent interpret simple numerical
explanations
tables, graphs. form.
(ex. 50%).
for answers.
Level 2
(226-275)
Below
Level 1 (0175)
Perform
basic tasks:
counting,
arithmetic
operations
with whole
numbers.
Level 3
(276-325)
Level 4
(326-375)
Understand
complex
abstract
mathematical and
statistical
ideas,
embedded
in complex
texts, draw
inferences;
arguments
or models;
justify,
reflect on
solutions or
choices.
42
The U.S. average digital problem solving* score (277) was
lower than the international average (283)
PS-TRE
Japan
Finland
Australia
Sweden
Norway
Netherlands
Austria
Denmark
Czech Rep.
Korea, Rep. of
Germany
Canada
Slovak Rep.
Flanders-Belgium
U.K.
Estonia
United States
Ireland
Poland
Italy
Spain
Cyprus
France
• Italy, Spain, Cyprus and France did
not include this domain in their
assessment
• Scores ranged from 294 (Japan) to
275 (Poland)
• U.S. scores were:
• Lower than in 14 countries
• Not significantly different than in 4
countries
• Higher than no other country
* Officially problem solving in technology-rich environments (PS-TRE)
43
A higher proportion of U.S. adults are at the
bottom levels of problem solving in
technology-rich environments
These descriptions of the PIAAC proficiency levels
for Digital problem solving define what adults
can do at each level.
Level 3
(341-500)
Below Level 1
(0-240)
Tasks are welldefined
involving use
of only one
function within
a generic
interface.
Level 1
(241-290)
Tasks require
little or no
navigation, and
only a few steps
to access
information for
solving the
problem. There
are few
monitoring
demands.
Level 2
(291-340)
Tasks require
some navigation
across pages and
applications for
solving the
problem.
Evaluating the
relevance, some
integration and
inferential
reasoning may be
needed.
Task may involve
multiple steps
and operators,
navigation across
pages and
applications.
There are
typically high
monitoring
demands, and
evaluation of
relevance and
reliability of
information.
45
The U.S. average literacy score in 2012 is not significantly
different from 2003, but both are lower than in 1994.
285
280
275
IALS:
273*
ALL:
268
270
PIAAC:
270
265
260
255
250
245
1994
2003
*p < .05. Average score is significantly different from PIAAC.
2012
46
The U.S. average numeracy score in 2012 is lower than
in 2003.
275
270
265
ALL:
262*
260
PIAAC:
253
255
250
245
240
235
2003
*p < .05. Average score is significantly different from PIAAC.
2012
47
U.S. adults at every income level scored lower in
numeracy than the international average
United States
Upper middle quintile
PIAAC international
average
Top quintile
Top quintile
293*
274*
Middle quintile
258*
Lower middle quintile
235*
Bottom quintile
245*
302
Upper middle quintile
286
Middle quintile
273
Lower middle quintile
260
Bottom quintile
263
0
50
100
150
200
250
*p < .05. U.S. average score is significantly different from PIAAC international average.
300
350
48
U.S. adults in every age group scored below the international
average for their age group in numeracy
250*
United States
247*
45-54
16-24
249*
PIAAC international
average
55-65
55-65
253
35-44
258*
25-34
260*
45-54
265
35-44
275
25-34
279
16-24
271
0
50
100
150
200
250
*p < .05. U.S. average score is significantly different from PIAAC international average.
300
49
PIAAC
International
average
United States
Young U.S. adults (16-24) performed lower in digital
problem solving than the international average of
their peers. Older U.S. adults (55-65) performed
higher than their peers.
55-65
45-54
35-44
25-34
16-24
267*
271
279*
283*
285*
55-65
45-54
35-44
25-34
16-24
259
272
285
295
295
0
50
100
150
200
250
300
*p < .05. U.S. average scores are significantly different from PIAAC international average.
350
50
Average literacy scores by parents’
education level
OECD average
United States
Neither parent has attained a high school credential (18%)
233*
At least one parent has attained a high school credential (44%)
270*
At least one parent has attained a college degree or higher
(38%)
290*
Neither parent has attained a high school credential (36%)
255
At least one parent has attained a high school credential (39%)
278
At least one parent has attained a college degree or higher
(25%)
295
0
50
100
150
200
250
300
350
*p < .05. U.S. average score is significantly different from OECD average.
51
In numeracy, the gaps in the U.S. are larger by
educational attainment, income, and by occupation.
Note that though the size of the gap is similar for
employment status, the U.S. scores are lower.
52
Only 18% of U.S. adults scored below level 2 in literacy – considerably better
than in numeracy. But there are still too many working adults whose literacy
skills (47%) do not adequately equip them for the increasing demands of the
workplace.
U.S. Pop. 16-65 = approx. 200
million
All U.S. 16-65
Employment status
Employed full-time
Employed part-time
Unemployed
In school
Retired
Permanently disabled
Looking after family
Other
Percent of subpopulation
Percent of Below
population Level 1 Level 1 Level 2
100
4
14
33
52
12
8
10
4
5
6
3
4
4
5
2
3
14
3
5
12
15
24
10
16
29
14
12
31
36
41
35
36
40
38
35
This slide lets us compare the skills of working adults with those of all U.S.
adults. Given that 30% of the adult population in the US scored below level 2 in
numeracy, and another 34% scoring no higher than level 2, we find that over
half of adults working full-time (57%) have low skills in numeracy.
U.S. Pop. 16-65 = approx. 200
million
All U.S. 16-65
Employment status
Employed full-time
Employed part-time
Unemployed
In school
Retired
Permanently disabled
Looking after family
Other
Percent of subpopulation
Percent of Below
population Level 1 Level 1 Level 2
100
10
20
34
52
12
8
10
4
5
6
3
7
9
18
6
10
28
9
10
17
22
32
21
21
28
22
18
33
36
34
37
36
31
39
34