PRAXIS II Determining Cut Scores for Content Exams Jami Goetz, Ph.D., CDE

PRAXIS II
Determining Cut Scores for Content Exams
a presentation to the Colorado State Board of Education
Jami Goetz, Ph.D., CDE
Terry Owens, Ph.D., Educational Testing Service
June 12, 2013
Recap of Last Month’s Decision
 At its May 2013 meeting, the Colorado State Board of
Education voted to replace two PRAXIS II content exams
(which are being removed from circulation) with two revised,
more relevant and more rigorous content exams, aligned to
the current academic standards for each content area:
 Mathematics: Content Knowledge #0061/#5061
Mathematics: Content Knowledge #5161
 English Language Arts, Literature & Composition #0041/#5041
English Language Arts: Content Knowledge #5038
2
About the Content Tests
 These content exams are designed to measure whether
prospective or entry-level educators have the standardsrelevant content and pedagogical knowledge, skills and
abilities believed necessary for competent professional
practice.
 These content exams are not designed to assess or determine
efficacy of teaching practice.
3
Cut Score Determination – Math
#5161
 To support the decision-making process of establishing
passing or cut scores, Educational Testing Service (ETS) designs
and conducts a multistate, standard-setting study for each
exam.
 For the PRAXIS II Mathematics: Content Knowledge #5161,
states and jurisdictions that currently use PRAXIS were invited
to recommend panelists with
 Experience as math teachers/college faculty who prepare math
teachers and
 Familiarity with the knowledge and skills required of beginning
math teachers
4
Cut Score Determination – Math
#5161
 24 states and Washington, D.C., were represented by 35
panelists:
Alaska (1)
Louisiana (1)
New Hampshire (2)
Rhode Island (1)
Arkansas (2)
Maine (1)
New Jersey (2)
South Carolina (1) Washington, D.C. (1)
Delaware (1)
Maryland (2)
North Carolina (2)
South Dakota (1)
West Virginia (1)
Idaho (2)
Mississippi (2)
North Dakota (1)
Tennessee (1)
Wisconsin (2)
Kentucky (1)
Nevada (1)
Pennsylvania (1)
Utah (2)
Wyoming (2)
5
Vermont (1)
Cut Score Determination – Math
#5161
 Participants were divided into two panels. After each
panelist took the test, they discussed it to arrive at a shared
understanding of what the test does and does not cover.
 Panelists were asked to remark on any content areas that
would be particularly challenging for entry-level teachers
and on areas that address content particularly important for
entry-level teachers to know.
 Panelists also determined the knowledge/skills that
differentiate a “just qualified” from a “not-quite qualified”
candidate.
6
Cut Score Determination – Math
#5161
 In Round One, panelists rated each question on a scale of
0 to 1.0 to indicate the likelihood that an entry-level teacher
candidate would answer the item correctly. (The lower the
value, the less likely the candidate would answer the item
correctly.)
 In Round Two, panelists discussed the rationale for their
ratings and were allowed to adjust their ratings if desired.
 Results of one panel were not discussed with the other panel.
 This process – known as a modified Angoff method– is both
psychometrically advanced and legally defensible.
7
Mathematics: Content Knowledge
#5161
 150 minutes
 60 selected response questions
 Number and Quantity, Algebra, Functions, Calculus
~ 41 questions / ~68 % of exam
 Geometry, Probability and Statistics, Discrete Mathematics
~ 19 questions / ~32 % of exam
 Panelists’ Recommended Cut Score: 160
Based on a recommended passing raw score of 32 (out of a possible 50).
When placed on a 100-200 scale, a raw score of 32 associates with a scaled
score of 160.
8
Mathematics: Content Knowledge
#5161
 To date, Mathematics #5161 has been adopted by 25 states
(including Guam and Washington, D.C.):
Alabama, Alaska, Arkansas, Colorado, Delaware, District of Columbia,
Guam, Hawaii, Idaho, Iowa, Kansas, Maine, Maryland, Montana, Nebraska,
Nevada, New Hampshire, New Jersey, North Dakota, Pennsylvania, South
Dakota, Utah, Virginia, West Virginia, and Wyoming
 12 adopted the MSSS panels’ recommended score of 160 (one
is pending final approval)
 12 states will determine their scores at upcoming board
meetings
 1 state adopted “NM” (no minimum score)
9
Cut Score Determination – English
#5038
 For the PRAXIS II English Language Arts: Content Knowledge
#5038, states and jurisdictions that currently use PRAXIS were
invited to recommend panelists with
 Experience as either English teachers of college faculty who
prepare English teachers and
 Familiarity with the knowledge and skills required of beginning
English teachers
10
Cut Score Determination – English
#5038
 22 states, Washington, D.C., and Guam were represented by
38 panelists:
Alaska (2)
Idaho (1)
Montana (2)
Pennsylvania (1)
Utah (2)
Arkansas (1)
Kansas (1)
Nevada (1)
Rhode Island (2)
Washington, D.C. (1)
Delaware (2)
Louisiana (1)
New Jersey (2)
South Carolina (1)
West Virginia (2)
Guam (1)
Maine (2)
North Carolina (2)
South Dakota (2)
Wisconsin (2)
Hawaii (1)
Mississippi (2)
North Dakota (2)
Tennessee (2)
11
Cut Score Determination: English
#5038
 Again, a modified Angoff method was employed: Two panels
of participants took the test, discussed it to arrive at a shared
understanding of what the test does and does not cover, and
remarked on what might be challenging for entry-level
teachers, as well as on what content was particularly
important for entry-level teachers to know.
 Panelists also determined the knowledge/skills that
differentiate a “just qualified” from a “not-quite qualified”
candidate, and rated each item on the 0 to 1.0 scale.
 Neither panels’ results were shared with the other panel.
12
English Language Arts:
Content Knowledge #5038
 150 minutes
 130 selected response questions
 Reading
 Language Use and Vocabulary
~49 questions / ~38 % of exam
~33 questions / ~25 % of exam
 Writing, Speaking and Listening
~48 questions / ~37 % of exam
 Panelists’ Recommended Cut Score: 167
Based on a recommended passing raw score of 79 (out of a possible 110).
When placed on a 100-200 scale, a raw score of 79 associates with a scaled
score of 167.
13
English Language Arts:
Content Knowledge #5038
 To date, English Language Arts #5038 has been adopted by 19
states (including Guam and Washington, D.C.):
Alabama, Alaska, Colorado, Delaware, District of Columbia, Guam,
Hawaii, Idaho, Kansas, Maine, Montana, Nevada, New Jersey, North
Dakota, Pennsylvania, South Dakota, Utah, Virginia, and West Virginia
 7 adopted the MSSS panels’ recommended score of 167
 1 state adopted a passing score of 162
 11 states will determine scores at their upcoming board
meetings
14
Review and Re-Setting Cut Scores
 Data showing state and national passing rates for these tests,
as well as the mean, median and range of these rates, will be
available a few months after the testing cycle starts this fall.
 Colorado-specific data will be available as soon as a
statistically reliable sample size is obtained.
 CDE staff will analyze pass rate data for the new exams when it
becomes available in late fall 2013, and inform the state board
accordingly, making additional recommendations as necessary.
15
Recommendation
Colorado Department of Education staff recommend the
Colorado State Board of Education adopt the cut scores
recommended by the MSSS panels for Mathematics #5161
and English Language Arts #5038.
Please note: Once an adequate number of test takers have taken
the test to provide impact data, the Colorado State Board of
Education may review and reset cut scores as it deems
necessary at a subsequent board meeting.
16
In Closing …
 The “outgoing” Math #0061 and English #0041 (paper) will be
offered one last time in Colorado at select testing centers on
July 20, 2013. Computer-based versions (Math #5061 and
English #5041) will be offered during the July 29 – Aug. 17,
2013 testing window.
 The “new” Mathematics #5161 and English Language Arts
#5038 exams – both of which are computer-based – will
replace these exams beginning in fall 2013.
17