Assessment Instrument Description: Galileo

AssessmentInstrumentDescription:Galileo
Element Instrument Name Vendor Purpose (Intended Use) Population When? How frequently? Content Area (s) Description Name of specific instrument (more than vendor name). Name of the company or organization that produces the instrument. The described purpose and appropriate uses of the instrument. Who (which students) could be assessed using the instrument. How frequently the instrument can be administered in a school year, and recommended or required administration windows. Content area or areas being assessed. Assessment Instrument Information Galileo K‐12 Online Assessment Technology Incorporated (ATI) Galileo K‐12 Online is a fully integrated, web‐based instructional improvement and instructional effectiveness system, intended to predict performance on the state’s annual achievement test. Assessments resources included in the Galileo system are aligned to the Colorado Academic and Common Core State Standards. Preschool Assessment: Birth‐5 years K‐12 Assessment: 6‐18 years ATI’s online Benchmark Planner allows the district to determine how many benchmark assessments they will administer during the school year and when benchmark testing will occur, which learning standards will be covered in the instruction and benchmark assessment, and how many items will be included for each learning standard. Typically, districts implement between 3‐5 benchmark assessments during the school year. ATI can provide district customized benchmark assessments aligned with the district pacing guides, statewide test aligned comprehensive benchmarks, or a combination of both depending on the needs of the district. When a district is using ATI pre‐post assessments as part of its educator effectiveness initiative, these are included as part of the 3‐5 assessment during the year. Galileo K‐12 Online also includes computerized adaptive assessment, formative assessment instruments, pre‐ and post‐test assessment for use in educator effectiveness initiatives, interim and end of course examination and screening tools. Math, reading/English language arts (ELA), science and writing. Assessment items and tests are also available in non‐state‐tested areas such as social studies, foreign languages, physical education, music, arts, career and technical education, to name a few. Page | 1 Element Learning Objectives Individual Metrics Individual Comparison Points (cut scores) Aggregate Metrics Description Specific learning objectives being assessed, at as detailed a level as is provided. This may be "topics" or categories or may be actual learning objective statements. The scores provided at the individual (student) level. Information provided regarding how good is good enough performance on the instrument. Comparison information should be available for every individual metric. This may be performance level ratings with specific cut scores. Scores provided at the group level. The group could be a grade level, school, district, or disaggregated groups (e.g. race/ethnicity, gender, IEP status, FRL status) Specify the group(s) and the score(s) provided. Assessment Instrument Information Standards are entered into the system using ATI’s online Standards Builder. Galileo can accommodate any set of standards as well as the underlying performance objectives associated with those standards, including student learning objective needed by a school district. Colorado’s version of the Common Core State Standards is in the system and ATI provides assessments aligned to the Colorado Academic Standards to Colorado school districts. Raw scores IRT scaled scores (IRT scale scores from different assessments are placed on a common scale to support the measurement of growth). If statewide test scores are available from a previously administered statewide test, cut points for standards mastery can be set to correspond to cut points on the statewide test using “equipercentile” equating Student scores are categorized as Advanced, Proficient, Partially Proficient or Unsatisfactory. Interim assessments can be aggregated by student, course, grade level, school or LEA, as well as student demographic filters and intervention groups, teacher and program variables. An Aggregate Multi‐Test Report and Categorical Growth Report, among others can be run at the LEA, school and class, and intervention group levels. Development summary reports include four scores: Standard Score, Percentile Rank, Normal Curve Equivalent Score, and Developmental Level (IRT score). The Aggregate Metric of each of these measures that are reported on the Developmental Summary are the score for each measure that is associated with the average IRT scale score on the Page | 2 Element Description Assessment Instrument Information test. For example, if the mean scale score at the school level is 1152 on a given test, then the Percentile Rank that is reported on the school‐level Developmental Summary report is the percentile rank that is associated with a scale score of 1152 in the score lookup table for that particular test (which, in turn, is based on the IRT parameters for that test). Aggregate Information provided Cut points are calculated for each assessment using equipercentile equating to align the student Comparison regarding how good is scores on the Galileo assessment to their scores on the statewide assessment (TCAP). Points (cut good enough scores) performance at the ATI encourages school districts to design and administer customized assessments that are aligned group level. to their curriculum. As a result, most Galileo assessments that are administered by Colorado school districts are unique. Although student scores on these assessments are placed on a common scale using IRT techniques, when it comes to forecasting student performance on a statewide assessment ATI has felt it is more prudent to conduct the equipercentile alignment to the statewide assessment for each district individually based on the performance of that district’s students on the most recent statewide assessment. Therefore, the cut scores that reflect alignment with the TCAP performance levels may vary bit between districts, reflecting the refined alignment that focuses on that district’s particular set of students. This approach has yielded high accuracy in forecasting student performance on statewide assessments. ATI is currently conducting research to determine whether a single set of cut scores can be applied to all school districts within the state and still yield the same level of forecasting accuracy to which Galileo users have become accustomed. Cut sores established for Aggregate 50th Percentile of % of Students Proficient & Advanced on Post‐
Requests to Reconsider Comparison Test Points (CDE Cut Grade Reading Writing Math Science Scores) K
71.65
53.52
70.89
N/A
1
71.65
53.52
70.89
N/A
2
71.65
53.52
70.89
47.53
3
71.65
53.52
70.89
47.53
4
71.65
53.52
70.89
47.53
5
71.65
53.52
70.89
47.53
6
71.43
57.77
52.48
48.00
7
71.43
57.77
52.48
48.00
8
71.43
57.77
52.48
48.00
Page | 3 Element Alignment Description Information provided by the vendor about alignment of this instrument to other instruments, standards, etc. Data Reports Description of data reports that are provided/available at the individual and aggregate level(s). Technical Quality Assessment Instrument Information ATI item banks currently contain over 110,000 items which are expanding at the rate of over 800 items per month. The items are constructed, reviewed, and certified to align to state standards via specifications that lay out the format, content, and target skill. The specifications both ensure high quality consistent items, and make it possible to readily deal with changes to standards. The adopted item development procedures are informed by industry standards outlined by the American Educational Research Association, the American Psychological Association, and the National Council on Measurement in Education in The Standards for Educational Testing, 1999. Development profiles, development summaries, multi‐test reports, testing reports (class profile and item analysis reports), administrative reports, test monitoring reports, dashboard reports for both instructional planning and educator effectiveness initiatives (class dashboard, benchmark and formative results reports) See technical manual. References: Benchmark Assessment in Standards‐Based Education Meeting Local, State, and Federal Instructional Improvement System Requirements. A Resource Document for LEAs: Illustrated through the Galileo K‐
12 Online Instructional Improvement System Quick Facts Sheet: Galileo Instructional Effectiveness Assessment System Technical Manual Benchmark Assessment Development in Galileo Educational Management System Page | 4