Results Driven Accountability Continuous Improvement Process 2014-2015 FAQs by Topic

Results Driven Accountability Continuous Improvement Process
2014-2015 FAQs by Topic
All FAQs published in 2014-15 are compiled here, organized alphabetically
by topic. Hyperlinks are provided at the beginning of the document to
facilitate the reader’s ability to navigate through the document by topic.
Click on the topic to move to that area within the document. Within each
topic area, the questions and responses are organized chronologically (i.e.,
by the months the questions were posed, beginning with the introduction
to the Continuous Improvement Process in July 2014).
AU Partners
Communication
Compliance
Continuous Improvement Process
Determinations
Enrich
ESSU Collaboration and Work
ESSU Data Management System
ESSU IEP Interchange
Fiscal
IEP Process
Parent Survey and Indicator 8
Performance Data
Post School Outcome Interviews
Record Reviews
Resources
Results Driven Accountability
Samples
Timelines
Trainings
Transition
Continuous Improvement Process
2015-16 Timelines






July-August 2015 – Record review
samples and parent survey
samples drawn and posted for
the 2015-16 school year.
September 15, 2015 – All post school outcome interview data
due in the ESSU Data
Management System
November 15, 2015 – Indicator
14 samples distributed, i.e.,
students noted as exited in EOY
December 1, 2015 – Indicator 13
record reviews due in the ESSU
Data Management System
January 15, 2016 – Fiscal selfaudits due in the ESSU Data
Management System
June 1, 2016 –
o All record reviews
completed
o Parent survey closes
o Begin post school outcome
interviews
Common Acronyms:
AU (Administrative Unit), ESSU (Exceptional Student Services Unit), CDE (Colorado Department of Education), CAP
(Corrective Action Plan), DMS (Data Management System), MTSS (Multi-Tiered System of Supports), SSIP (State
Systemic Improvement Plan), SiMR (State identified Measureable Result), OSEP (Office of Special Education Programs),
IEP (Individualized Education Program), TA (Technical Assistance), PD (Professional Development), SDLT (State
Director’s Leadership Team), EDAC (Educational Data Advisory Committee), IDEA (Individuals with Disabilities
Education Act), ECEA (Exceptional Children’s Educational Act), RDA (Results Driven Accountability), PLAAaPF (Present
Levels of Academic Achievement and Functional Performance), ELA (English Language Arts), CAS (Colorado Academic
Standards), EEO (Extended Evidence Outcomes), CIMP (Continuous Improvement Monitoring Process), IdM (Identity
Management), YODEL (Your On Demand Educational Library)
JULY 2015
RESULTS DRIVEN ACCOUNTABILITY: CONTINUOUS IMPROVEMENT PROCESS FAQs
2
AU Partners
July 2014
Is there a process if I would like a different AU partner from CDE?
At this point, we have not yet put into place a process for requesting a different AU partner. We will discuss this with
the SDLT to determine what process, if any, should be put in place (e.g., using the ESSU Organization Chart and making
a request through the individual’s supervisor).
Will the AU Partner replace the work of the CDE ESSU consultant who has previously been working with the
AU on various initiatives?
The AU Partner will not assume responsibility of all the activities that are occurring in an AU with other ESSU
consultants. Rather, the AU Partner will become familiar with those activities and may join the work of those activities
but will not assume the primary responsibility.
October 2014
When will technical assistance be provided? Will the AU Partner assist the special education director during the
review of files to help determine where the technical assistance is needed, e.g., combine TA with file reviews?
Training on the record review process occurred at the beginning of October 2014. Trainings were provided to the
Directors groups (e.g., Metro Directors, Northern Directors, Outback, etc.) The AU Partners were invited to these
trainings. AU Directors were able to send a total of 5 team members to the trainings, including themselves. Following
these trainings, the AU Directors may request that the AU Partner facilitate the provision of technical assistance, such as
for inter-rater reliability checks.
How are the new AU partners going to provide more support than what is already given (and appreciated) by CDE
personnel?
The AU Partner will be the point of contact for document needs, issues, and strengths specific to the AU. They will
represent their AUs in ESSU-based meetings. The AU Partners for all AUs communicate with each other through
weekly meetings at the ESSU. Based on identified needs, the AU Partners will brainstorm solutions and bring those
suggestions to the AU. In partnership, the ESSU and AU will identify solutions. Once established, the AU Partner will
serve as the conduit that enables the AU to access any needed technical assistance.
December 2014
What is the role of the AU Partner in an AU’s record reviews?
The AU Partner would not be expected to conduct the reviews for an AU. If the AU requests assistance or guidance in
completing the reviews, the AU Partner may either provide that assistance or request support from the ESSU
Accountability Specialists whose primary responsibilities include monitoring.
JULY 2015
RESULTS DRIVEN ACCOUNTABILITY: CONTINUOUS IMPROVEMENT PROCESS FAQs
3
Communication
January 2015
What will be the process for responding to questions from the field?
The ESSU AU Partners typically meet on a weekly basis. A standard agenda item of these meetings is to gather questions
from the field. The following process for managing questions was recently established and includes the following
procedures:
 When a question from an AU Partner is specific to a content area, the question will be presented to the ESSU
personnel whose responsibilities are most closely linked to that topic area. That expert will either provide a
response directly to the AU Partner who will then contact the Director with the response or the expert will
respond to the Director directly, with the AU Partner informed or included in the communication. The content
expert will then determine whether the question should be included in the FAQ.
 If the content expert determines a question from the field should be included in the FAQ, (i.e., it most likely has
widespread relevancy to the field), s/he will inform the AU Partner to bring the question to the AU Partners’
next meeting so the question can be officially included in the monthly FAQ.
 At times, questions may require deeper consideration by ESSU staff and/or with SDLT members in order to
establish a consistent message in the field. In those circumstances, the question will appear in the FAQ with a
notation that further consideration is needed prior to a response being posted.
 Questions from the current month will be gathered from the minutes of AU Partner meetings on the third
week of the month (e.g., the third week of December).
 Responses will be posted in the Special Education Directors’ Corner on the first of the following month (e.g.,
the first of January). See Results Driven Accountability FAQs.
February 2015
Why is different information given from different groups out of CDE?
The ESSU is implementing a variety of routines in an effort to unify any messages that are given to the field from ESSU
staff. If an AU Director feels that s/he has been provided with conflicting information from members of the ESSU, please
report these inconsistencies to the AU Partner so the ESSU may clarify and establish a unified message for all AU
Partners and staff regarding the appropriate response.
May 2015
How does the ESSU communicate with all AU Directors?
Any announcements that will be delivered to ALL AU Directors will be sent via email from Jason Baggs
([email protected])
What will be the process that the ESSU will use to follow up on concerns received from parent calls?
Currently, all parent calls are addressed by Joyce Thiessen-Barrett. In the near future, the ESSU will be instituting a new
procedure which will direct parent calls to the appropriate AU Partner. The process for responding to the calls will
mirror the current process. The content of these calls will be posted to the Communication log on the Family-School tab
JULY 2015
RESULTS DRIVEN ACCOUNTABILITY: CONTINUOUS IMPROVEMENT PROCESS FAQs
4
in the ESSU Data Management System (DMS). The AU Partner will contact the AU Director to alert him/her to the fact
that a call had been received and will note that the call is documented on the Family-School tab in the DMS. If the
concern is specific to a particular child, the ESSU DMS is “behind” CDE’s Identity Management program and is thus a
secure location for exchange of information between the ESSU and the AU. The AU Director may then follow up locally
and document any communication in the DMS.
What will be the process that the ESSU will use to follow up on concerns from the field?
The ESSU occasionally receives calls of concern from the field (e.g., agencies, advocates). The AU Partner will document
the stated concerns in the Communication area of the Family-School tab in the DMS. The AU Partner will then contact
the AU Director and share the concerns heard from the field and report any supportive documentation. The AU Director
will respond by providing any information specific to the area of concern. This information will then be posted to the
Communication log of the Family-School tab. Follow-up communication may or may not be needed after the initial cycle
of communication; all communication will be posted via logs.
JULY 2015
RESULTS DRIVEN ACCOUNTABILITY: CONTINUOUS IMPROVEMENT PROCESS FAQs
5
Compliance
October 2014
When will Corrective Action Plans (CAPs) be required?
CAPs are required when a state complaint decision indicates the need for one. The ESSU has a Family-School Liaison who
works with the AU, when she is required, to ensure all the requirements for the CAP are met and all actions in the CAP
are completed within required timelines. With regard to the record review process for the Continuous Improvement
Process, the AU will conduct the reviews of the random samples and analyze the results to determine if there are areas
where training is required. If training appears to be necessary, the AU will collaborate with the AU Partner(s) to
determine how the technical assistance will be provided to staff. The ESSU and the AU will work together over a period
of 18-24 months to improve the AU’s compliance. This process will hopefully obviate any issues identified by the AU
without the need for official findings and subsequent CAPs to be developed.
Why does it seem that there is a focus on compliance when the plan is to focus on performance?
This new Continuous Improvement Process includes a focus on both compliance and performance; however, the
compliance activities will focus on those components of the IEP process and documentation that were determined to
appear to have the most direct effect on student outcomes. The file reviews for each AU will not exceed 50 reviews per
year (5 per month), unless the AU is currently in a position of non-correction of noncompliance.
JULY 2015
RESULTS DRIVEN ACCOUNTABILITY: CONTINUOUS IMPROVEMENT PROCESS FAQs
6
Continuous Improvement Process
July 2014
Was this reviewed at SDLT or EDAC? Is it in IDEA or ECEA requirements?
The Continuous Improvement Process has been an agenda item for the SDLT meetings since the fall of 2013. In
February 2014, ESSU presented a general overview of the Continuous Improvement Process being developed to the
Educational Data Advisory Committee (EDAC). In March 2014, EDAC provided their stamp of approval. ESSU will
continue to keep the EDAC informed as the development of the state level application (i.e., the ESSU Data
Management System) continues.
January 2015
Please provide clarification on all components for the Results Driven Accountability (RDA) Continuous Improvement
Process.
The Exceptional Student Services Unit (ESSU) must ensure that Administrative Units (AUs) are in compliance with and
satisfy the requirements of the Individuals with Disabilities Education Improvement Act of 2004 (IDEA 2004, Part B,
§300.600 (a)) and the Exceptional Children’s Education Act (ECEA). Consistent with these requirements, Colorado’s
Continuous Improvement Process will involve a three-tiered model designed to provide individualized supervision and
technical assistance of “Every AU, Every Year.”
The bodies of evidence that will be gathered as a component of the Continuous Improvement Process include the results
of the IEP record reviews that have been streamlined to place a stronger emphasis on those IEP components considered
most closely related to student outcomes as well as data that have been gathered through established data collections
for Indicator reporting: the parent surveys reflecting an AU’s family‐school partnering and involvement; staff
qualifications gathered from the December snapshot data; fiscal self-audits and reporting; special education discipline;
AU policies and procedures documented in the Comprehensive Plan; AU academic achievement data; secondary
transition; disproportionate representation; and professional development.
Colorado Continuous Improvement Process Objectives
1. Ensure a meaningful and continuous process that focuses on improving academic performance and outcomes
for students with disabilities by linking AU data, including indicator data, to improvement activities.
2. Partner with AUs to ensure compliance with IDEA and ECEA regulations.
3. Connect AU‐level and school‐level improvement activities with IDEA and ECEA regulations.
4. Support each AU in the process of self‐audit, evaluation, and improvement of instructional effectiveness and
compliance to ensure growth in student academic performance and outcomes.
5. Link improvement activities with long term, multi‐year professional development to support capacity building
and sustainability of compliance and instructional effectiveness.
Three Tier System
JULY 2015
RESULTS DRIVEN ACCOUNTABILITY: CONTINUOUS IMPROVEMENT PROCESS FAQs
7
In the tradition of the Multi-Tiered System of Supports (MTSS) the CDE has embraced a three-tiered MTSS logic to
provide supervision and technical assistance to AUs. The majority of AUs are anticipated to need support at the
Universal Tier, with only a handful of AUs identified as needing support at the Targeted and Intensive Tiers.
When is the UIP due?
NOTE: This is the exact form of the question from the field. It is being assumed that there may be confusion between
the UIP process and the AU Improvement Plan. See below for clarification of the processes.
Currently, the future AU Improvement Plan is not a component of CDE’s Unified Improvement Process (UIP). The AU
Improvement Plan will be directly related to our new Indicator 17, the State Systemic Improvement Plan (SSIP) and
State-Identified Measurable Result (SiMR). The ESSU will report the Identified Measurable Result to the Office of Special
Education Programs (OSEP) in April 2015. The identification of the SiMR will also include specific state targets that will
be measured through 2018. It is expected that the AUs will implement improvement planning and practices that will
include a focus on this identified target for the targeted population of K-3 students who have both IEPs and Read Plans.
JULY 2015
RESULTS DRIVEN ACCOUNTABILITY: CONTINUOUS IMPROVEMENT PROCESS FAQs
8
Determinations
October 2014
How will determinations be made? How will determinations affect an AU’s tiered system of support?
OSEP released to states in June 2014 determinations based on a matrix of 50% Compliance and 50% Results. The ESSU
used this matrix to demonstrate what an AU’s determination would look like under this new system. These
determinations were distributed to AU Directors at the Fall Directors’ meeting.
The ESSU will not use the new determination matrix for AU determinations for 2013-14. These determinations will be
based strictly on compliance activities and will hopefully be released in November 2014. Colorado is in the Meets
Requirements category for Compliance areas and the Needs Assistance category of the Results areas. This puts Colorado
in the Needs Assistance category for a second consecutive year.
The ESSU has established a work group, comprised of internal and external stakeholders, to address the matrix for the
Results areas. Directors across the state are concerned that achievement gaps alone cannot be the sole criterion for this
area. The work group will review Colorado’s ESEA Flexibility Waiver to see if some segment of student growth can be
identified and proposed to OSEP to be used in our future results determinations.
The graduation rate based on a 4-year cohort penalizes Colorado – especially when, by law, we must educate students
through the age of 21 if determined as a need on their IEPs. Students who have died should not be in the formula.
The ESSU realizes the issues in the current definition of graduation rate, as determined by OSEP. The ESSU submitted
comments regarding the need to adjust specific definitions, including the graduation rate definitions. We realize there is
a need to continue to comment to OSEP on these and other definitions.
With OSEP willing to partner with CDE, can this be considered a hold-harmless year?
The plan is to consider this year a transition year for the new OSEP determination matrix. Except in cases where there
are multiple findings via state complaints, due process, or a continuation of non-correction of previous non-compliance,
the goal of the ESSU is to partner with each AU in building infrastructures to be able to integrate solid compliance
practices while improving the achievement results of students with disabilities.
JULY 2015
RESULTS DRIVEN ACCOUNTABILITY: CONTINUOUS IMPROVEMENT PROCESS FAQs
9
Enrich
November 2014
Why is there duplication of the postsecondary goals on the IEP?
Enrich is the state’s chosen IEP system, developed by the ESSU in conjunction with the AUs who served as “early
adopters”. As with any IEP system, Enrich follows the model forms that are posted on the CDE website. As per the
model form, the Transition IEP restates the student’s postsecondary goals throughout the development of the IEP so the
focus remains on the student’s goals. The postsecondary goals are initially described in the student’s Present Levels of
Academic Achievement and Functional Performance. They are then repeated at the beginning of the transition planning
where the course of study and transition services are developed.
February 2015
When will Enrich have the updated Colorado Academic Standards uploaded?
This response was provided by the CDE Standards Project Director when provided with the question noted above:
Since the adoption of the Common Core in math and ELA (in 2010) there have been no changes to the Colorado
Academic Standards (CAS). Any proposed changes to the Colorado Academic Standards would go through an extensive
and public process and would require State Board of Ed approval before adoption. CAP4K does allow for the revision on
or before 2018, but that process would be public, involve multiple constituencies, and would require Board approval.
The coding on the standards dates them as 2009 and 2010, because those are the years they were written and adopted.
Those dates may lead people to believe we have new standards as implementation took place in 2013. Some people
may think the implementation date of 2013 means that there were new standards, when in actuality the standards
written and adopted in 2009/10 were the ones implemented in the 2013/14 school year.
Note from Cindy Millikin: Enrich is currently populated with the CAS as written. In addition, the standards from the
Colorado Early Learning Development Guidelines have been added for preschool.
How does one access reports in Enrich?
Access to reports in Enrich is role-based, meaning that the system administrator would have checked that you are able
to create, view, and/or delete reports based on your Enrich access. When you click on Create/Manage Reports, you will
see four categories for reports: Saved Reports, Shared with Me, Report Templates, and Blank Reports. There is a sample
view of the report templates that displays the layout of a report. You can choose a report template and then Edit the
Criteria to add or delete elements.
April 2015
What help features are there in Enrich?
JULY 2015
RESULTS DRIVEN ACCOUNTABILITY: CONTINUOUS IMPROVEMENT PROCESS FAQs
10
There are a variety of help features within Enrich. Some of these features are built into the system and are there upon
implementation. For example, there are approximately 25 videos within Enrich. Videos are found when you click on
Help in the banner bar. To view one of the video options in the IEP, click on Help for Goals:
Videos available in the goal
area
There are district guidelines which contain information created by the district/AU in order to guide staff in completing
that section of the document. When an AU creates guidelines, that text is also viewed directly on the screen by the staff
without having to click on Help. The text is located directly under the banner, as in the example, below:
The state guidelines are drawn directly from the IEP Procedural Guidance document, parsed out by the specific content
area in which it is located. Finally, the Federal Regulations tab in the Help area includes the regulations from IDEA that
are applicable to that area.
There are other features in Enrich to support the staff in completing their IEP-related documentation. The navigation
bar enables the staff member to move quickly between sections. The up arrow in the upper right corner of the screen
operates as a [Top] tool.
Text Assistant is a feature in every memo box where the AU system administrator team is able to design stem
statements and other helpful text to assist staff in ensuring that all components needed in a specific area are included.
In the first example below, the stem statement inserts the student’s name to personalize the comment.
JULY 2015
RESULTS DRIVEN ACCOUNTABILITY: CONTINUOUS IMPROVEMENT PROCESS FAQs
11
Another critical feature that provides help to the staff is the Validation tool. When a staff member feels that s/he has
completed all components of the IEP, clicking on the Validation tool will identify if any areas have been missed. The
items that have been missed turn red in the navigation bar and are noted in red in the data entry area.
These are just some of the features that are included in Enrich to assist the staff member in maneuvering within the
system and in completing all critical IEP-related documentation. Many of these features are built once the initial
implementation is underway.
How does one access the standards in Enrich?
The standards are located in the goal area of Enrich. The IEP team will have considered the student’s strengths and
needs and the impact of the disability on the student’s progress in the Colorado Academic Standards and determined
those standards in need of specialized instruction. It is important that the IEP team is familiar with the layout of the
components of the standards (see below).
When the IEP team member clicks Standard in Enrich, the following sequence occurs:
Click on Add Standard:
Click on the desired Content Area:
JULY 2015
RESULTS DRIVEN ACCOUNTABILITY: CONTINUOUS IMPROVEMENT PROCESS FAQs
12
Click on the Grade Level:
Click on the desired standard:
Click on the concepts and skills area:
JULY 2015
RESULTS DRIVEN ACCOUNTABILITY: CONTINUOUS IMPROVEMENT PROCESS FAQs
13
Click on the component of the standard’s concepts and skills that best reflect the standards-based focus of the goal:
May 2015
Have the DLM and PARCC accommodations for Enrich been released?
The most recent update to Enrich includes the specific accommodations that are allowable for DLM, PARCC, etc. These
accommodations were reviewed and verified by the Assessment Unit for accuracy. AUs should remind their staff that
students who meet the participation requirements for the alternate assessment should NOT select accommodations for
PARCC but should review the student’s needs for accommodations for DLM.
June 2015
Model Forms and Enrich: Which comes first?
Enrich follows the model forms as all vendors do. Recently, a few changes were inadvertently made to Enrich without
the vetting process for changing model forms. Those changes were therefore reversed.
Will Enrich have new transfer forms?
There are two transfer forms posted to the ESSU’s website for the model forms: the Transfer Within State and Transfer
Out of State forms. The out-of-state transfer form is used when a student transfers during the school year into a
Colorado school from out of state; it provides for the IEP team’s determination regarding the alignment of the other
JULY 2015
RESULTS DRIVEN ACCOUNTABILITY: CONTINUOUS IMPROVEMENT PROCESS FAQs
14
state’s criteria for eligibility to Colorado’s criteria. If an AU is unsure of the alignment of the other state’s criteria to
Colorado’s eligibility criteria, they may choose to conduct a reevaluation of the student (referred to as an “initial
evaluation” for Colorado) and eligibility determination, based on Colorado’s rules. While conducting the reevaluation,
the AU would provide comparable special education and related services to those written in the out-of-state’s IEP to
ensure the continued provision of FAPE.
JULY 2015
RESULTS DRIVEN ACCOUNTABILITY: CONTINUOUS IMPROVEMENT PROCESS FAQs
15
ESSU Collaboration and Work
October 2014
What level of inter-rater reliability training will occur within CDE personnel?
CDE personnel will be conducting records reviews that will be compared to the AUs’ record reviews for inter-rater
reliability following multiple trainings/meetings with a focus on compliance. Where there may be discrepancies in
ratings, routine meetings that will focus on these reviews will identify the discrepancies and establish shared criteria
among staff members.
November 2014
Will there be feedback from CDE on the AU’s record reviews?
There are no plans for providing specific feedback to the AUS on their record reviews. However, the AU Director
may request that the ESSU provide support to this process, either by conducting inter-rater reliability checks or by
supporting training needs that are discovered through this process.
Are the teams at CDE going through the same training?
Many of the CDE ESSU’s AU Partners attended the trainings with their AU Directors that were conducted across the
state in October. In addition, most of the staff at the ESSU have been involved in previous monitoring activities which
involved reviews of IEPs using the comprehensive record review checklist. Finally, the ESSU provides ongoing internal
professional development to support the work of the AUs and the ESSU.
December 2014
Will ESSU staff enter the data from the record reviews into the DMS since it was not “up and running” by the
expected “go live” date?
The deadline date for the record reviews of the Transition IEPs has been extended to December 15, 2014. This should
provide sufficient time for the AU teams to enter their data into the DMS. However, if this timeframe is a hardship for
any of the AU Directors, in a spirit of partnership, the AU Director may contact Cindy Millikin at the ESSU
([email protected]) and request assistance for entering those data from the reviews conducted on paper into
his or her AU’s section of the DMS system.
JULY 2015
RESULTS DRIVEN ACCOUNTABILITY: CONTINUOUS IMPROVEMENT PROCESS FAQs
16
ESSU Data Management System
July 2014
Why am I uploading data that has been uploaded in December report when CDE already has the data?
This question refers to the process of uploading the signed reports that are generated from the data collections. The
ESSU Data Management System will provide the ability to archive all of the documents that are shared from AUs to the
ESSU. Included in this archive will be the signed reports that are generated from the December count. There are a
variety of reports that are generated from data collections, based on the data submitted. Each AU Director is required to
verify the data submitted by reviewing these reports and putting his or her signature on the report as an assurance to
the ESSU that the data are valid and accurate. The ESSU will use the ESSU Data Management System to store or archive
those signed assurances.
Will the new CIMP process data collection replace the data collection needed for individual district evaluation
plans?
The AU’s Comprehensive Plan includes a requirement for an evaluation by the AU of its programs and services. The data
documented in the ESSU Data Management System would support that evaluation process.
November 2014
What is the new state system called and what are its components?
The ESSU has built a state level application intended to serve as a tool to both the Administrative Units (AUs) and to the
state. This system is called the ESSU Data Management System (DMS). The system includes the following areas: Profile,
which will include document specific to the AU, such as the Comprehensive Plan, procedural manuals, and data reports
that have been verified and submitted by the AU Director at the end of data collections; Compliance, which will provide
the tool for conducting record reviews as well as dashboards for at-a-glance summaries of compliance across the AU,
both at an individual student record level and at an aggregate level; Fiscal, which will provide a dashboard summary of
the AU’s fiscal self-audit; Family-School Partnering, which will provide the results of the parent surveys for Indicator 8 as
well as the results of parent surveys for the AU as a whole if the AU has opted to disseminate surveys to all families;
Performance, which will include dashboard summaries of the results of the Post-School Outcome Interviews; and
Improvement Planning, which will house the action planning of the AU for improved student outcomes. Each area also
includes a communication log and the ability to upload attachments specific to that area.
December 2014
What is the purpose of the Data Management System?
The ESSU Data Management System (DMS) was created to serve multiple purposes. Primarily, it is designed to serve
and support the managerial needs of the Directors of the Administrative Units (AUs) as well as meet the monitoring and
enforcement requirements of any entity (CDE, AUs) that receives funds to serve students with disabilities under the
Individuals with Disabilities Education Act (IDEA). There are multiple ways in which the DMS will meet these
expectations. For example, the DMS will function as a repository for documentation required from each AU, e.g., the
Comprehensive Plan, signatures of verification for data collections, assurances, local performance data summaries, etc.
JULY 2015
RESULTS DRIVEN ACCOUNTABILITY: CONTINUOUS IMPROVEMENT PROCESS FAQs
17
It will provide a method of delivery and exchange of performance data summaries from the ESSU to the AU, e.g., the
performance data presentation slides, and from the AU to the ESSU as a body of evidence supporting the results of local
improvement planning. It will summarize the following data: IEP record reviews, including the data to be submitted for
Indicator 13; responses from the Post-School Outcomes interviews; responses from the parent surveys; and, the results
of the Fiscal Self-Audit. It also offers a secure communication mechanism for the exchange of information between the
AUs and the ESSU. As the DMS becomes fully implemented, other types of support will most likely also be realized.
How is security managed in the ESSU DMS?
Data and documentation within the DMS is secured via the Single Sign-On Identity Management (IdM) program of the
Colorado Department of Education (CDE). Only individuals who have been authorized by the AU Directors will be able to
access the DMS. Security is role-based, with access to the various sections of the DMS defined by these predetermined
roles. When an individual enters his or her username and password, the DMS authenticates against the IdM before the
individual is passed through to the DMS. Finally, CDE’s IdM program is the same pathway currently used by data
respondents for submitting data securely to the CDE for required data collections; it has a proven track record as a
secure environment through which to pass data and other documentation.
February 2015
Who has access to the Data Management System within an AU?
The AU Director makes any decisions regarding access to the ESSU Data Management System. The following list
provides descriptions of the current roles available at the AU level:
AU Monitoring Director: This role is intended for Directors. They have the ability to engage in all actions within the
ESSU Data Management System (DMS). They will set up any teams they may have within the areas of the DMS. The
Directors with this role as stated do NOT submit files within the IEP Interchange; if a Director also submits data to the
IEP Interchange, the role is a combined role, i.e., AU Monitoring Director and LEA User.
AU Monitoring Manager: This role is intended for people who serve as essential support to the Directors. These might
be mid-level management staff who will be able to do all actions across all areas of the ESSU Data Management System
EXCEPT designing/selecting teams. If the AU Monitoring Manager also submits data to the IEP Interchange, the role
would be combined with LEA User.
AU Monitoring Record Reviewer: This role is intended for staff that the Director assigns to help with the record reviews,
only. They will not be able to view other areas of the ESSU DMS.
AU Monitoring Post-School Interviewer: This role is intended for staff who are assigned by the Director to enter data
from the Post-School Outcome interviews only. People with this role will not be able to enter other areas in the ESSU
DMS.
AU Monitoring Record Reviewer and Post-School Interviewer: This role is intended for staff who are assigned by the
Director to conduct record reviews AND complete post-school outcome interviews. They will be able to conduct record
reviews. They will be able to enter data from the post-school outcome interviews. However, they would not be able to
enter other areas in the ESSU DMS.
JULY 2015
RESULTS DRIVEN ACCOUNTABILITY: CONTINUOUS IMPROVEMENT PROCESS FAQs
18
What is the function of the Profile tab in the data management system?
The Profile tab will hold data that summarize the profile of the AU. This is where the AU Director will attach the signed
reports as verification of data for collections. This is also the location where the AU’s Comprehensive Plan will be
attached. The CDE ESSU will attach data charts and graphs that summarize the AU’s student profile.
If new reports are added to the data management system, how long does it take for a new report to be completed?
Currently, the only reports that have been written for the DMS are the fiscal reports ordered by Barb Goldsby and Vicki
Graham. However, the ESSU will be building reports for Compliance, parent surveys, and post-school outcome
interviews. Some of these reports will be in the aggregate for the ESSU. Other reports will be created and posted at the
AU level to support the AU Directors in their administrative decision making and evaluation activities.
April 2015
I am having trouble accessing the DMS. Who can help me?
If you or any member of your team is having trouble accessing the DMS, contact Cindy Millikin at CDE.
([email protected])
What is the status of the AU Improvement Plan?
The AU Improvement Plan is currently under construction in the DMS. Once developed, it will provide a means for the
AU to develop an improvement plan based on an analysis and summary of the trend data across the AU, a description of
root cause analyses, and then action steps documented to specify the activities for the plan. This Plan was originally
intended to be created in January, 2015; however, with the other initiatives and components in the DMS, the
requirement for the Plan to be developed in 2014-15 was removed. This functionality will most likely be amended to
integrate with and reflect the work of the Results Work Group as a foundation for improvement planning.
What is the purpose and how does one initiate “real-time” reviews?
There is a new functionality available in the ESSU DMS that allows the AU to conduct record reviews of IEPs that have
currently been developed and are not in the predetermined sample set provided by CDE. With this functionality, the AU
would upload the IEP to the DMS and conduct a standard review. Lifesavers will represent the status of all IEPs
reviewed as “real-time reviews”, i.e., there would be two columns of lifesavers in the compliance area – one for the
reviews conducted from the sample and one for the reviews that are being conducted in real time.
Must the AU be an Enrich user in order to conduct real-time reviews?
Real-time reviews can be conducted by any AU whether or not they use Enrich. If the AU is using Enrich for their IEP
system, the upload of the IEPs to the DMS would occur electronically with a command on the students’ Programs tab
(Enrich version 10.2). If the AU’s version of Enrich is not at 10.2, the IEPs can be uploaded through the zipped file
process (see handouts from Fall 2014 on Uploading Documents to the DMS) or contact Cindy Millikin.
How do I start doing real-time reviews?
JULY 2015
RESULTS DRIVEN ACCOUNTABILITY: CONTINUOUS IMPROVEMENT PROCESS FAQs
19
If the AU Director would like to initiate real-time reviews in the DMS, contact Cindy Millikin and she will set up the
Compliance area to include that functionality.
May 2015
How will the ESSU store data for and document communication with the AUs?
In the ESSU Data Management System, all information around students, staff and data collections will be located on the
Profile tab. Data samples will be uploaded and attached in the Attachments area. AU Directors’ signed verifications
from data collections will be uploaded by the AU to the Attachments area on the Profile tab. In addition, any reports
from the Office of Civil Rights, received by the ESSU, will be uploaded to the Attachments area of the Profile tab.
AUs will see a Dispute Resolution tab (not seen by the staff at the ESSU outside of the Dispute Resolution team). This
tab will be used by the Dispute Resolution team to exchange files and documents with the AU when needed. Any
communication with the AU regarding current cases of dispute resolution will also be documented on the
Communication log. If/when a complaint or dispute is resolved, any public facing reports will be posted on the website
and also to the Attachments area of the Compliance tab for the AU, thus enabling the AU Partners to be able to view the
reports within the DMS.
The Compliance tab will include logs of any communication between the ESSU and the AUs with regard to the work of
following up on compliance.
Inquiries regarding fiscal matters will be documented on the Communication log of the Fiscal tab.
All inquiries or calls from families or from the field (e.g., agencies, community representatives, AU local staff) will be
documented in the Communication log of the Family-School tab. When a call is received from a family, all attempts will
be made to gather permission from the family to share their identity with the AU Director, thus providing the means for
the AU to resolve issues directly. If a family does not wish to identify themselves, the AU Partner will log the call with an
anonymous inquiry, documenting a summary of the details of the call. Professionals who call the ESSU with inquiries or
reports will need to identify themselves so the AU Directors are able to address any issues directly. Their inquiries or
input will be documented in the Communication log of the Family-School tab.
Any inquiries from the AU regarding the Indicator 14 post school outcomes interviews or regarding performance data
will be documented in the Communication log of the Performance tab. The ESSU Data Team will also post performance
data files in the Attachments area of the Performance tab.
Calls specific to improvement planning will be posted in the Communication log of the Improvement tab. Any
documentation around an AU’s body of evidence for an improvement plan would be uploaded to the Attachments area
of the Improvement tab.
JULY 2015
RESULTS DRIVEN ACCOUNTABILITY: CONTINUOUS IMPROVEMENT PROCESS FAQs
20
ESSU IEP Interchange
November 2014
Do we still have access to the ESSU IEP Interchange?
The ESSU IEP Interchange is still accessible to all AUs.
Will there be a change in the current submission process to the ESSU IEP Interchange?
In the near future, the automatic transmission of data from Enrich to the IEP Interchange (via a “Send” command
within Enrich) will be electronically redirected to go directly to the Pipeline’s IEP Interchange and will no longer go
through the ESSU IEP Interchange. This new development will ensure that Enrich users will be able to continue to
submit their data electronically without having to create file layouts to submit their data to the Pipeline. This should
also speed the transmission process for data submissions during data collections.
JULY 2015
RESULTS DRIVEN ACCOUNTABILITY: CONTINUOUS IMPROVEMENT PROCESS FAQs
21
Fiscal
January 2015
Will there be fiscal audit training?
We will not be conducting a formal fiscal self-audit training, as all questions are self-explanatory with yes/no responses.
Special education directors and their special education fiscal partners should collaborate with all appropriate members
of their AU to complete the self-audit. A hard copy of the self-audit has been provided to all special education directors
and their fiscal partners, in the case that the self-audit is not ready to go "live" in the statewide data management
system. It should take approximately 1-2 hours to complete the self-audit. You will be notified as soon as the self-audit
goes "live".
Can their report from auditors be used as part of their fiscal audit?
In order to provide for consistent measures across all AUs, each AU will be expected to complete the self-audit. If an AU
is interested in supporting the documentation of the Fiscal self-audit by providing a copy of their report from auditors,
there is functionality in the Fiscal area (i.e., Attachments) for uploading relevant documents.
March 2015
Our fiscal self-audit is complete. What happens next?
The ESSU’s fiscal team will be analyzing the data from the fiscal self-audit, e.g., looking at baselines, and will then work
with the AUs and ESSU’s federal partners to determine next steps regarding technical assistance and training.
My special ed numbers are down by about 50 students for this December count. We are now at an identification rate
of 11.7%. vs. 12.5% last year. Would 50 students be enough of a decline to make a difference in our Maintenance Of
Effort (MOE)?
As with many intricate questions, the answer to this question is not a simple yes or no, but a maybe. Here is why. While
a reduction in students is an allowable exception when there is a failure of MOE, it cannot necessarily be viewed as a
one to one correlation. This is because every student with disabilities is unique and therefore meeting his/her
educational requirements as spelled out in the IEP does not necessarily cost the same for each student. So, while there
may have been a decrease in student count of 50, this number may not accurately portray the current costs of the
remaining population. For example, the change of 50 students may have occurred when 75 students with mid-range
excess costs left while 25 new extremely high-range excess cost students enrolled; therefore, the changes in student
population by 50 students may actually have resulted in the same or higher costs serving the lesser number of students
and not necessarily that lesser costs are to be expected from a drop in student count. The requirement of Maintenance
of Effort is twofold. The first part is to determine eligibility and requires the AU to budget, on either a per capita or on an
aggregate basis, at least as much as what was expensed in the most recent year for which data are available (IDEA §
300.203 (b)). The second part is to determine compliance and this requires that the AU not reduce the level of
expenditures for the education of students with disabilities below the level of those expenditures from the preceding
year (IDEA § 300.203 (a)). Exceptions to Maintenance of Effort are found in IDEA § 300.204.
JULY 2015
RESULTS DRIVEN ACCOUNTABILITY: CONTINUOUS IMPROVEMENT PROCESS FAQs
22
April 2015
If we have areas identified in our Fiscal Self-Audit as areas of need (i.e., red lifesavers), should we be working on
those areas now? Or wait for training?
The ESSU’s fiscal team has been analyzing the data from the fiscal self-audits and will bring these results to the May
fiscal training being provided to the AU Directors. AUs should wait until May when next steps will be announced at that
time. For added information, the ESSU Fiscal Team presented the following Talking Points to the Fiscal Practices &
Procedures Advisory group in March, describing in detail the Fiscal Self-Audit process and results.
Why?
 Monitoring of IDEA Part B – Subpart C – Local Education Agency (LEA) Eligibility
 34 CFR §300.200: An LEA is eligible for assistance under Part B of the Act for a fiscal year if the agency
submits a plan that provides assurances to the State Education Agency (SEA) that the LEA meets specific
conditions
 34 CFR §300.201: The LEA, in providing for the education of children with disabilities within its jurisdiction,
must have in effect policies, procedures, and programs that are consistent with the State policies and
procedures
 34 CFR §300.211: The LEA must provide the SEA with information necessary to enable the SEA to carry out its
duties under Part B of the Act
 2013 - Developing process for fiscal monitoring as part of CIMP (Continuous Improvement Monitoring Process) when
the move to RDA and changes to circulars required us to rethink the format/process
 New Uniform Grants Guidance (UGG) consolidated OMB Circulars A-21, A87, A-89, A-102 and A-110, A-122,
A-133 into a uniform set of rules
 Release of newly updated EDGAR – December 19, 2014
 Effective with new federal funding (for IDEA – July 1, 2015)
 To increase efficiency and effectiveness of federal awards and strengthen oversight of federal funds to
reduce the risk of fraud, waste and abuse
 SEA is required to do a Risk Assessment for all LEAs (Administrative Units - AUs)
 Single audits do not provide the level of information needed to ensure compliance with monitoring of IDEA
funds, and are not required for AUs receiving less than $500K in Federal funds (threshold is increasing to
$750K on July 1, 2015)
How?






Collaboration with our Grants Fiscal Management Office and ESSU’s Results Driven Accountability (RDA) team
Consultation with Brustein and Manasevit – our DC attorneys
AGA (Association of Government Accountants) Monitoring Tools – 1) Risk Assessment, 2)Financial and Administrative
Federal Register (Vol. 78, No. 248) Part III Office of Management and Budget
The Administrator’s Handbook on EDGAR, 2nd Edition
Federal Register Part II Department of Education - IDEA
When?
 January 2014 – Fiscal Self-Assessment draft
JULY 2015
RESULTS DRIVEN ACCOUNTABILITY: CONTINUOUS IMPROVEMENT PROCESS FAQs
23
 March 2014 – Fiscal Self-Assessment final to EDAC
 April 2014 – Explained role of Fiscal Self-Assessment in RDA to special education directors at state meeting
 May 2014 – Unveiled “IDEA and ECEA Fiscal Checklist” at the ESSU/Grants Fiscal joint training – allowed time for AUs
to dig in
 August 2014 – Finalized “IDEA and ECEA Fiscal Checklist” in the Statewide Data Management System
 January 16, 2015 – 100% of AUs completed the checklist on time
100
Internal Controls
95
Accounting
90
Record Retention and Access
85
Grant Management and Administration
80
Payroll/Time Distribution
Procurement
75
Self-Audit
Indirect Costs
Property Management
Next Steps?
 CDE/ESSU will be receiving TA and training from:
 OSEP
 Brustein and Manasevit
 Center for IDEA Fiscal Reporting (CIFR)
 WestEd
 CDE/ESSU will then:
 Dig into data further
 Use baseline data from IDEA and ECEA Fiscal Checklist to provide TA and training to AUs
 Measure annual progress with AUs
JULY 2015
RESULTS DRIVEN ACCOUNTABILITY: CONTINUOUS IMPROVEMENT PROCESS FAQs
24
IEP Process
February 2015
If there are no special evaluations, why is this choice on the paperwork?
Under the IDEA, there are two types of evaluations: an initial evaluation and a reevaluation. The regulations require
reevaluations to be conducted at least every three years. Some AUs have implemented a process for reevaluations that
they term "special evaluations" and that have specific requirements regarding process and documentation for
reevaluations for which the AUs do not want to reset the three-year reevaluation timelines. Other AUs do not make any
distinctions for these reevaluations. To accommodate the procedures of all AUs in the state, a Colorado State Advisory
Group that was formed to develop a document for IEP guidance in 2008 included the use of this procedure. This was
also reflected on the model IEP forms developed by this Advisory Group that are posted on the CDE website.
Will a clear definition of a transfer student be provided in writing?
The IEP Guidance document is currently under revision. The revised edition will provide guidance regarding transfer
processes. The revision is expected to be posted to the CDE website by the end of February 2015.
Would CDE provide a written explanation of why out of state IEPs should not be accepted?
Every AU has the responsibility to provide a free appropriate public education to all eligible students with disabilities. If
a student with a disability moves into an AU from another state, the AU will have to provide appropriate special
education and related services to that student, consistent with both federal law and Colorado’s Rules for the
Administration of the Exceptional Children's Educational Act (ECEA), including determining whether the student is
eligible under Colorado’s eligibility criteria, and whether the student’s out-of-state IEP provides the appropriate special
education and related services to provide that student with FAPE. As with every eligible student with a disability, this
will always be an individualized determination based upon the specific circumstances of each child.
If the student’s out of state information, including evaluative information and the IEP itself, satisfy the Colorado IEP
team and meet the requirements of Colorado’s rules, then the IEP team would be free to accept or adopt that IEP as its
own. If the Colorado IEP team is not satisfied that the sending state’s eligibility criteria are consistent with Colorado’s,
or that the out of state IEP offers FAPE, then the IEP team is free to conduct its own evaluations to determine eligibility
and then develop its own IEP, as appropriate.
Will CDE provide model IEPs?
CDE will not be creating model IEPs. However, the ESSU is recruiting redacted exemplars of IEPs from the AUs as they
conduct their local record reviews. These redacted exemplars (whether considered positive or negative examples) will
be entered into the CDE Enrich Sandbox so they are connected to fictional students and will have no connection to their
original student data or their AU. These exemplars will be available to the AUs so they may use these IEPs for training
purposes.
What does a good PLAAFP look like?
JULY 2015
RESULTS DRIVEN ACCOUNTABILITY: CONTINUOUS IMPROVEMENT PROCESS FAQs
25
The ESSU has collaboratively produced a guidance document titled, “Writing Standards-aligned ALPs and IEPs: A
Supplemental Guidance Document for Designing Effective Formal Educational Plans” Dec. 2014
The IEP section of the document is now located on the CDE IEP Forms page:
http://www.cde.state.co.us/cdesped/iep_forms
The guidance document presents a Seven Step Process for practitioners as they consider the student’s present level of
academic achievement and functional performance as it relates to the requirements of the Colorado Academic
Standards. Due to the fact that each child’s individualized education program is unique to the strengths and needs of
that particular student, there can be no “model” IEP. However, there are examples provided within the narrative of
Step 2, Gather Data, and Step 3, PLAAFP. In addition, a variety of constructed examples are included in the Through the
Lens… section. These scenarios offer various ways an IEP team may draft a student’s PLAAFP in order to provide a clear
picture of the student’s abilities as they relate to the academic standards and the impact of the disability. While there is
no formally prescribed format, and local policies and procedures of an Administrative Unit may further define the
structure for a PLAAFP statement, IDEA requires the following elements:
 the most recent evaluation data CRF §300.324(a)(iii)
 statement of the child’s present levels of academic achievement and functional performance §300.320(a)(i)
 academic and developmental, and functional needs of the child §300.324(a)(1)(iv)
 strengths, preferences and interests CRF §300.324(a)(1)(i)
 impact of disability statement §300.320(a)(1)(i) and (ii)
 concerns of the parent CRF §300.324(a)(1)(ii)
April 2015
Please provide clarification about which students meet participation requirements for EEOs/alternate assessment.
The Colorado Academic Standards’ alternate academic achievement standards are called the Colorado Academic
Standards/Extended Evidence Outcomes (EEOs). In order for a student’s IEP Team to consider alternate standards for
instruction and assessment, an optional worksheet has been designed to assist the IEP Team as they work through the
process. These are called “participation guidelines” and are posted
here: http://www.cde.state.co.us/cdesped/instructionalstandards . There is also a companion document and
PowerPoint to give more information.
The IEP Team considers the body of evidence and evaluations that support the existence of a cognitive disability and
then determines whether that cognitive disability can be considered to be in the “significant” cognitive disability
range. The definition of significant cognitive disability is outlined in Colorado statute
http://www.cde.state.co.us/cdesped/iep_forms.asp#eligibility in the Intellectual Disability category. (Significant
cognitive disability may co-exist with other disabilities or be considered a multiple disability; but the eligibility criteria
checklist outlines the requirements. There are also guidelines:
http://www.cde.state.co.us/cdesped/guideliensfordeterminationeligibility_id_md
*Just as a note, there are some disability categories that exclude the existence of an intellectual disability in the
eligibility requirements, such as Specific Learning Disability (SLD) and Serious Emotional Disorder (SED). The Speech
Language Impairment category (SLI) does not include the existence of a significant cognitive disability as a qualifier.
Therefore, in these disability categories, the IEP Team would not consider the use of alternate standards for instruction
or participation in alternate assessment.
JULY 2015
RESULTS DRIVEN ACCOUNTABILITY: CONTINUOUS IMPROVEMENT PROCESS FAQs
26
If the student’s characteristics as a learner consistently indicate that the impact of the disability is indeed in the
significant cognitive disability range, at that point, the IEP Team can consider which academic achievement standard is
the most appropriate for instruction, and then for assessment built upon that academic achievement standard. Just
because the student is considered to have a significant cognitive disability, that is not an automatic determiner that the
student MUST receive instruction on alternate standards; they may, but they are not required to. The IEP Team will
make that determination based upon the body of evidence. (Criterion #4) If the team determines that the impact of the
disability is such that it is most appropriate for the student to receive instruction on alternate academic achievement
standards and their progress evaluated using alternate assessment based upon alternate academic achievement
standards, then the IEP Team designates alternate standards for instruction/assessment. Such a decision indicates that
the student’s instruction will be modified and is based upon the alternate enrolled grade-level expectations through the
EEOs, which reflect a different level of complexity, content, and rigor. The student’s progress toward those standards is
then evaluated by the Colorado Measures of Academic Success Alternate assessments in English Language Arts/Literacy,
Mathematics (CoAlt: DLM) and CoAlt: Science and Social Studies. If the student meets participation requirements for
alternate standards and assessment, then ALL district/state assessment is taken in the alternate form. (Alternate
ACCESSS) For students enrolled in 11th Grade, this includes the 11th Grade Alternate Assessment for the Colorado ACT.
The Standards Side-by-Side Reference Tools have been created and designed to help teachers and parents see the
relationships between the enrolled grade-level expectations (Evidence Outcomes), the Extended Evidence Outcomes,
and the DLM Essential Elements for English Language Arts/Literacy and Mathematics (The Essential Elements
correspond with the CCSS references in the Evidence Outcomes and can serve as the “learning progressions” or
foundational skills a student would need to make progress toward the EEOs. (On an IEP, you would reference the
CAS/EEOs.) The Writing Standards-aligned IEP Guidance document outlines a Seven Step Process for designing IEPs that
are aligned to academic standards.
For questions about alternate participation requirements, please contact Linda Lamirande [email protected]
or call 303-866-6863.
When are standards included in the IEP?
For the seven step process on writing standards-aligned IEPs, Step 1 is reviewing the standards, step 2 is collecting data,
and step 3 is the Present Levels of Academic Achievement and Functional Performance. Thus, the recommendation is to
include standards before developing the goals, step 4.
June 2015
Will there be new model forms?
The ESSU is currently undergoing a vetting process to make a few modifications to the IEP. These changes will be made
through the collaboration of the ESSU with the State Director’s Leadership Team (SDLT). In addition, the current model
form for the Notice of Meeting includes language that has become outdated, i.e., the form references “present levels”
instead of “present levels of academic achievement and functional performance” and references “goals and objectives”
when objectives are optional unless the student’s instruction is based on expanded evidence outcomes. Those two
changes will be made to the model form for the Notice of Meeting.
JULY 2015
RESULTS DRIVEN ACCOUNTABILITY: CONTINUOUS IMPROVEMENT PROCESS FAQs
27
Parent Survey and Indicator 8
July 2014
Will the family survey be added to Enrich?
At this point in time, there are no plans to incorporate the family survey in the ESSU Data Management System for
completion by families. It will include the capability of entering the data, either through a file import from the CDE’s
online survey tool or from data entry of paper‐based surveys mailed to the CDE. CDE ESSU staff will enter the data from
any paper‐based surveys received.
March 2015
Indicator 8
The Exceptional Student Services Unit (ESSU) reports the results of the various indicators to the Office of Special
Programs (OSEP) every year as a reflection of their annual performance report (APR) for the State’s Performance Plan
(SPP). There are compliance and performance indicators. Indicator 8 is a performance indicator. Specifically, the ESSU
provides to OSEP the “percent of parents with a child receiving special education services who report that schools
facilitated parent involvement as a means of improving services and results for children with disabilities.” (20 U.S.C.
1416(a)(3)(A))
How did Colorado determine a definition for family involvement?
In May 2010, under the guidance of Cindy Dascher, Supervisor of Family-School Partnering, the ESSU invited a
stakeholder group to define parent involvement within the context of Indicator 8. The stakeholder group included
school- and district-based educational administrators and practitioners, representatives from the Parent Training and
Information Centers, parents of students on IEPs, and ESSU personnel. The meeting was facilitated by the Mountain
Plains Regional Resource Center. The stakeholder group created the following definition for parent involvement:
Family involvement for improving services and results for children with disabilities means that:
• Students are the center of all problem-solving.
• Family input is actively sought and valued.
• Representation of families from diverse backgrounds is evident at all levels of decision-making at the school and
district level.
• All families and stakeholders (e.g., educators, other school staff, administrators, community members, etc.) have
access to relevant and useful information in a variety of formats, e.g., meetings, phone calls, emails, interpreted
language.
• Effective, ongoing relationships between families and schools are based on mutual trust, respect and acceptance.
• Families and professionals seek to understand and use the different perspectives and experiences they bring to
the table.
How was the parent survey developed?
Since the beginning of the parent survey process for Indicator 8, a variety of sources have been utilized for item
development. In all cases, an Indicator 8 stakeholder team has convened for collaborative decision-making. In the
JULY 2015
RESULTS DRIVEN ACCOUNTABILITY: CONTINUOUS IMPROVEMENT PROCESS FAQs
28
earliest versions of the survey, most items on the survey were compliance based. After the 2010 meeting where the
focus group defined family involvement, the Indicator 8 team reconvened and amended the survey to include items that
reflected this new definition. In addition, the scoring moved form a yes/no (all or nothing) scoring system to a Likertbased scale, with items weighted to align with the new definition (i.e., items that more closely reflected the
characteristics noted above were weighted heavier in the scoring results).
How do I access the results of the AUs’ parent survey returns?
Families have been sending their surveys to the ESSU using various options, i.e., an online survey, by mail, fax, or
telephone. For submissions not entered online through the online survey, ESSU staff have entered the results of the
parent survey into the ESSU Data Management System (DMS). Online results are in the process of being imported as of
this publishing date. AU staff who have been provided access to the DMS and, more specifically, to the Family-School
tab within the DMS will be able to view the results received thus far.
How do I interpret the results that I see in the DMS? What do the different scores mean?
1
5
2
4
3
To assist the discussion of the various metrics for the parent survey, an image of the results is presented, above.
When one looks at the detailed results from the surveys, each record is presented horizontally, with keywords for the
items within the survey listed vertically. An AU is able to determine the number of responses they have received by
noting the number in parentheses to the right of the word, Record (see #1).
Under each record number is a summary of that family’s responses (#2). The results represent the percentage of the
items scored with a 4 or greater by the family as well as the mean Likert across the items scored by the family. Some of
the results are encased in a filled square (see Record 9); whereas, other records do not have a square surrounding the
results (see Record 8). The square indicates those records where the family completed each item in the survey;
whereas, the open results such as Record 8 indicate that the family did not respond to one or more of the items in the
survey.
Across from the main header, Survey, and under the summary results for each record (#3) is the average Likert score of
the family across all items to which they provided a response. In record 9, above, the average Likert score is 4.2. This
result is also noted in the summary square for the record, i.e., 4.2 / 5, meaning that the family provided the rating of 4.2
out of 5 for the school’s facilitation of the family’s involvement as a means of improving services and results for students
with disabilities.
JULY 2015
RESULTS DRIVEN ACCOUNTABILITY: CONTINUOUS IMPROVEMENT PROCESS FAQs
29
In the left column of the results screen (#4), the average Likert score across all respondents is noted for the Survey,
overall, as well as for each subcategory of the survey. In this example, the current results yield an overall 4.0 Likert score
for the survey, with 4.2 on the item which notes the families’ ability to provide input to their students’ assessment plans,
a 4.4 for the school’s teams writing evaluation reports in terms that the families understand, and a 4.5 for sending timely
notices to families.
Finally, the “overall” score in the upper left hand corner (#5) indicates the percentage of families who scored their
surveys with 75% (or greater) of the items with a 4 or 5.
What is reported to OSEP?
The ESSU reports the statewide percentage of families who reported their schools to be facilitative of their involvement
as a means of improving services and results for students with disabilities (i.e., indicated by family scores of 75% or
greater – reflected by #2 in the above sample).
ESSU recently reported that the AUs will no longer be required to report the parent addresses through the data
collections. How did that decision occur?
In the past, AUs would report the addresses for all of their parents if they were “in the sample” for the parent surveys
the following year. Four of the AUs reported addresses every year because they were always “in the sample”. This year,
every AU is “in the sample”, with either 100 or 200 families being surveyed. This would have meant all AUs reporting
addresses for their selected families, placing a considerable burden on the AUs. To ease that burden, the ESSU decided
to use the highest response rate for surveys from the past (i.e., 16%) as a minimum threshold for the AUs who would
need to report addresses. However, this continued to put a potential burden on AUs so through discussion with our
State Director’s Leadership Team, it was decided that no AUs will report addresses through data collections to Data
Pipeline. Rather, if an AU does not meet the minimum threshold for the response rate for the surveys (16%), the AU, in
collaboration with the ESSU, will develop an action plan for increasing their response rates.
Is there a requirement for a specific method to be used by the AU in order to distribute the surveys to their families,
e.g., must the AU mail them?
Each AU will determine the method that best meets their needs for distributing the surveys to their families. This year,
some AUs mailed them to each family in the sample; other AUs used their IEP teams to distribute the surveys in person
to their families; another AU utilized their social workers and interpreters to make home visits to families who do not
speak English. ESSU may conduct an internal study after the close of the survey period to inform AUs on which methods
or strategies appear to result in greater response rates.
April 2015
What does it mean if I have 16% of my parent surveys returned?
If you have 16% of your parent surveys returned, you have met the minimum threshold for the response rate for parent
surveys. This means that you will not need to develop an action plan for 2015-16 designed to increase your response
rate.
Should I aim for a response rate higher than 16% for my parent surveys?
JULY 2015
RESULTS DRIVEN ACCOUNTABILITY: CONTINUOUS IMPROVEMENT PROCESS FAQs
30
With regard to survey data, the higher your response rate, the more you will be able to rely on the data submitted.
With the parent survey data representing a sample of your population (i.e., 100 or 200 families of your total population
of families with children on IEPs), the survey itself is already a proportion of your total population (e.g., if your
population of students on IEPs is 500 then 100 families of 500 total families would mean 20% of your total population).
If your sample’s response rate is currently 50%, it means you have received 50% of the total number of surveys
distributed. In the example presented here, a 50% response rate for a sample that represents 20% of the families in the
AU would mean that you are receiving responses from 10% of the total population (i.e., 50 families). Therefore, when a
survey represents a sample of your total population versus your entire population, it is even more important to have the
highest response rate possible in order to be able to have the results inform your work.
The ESSU will support the AUs by analyzing the effectiveness of various strategies that are currently being used across
AUs to assist in determining which actions appear to result in achieving better response rates.
We requested a second set of the parent surveys to disseminate to our families. When will we receive these surveys?
The ESSU will be sending the second set of surveys within the next week to those AUs that had requested them. The
AUs should be receiving them shortly if they have not already received them.
May 2015
Indicator 8
The Exceptional Student Services Unit (ESSU) reports the results of the various indicators to the Office of Special
Programs (OSEP) every year as a reflection of their annual performance report (APR) for the State’s Performance Plan
(SPP). There are compliance and performance indicators. Indicator 8 is a performance indicator. Specifically, the ESSU
provides to OSEP the “percent of parents with a child receiving special education services who report that schools
facilitated parent involvement as a means of improving services and results for children with disabilities.” (20 U.S.C.
1416(a)(3)(A))
For the parent-school survey, an AU must have at least a 16% return rate. I noticed that the DMS is calculating
percentages of results. Is there a requirement for a specific percentage for overall results? If not, will this be a
requirement in the future?
The ESSU reports the statewide percentage of families who reported their schools to be facilitative of their involvement
as a means of improving services and results for students with disabilities (i.e., indicated by family scores of 75% or
greater). The percentage in the upper left corner of the Survey results indicates the percentage of families who
provided that score via their survey results.
June 2015
How was the weighting of the questions determined?
The following description of the process was first written in the March 2015 FAQ:
JULY 2015
RESULTS DRIVEN ACCOUNTABILITY: CONTINUOUS IMPROVEMENT PROCESS FAQs
31
In May 2010, under the guidance of Cindy Dascher, Supervisor of Family-School Partnering, the ESSU invited a
stakeholder group to define parent involvement within the context of Indicator 8. The stakeholder group included
school- and district-based educational administrators and practitioners, representatives from the Parent Training and
Information Centers, parents of students on IEPs, and ESSU personnel. The meeting was facilitated by the Mountain
Plains Regional Resource Center. The stakeholder group created the following definition for parent involvement:
Family involvement for improving services and results for children with disabilities means that:
• Students are the center of all problem-solving.
• Family input is actively sought and valued.
• Representation of families from diverse backgrounds is evident at all levels of decision-making at the school and
district level.
• All families and stakeholders (e.g., educators, other school staff, administrators, community members, etc.) have
access to relevant and useful information in a variety of formats, e.g., meetings, phone calls, emails, interpreted
language.
• Effective, ongoing relationships between families and schools are based on mutual trust, respect and acceptance.
• Families and professionals seek to understand and use the different perspectives and experiences they bring to
the table.
How was the parent survey developed?
Since the beginning of the parent survey process for Indicator 8, a variety of sources have been utilized for item
development. In all cases, an Indicator 8 stakeholder team has convened for collaborative decision-making. In the
earliest versions of the survey, most items on the survey were compliance based. After the 2010 meeting where the
focus group defined family involvement, the Indicator 8 team reconvened and amended the survey to include items that
reflected this new definition. In addition, the scoring moved form a yes/no (all or nothing) scoring system to a Likertbased scale, with items weighted to align with the new definition (i.e., items that more closely reflected the
characteristics noted above were weighted heavier in the scoring results).
JULY 2015
RESULTS DRIVEN ACCOUNTABILITY: CONTINUOUS IMPROVEMENT PROCESS FAQs
32
Performance Data
July 2014
Outcomes are very much more of a general education element – like graduation. How can we really impact
outcomes without general education?
The ESSU is hopeful that the process of reviewing data and analyzing root causes would provide an incentive for
conducting these analyses in partnership with general education leadership. If not the actual data and root cause
analyses, the determination of whether the AU Improvement Plan converges with or diverges from the district’s
(districts’) Unified Improvement Plan may also serve to initiate conversations with the district leadership.
October 2014
How do we balance the criteria for identifying a student with Specific Learning Disabilities against the need to
demonstrate improved student outcomes? Can we include partially proficient in our calculations of “proficient”? In
addition, how might we incorporate our successes when students are proficient and are staffed out of special
education?
OSEP requires that the states whose accountability is determined by an ESEA Flexibility waiver (e.g., Colorado) use the
Annual Measureable Objectives (AMOs) as their measures of achievement for Indicator 3 via set targets. As a
component of our Flex waiver, Colorado no longer uses partial proficiency as a component of proficiency. Therefore, the
ESSU is bound to use terms that define proficiency as proficient or advanced for all eligibility categories. The ESSU will
consider including exited students into the Results Matrix when the work group meets.
How will AUs be able to access the needed data? Some AUs have more difficulty than others in gaining access to
performance data. Will there be a formal process developed?
The ESSU has recently created a position of Supervisor for Data Achievement and Accountability on the Results Driven
Accountability team. A process is being developed for providing access to data summaries for state assessments in the
ESSU Data Management System. Data portfolios will be uploaded to the Performance area in this system. In the
meantime, the ESSU is providing data to those AUs that require data or additional technical assistance. Finally, the state
is in a period of transition of their state assessments to PARCC and Dynamic Learning Maps; therefore, it will be
imperative that each AU identifies local sources of achievement and growth data that will provide the means to monitor
progress over time.
How will AUs access technical assistance in reviewing and analyzing achievement data?
The ESSU will actively seek resources to share with AUs. The AU Director and AU Partner will work together to identify
what technical assistance is needed and the AU Partner will bring these priorities back to the ESSU for consideration. The
ESSU will gather all these needs and develop a statewide process for responding to priority needs.
JULY 2015
RESULTS DRIVEN ACCOUNTABILITY: CONTINUOUS IMPROVEMENT PROCESS FAQs
33
Consider a task force with regional representation for student achievement – focused on evidence-based practices
and quality instruction.
The ESSU utilizes a variety of stakeholder groups and organizations to serve as task forces in the area of the Continuous
Improvement Process. The State Director’s Leadership Team (SDLT), the Reinventing Special Education group, and the
Colorado Special Education Advisory Committee (CSEAC) comprise three of the various groups that regularly work with
the ESSU. In addition, a variety of other stakeholder groups will be involved as a component of the development of the
State Systemic Improvement Plan. For example, the ESSU is establishing a new work group that will develop a Results
Matrix to include growth data considerations.
December 2014
Will someone from the ESSU be able to help an AU in looking at their data?
If an AU feels that technical assistance is needed in order to review their state level performance data, the AU Director
may reach out to the AU Partner to make that request. It would be helpful to provide as much specificity as possible
with regard to the needs with data interpretation so the appropriate technical assistance can be provided.
February 2015
What support can CDE give to AUs, specifically BOCES, for data?
CDE provides data to the public via various outlets. Enrollment and demographic reports specific to students with IEPs
are available at http://www.cde.state.co.us/cdesped/sped_datareports. Achievement data are available at Data Center
and Data Lab at School View (http://www.cde.state.co.us/schoolview). If you need data unavailable at these publicly
facing tools, please feel free to contact the data team directly through the supervisor of the data team, Miki Imura;
[email protected].
Once the Data Management System’s performance tab is available, the data team will post PowerPoint slides that
summarize each AU’s performance on the state assessments. If you need further assistance in understanding the data or
would like more detailed data, please do not hesitate to contact Miki. Some BOCES have data sharing agreements
between the BOCES and their member districts. In such cases, the data team will provide data that contain n < 16. If the
BOCES does not have the signed data sharing agreement, CDE ESSU regretfully cannot provide deeper level data to the
BOCES (i.e., no district, school, or student level data).
How do we deal with the fact that the ‘n’ size really impacts data outcomes?
When a group contains a small number of students, it can be more easily influenced by outliers (e.g., one student
scoring with an extremely high or low score). Attempts to make generalizations from the data when the sample size is
so small are typically prone to error. In an attempt to avoid this problem and also to protect students’ privacy, CDE has
the n<16 rule – i.e., CDE does not provide data to the public that contains less than 16 students. The ESSU will be able to
provide data to the AU Director when the sample sizes are less than 16; however, these data may not be shared in any
publicly facing way. If the AU is a BOCES, a data sharing agreement must be in place. If the AU is a district or school
with a small number of students, it is recommended that student data be examined individually rather than as in the
aggregate, as such examination should be more meaningful and free of error.
JULY 2015
RESULTS DRIVEN ACCOUNTABILITY: CONTINUOUS IMPROVEMENT PROCESS FAQs
34
JULY 2015
RESULTS DRIVEN ACCOUNTABILITY: CONTINUOUS IMPROVEMENT PROCESS FAQs
35
Post School Outcome Interviews
July 2014
Will CDE continue to work with the University of Oregon and Post-School Outcomes for data analysis? Timeframe is
too narrow for staff who come back to work August 15th to get calls done by September 15th. How and when will we
get our data? Concerned about how long it will take to get results back from CDE.
The CDE ESSU is currently participating in a project with the National Post‐School Outcomes Center (NPSO) and is in the
final year of this project, in which the NPSO Center is providing technical assistance to the CDE. The ESSU plans to
incorporate the technical assistance from the NPSO Center in providing training and support this fall to the AUs in
conducting the post‐school outcomes interviews using Indicator 14 required procedures.
April 2015
When can I start the Post School Outcome interviews?
The Post School Outcome interviews occur at least one year from the date of exit of the student. If a student exited in
December, s/he could be interviewed now. If the students exited in June 2014, the Post School Outcome interviews
should proceed once the anniversary date of exit has passed.
JULY 2015
RESULTS DRIVEN ACCOUNTABILITY: CONTINUOUS IMPROVEMENT PROCESS FAQs
36
Record Reviews
July 2014
How does CDE check that the AU is reviewing IEPs correctly? Will CDE check every IEP as well?
Before the AUs will conduct their record reviews, the CDE ESSU will provide training in the process of conducting a record
review. After the initial training, when the AU begins their record reviews and enters the data in the ESSU Data
Management System, they may request that CDE ESSU work with them to establish inter‐rater reliability between their
analysis and ESSU regarding the required elements of the review.
October 2014
When does the inter-rater reliability training occur? Before or after directors review their files?
The ESSU provided training in October 2014 for conducting record reviews as a component of the Continuous
Improvement Process. AUs had been provided with their lists of randomly generated students for these record reviews.
NOTE: Transition IEPs must be reviewed by December 1 to allow for Indicator 13 reporting on the State Performance
Plan. All other record reviews are to be completed by June 2015.
With regard to inter-rater reliability checks, AU Directors may request to have the ESSU conduct activities to establish an
inter-rater reliability agreement between the AU and the reviewing team from the ESSU. If requested, the ESSU will
review the records from the AU’s random sample and the AU, along with the AU Partner, will analyze the results to
establish the inter-rater reliability agreement and determine if there is need for additional technical assistance.
Will CDE review the AUs’ record review results?
There will not be repercussions for the results of AU reviews; rather, the ESSU will partner with the AU to provide any
needed training or technical assistance that might be identified through the AU’s review process. The AU Partner will
discuss the results of the AU’s review and collaborate with the Director to support any potentially needed technical
assistance. The AU may opt to provide trainings on their own or they may request the support of the ESSU. If there are
areas of need, the AU Partner and Director will monitor the results of trainings to determine if areas of need are
resolved.
If an AU opts to conduct their record reviews on all areas of compliance rather than the smaller number of items
required for submission, the AU will have 2 sets of results on its dashboard, i.e., the percentage for compliance on items
required for submission (Standard review) and the percentage for compliance on all items (Comprehensive review).
There will not be “repercussions”; rather, the AU Partner will work with the AU to assist staff in meeting any identified
needs as a result of those comprehensive reviews.
What if an AU wishes to “opt out”? Will there be repercussions?
There is no “opt out” option for this process. AUs may face repercussions to be determined by the CDE ESSU and which
may include an impact on IDEA funding.
JULY 2015
RESULTS DRIVEN ACCOUNTABILITY: CONTINUOUS IMPROVEMENT PROCESS FAQs
37
Does CDE have an estimate of time it will take to complete reviews? Approximately how long to complete one IEP
review – 30 mins, 2 hours, 3 hours…?
There has not been a baseline determined for the time required to conduct reviews. It is expected that this timeframe
will vary, based on the IEP.
Concerns regarding the continuity of file reviews from AU to AU. What will the process be for ensuring reliability?
Trainings will be provided to Directors groups. Group training should increase the probability of continuity. In addition,
the ESSU will institute a process for establishing inter-rater reliability among the AU Partners. AUs may request to have
inter-rater reliability checks with the ESSU and there may be requests made to the SDLT to assist in establishing certain
baseline criteria.
November 2014
Will the checklist be aligned with the flow of the IEP?
As a result of the initial training in the record review process, the sequence of the questions within the Compliance tab
have now been restructured to more closely align with the flow of the IEP document. This will aid the review of the
record, reducing the need to move back and forth within the IEP in order to respond to the questions.
Can we use current IEPs? Or do we have to use last year’s IEP?
Except for the completion of the record reviews for Transition IEPs, which are due December 1 in order to include the
data for Indicator 13 in the State’s Performance Plan, the AU has the school year to complete the reviews for those
records randomly selected for the 2014-15 school year. Therefore, the AU may opt to conduct reviews of IEPs that were
completed during the current school year rather than using last year’s IEPs. The choice of which IEPs to review is at the
discretion of the AU Director.
What is the timeline for the ability to do real-time IEP reviews?
The ability to do real-time IEP reviews is the next planned development for the Record Reviews in the ESSU Data
Management System. It is expected that this option will be available to the AUs within the next couple of months.
How do we define the quality criteria?
As the AU reviews IEPs, there are questions presented for each section of the IEP. There are also appendices which allow
for more specific analyses beyond the IEP, e.g., evaluation reports. The AU Director and his/her team of reviewers will
establish the criteria for each question with regard to a reviewer assigning “yes” or “no” to each question. The Director
may choose to have the criterion be “meets compliance” or the Director may choose to develop quality criteria that
include compliance PLUS the added expectation for specific content that will have a greater likelihood of accelerating
student outcomes.
Will there be state standards for each section of the IEP?
At this point in time, there are no expectations to develop state standards, beyond compliance, for each section of the
IEP.
JULY 2015
RESULTS DRIVEN ACCOUNTABILITY: CONTINUOUS IMPROVEMENT PROCESS FAQs
38
Where can we find the handout for the Compliance training for uploading documents?
The ESSU has posted this document to the Special Education Directors’ Corner on the CDE website.
Is there a maximum file size for the zipped file for upload?
At this point in time, there is not a maximum file size for uploading the zipped file to the ESSU Data Management System.
For people who have recently come on board with Enrich and had their data cross-walked from Encore into Enrich,
should we use the cross-walked IEP, i.e., CW IEP?
The CW IEP is a complete IEP. The crosswalk also included a pdf of each IEP. The AUs with cross- walked IEPs may
choose to use those documents for their reviews. As noted previously in this FAQ, the AU may opt to use last year’s
IEPs which in this case would be the CW IEPs or they may choose to do IEPs completed in the current school year.
Will there be Tip Sheets like the Indicator 13 Compliance Tips?
At this point in time, the ESSU has not developed additional resources such as the Indicator 13 Compliance Tips.
However, this is a great suggestion and will be considered for future enhancements and support of this process.
Will there be step-by-step tutorials for using the system?
The ESSU has plans to develop a library of tutorials or resources for the AUs, called Your On Demand Educational
Library (YODEL). This library will offer a variety of short video tutorials. These tutorials will be developed to
demonstrate how to use the ESSU Data Management System.
At some point, will there be exemplars?
There are always pros/cons to posting exemplars with regard to IEPs. As an individualized process, each IEP is
developed specifically for the student for whom it is intended. What is exemplary for one student’s IEP may not apply
to another student’s IEP. On the other hand, there is value in having “exemplars” of well-written and subpar IEPs
available to the AUs to use when training reviewers and service providers. This is an option that the ESSU may
explore for further support of the AUs.
December 2014
How would an AU send completed paper record reviews to the ESSU? What identifying information would be
needed?
As noted above, the AU Director would need to contact Cindy Millikin to request this assistance. The transfer of the
paper-based record reviews would be accomplished via a secure transfer exchange, e.g., either through the CDE File
Transfer Protocol (FTP) site or via the attachment feature within the DMS itself. The required identification elements
would be determined during that communication exchange.
Do the participants on the Notice of Meeting need to match the participants page, i.e., who attended the meeting?
JULY 2015
RESULTS DRIVEN ACCOUNTABILITY: CONTINUOUS IMPROVEMENT PROCESS FAQs
39
For compliance purposes, the Notice of Meeting must report the titles or positions of those individuals who will be at
the intended meeting, not who might be at the meeting. Therefore, the participants expected should match those who
attended. As per the procedures in place when there is an absence of expected participants at a meeting, the IEP team
should work with the family. Documentation should be maintained that reflects any agreements made between the
family and the AU regarding team participation and excusals.
If we only need the 8 transition questions for Indicator 13, are we supposed to do the whole Standard Review or only
the 8 questions?
With regard to conducting reviews of students with Transition IEPs, the entire Standard Review is required by May. The
reviewers may find it difficult to restrict responses to only the 8 Indicator 13 questions; however, it is up to the AU
Director to make that decision. If only the responses to the 8 questions are provided by December 15th, the rest of the
Standard Review is expected to be completed by the end of May.
When we were doing our Transition IEP reviews, we found statements like this for the transition services area:
"Special Education and General Education staff will support "student" in ....etc...". We understand that technically
this may meet the transition service requirement, but when the ESSU provided our training for record reviews, it was
emphasized that IEP teams report these transition services responsibilities specifically by professional. This may not
need an amendment, but we feel the need to communicate this change to our staff. What should we do?
The AU will not need to amend the IEPs with statements like the one described; however, it would be important to
provide follow-up and training to the teachers who are not identifying the transition services as clearly and specifically
as they should. Feel free to use the Indicator 13 Compliance Checklist and the Indicator 13 Tips sheet to provide
examples and training to your staff.
Which students receive modification on their IEPs, e.g., only students with significant cognitive delays?
As per IDEA, any student may require modifications as per his or her IEP team. There are important factors for the IEP
team to consider when determining if a student needs specific instructional adaptations to ensure accessible and
meaningful engagement with enrolled grade-level topics. The vast majority of learners with IEPs (99%) receive their
instruction based on grade-level Colorado Academic Standards (CAS). As such, the provided adaptations will not
significantly change content, complexity, and rigor. In this scenario, the student is still expected to achieve the gradelevel CAS.
Modified instruction is necessary for students with a significant cognitive disability who meet participation requirements
to receive instruction based on alternate academic achievement standards (Colorado Academic Standards/Extended
Evidence Outcomes). In this situation instructional content is modified significantly for content, complexity and rigor to
allow the student to engage with enrolled grade-level topics based on the alternate achievement standards. These
modifications change the depth of what the student is expected to learn and the academic achievement standard by
which the student is evaluated.
Why do we need to scan the IEP documents into the DMS?
The DMS was designed to provide side-by-side presentation of the record review questions with the actual documents
being reviewed. Procedures were developed for the process for loading the documents, both by AUs implementing
Enrich as well as those AUs who have not opted to implement Enrich. With this design, the ESSU will be able to assure
the Office of Special Education Programs that the ESSU has the opportunity to verify any data that are entered into the
JULY 2015
RESULTS DRIVEN ACCOUNTABILITY: CONTINUOUS IMPROVEMENT PROCESS FAQs
40
system. In addition, the ESSU may also need to review AUs’ documents when there are periods of noncorrection of noncompliance within a timely manner. Finally, at the request of the AUs, the ESSU will be able to conduct documented
inter-rater reliability checks regarding the review of the uploaded documents.
On a side note, the IEP documents may not need to be scanned. The upload requires a pdf; therefore, if the IEP system
being used by the AU produces a pdf of the IEP, the file is what is uploaded to the DMS rather than the document itself
requiring scanning, per se.
Do we need to upload the most recent IEP, the latest re-evaluation, and the initial?
The AU Directors may choose to use last year’s IEPs or the most recent IEPs. When the IEP includes an eligibility
meeting, the evaluation/reevaluation documentation should be uploaded and the appendix for evaluation/reevaluation
should be completed. When responding to the first two questions of the Standard Review, i.e., where the questions ask
if the IEP was reviewed within one year of the prior IEP and the evaluation was conducted within 3 years of the prior
evaluation, the AU does not need to upload the prior IEP and the prior evaluation but will simply respond to the
question after referring to their locally stored documentation.
How do I enter the results of the record reviews into the DMS?
When the AU team member enters the DMS, s/he chooses the Compliance link on the right side of the opening screen,
under the banner that reads “State Monitoring”. Upon reaching the summary screen, with the year’s Standard Review
written in blue and the categories within the review listed down the left side, the individual should click on the words,
Standard Review, to move to the record review area.
If the AU is an Enrich AU, the documents should have been previously loaded from their Enrich AU system. If this step
has not been completed, exit the DMS and open your Enrich system. On the right side of your home screen in Enrich,
under State Monitoring, click on Upload Documents. At the next screen, click on the hyperlink referencing the CDE
review. You will then see the SASID list on the left side of the screen and the individual students on the right side of the
screen, with the documents currently stored in Enrich for those students. Click on the + sign by the records you wish to
upload for your review then click Upload at the bottom of the screen. Return to the DMS, following the procedures
noted above. Once you reach the record review area, the records will be represented by numbers at the top of the
columns. If there are more than 10 records in your data set, there is a horizontal scroll bar at the bottom of the screen
to move through your records in groups of ten. To locate a specific record, hover your mouse over the record number
and the SASID will appear; or, you can click on Select Records which will provide a list of the records in your reviews in
order of their appearance in your display. Once the desired record is located and the number of the record noted,
return to the horizontal list of the records in the review area and click on the hyperlink for the record number that
matches the desired record.
If the AU is a non-Enrich User, click on Provide Documents. At the top of the dialogue window is a command to Upload
Documents. Follow the directions on the screen for pointing the system to your compressed zipped folder that contains
your folders of files for each student (see handout from October training for uploading documents). Once the
documents are uploaded, you will see the records represented by numbers at the top of the columns. If there are more
than 10 records in your data set, there is a horizontal scroll bar at the bottom of the screen to move through your
records in groups of ten. To locate a specific record, hover your mouse over the record number and the SASID will
appear; or, you can click on Select Records which will provide a list of the records in your reviews in order of their
appearance in your display. Once the desired record is located and the number of the record noted, return to the
JULY 2015
RESULTS DRIVEN ACCOUNTABILITY: CONTINUOUS IMPROVEMENT PROCESS FAQs
41
horizontal list of the records in the review area and click on the hyperlink for the record number that matches the
desired record.
Click on the Compliance tab to return to the overall Summary of the reviews’ results.
Will the ESSU set criteria for the questions on the record reviews?
Each AU Director will determine the criteria to be used when considering the Yes/No responses to the questions in the
record reviews as adequate (Yes - green) or inadequate (No - red). Some AU Directors have reported using a bottom bar
of compliance for their first analyses of reviews. Once compliance is found across the board, some Directors have
reported that they plan to raise the bar and move the criteria to other qualitative indicators (determined by the AU
Director). At this point in time, there are no plans to have ESSU develop criteria beyond the expected compliance.
Will we be able to conduct inter-rater reliability checks in the DMS?
Upon request, the AU Directors may request to have an inter-rater reliability check conducted with the Accountability
Specialists at the ESSU whose primary responsibilities include monitoring and compliance.
Do we conduct record reviews if the student is at an eligible facility?
The AU is the responsible entity for the IEPs developed for students attending CDE-approved facility schools. As such,
they should conduct the reviews of any IEPs in their samples that were written by staff at the facility schools. The AU
should collaborate with the IEP team at the facility school if there is a need for corrections or amendments.
February 2015
Some of the questions on the paper checklist do not match the wording for questions in the data management
system. Have these items been revised to reflect the information requested?
With the delay in the initial development of the Compliance functionality in the ESSU DMS, the ESSU created a paper
version of the checklist so that AUs could begin conducting their record reviews on paper (i.e., to be transferred into the
DMS once “live”). When the DMS Compliance functionality went “live” and AUs were entering their data into the
system, it became apparent that there were a few questions that needed adjustments or clarification.
With regard to the local vs individual standards item, it became apparent that the language was too vague; therefore,
two more explicit items were posted to replace that original item.
With regard to the item that referenced whether the data found in a reevaluation was in contradiction to the
information in the Evaluation report or IEP, it became apparent that the question’s polarity was reversed, i.e., if the AU
answered with the appropriate “no” response, the “lifesaver” for that item was red, indicating a negative value.
Therefore, this question was re-worded in the DMS so that it reads in the reverse, i.e., the data found in the reevaluation
was consistent with the information in the Evaluation report and the IEP.
As of this publication, the ESSU DMS is “live,” so record reviews should be entered directly into the DMS to avoid
duplicative work. Access is available at all times from any browser, iPad, etc..
JULY 2015
RESULTS DRIVEN ACCOUNTABILITY: CONTINUOUS IMPROVEMENT PROCESS FAQs
42
March 2015
Is there a way to do a query to find out trends related to the “red” portion of the lifesavers without having to open
each individual record, i.e., we need usable data to identify potential concerns related to the offering of FAPE.
As the ESSU has completed the development of the ESSU Data Management System, it is now time to review the data
that are generated in the various areas and decide which data are needed in summary form to inform the work of the
AUs and the ESSU. Over the next 6 months or so, the ESSU will be developing specifications for reports that will be
required from the data in the DMS. The AUs are encouraged to view the data that are in the ESSU DMS and make
specific requests to Cindy Millikin regarding reports they would like to see developed. There will be reports designed to
support the work of the ESSU but there will also be reports that will be available to each AU to summarize their data.
April 2015
When do we need to have all of our record reviews completed?
The Transition IEP record reviews were completed in December 2014. The record reviews for the remainder of the
sample need to be completed by June 1, 2015.
Is an AU able to request assistance from ESSU for the record reviews? What type of technical assistance is available?
The Accountability Specialists at the ESSU are able to work with any AUs that request technical assistance. The AU
Director may make requests such as having a team from ESSU conduct review records with the AU team; requesting an
inter-rater reliability check be conducted by the ESSU, i.e., where the ESSU reviews a sample of the AU’s records without
initially viewing the AU’s responses to determine inter-rater agreement; or, the AU Director may request specific
technical assistance around a particular question from the record review. To request technical assistance, please
contact your AU Partner.
May 2015
Can an AU still use the paper checklist that was distributed in the fall of 2014 for conducting record reviews?
The paper review checklist that was distributed last fall should no longer be used. That checklist was developed as an
interim process for November and December 2014, when the ESSU DMS was late in opening and there was a need to
have the data from the Indicator 13 Transition IEP reviews for federal reporting. In addition, documenting reviews on
paper results in duplicative work and increases the potential for error when responses are transferred into the DMS.
Record reviewers may use a variety of portable technology to conduct record reviews directly in the DMS, e.g., iPads,
iPhones, Macs, thus removing the need for a paper process.
What is the plan if an Administrative Unit does not complete the student record reviews?
JULY 2015
RESULTS DRIVEN ACCOUNTABILITY: CONTINUOUS IMPROVEMENT PROCESS FAQs
43
The record reviews are due June 1st. In the meantime, the ESSU AU Partners will be reaching out to their AUs if their
percentages for completed record reviews are low. The ESSU will be offering their support to the AUs in completing
these processes, if requested.
When should amendments be conducted if the AU’s record review finds procedural errors that may indicate a
violation of FAPE?
When the AU’s record reviews find omissions or errors in the IEPs that may indicate a denial of FAPE, the AU must
conduct an IEP meeting or complete an IEP amendment as soon as possible after discovering the error, in alignment
with the AU’s policies and procedures.
June 2015
How will the ESSU use the data from the record reviews in the DMS?
In a recent update to the ESSU Data Management System, the developers built statewide “lifesavers” for each area
within the DMS. These lifesavers will enable the ESSU to see “at a glance” the trends and patterns across the state,
whether it be record reviews, parent surveys, or the fiscal self-audit results. By having these statewide comparisons, the
ESSU will now be able to develop tools and trainings targeted to support the needs of the AUs, starting with those areas
that appear to have the greatest need.
With regard to the compliance reviews, specifically, recorded trainings will be developed in Your On Demand
Educational Library (YODEL) and posted to the website. Future plans include the ability to document the time spent in
reviewing these trainings in order to earn renewal credit for licensing or certifications.
Finally, beginning in July 2015, the DMS will open two reviews for each record, one called Standard Review to record
whether records have met the compliance standards for each area and the other called Quality Review where an AU
team could use this review to determine whether a record meets the local AU’s standards for quality.
JULY 2015
RESULTS DRIVEN ACCOUNTABILITY: CONTINUOUS IMPROVEMENT PROCESS FAQs
44
Resources
July 2014
As our district experiences declining enrollments and significant budget cuts, how do you expect districts with
no staff (other than the Special Ed Director) on staff during summer to complete this?
None of the activities is required to be completed in the summer. The interviews of exited students occur one year
following the student’s date of exit and are reported to CDE between July and September. Record reviews are to be
completed by May of the school year. The AU Improvement Plans and Fiscal Self‐Audits are due in January. It is
suggested that parent surveys be distributed at the IEP meetings.
AUs are being asked to pick up more – we use more FTE to manage. Will there be more flow- through dollars to
help AUs manage this system?
The ESSU is considering a variety of options to support AUs in completing these activities.
Are there costs to the progress monitoring?
The AUs will have flexibility in designing their processes for determining how progress is being made for their students
on IEPs. Some strategies might have costs associated with them; other strategies may be utilizing tools already
developed or purchased for the district.
December 2014
Will there be an operational manual?
The ESSU has plans to develop a variety of tools to support the AUs in learning to navigate successfully within the DMS.
For example, the ESSU has plans to develop a library of tutorials or resources for the AUs, called Your On Demand
Educational Library (YODEL). This library will offer a variety of short video tutorials. These tutorials will be developed to
demonstrate how to use the ESSU Data Management System.
Will face-to-face trainings be necessary for all parts of the DMS?
Once the AU has successfully navigated the system to enter the compliance data into the DMS, it is expected that faceto-face trainings will most likely not be needed for the other areas in the system. The ESSU may publish guidance for the
other areas and offer other support or technical assistance options. However, at the point in time, no plans are being
made for further face-to-face trainings. If an AU Director feels s/he requires technical assistance, s/he may request
support through the AU Partner so technical assistance can be considered.
February 2015
What support can CDE give to uploading IEPs into the Data Management System?
Cindy Millikin will provide direct support to any AU that needs assistance in the process for uploading IEPs into the Data
Management System, whether the upload will occur via Enrich or through uploading a zipped file of records from other
JULY 2015
RESULTS DRIVEN ACCOUNTABILITY: CONTINUOUS IMPROVEMENT PROCESS FAQs
45
IEP systems. There is also a handout with step-by-step directions for the upload process, including screen shots of the
process. Please email Cindy if you need assistance: [email protected]
JULY 2015
RESULTS DRIVEN ACCOUNTABILITY: CONTINUOUS IMPROVEMENT PROCESS FAQs
46
Results Driven Accountability
July 2014
The new process for Results Driven Accountability being initiated by the Exceptional Student Services Unit (ESSU) is in
response to the shift in focus by the Office of Special Education Programs (OSEP) to improve student outcomes. The ESSU
wishes to accomplish this change through a process that will also build and/or strengthen our relationships with the
Administrative Units (AUs) so we can work together to accelerate student growth in the state. To improve
communication, we will also publish FAQ documents whenever we have the opportunity to gather questions from the
field regarding these changes.
JULY 2015
RESULTS DRIVEN ACCOUNTABILITY: CONTINUOUS IMPROVEMENT PROCESS FAQs
47
Samples
July 2014
For districts with fewer than 100 students, what census data do you need?
When the language of the sampling plan refers to “census” data, this means that all students would be in the sample.
For example, with the Post‐School Outcomes interviews, if there were fewer than 100 students ages 16+ years who
exited in the prior year, all of those students would be contacted for the interviews (i.e., a “census” would be taken).
What was CDE’s percent of return on calls and Indicator 14 data collection attempts? What will the percent return
be for districts? Will this influence status of AUs?
With the parent surveys, the response rates have been 6‐12% over the years. With Indicator 14, the reported response
rate is not reflective of the # of students interviewed / # of exited students ages 16+ years of age so it is difficult to
provide an exact percentage. For our last reporting, 470 students were interviewed as “representative” of the state’s
exited students’ post‐school outcomes.
When sampling response rates are low, it is difficult if not impossible to rely on the data to inform practice. As we place
a greater emphasis on improved student outcomes, it will be critical to be able to reference these results. Research has
demonstrated that student outcomes improve when families are engaged in their students’ achievement efforts. The
parent survey was designed to include items that operationalize a definition for “parent participation” that was
developed a few years ago as a result of a specific focus group. Post‐school outcomes reflect the ultimate outcome of
our students and also comprise the data required for submission to OSEP for Indicator 14. Therefore, response rates
need to have a greater degree of representativeness than has previously been obtained from current practice.
October 2014
Will the CDE random sample actually be a review of the files that the AUs have already reviewed or areas that the AU
director (and team) has determined to be areas of need through their self-assessments?
The random sample distributed to AUs for the 2014-15 school year represents the SASIDs of students randomly selected
whose IEPs will be reviewed, whether developed in 2013-14 or 2014-15. These reviews will form the basis for hands-on
practice of the record review process developed within the ESSU Data Management System. AUs will also be able to use
the Data Management System to conduct real-time reviews of IEPs being developed by their staff. As examples,
Directors may decide to conduct reviews of particular staff members’ documentation or may use the tool for all realtime IEPs.
December 2014
Do we need to replace names/SASIDs for students who are no longer there?
We will no longer be replacing SASIDs in the samples. If an AU has SASIDs of students who have graduated or moved,
they must report the SASIDs of those students to Miki Imura along with the reasons why the students’ record reviews
cannot be completed.
JULY 2015
RESULTS DRIVEN ACCOUNTABILITY: CONTINUOUS IMPROVEMENT PROCESS FAQs
48
Which SASIDs can be excluded?
At this point in time, for record reviews, we are only excluding SASIDs of students who have been staffed out of special
education, died, graduated, or moved.
For parent surveys, we are excluding SASIDs of students who have died or moved. If the students have been staffed out
or graduated, it might be to the AU’s advantage to mail the survey to those families, i.e., they may report positive results
of the program.
For post-school outcome interviews, we are excluding SASIDs of students who died, moved, returned to general
education, or transferred to the Colorado Department of Corrections or Division of Youth Corrections.
JULY 2015
RESULTS DRIVEN ACCOUNTABILITY: CONTINUOUS IMPROVEMENT PROCESS FAQs
49
Timelines
July 2014
Why choose this year when there is more than enough to immerse ourselves in on a daily basis?
There is an urgency to improve student outcomes for students with disabilities. In the last year, statewide data analyses
found that improvements in achievement and growth occurred in all subgroups across the state except for students on
IEPs. Performance of students on IEPs reportedly declined.
There was a request from the field that CDE produce a master calendar for all the many “due” dates involved in the
Continuous Improvement Process.
We will be creating a “Year at a Glance” document and these dates will be included in that document.
What are the timelines for CDE to share data with AUs? More regular feedback or given to us prior to January
2015 for our AU Improvement Plan?
CDE ESSU is anticipating being provided with state assessment performance data in mid‐late August. CDE is
implementing a new process for data security and sharing and the development of information reflecting performance
data results will be subject to these new guidelines. Districts receive state performance data in July.
JULY 2015
RESULTS DRIVEN ACCOUNTABILITY: CONTINUOUS IMPROVEMENT PROCESS FAQs
50
Trainings
January 2015
Is it possible to have a master calendar with statewide trainings listed?
The ESSU is in the process of building a comprehensive calendar of events that will summarize all ESSU training events
that are scheduled.
See http://www.cde.state.co.us/cdesped/calendar for the current calendar.
Why do trainings overlap?
At times, the overlap of training events has occurred due to an inconsistent use of calendar tools that are available
within the ESSU. At other times, the overlap of ESSU trainings in an AU or region may have been acceptable because the
target audiences were unique to each event and did not overlap. The ESSU is committed to building routines that will be
implemented with consistency to avoid the overlap of trainings that target similar audiences.
Will the calendar have a registration link?
The calendar will reference the location for registrations.
Will they have at least 30 day notice for all trainings?
The current plan for the submission to the calendar for ESSU events is to have events posted as soon as possible. There
will be a minimum expectation of events being posted in the month prior to their expected occurrence.
JULY 2015
RESULTS DRIVEN ACCOUNTABILITY: CONTINUOUS IMPROVEMENT PROCESS FAQs
51
Transition
February 2015
Is there a decision tree to assist in designing programming for 18-21 year olds?
There is no specific decision tree to assist in designing services for students 18-21 years of age. School districts in
Colorado have developed creative ways of providing transition services to students with disabilities, ages 18-21, who
require services beyond the typical senior year of high school. Frequently, these services are delivered through
community-based or school-based programs that do not fit the mold of a typical classroom or center-based program.
These services are part of the continuum of transition services (1821) that must be individualized and linked to the
students’ identified postsecondary goals. The decision to have a student continue beyond the traditional senior year is
determined by the IEP team and should be based on continued academic and/or functional needs documented in the
IEP. Annual goals linked to the postsecondary goals should drive the services, and progress/mastery of the annual goals
as well as linkages determine the time of graduation/exit from K-12 Services. It is important to have clearly defined
processes for determining schedules, credits, attendance, and designating responsibilities for service delivery.
The contents of this handout were developed under a grant from the U.S. Department of Education. However, the content does not necessarily
represent the policy of the U.S. Department of Education, and you should not assume endorsement by the federal government.
Colorado Department of Education, Exceptional Student Services Unit
1560 Broadway, Suite 1175, Denver, CO 80202
(303) 866-6694 / www.cde.state.co.us/offices/exceptionalstudentservicesunit
Dr. Cindy Millikin, Director of Results Driven Accountability, 303-866-6619, [email protected]
JULY 2015