Each year, the Office of the State Superintendent of Education (OSSE) reviews statewide assessment administration to ensure the validity and reliability of the assessment results. As part of this process, OSSE reviews test security protocols and data to ensure the tests were administered with fidelity. OSSE takes breaches of test security very seriously, as violations of any nature can jeopardize confidence in the accuracy of our assessment results. At the same time, it is important to note that violations are intended to capture a broad range of issues in test administration and do not necessarily indicate wrongdoing or intentional misconduct.
For the 2017 administration, OSSE implemented a process for an OSSE-initiated and LEA-led test integrity investigations process. Each year, OSSE reviews test administration practices across schools in the District. This review includes analysis of observation reports from OSSE Auditors from onsite test administration auditing, review of OSSE support tickets submitted by LEA/School Test Coordinators, and analysis of during test administration practices for potential test integrity violations. Additionally, OSSE partnered with Caveon Test Security, a nationally recognized firm with substantial experience in conducting test integrity investigations. Caveon conducted data forensics analysis of test response data to identify anomalous results, and conducted post-administration investigations to attempt to understand what may have caused statistical anomalies identified in the test response data. Lastly, OSSE consulted with subject-matter experts in the test security field to ensure that the methodology used to identify schools was sound and up to industry and professional standards.
Schools may be identified for investigation based on one or more of the following sources: anonymous tip/whistleblower; media monitoring (breach of security); data forensics flagging; monitoring reports; violation of test security through a review of test administration practices (incident reports, OSSE support tickets, etc.); and random identification.
The methodology used to flag schools using data forensics included considering similarities in individual students’ response patterns; looking at levels of student response changes (commonly known as “erasure analysis” or “response change analysis”); and looking at unusual changes in scores, year to year. The onsite post-administration investigation led by LEA investigators included site visits, document reviews, and one-on-one interviews.
Test integrity violations are classified in the following manner:
- No findings: No evidence of test security violations, wrongdoing, or error
- Minor: Minor misadministration errors (e.g., incomplete or missing test documents, inconsistencies in the application of administration procedures)
- Moderate: Moderate misadministration errors (e.g., misinterpretation of requirements in the Test Coordinator/Administrator Manuals not related to test tampering or academic fraud; failure to report certain types of incidents; failure to distribute, collect, or refusal to sign Non-Disclosure Agreements)
- Significant: Substantiated isolated test security violations; test tampering or academic fraud (e.g., educator coaching; providing students with answers; allowing student use of calculators or other technology when prohibited)
- Critical1: Substantiated systemic test security violations; school- or LEA-led test tampering or academic fraud (e.g., school- or LEA-level coordination of educator coaching or test tampering; school-wide test fraud)
Please see below the results of the statewide test integrity investigations:
2016-17 Test Integrity Investigation Reports
2015-16 Test Integrity Investigation Reports (no substantiated findings)
2014-15 Test Integrity Investigation Reports (no substantiated findings)
2013-14 Test Integrity Investigation Reports
2012-13 Test Integrity Investigation Reports
2011-12 Test Integrity Investigation Reports
2010-11 Test Integrity Investigation Reports