Procedures and Guiding Principles for Assessment Decision Making to Insure Consistency, Reliability, and Validity
Inspired by Barbara S. Plake
The following procedures and strategies were created to use for the creation, implementation, evaluation, and reporting of valid assessment.
Procedure to create assessments
A diverse team composed of the following members met once a week for two hours during a school year to create learning outcomes, concepts, and assessment items, for the express purpose of using them to create assessments to measure student achievement and progress on the school and state standards: curriculum director, external consultant, classroom teachers from the grade for which the assessments were being developed and classroom teachers above and below that grade level.
The team considered the following principles when making decisions in an attempt to assure the possibility of maximizing the validity of each item and instrument created. The principles are organized by tasks: development, implementation, scoring and evaluation, and reporting.
Development of assessment instruments
- The construct (what is being assessed) is understood and can be written as concepts and or skills (how the information is expressed mentally) and outcomes (artifacts created by the students to demonstrate their level of understanding).
- The assessment item is appropriate for the construct being measured.
- The assessment item is appropriate for the developmental level of the students.
- Other reasons the student might do well on the construct have been searched for and ruled out.
- Reasons the student might do poorly on the construct, not related to the construct, have been searched for and ruled out.
- Students’ learning of the related information will be facilitated in a similar manner (materials used, kinds of questions asked or tasks given, types of answers expected, time allowance, …) as it will be assessed.
- All necessary and sufficient information needed by the students to successfully complete each item is or will be taught.
- All students have sufficient opportunities of access to learn the necessary and sufficient information needed to be successful on each item.
- The number of items in different areas are proportional to what is emphasized during instruction.
- Cognitive demands of items match the intended interpretations of the assessments.
- Tasks similar to the selected tasks would give similar results.
Implementation of assessment instrument
- Students have been informed of the assessment and feel confident they are ready for the challenge.
- Students have been given opportunities to ask questions about the test.
- Students are motivated to take the test.
- Instructions have been clearly explained to all students in a similar manner.
Scoring and evaluation of assessment instrument
- The scoring key has been validated.
- The scoring process is clear.
- The scoring rubric is clearly understood by all evaluators.
- Performance level descriptors are meaningful.
- Performance level indicators are developmentally appropriate.
- The rubric fits the construct.
- The rubric is congruent with the instructional emphasis.
- The scores reflect the students’ abilities and skills and are not due to scorer bias.
- Scoring is replicable and different raters are confident that similar abilities and skills are scored similarly.
- It is evident that students’ scores indicate that the instructions were clear.
- It is evident that students knew what they needed to do to be successful on the task.
- Performance category and cutscore decisions are sound and defensible.
- Results are consistent with teacher expectations.
- There are previous results to suggest that these results are probably accurate.
- Students’ individual scores on one particular assessment relate to their average scores on other similar assessments. Any performance that isn’t consistent does not relate to the construct.
- Students’ performance on a particular assessment will be helpful to make instructional decisions to facilitate students’ learning.
Reporting of assessment results
- Students understand their score and its relationship to levels of performance.
- Students understand how the scores will be used.
- Score reports are clear.
- Reporting of growth is comparable across time.
- Reports are consistent with the precisions reported or suggested in the report and the actual precision level of the assessment.
- There will be an increase in teacher collaboration until it becomes a significant element of the school culture.
- There will be an increased reporting of students’ ability to learn until it becomes a significant element of the school culture.
- Reporting of decreased behavioral interactions until they become minimal.
- Reporting of decreased referrals for alternative and remedial programs until they become supportive mainstream programs.
- Reporting in decrease of student completion of task completion until virtually nonexistent.
- Reporting of teacher satisfaction in program development and curriculum decisions.
- Reporting of positive teacher and student morale.
- Reporting of student support for the curriculum.