Accountability is More Than Test Scores
By Douglas B. Reeves
1998
Test scores represent only one piece of the accountability data. These test scores should be interpreted in the context of other information about what schools are doing.
THE ACCOUNTABILITY IMPERATIVE
Accountability is an imperative facing every school administrator and board member. Unfortunately, what passes for "accountability" is frequently no more than a recitation of test scores. Such test scores may or may not reflect the accomplishments of students, teachers, and leaders in a school system. Unfortunately, these numbers often influence the jobs of superintendents and principals, property values in a neighborhood, and the confidence of the public in our educational system.
EXCELLENCE AND EQUITY
Some districts view accountability solely as a compilation of test scores. This can become a thinly veiled mechanism to threaten administrators and teachers. Other school districts have found a better way. These districts have transformed accountability into a constructive strategy for informing the public and reforming schools. They have focused attention not only on test scores, but also on the "antecedents of excellence." These antecedents of excellence are those activities in schools that lead to the twin goals – excellence and equity.
FIVE KEY PRINCIPLES
There are five key principles for effective accountability systems: congruence, specificity, relevance, respect for diversity, and continuous improvement. Although many accountability systems incorporate some of these principles, it is rare to find one that includes all of them.
1. Congruence - Standards, assessments, and evaluations of teachers and administrators must be congruent with the accountability system. Many teachers are frustrated when school administrators establish academic standards and then hold teachers accountable for test scores that are only marginally related to those standards. The same frustration afflicts superintendents when a district or state accountability system is announced with considerable fanfare, but the evaluations of administrators and teachers, as well as the assessments of students, fail to match the goals of the accountability system. Accountability is nothing more than a paper tiger without congruence.
2. Specificity - Educational reformers are sometimes better able to articulate grand goals than to identify specific expectations. "Improve reading achievement" is a wonderful goal, but it gives little in the way of specific advice for teachers and principals. A better goal is to "increase by ten percent the proportion of students who receive a score of "proficient" or better on the district reading assessment." Teachers and administrators know precisely what is expected of them and how to prepare students for this challenge.
3. Relevance - Principals and teachers need accountability goals that are directly related to their individual schools. Test scores are related to effects. Meaningful accountability systems have goals that are relevant to causes. In the Milwaukee Public Schools, for example, a nationally recognized effort was undertaken to improve math performance by students. Schools were not only recognized for the achievement of high math scores. They were also recognized for achieving the antecedents of excellence – improving calculator proficiency, establishing "math buddies" programs, increasing parent tutor training, and a host of other accountability indicators – that were directly relevant to the needs of the schools and the overall goal of improved math scores.
4. Respect for Diversity - Effective districts have high expectations for all students — rich and poor, minority and majority. All students are expected to achieve at high levels. Effective accountability systems should provide for some indicators that are uniform throughout the district and others that are unique to each school. A system that combines "system-wide" indicators (such as achievement in math, reading, and science) with "school-based" indicators, can provide this critical balance. In Milwaukee, the accountability system included six system-wide measures and five school-based measures. The school-based measures were chosen by teams of administrators, teachers, and parents. The diversity of the district was reflected in the diversity of the school-based goals. They range from an emphasis on discipline to an emphasis on participation in Advanced Placement calculus classes. All of these school-based indicators had one thing in common: they addressed the individual needs of that school and represented a specific mechanism for helping that school achieve the system-wide goals of higher student achievement.
5. Continuous Improvement - The fundamental idea is this: accountability systems must themselves be accountable. School leaders want to ensure that the focus remains on high student achievement, the expectations of rigor and challenge are consistent, and there is congruence between the accountability system and the expectations of the community. Thus the accountability system cannot be regarded as holy writ, exempt from continuous review and improvement. Stakeholders from throughout the community should regularly review the accountability indicators to ensure these measures continue to meet all of the previous four principles and serve the needs of the community.
THE TYPICAL SYSTEM
Many superintendents will insist, "We already have an accountability system." This is certainly true. Unfortunately, many of these systems share the same characteristics that inhibit their effectiveness.
ERRONEOUS PRESUMPTIONS
First, there is an erroneous presumption that accountability is the same as a compilation of test scores. We would be shocked if our physicians recommended major surgery based on our body temperature, weight, and a few other scattered indicators. Yet we recommend major surgery for our school systems based on test scores of dubious quality and relevance. I am not an apologist for low test scores. However, I do maintain that these scores represent only one piece of the accountability data. Therefore, these scores should be interpreted in the context of other information about what schools are doing.
NO COMPARABILITY
The second characteristic of typical accountability systems is a lack of comparability. The typical "spring-to-spring" comparisons assume a homogeneity between grades that does not exist in many school districts, particularly those with high mobility rates. Even teachers in more stable rural areas are well aware that testing students in a single grade and comparing the fourth graders of 1995 to the fourth graders of 1996 is not a reflection of academic achievement, but rather a reflection of the characteristics of very different students. In Milwaukee, schools were given the choice of making "spring-to-spring" comparisons or "fall-to-spring" comparisons. In schools with high mobility rates, the latter provided a much more accurate indicator of school improvement.
QUANTITATIVE DATA
The third characteristic of typical accountability systems is that they are limited to quantitative data. I have taught multivariate statistics to graduate students and would summarize my knowledge of statistical analysis as follows: numbers alone never tell the whole story. A narrative description of the challenges and opportunities from each school would create a lens through which quantitative data can be viewed. Without this lens of descriptive information, the numbers are sterile and devoid of context.
FIXATION ON THRESHOLDS
The fourth characteristic of typical accountability systems is that they are fixated on thresholds and not improvements. Schools are often given pejorative labels such as "Schools In Need of Improvement" or its unfortunate acronym "SIN Schools." In addition to feeding the dogmatism of those who enjoy attacking public education, these labels are imprecise and inaccurate. They imply a binary system in which schools are either "OK" or "Not OK" and fail to investigate the extent to which schools are making progress and addressing the antecedents of excellence. Two schools might have identically low test scores. However, one that is making substantial progress on the antecedents of excellence (reported in its accountability system) has a significantly different profile than the school that cannot render such as report. Policy makers and the public ought to have sufficient information to distinguish between these two schools.
A comprehensive accountability system is more rigorous than a list of test scores
A BETTER WAY - THE COMPREHENSIVE ACCOUNTABILITY SYSTEM
There is a constructive alternative to the typical accountability system. The Comprehensive Accountability System, such as that implemented by the Milwaukee Public Schools, includes three separate tiers.
First Tier The first tier consists of six "system-wide" indicators such as math, reading, writing, science, grades, and attendance. For each of these six indicators, schools can earn from zero to ten points based on their achievement of the goals established by the Board of School Directors. These high goals are the same for every school in the district.
Second Tier The second tier consists of five "school-based" indicators. In these indicators, each school has identified its own unique antecedents of excellence. These indicators reflect the diversity of the district. Some schools choose to focus on technology, staff development, discipline, or parental involvement. All schools, however, choose these indicators so they can improve their performance on the system-wide indicators. For each of these five indicators, schools can earn from zero to ten points based on their improvement from the previous year.
Third Tier The third tier is a narrative description of the challenges and opportunities facing the school. This page-long narrative allows the reader to better understand the test scores and other data that are contained in the first and second tier. The Comprehensive Accountability System creates balanced incentives, encouraging all schools to meet uniformly high standards while at the same time allowing each individual school to pursue goals that are uniquely relevant to its needs.
POLITICAL REALITIES MUST BE CONFRONTED
There are clearly some political realities that must be confronted by districts attempting to move beyond the "report card" (or a list of test scores) to an improved Comprehensive Accountability System. These realities can be anticipated and the objections they imply can be answered.
RIGOR AND ACCOUNTABILITY
The first and most important reality is that the public demands rigor and accountability. The Comprehensive Accountability System is more rigorous than a list of test scores. It holds schools accountable not only for test results but also for specific activities that are necessary in order to achieve higher scores.
CONSEQUENCES AND INCENTIVES
The second reality is that the public demands consequences and incentives associated with school accountability. A Comprehensive Accountability System, fully congruent with administrator evaluations, will provide this link between public accountability and personal responsibility. On the other hand, it also ensures fairness to superintendents and principals by preventing policy makers from enunciating an accountability policy and then issuing evaluations that are only remotely related to the accountability goals.
FASTER
The third reality is that a Comprehensive Accountability Plan provides faster reporting and analysis of information. The school-based indicators are half the system. These can be gathered and analyzed during the school year. Rather than waiting months for test scores, schools can make "mid-course corrections" that are at the heart of an effective data-driven decision-making process.
CONCLUSION
Demands for accountability are here to stay. Superintendents and school board members must choose whether to allow accountability systems to be a "gotcha!" used by public education critics or to create accountability systems that are rigorous, relevant, fair, and meaningful. The Comprehensive Accountability System is a step in the right direction.
NOTE: This article is based on a presentation to the national convention of the American Association of School Administrators (AASA) in 1997. The author gratefully acknowledges the assistance of Mr. Robert Jasna, Supt.; Mrs. Barbara Horton, Deputy Supt.; Dr. Deanna Housfeld, Assist. Deputy Supt.; and the principals, teachers, parents, and community who served on the Accountability Task Force in Milwaukee.