Data: Meaningful Analysis Can Rescue Schools from Drowning in Data
By Douglas B. Reeves
August 2011
Learning Forward’s Standards for Professional Learning have the potential to influence educational policy and practice in profound ways for the systems that are courageous enough to take them seriously, and the Data standard is a critical element systemwide. Schools are overwhelmed with data warehouses, colorful charts and graphs, and endless PowerPoint presentations. The millions of dollars that governments at all levels are investing in data systems will be wasted unless significantly greater attention is paid to the systematic evaluation of teaching and leadership decisions based on data. However, in many schools, the availability of data is inversely proportional to meaningful analysis. The reality is that many common practices substitute the appearance of data analysis for the reality of substantive analysis
To realize the achievement of the Data standard, we offer three imperatives for school leaders and policymakers. First, close the implementation gap for professional learning standards. To close the gap between the aspirations expressed in the standards documents and the reality of educational systems, leaders at every level must hold themselves accountable for the implementation of the standards. Second, change accountability from an evaluation system, linked to punishments and rewards to a learning system. Feedback for improved performance has a greater impact on morale and productivity than the use of the same data for financial incentives alone. We recognize the present political reality that data will be used for economic incentives; we are suggesting, however, that the massive investment that educational systems are making in data systems could be used for far more constructive purposes. Third, change
Professional learning that increases educator effectiveness and results for all students uses a variety of sources and types of student, educator, and system data to plan, assess, and evaluate professional learning.
data system investment strategy from one that disproportionately allocates resources to hardware, software, and data warehouses to new strategies that disproportionately allocate resources of money and time to data analysis and decision-making processes. With these emphases, the Standards for Professional Learning will have the opportunity to influence student learning and improve teaching and leadership effectiveness. Without these imperatives, however, teachers and leaders will continue to be drowning in data but failing to have the time, professional learning, and leadership support to use data to improve teaching and learning.
CLOSING THE IMPLEMENTATION GAP
Consider the fate of academic content standards over the past two decades. In some schools, standards formed the basis of new curricula, teaching methods, assessments, and grading systems. When the work of students was compared to a clear and objective standard rather than to that of other students, both academic achievement and educational equity improved. Standards-based education allowed researchers from multiple perspectives to document sustained improvements in a variety of schools. Marzano (2007) and Hattie (2009) provide meta-analytical approaches that offer compelling evidence of the impact on student achievement when students have learning goals that are explicit and teachers provide accurate and specific feedback to improve performance related to those learning goals. Hattie in particular describes the power of feedback from formative assessments. Teachers use the formative assessments to provide meaningful recommendations for improved performance to students as well as using that feedback to understand the effectiveness of their instructional practices. Hargreaves and Shirley (2009) and Fullan (2010) complement that research with case studies of sustainable system reform, while Anderson (2010) links specific gains in student achievement to comprehensive and consistent data analysis. Certainly the standards movement alone was not responsible for all of these improvements; when the right “constellation of practices” (Reeves, 2011a) came together, improvement was significant and sustained. The last study, including an analysis of student results over three years in more than 2,000 schools, suggested that of 21 teaching and leadership practices observed, effective monitoring of student, teacher, and leadership data was significantly more powerful than other variables, particularly when effective monitoring was combined with leadership focus and teacher efficacy. The research suggests that student success is possible with the right combination of teaching and leadership strategies, and standards for professional learning play an integral role.
STANDARDS ARE NOT ENOUGH
Unfortunately, these success stories are overshadowed by the number of instances in which standards were merely adopted by governing boards and never implemented at the classroom level. Two decades after the dawn of the voluntary standards movement and one decade after No Child Left Behind required all states to have academic content standards supported by standards-based assessments, there remain an astonishing number of schools where instruction and assessment are indistinguishable from 1991. Despite a blizzard of standards, pacing guides, and mandates from federal, state, and local education policymakers, the fact remains that the same performance by the same student can yield wildly different evaluations based solely upon the idiosyncratic judgment of individual classroom teachers (Reeves, 2011b), the antithesis of what standards-based assessment should be. Data about the effectiveness of professional development strategies and the implementation of academic standards were the missing links. Leaders and policymakers have the opportunity to learn from the past and immediately begin monitoring the effectiveness of data analysis practices, to begin using data on data to improve learning.
Similarly, data systems have proliferated. Indeed, it is difficult to find a school that does not profess to have teachers and administrators “looking at data.” Given the avalanche of data coming from state and local sources, one cannot avoid “looking at” the data. The question is what teachers and administrators are doing with it.
IMPLEMENTING PROFESSIONAL LEARNING STANDARDS
Consider this sample scoring guide for application of the Standards for Professional Learning for data analysis.
1. Not meeting standards: Meetings are inconsistent and haphazard. There are no agendas and little reference to data. Teachers and administrators are preoccupied by other concerns, including discipline, parent complaints, and policy disputes. While information about student achievement is available, school leaders complain that teachers are “not ready” for this sort of analysis.
2. Progressing: There is a sincere attempt to look at data, but only in the most general form. The threat of complaints prevents any classroom or student-level analysis, so the data analysis leads only to platitudes about “working smarter” and there are not explicit instructional or leadership decisions that emerge from the meetings.
3. Proficient: Teachers and administrators meet weekly to consider a variety of data sources, including formative and summative assessments as well as teacher observations. Each meeting has written records of decisions and commitments, with explicit teaching and leadership decisions based on clearly identified student data.
4. Exemplary: In addition to all of the characteristics of “proficient” performance, teachers and administrators regularly share their insights with their colleagues, benefitting not only their colleagues within the school, but the entire system. There is clear and compelling evidence that best practices are replicated and ineffective practices are discontinued. We have created other scoring guides for data analysis that are more detailed (available as free downloads at www.LeadandLearn.com), but our experience suggests that when it comes to creating rubrics for professional practices, specificity, clarity, and brevity beat complexity every time.
This brief example illustrates how schools can transform standards for professional learning into practical guidelines. Each staff meeting could conclude with an objective analysis of performance. My colleagues at the Leadership and Learning Center have field-tested scoring guides like this using a combination of direct observation, interviews with teachers and principals, focus groups, and anonymous and confidential surveys. The schools we observed had the same data systems, same professional development seminars, and same state and district mandates for data analysis. Nevertheless, they varied widely in their actual implementation of data analysis standards.
ACCOUNTABILITY AS A LEARNING SYSTEM
The Standards for Professional Learning make a trenchant ethical point that the data that are to be analyzed by schools must include student, educator, and system performance. In brief, effective data analysis must include much more than test scores. The application of a scoring guide for professional practices allows a system to take the standards seriously, examining the interrelationship between professional practices surrounding data analysis and gains in student achievement. Analyzing data from more than 600,000 students in more than 700 schools, we plotted the relationship between effective data analysis and gains in student reading and math scores. The results offer good and bad news. The good news is that there is a clear and consistent relationship between deep implementation of professional practices surrounding data analysis and gains in student achievement. We have found that to be true not only with regard to deep implementation of data analysis, but also other instructional initiatives, such as professional learning communities, positive behavioral support, effective instructional practices, and instructional coaching. The bad news is that in almost every case, the relationship between implementation and student achievement is nonlinear. That is, the impact of implementation on student results does not proceed in a stair-step like fashion, with each increment of improvement in implementation associated with a gain in achievement. In fact, middle levels of implementation — which demand a good deal more effort by teachers and students — have no better results than low levels of implementation. Unless leaders and educators are committed to deep implementation of a relatively small number of instructional initiatives, then they will never have the time and energy to get to deep levels of implementation required in order to influence student achievement in a meaningful way.
Systems that focus exclusively on test scores would be like an initiative to combat student obesity by posting the annual weight scores of every student and exhorting teachers to improve the scores. But with an exclusive focus on weight loss, neither parents nor policymakers would ever know if weight loss was associated with improved diet and exercise or with eating disorders and drug abuse. After all, all we care about is the score. Similarly, the best way to improve average SAT and ACT scores in any high school is to limit the number of students who take those assessments to those with the best academic preparation. Any principal and faculty member who seeks to encourage the broadest level of post-secondary opportunity by increasing the number of students taking the SAT and ACT will almost certainly be punished by an accountability system that focuses exclusively on the average test score. In brief, we must consider causes — teacher and leadership actions — not just effects — student scores. The Standards for Professional Learning make an important ethical statement when they conclude that student test data, without data about inputs such as instructional practice and professional development, are insufficient to improve system performance or inform decisions about professional learning.
Data analysis requires time and practice. Schools that bring in an inspirational speaker to address the faculty on “Data Day” are doomed to disappointment. Only schools that are willing to commit to a consistent and rigorous discipline that includes an examination of data at every level — student, teacher, administrator, and system — will make the leap from intent to impact.
FROM EVALUATION TO ASSESSMENT FOR LEARNING
The reason that so few teacher and administrator evaluation systems provide any opportunity for accountability to serve as a learning tool is that the words “needs improvement” are both rare and an invitation to litigation. As DuFour and Marzano (2009) demonstrated, evaluation scores are so disconnected from reality that they cannot be used as a tool for feedback and improvement. The picture for administrators is even worse (Reeves, 2008), with many leaders never receiving an evaluation and the content of the evaluations deteriorating as experience and placement in the hierarchy increases. We know what to do. Stiggins (2007) has long demonstrated that accountability for learning is the best practice in providing feedback to students. Marshall (2010a &2010b) has demonstrated that rubric-based observations can be provided for teachers and administrators in a way that leads to improved performance through accurate, consistent, frequent, and meaningful feedback. Amabile & Kramer (2011) documented that frequent feedback to improve performance is associated with employees feeling that they are having their best days at work. Strikingly, annual performance reviews, financial rewards, and public recognition weren’t nearly as powerful as frequent and specific feedback. Improved educator performance stems directly from open and honest data on their professional practice.
ESSENTIAL QUESTIONS IN ACCOUNTABILITY FOR LEARNING SYSTEMS
There are two essential questions for any accountability system. First, which specific teaching practices are associated with improvements in student learning? Second, which specific leadership practices are associated with improvements in teaching? As the Standards for Professional Learning suggest in their description of input data, only an accountability system that includes student, educator, administrator, and system performance data will be able to address those questions.
POLITICAL REALITIES AND FALSE DICHOTOMIES
The political reality is that many jurisdictions have made a decision to evaluate and compensate teachers and administrators based solely upon changes in student test scores. Many researchers, Pink (2009) among them, have made the point that extrinsic motivation can be counterproductive and, particularly in the case of student test scores, can lead to a host of unintended consequences. Nevertheless, just because a data system is misused in some areas should not lead to the conclusion that the system is worthless in all areas. As the data from implementation audits demonstrate, it is possible to link professional practices, or input data, with student learning in a constructive manner, even if the same data are misused in ill-advised reward and punishment schemes.
FROM “RESPONSIBLE FOR DATA” TO RESPONSE TO DATA
There is, we believe, a way out of this conundrum. The Standards for Professional Learning suggest that comprehensive data analysis includes not only test scores, but also system, teaching, and leadership observations, as well as a variety of student demographic data. The last of these data elements do not excuse poor student performance, but rather help teachers and school leaders understand potential trends and suggest essential interventions to support student success. The subtle but essential shift in data-based conversations with teachers is a move from the contention that teachers are “responsible for data” — an indefensible position when that data includes multiple factors beyond the control of the teacher — to the contention that teachers and administrators are responsible for their “response to data.” When a student arrives in 9th grade reading on a 4th-grade reading level, that is not the fault of the 9th- grade teachers and administrators, any more than those teachers and administrators are responsible for the height, weight, home life, or housing of that student. Not all of these students arrive in 9th grade with red flags waving, screaming the message, “I need intervention right now!” Some of these students have the social and political skills to finish middle school with C and D grades, with the occasional B because they are a “pleasure to have in class.” Therefore, their needs are not immediately obvious to 9th-grade counselors and teachers looking only at previous transcripts.
However, 9th-grade teachers, administrators, and the educational systems that support them are responsible for how they respond to this situation. When I ask 9th-grade teachers and administrators, “How will the curriculum, schedule, teacher contract, administrative support, and instructional strategies be different for a student who is significantly below grade level in reading?” the most common response I receive is, “It won’t be — the schedule is set.” By contrast, there are high schools that assess literacy for every incoming student within the first two weeks of school and make interventions where necessary, providing double or even triple time for literacy in those critical early months of high school to avoid multiple failures later in life. Data analysis is not about “looking at data” but about responding to data with decisive actions in teaching, leadership, professional learning, and allocation of instructional resources.
BALANCE DATA SYSTEM INVESTMENTS
Data systems are powerful and necessary tools to improve teaching, leadership, and learning. However, data systems by themselves are insufficient for educational improvement. Technology vendors and advocates, along with professional development leaders, must make a fundamental shift in their focus on professional learning from how to use data systems to how to make better instructional decisions based on data contained in those systems. These are two distinctly different skills, and the latter has received short shrift in the past several years. We have found that the best (and most ethical) practice is to make the data analysis training “agnostic,” in the words of one of our colleagues — that is, divorced from vendor of the equipment and software. While data systems may come and go, schools need consistent disciplines (Anderson, 2010) to link data to decision making. Although we have observed school systems that claim to devote one-third of their technology and data analysis budget to professional learning, we find that to be consistently insufficient. The demands of the Learning Forward Data standard do not call for a one-time investment. The links between student data and practices are complex and varied, and therefore schools must be willing to invest time, resources, and intellectual energy on a continuous basis to gain maximum value from their significant investment in the hardware and software to support data systems.
LONG-TERM CHALLENGES
Many school systems are facing their greatest financial crises since the Great Depression. These challenges are not short- term. In many areas the decline in property values will lead to a long-term decline in school system revenues. Federal funds, which mitigated some of the worst financial damage in some schools in the past year, will soon evaporate, returning to their pre-2008 levels or lower. So what happens when the money runs out and the mandates expire? If data analysis for improved teaching, leadership, and learning is based solely on the external stimuli of money and mandates, then it was all a pipe dream, an evanescent vision of what might have been. But if these professional standards have the moral foundation that we believe that they do, then the standards will outlast transitory political and financial conditions and form the basis for generations of improved opportunities for students.
REFERENCES
Amabile, T. & Kramer, S. (2011, May). The power of small wins. Harvard Business Review, 85(5).
Anderson, K. (2010). Data teams: Success stories (Vol. 1). Englewood, CO: Lead + Learn Press.
DuFour, R. & Marzano, R.J. (2009, February). High- leverage strategies for principal leadership. Educational Leadership, 66(5), 62-69.
Fullan, M. (2010). All systems go: The change imperative for whole system reform. Thousand Oaks, CA: Corwin Press.
Hargreaves, A. & Shirley, D. (2009). The fourth way: The inspiring future for educational change. Thousand Oaks, CA: Corwin Press.
Hattie, J. (2009). Visible learning: A synthesis of over 800 meta-analyses relating to achievement. New York: Routledge.
Marshall, K. (2010a). Principal evaluation rubrics. Available at www.marshallmemo.com/articles/Prin%20 Eval%20Rubrics%20Jan%2026,%2010.pdf.
Marshall, K. (2010b). Teacher evaluation rubrics. Available at www.marshallmemo.com/articles/%20KM%20 Teacher%20Eval%20Rubrics%20Sept%202010.pdf.
Marzano, R.J. (2007). The art and science of teaching: A comprehensive framework for effective instruction. Alexandria, VA: ASCD.
Pink, D.H. (2009). Drive: The surprising truth about what motivates us. New York: Riverhead Books.
Reeves, D.B. (2008). Assessing educational leaders: Evaluating performance for improved individual and organizational results (2nd ed.). Thousand Oaks, CA: Corwin Press.
Reeves. D.B. (2011a). Elements of grading: A guide to effective practice. Bloomington, IN: Solution Tree.
Reeves, D.B. (2011b). Finding your leadership focus: What matters most for student success. New York: Teachers College Press.
Stiggins, R.J. (2007). An introduction to student-involved assessment for learning (5th ed.). Upper Saddle River, NJ: Prentice Hall.