On data driven accontability

Data is powerful because it has the ability to provide the objective evidence we use to support claims valued by society.    Want to know if we are making progress?  Simply monitor the right data.  For example- Our schools are failing- just look at the most recent PISA results from reading and mathematics.   Want to know if students in Montgomery County Public Schools are "college ready?"   There is data for that too.  These selective still frames of evidence help us to build an argument- or in essence tell a story about successes and failures- from which we can hopefully make more thoughtful and socially optimal decisions.   It is thus helpful to think of data as a form of evidence that helps formulate an argument. 

No Child Left Behind has recently made famous this use of evidentiary argument.  If schools fail to meet certain performance thresholds across various demographic groups in reading and math, schools are labeled as "failing."  For this reason, many have hailed NCLB as the beginning of the accountability movement.    The result, of course, is that we now have thousands of failing schools across the United States.

However valuable data can be- and however useful it is in helping to guide decision making, the use of data also has serious though often ignored limitations.   We obsess not over solutions to serious problems, but over the most recent release of data points of dubious value.   As if that one data point could make or break a school system, school, principal, or teacher.  If the data can be moved, the reasoning goes, there must be progress.  Decision makers therefore attack data points, and not the underlying condition or problem which a data point suggest might exist.   Of course, we cannot simply conclude that policy decisions which target specific data points are all wrong all the time;  in fact, very often these decisions are accompanied by positive externalities.  However, it is also true that this type of decision making can prove particularly superficial, if not entirely problematic, when attempting to answer our most pressing issues.

As I was reading All the Devils Are Here: The Hidden History of the Financial Crisis, I was struck by the type of catastrophic mistakes that can result when we try to fix data rather than problems.   By now, we are well aware of the housing crisis that led to the "great recession" of our generation.   In part, I believe this was caused by an obsession with data points.  As the book reveals, in 1980 the home ownership rate in the United States reached a temporary peak of 65.6% and by 1990, this number had fallen to 63.9%.    The country, according to the data, was failing.  A few years later, then President Clinton embarked on a concentrated effort to reverse this trend- announcing the specific goal of increasing home ownership by 8 million families.   The goal was noble- raise the percentage of people who own their homes.   How could we go wrong?

When we attack data points rather than underlying problems- we ask ourselves an inherently different set of questions than when we attack problems.  For instance, when we attack data like home ownership rates, we ask how it might be possible to move one more family, or one million more families, toward home ownership. We concentrate on the result- understanding that by increasing home ownership we realize success.  However, if our recent past is any indicator, this was indeed the wrong question.   To move the data point we did not have to increase the ability of people to afford homes, we could simply relax the  lending standards necessary to qualify for a loan.   Indeed, home ownership at the end of 2009 stood at 67.4%; the past twenty years verify the trend.  But how many people would dare declare we are now better off now than we were twenty years ago?   And how many people who own homes today wish they did not because they are not house rich, but pitifully house poor?

Home ownership is not an end- but one piece of evidence that could support a claim about standards of living.   To truly attack this problem, we must do something other than move an economic needle.   We must address a wide range of economic indicators of well being.  Not a simple chore- nor one that could be accomplished in the short political life cycle of an individual politician.

The same type of data driven decision making is made in education today.  Let me start by offering a common example of how principals might use data to evaluate a teacher.  As we know, a successful teacher will most often have a well behaved classroom. In education we call this skill behavior management. It thus makes sense to use office referrals as an indicator of behavior management.  The premise is simple, a teacher who refers many students to the office over the course of a year has a behavior management issue in his or her classroom.   Likewise, a teacher who refers very few students to the Principal's office has no such issues.   Hopefully by now, you see where this is leading.   A teacher who wishes to appear "successful" simply must forgo sending students to the office; in so doing the teacher eliminates the appearance of behavior management issues.  The principal need not spend valuable minutes or hours reviewing behavioral management techniques with the teacher.   And the teacher need not attend any professional development classes.   The problem- or I should say the appearance of a problem- has disappeared.

This type of decision making, decision making which targets data and the perceptions the data creates- is not limited to classroom teachers or principals.  In fact, I believe it is fully integrated into the culture of school systems nationwide.  To return briefly to the example of No Child Left Behind, we might imagine the following scenario: a middle school fails to meet the testing threshold for NCLB (known as Adequate Yearly Progress) in 8th grade reading.  The school therefore faces a probationary period before being added to the list of  "failing schools."  If the the school moves enough 8th grade students in the right demographic groups across the threshold, the school can once again be considered a darling.   So the school spends time, money, and resources targeting all students that are most likely on the precipice of passing the statewide 8th grade reading test.   The school, in order to avoid being taken over, smartly chooses to ignore students who are above grade level in reading, and furthermore decides to ignore students substantially below grade level (reasoning that no matter what interventions take place- those students are likely to fail).  The school then pumps resources into a group of 20 students in 8th grade reading that will likely make the difference between a "successful" school year and a "failing" one.   The school passes, and the principal and school announce their success.  All the while, there has been no change in the underlying problems that the school community may face.   Nonetheless, system superintendents continue to target data points that create favorable perceptions of their performance, and principals race furiously to make those superintendents happy.

The question is whether the data driven accountability movement will likely lead to the decisions that improve our educational system.   Unfortunately, most of those who make the decisions that affect students are not around long enough to see their impact. As far as careers go, it is much more effective to appear successful immediately.  With only three to  five years before the next big promotion, decision makers must act quickly (just enough time to move data).

Furthermore, adding more data points as evidence of success will do little to address the underlying problems faced by educators today.   Rather, the use of additional data only serves to occupy even more people with the time consuming task of figuring out how to move all the needles in the correct direction.   That is not to say that data cannot be useful as feedback or promising as a tool when considered holistically.  But the current trend that sends teachers and principals after nuggets of pseudo success will only result in more educational stagnation. 

These solutions fix data, but not education.  And we cannot afford to be underwater on our future.