Wednesday, August 10, 2011

When statistics lie- MCPS and the SAT

MCPS reported more than just a record for average SAT scores in 2010.   They also reported a new record in the percentage of students earning a "college ready score."    More than 50% of test takers received a college ready score of 1650 or better on the SAT.   The Office of Shared Accountability used this data point as further proof that reforms implemented in Montgomery County were producing results.  But when compared to the percentage of graduates who scored a 1650 or higher we see something different.

We see stagnation.

So what happened?   How did MCPS raise the percentage of test takers who scored a 1650 or better without improving the percentage of graduates who scored a 1650 or better?    It seems implausible- until you consider the participation drop off that occurred.

 If (hypothetically) one were to encourage lower scoring test takers not to take the SAT, one might see the kind of data produced in 2010.   One might a series of decling SAT scores become records.   One would see more impressive gains in the groups that were encouraged not to take the SAT. Scoop out the lower performers, and the average rises.  Likewise, when lower scoring students do not take the test- the percentage of test takers who score high (say above 1650) also must go up.  The statistic that will not lie, however, is the percentage of graduates who perform well.   This weeds out flucuations in participation on the SAT. 

Let us now consider Hispanic performance on the SAT over the last five years.   Remember that MCPS reported impressive gains.  They explained the average rose in one year, by more than 50 points.  They further explained that impressive gains were made in the percentage of Hispanic test takers who scored above 1650.   The achievement gap was closing.  Yet the data below leaves serious questions.

Is Montgomery County Public Schools truly the model we have been told it is?  Or does it face some very serious problems that deserve honest and open discussion?   When I look at this data I'm reminded of a story from 2003.

More to come.