Starr vs. Weast: The line in the sand

Joshua Starr, the new superintendent of Montgomery County Public Schools, seems to get it.     From the Gazette:

I’m struck by how outcomes-based the organization has been — and I am a believer in outcomes. You have to be accountable and responsible for outcomes. I learned after a few months that there was an orientation toward: Get the result. The shift that I am trying to make is to have the conversation about how we are getting the result. I still expect the result. In fact, I want a better result. ... So it goes back to how do we do it. ... If we orient ourselves to what do we do every day... to make sure that our kids are fully engaged in high quality instruction, what does that mean with our daily work, with teachers, with principals, with kids? And that is the change. I am not big into initiatives. I am not big into naming things and rolling out the banner and saying here is the big thing we are going to do now. It just doesn’t work. What works is having great instructional leaders leading in every building, having good curriculum and professional development and helping teachers and other folks in the system get better at what they do every day with the kids. That is what works...

This is the ideology behind real progress.   Almost makes me wish I still taught in Montgomery County.

Why we lie

Dan Ariely's new book, The Honest Truth About Dishonesty, is on its way out and it is one that all those interested in public policy should consider reading.    "Little lies" bombard us from both sides of the political isle, impeding the pursuit of truth, and in turn, progress.   Dan Ariely considers why most of us lie- at least just a little.   He gives a glimpse into his book in a Wall Street Journal article that is worth a read.

Data driven decision making- The Big Whoops

Two Superintendents improving education
by: mmccabe4724

A sophisticated decision: MCPS creates record SAT scores

The message in 2009 was much more sophisticated than in 2003.  This time, principals would not be asked how they could decrease participation on the SAT in order to raise the averages.  This time, MCPS would just ask a simple question:  "which test is best for you?"      It was so benign, yet so brilliant.   Have a meeting with the administration of every high school, and ask them to communicate to parents and students that they had a choice:  take the ACT or the SAT.   Just use criteria like the one below to help students make their decision:SAT ACT Comparison Checklist

Nothing to it.  This SAT/ACT action plan would then be "customized to meet the needs of individual schools."

Clarksburg High School created a page.

Walter Johnson had a page.

Northwood did this.

Damascus High School made a Power Point.

The plan was perfect.  In 2010, twenty-four of twenty five high schools in Montgomery County saw their percentage of SAT test takers drop.  Participation dropped, average scores rose, and Dr. Weast could continue to promote the MCPS brand name while building his consulting career in- get this- helping other districts use data to increase student achievement.

Want to know what is wrong with our educational system?   Folks- MCPS is a national model for its use of data.   But no one cares about what the data is designed to measure.   We just care about moving the data- which is no way to run an educational system.

An SAT meeting with results that matter

Dr. Jerry Weast likely learned a very important lesson in 2003.   Messaging is important.    By 2009 the message was far more refined, but the results would be no less contrived.  Every high school in Montgomery County would be invited to attend "Leadership Training" on the SAT, ACT, and Accuplacer tests- the three college placement exams.   What would go on at this meeting?  It would  certainly not be a repeat of 2003.

SAT ACT Accuplacer Confirmation


However, the results would be instantaneous, and just in time for Jerry Weast's retirement.   The year after the "Leadership Training" in 2009, 483 fewer students in Montgomery County Public Schools would take the SAT.  Of those 483 students, a disproportionate number would be minority students.   Hispanic participation would drop by 18% in 2010, African American participation would drop by 12%, while white participation would drop by a relatively modest 4%.

 The MCPS Office of Shared Accountability described the decline in this way:

For the MCPS class of 2010, slight declines in SAT participation were balanced by record increases in the number of students who took the ACT in lieu of the SAT.

This was the full extent of the MCPS analysis.    There would be no discussion of the who, why, or how.  There would be no detailed analysis of the massive drop in participation.  The analysis simply claimed that the 483 student decline in SAT participation was balanced by a 267 student increase in those students who only took the ACT.    An odd balancing act to say the least.

But what remained were legitimate questions.   Were the record SAT scores in 2010 worthy of celebration?  Was it a sign that all that had been done in Montgomery County Public Schools over the tenure of Jerry Weast had finally paid dividends?  Or was it as current Board of Education member and then principal, Michael Durso, explained in 2003, a way to make it on 60 Minutes.   Sometimes, the simple answer is the more likely one.

The Montgomery County Public Schools' Fudge Factor: Record SAT scores

Fudging data is now the culture of schools across the United States.   It is not a conspiracy theory.  It is the way that districts and the principals within those districts that "honor" the data, attempt to make themselves look great.   And as a general rule of thumb, the better the"fudger," the greater the reputation that accompanies it.  Let's just take former superintendent of Montgomery County Public Schools, Jerry Weast, for example, the man Jay Matthews once called one of the top ten superintendents in the country.  In 2010, the year he retired, he claimed as one of his top ten achievements as a superintendent, record SAT scores.   Of course, these SAT scores were accompanied by record drops in SAT participation.   Think this was just  an accident?   Or could this be a similar but more sophisticated version of what occurred in 2003, when a Montgomery County Public School's newspaper, Blair High School's SilverChips, reported that principals were actually told by Weast to contrive their SAT scores through participation rates:

At an Oct 2 meeting with high school principals, Weast suggested, according to Principal Phillip Gainous,that students who could not help out SAT scores [for MCPS] and were not ready to take the SATs should be discouraged from taking them," said Gainous.  

Both Principal Daniel Shea of Quince Orchard High School and Principal Michael Durso of Springbrook High School corroborated this statement. According to Shea, Weast wanted schools to *examine the level of students taking the SATs to appropriately limit exposure." The superintendent*s message was clear, said Gainous: *We were told to do it. And the expectation was that we would all go back and do it. 

Elimination of low-scoring students from the general test-taking pool will automatically boost average SAT scores, explained Durso. If we have a certain number of students not take the SATs, then we*ll be on Sixty Minutes because of our [high] scores," he claimed.  

"It's morally wrong. It's illegal," said Blair PTSA President Valerie Ervin, who works for the County Council, regarding Weast*s suggestion.

In 2010, the media picked up the the story and touted  Montgomery County's record, just the way Jerry Weast knew they would. But perhaps the record was not quite what it appeared to be on first glance.   Perhaps we were closer to what Valerie Ervin concluded- something morally wrong.

When statistics lie- MCPS and the SAT

MCPS reported more than just a record for average SAT scores in 2010.   They also reported a new record in the percentage of students earning a "college ready score."    More than 50% of test takers received a college ready score of 1650 or better on the SAT.   The Office of Shared Accountability used this data point as further proof that reforms implemented in Montgomery County were producing results.  But when compared to the percentage of graduates who scored a 1650 or higher we see something different.

We see stagnation.

So what happened?   How did MCPS raise the percentage of test takers who scored a 1650 or better without improving the percentage of graduates who scored a 1650 or better?    It seems implausible- until you consider the participation drop off that occurred.

 If (hypothetically) one were to encourage lower scoring test takers not to take the SAT, one might see the kind of data produced in 2010.   One might a series of decling SAT scores become records.   One would see more impressive gains in the groups that were encouraged not to take the SAT. Scoop out the lower performers, and the average rises.  Likewise, when lower scoring students do not take the test- the percentage of test takers who score high (say above 1650) also must go up.  The statistic that will not lie, however, is the percentage of graduates who perform well.   This weeds out flucuations in participation on the SAT. 

Let us now consider Hispanic performance on the SAT over the last five years.   Remember that MCPS reported impressive gains.  They explained the average rose in one year, by more than 50 points.  They further explained that impressive gains were made in the percentage of Hispanic test takers who scored above 1650.   The achievement gap was closing.  Yet the data below leaves serious questions.

Is Montgomery County Public Schools truly the model we have been told it is?  Or does it face some very serious problems that deserve honest and open discussion?   When I look at this data I'm reminded of a story from 2003.

More to come.