An SAT meeting with results that matter

Dr. Jerry Weast likely learned a very important lesson in 2003.   Messaging is important.    By 2009 the message was far more refined, but the results would be no less contrived.  Every high school in Montgomery County would be invited to attend "Leadership Training" on the SAT, ACT, and Accuplacer tests- the three college placement exams.   What would go on at this meeting?  It would  certainly not be a repeat of 2003.

SAT ACT Accuplacer Confirmation


However, the results would be instantaneous, and just in time for Jerry Weast's retirement.   The year after the "Leadership Training" in 2009, 483 fewer students in Montgomery County Public Schools would take the SAT.  Of those 483 students, a disproportionate number would be minority students.   Hispanic participation would drop by 18% in 2010, African American participation would drop by 12%, while white participation would drop by a relatively modest 4%.

 The MCPS Office of Shared Accountability described the decline in this way:

For the MCPS class of 2010, slight declines in SAT participation were balanced by record increases in the number of students who took the ACT in lieu of the SAT.

This was the full extent of the MCPS analysis.    There would be no discussion of the who, why, or how.  There would be no detailed analysis of the massive drop in participation.  The analysis simply claimed that the 483 student decline in SAT participation was balanced by a 267 student increase in those students who only took the ACT.    An odd balancing act to say the least.

But what remained were legitimate questions.   Were the record SAT scores in 2010 worthy of celebration?  Was it a sign that all that had been done in Montgomery County Public Schools over the tenure of Jerry Weast had finally paid dividends?  Or was it as current Board of Education member and then principal, Michael Durso, explained in 2003, a way to make it on 60 Minutes.   Sometimes, the simple answer is the more likely one.

The Montgomery County Public Schools' Fudge Factor: Record SAT scores

Fudging data is now the culture of schools across the United States.   It is not a conspiracy theory.  It is the way that districts and the principals within those districts that "honor" the data, attempt to make themselves look great.   And as a general rule of thumb, the better the"fudger," the greater the reputation that accompanies it.  Let's just take former superintendent of Montgomery County Public Schools, Jerry Weast, for example, the man Jay Matthews once called one of the top ten superintendents in the country.  In 2010, the year he retired, he claimed as one of his top ten achievements as a superintendent, record SAT scores.   Of course, these SAT scores were accompanied by record drops in SAT participation.   Think this was just  an accident?   Or could this be a similar but more sophisticated version of what occurred in 2003, when a Montgomery County Public School's newspaper, Blair High School's SilverChips, reported that principals were actually told by Weast to contrive their SAT scores through participation rates:

At an Oct 2 meeting with high school principals, Weast suggested, according to Principal Phillip Gainous,that students who could not help out SAT scores [for MCPS] and were not ready to take the SATs should be discouraged from taking them," said Gainous.  

Both Principal Daniel Shea of Quince Orchard High School and Principal Michael Durso of Springbrook High School corroborated this statement. According to Shea, Weast wanted schools to *examine the level of students taking the SATs to appropriately limit exposure." The superintendent*s message was clear, said Gainous: *We were told to do it. And the expectation was that we would all go back and do it. 

Elimination of low-scoring students from the general test-taking pool will automatically boost average SAT scores, explained Durso. If we have a certain number of students not take the SATs, then we*ll be on Sixty Minutes because of our [high] scores," he claimed.  

"It's morally wrong. It's illegal," said Blair PTSA President Valerie Ervin, who works for the County Council, regarding Weast*s suggestion.

In 2010, the media picked up the the story and touted  Montgomery County's record, just the way Jerry Weast knew they would. But perhaps the record was not quite what it appeared to be on first glance.   Perhaps we were closer to what Valerie Ervin concluded- something morally wrong.

When statistics lie- MCPS and the SAT

MCPS reported more than just a record for average SAT scores in 2010.   They also reported a new record in the percentage of students earning a "college ready score."    More than 50% of test takers received a college ready score of 1650 or better on the SAT.   The Office of Shared Accountability used this data point as further proof that reforms implemented in Montgomery County were producing results.  But when compared to the percentage of graduates who scored a 1650 or higher we see something different.

We see stagnation.

So what happened?   How did MCPS raise the percentage of test takers who scored a 1650 or better without improving the percentage of graduates who scored a 1650 or better?    It seems implausible- until you consider the participation drop off that occurred.

 If (hypothetically) one were to encourage lower scoring test takers not to take the SAT, one might see the kind of data produced in 2010.   One might a series of decling SAT scores become records.   One would see more impressive gains in the groups that were encouraged not to take the SAT. Scoop out the lower performers, and the average rises.  Likewise, when lower scoring students do not take the test- the percentage of test takers who score high (say above 1650) also must go up.  The statistic that will not lie, however, is the percentage of graduates who perform well.   This weeds out flucuations in participation on the SAT. 

Let us now consider Hispanic performance on the SAT over the last five years.   Remember that MCPS reported impressive gains.  They explained the average rose in one year, by more than 50 points.  They further explained that impressive gains were made in the percentage of Hispanic test takers who scored above 1650.   The achievement gap was closing.  Yet the data below leaves serious questions.

Is Montgomery County Public Schools truly the model we have been told it is?  Or does it face some very serious problems that deserve honest and open discussion?   When I look at this data I'm reminded of a story from 2003.

More to come.

How records are made

The MCPS press release was straight to the point.  The results were nothing short of miraculous:

Students in the Montgomery County Public Schools Class of 2010 produced the district’s highest-ever composite score on the SAT and dramatically outperformed their state and national peers, according to data released today by the College Board...

...African-American and Hispanic test-takers posted the strongest gains in MCPS, improving on last year’s average composite scores by 49 and 54 points, respectively. Those increases outpaced the growth in scores for Asian students (up 21 points) and White students (up 15 points), and further narrowed the district’s racial and ethnic gap in SAT performance.

It was an exclamation point on the tenure of Superintendent Jerry Weast.   What was at one time falling SAT scores, had turned on a dime, and turned hard.   The proof was in the preverbial pudding.   The accountability department published the remarkable data.   

From: http://www.montgomeryschoolsmd.org/info/pdf/Class2010SAT-MCPS-MD-Nation-9-13-10-final.pdf

The achievement gap was closing on perhaps the most educationally important data point there was, the SAT.   But what was the cause?

Was it excellent teaching?   Fierce Leadership?   I was skeptical.  My search started with the discovery of the  lowest participation rate in ten years, but that would just be the beginning.   It was even more important to figure out who did not participate.   The answer I found was unfortunately, not surprising.   In statistics circles it is well known that as participation increases, averages tend to fall, and vice versa.  

Fantastic rises in achievement were accompanied by equally fantastic decreases in SAT participation.   African American and Hispanic sub-groups led the decline in particpation.   The accountability office said it this way:

Examination of trends in SAT participation and performance provide evidence that MCPS is making progress toward the strategic plan of ensuring success for every student.

I suppose I saw it differently. What I saw as a problem, MCPS saw as evidence of success. Jerry Weast was no different

Superintendent Jerry D. Weast credits the record-breaking SAT results to talented teachers, committed staff, and motivated students as well as to the reform plan implemented 11 years ago.

“We believe in high standards, high expectations and high performance,” Weast said. “We’ve followed a clear path that works and produces exceptional results for our students at every level.  These SAT scores are extraordinary and are something that our entire community should take pride in."

 The participation data was even more disconcerting when disaggregated by service group.

Records were indeed set in 2010.   I'm just not sure they were the kind of records that should be making headlines.   And if this is how "records" were made in Montgomery County- we needed to do some real and honest soul-searching.

A Stomach Churning Letter

My wife thought it was of little consequence.  

However, as I read congratulatory emails sent out to staff and watched videos celebrating Montgomery County Public Schools' record high SAT scores, my stomach churned.   "What do you expect them to do?" my wife chimed.  I suppose I took it personally.  Yes, the mean SAT in MCPS had risen to its highest level ever- but the conclusions circulated by MCPS were nothing short of complete misrepresentations.    I couldn't help but feel insulted as I read the email from Superintendent Weast to the staff:

The College Board released SAT scores for the Class of 2010 this morning and MCPS students set an all-time record. Our 2010 graduates scored an average of 1653, which is our district’s highest score since the “new SAT” was implemented in 2006 and represents a one-year increase of 38 points. MCPS graduates outscored their Maryland peers by 151 points and the nation’s 2010 graduates by 144 points. Students in all racial subgroups improved over last year, but African American and Hispanic students made the biggest gains, further narrowing the achievement gap. The best news of all is that 51% of our students scored a 1,650 or higher, meeting the 7th Key to College Readiness—again, an all-time record.


Of course, these results did not happen in one year, or even in four years. The students whose achievements are described in this report were second graders when we began working together in 1999 and made a firm commitment that we would give all students access to an outstanding education. From elementary school, through the middle grades, and into high school, you provided our students with the opportunities and support they needed to be successful. The SAT results released today are the culmination of all the work done by you and your colleagues since these students entered MCPS.

Yes, insulted.   MCPS had long told me about the importance of data.  It had long emphasized how to use that data to guide my instruction.   Yet now I was being told that the most recent "record" SAT scores were evidence of MCPS's unparalleled success.

I finally replied to my wife, "I expect them to be reflective practioners.  I expect them to have an honest dialogue about the progress of MCPS."     

I showed my wife the data that initiallly raised an eyebrow.

Math and Verbal combined SAT scores for MCPS (writing excluded)

My wife was unconvinced.

I would need to spend more time proving my point.   The achievement gap hadn't closed.  The mean SAT was no more a record than had I served as Superintendent and asked only the top 1% of students to take the SAT.   Yet Weast would tell Bethesda Magazine that this data point was one of his top ten accomplishments as Superintendent of MCPS.

It couldn't be.  It wasn't even an accomplishment.

I dangerously decided to see if I could make my wife see it my way.

What's in a name

On May 5th 2011 I received my 3rd email invitation in three months. I was cordially invited to attend a celebration of 12 years of service for outgoing Montgomery County Public Schools Superintendent, Dr. Jerry Weast. It was now a model of communication for which I had become all too accustomed- whether through pat-on-the-back celebratory emails or via holiday robo-calls reminding me of my dedication to the school district. The invitation was titled, “A Legacy of Excellence” and directed readers to the www.weastlegacy.com where ticket purchases could be made for $38. The invitation explained that one could also make a donation to the newly established Jerry Weast Book Project. This project would “enhance the book collections in all 131 MCPS elementary school media centers.” The books purchased would also include a bookplate indicating it was purchased by the Jerry Weast Book Project. It was a fitting end, and we could now say our “collective thank you” to the tenure of Jerry Weast. The invitation was signed, “Patricia O’Neill, Member Board of Education.”




In a vacuum, the invitation was unassuming. It was a pleasant send off for the leader of one of the largest school districts in the United States, and Dr. Weast would do it in class- establishing a small foundation as a way to say thank you to the students he served. It was an appropriate way to say good-bye.


However as a teacher who knew nothing but the Weastian way of doing business, I could not help but view the invitation an entirely different light. After ten years of teaching in Montgomery County, my mind had turned what appeared to be a simple charitable act into a self-serving marketing event. With a certain amount of guilt, I imagined thousands of book plates with the Jerry Weast name affixed- now circulating  ad finem through MCPS libraries. It had become the way things were done in Montgomery County. Of course, I was not surprised by the invitation- but my vision of the “Weast Legacy” was probably much different than what Board of Education member Patricia O’Neill imagined.


The invitation- to me- was a symbol of what had become of the Montgomery County school system; one that was more concerned with creating a perception and protecting a reputation than in instituting meaningful reforms. It was a county that meticulously gathered data about student performance. It paid special attention to those metrics that were gathered by outside organizations- Newsweek, AP Board, and Education Week to name a few. It then focused its instructional programs and decision-making on ways to move this data.


Consider for a moment- the release of the Montgomery County 2010 SAT scores. The MCPS press release read the following: “MCPS class of 2010 scores at record levels on the SAT.” When the local county Gazette picked up the results they explained “Montgomery County Public Schools students… achieved the highest average SAT score in the school system's history, while black and Hispanic students posted the biggest improvement in SAT scores.” And the Washington Post could not have been more favorable in their coverage, reporting that “Montgomery County public school students posted record-high scores of 1653, and the school system took steps to narrow the persistent achievement gap.” However, these headlines and articles often concealed facts that were less than flattering. But they were facts buried in raw data that few reporters or even educational experts had time to review. The headlines were simple and celebratory. It was news that made everyone feel good- so they were easy to accept as fact.

However when we deal with large organizations- the stakes of this type of misrepresentation- the type of misrepresentation that I will show occurred with the 2010 SAT results- are substantial.   They have large ramifications because school districts accepted as successful by their peers  serve as models for others.   When we read that Montgomery County closed the achievement gap- we conclude that the district has done something worthy of repetition.   But if these types of statements are merely window dressing- then what is copied is really of no social value.   In fact- it is a waste of valuable resources.  Unfortunately, I believe that Montgomery County Public schools has done much in the last ten years to increase their brand name- to increase their national and international reputation- but little to institute reforms worthy of the brand it created.


What may sound like an indictment of MCPS is really not.   MCPS plays by the rules and the incentives valued by our culture- they just happen to do it extremely well.   Under Weast, MCPS created a thesis and advanced it through what grew to be a ten million dollar communication and outreach budget.   And it was a thesis which united unions, parents, corporate leaders and academics behind it.    The thesis- however- that MCPS has systematically narrowed the achievement gap or in some other way improved the quality of teaching within the county is simply put- not supported by the data in any meaningful way.


The stakes are thus considerable.    If our models of reform have it wrong, then our educational system will not provide the human capital ready and able to compete in  (as Friedman puts it) a flattening global environment.     We must pick our models of reform well.     


My hope is to help us do just that.

First in schools but last in fiscal responsibility

An article from the Maryland Public Policy Institute:

The Pew Center ranks Maryland at the bottom of the pack when it comes to funding pensions, along with 18 other states. The center also says the state "needs improvement" when it comes to paying for OPEBs, although 39 other states share that label.

"This puts us in a tier with states known for serious fiscal problems like Illinois and New Jersey," said Gabriel Michael, a fellow at the Maryland Public Policy Institute, which sponsored the event.