Quotes of the Day

The new SAT scores are out, and buried in them is a sign of hope for American education. True, the scores are actually a bit lower than last year’s — the combined average for the SAT’s math and reading sections fell seven points, to 1,021, the biggest single-year decrease since 1975, when the score dropped 16 points, to 1,010. But statistically speaking, a seven-point decline (out of a possible 1,600 on those two sections) isn’t much. It’s less than the value of a single question, which is about 10 points. Also, the SAT radically changed last year. The College Board made it longer and added Algebra II, grammar and an essay. Fewer kids wanted to take the new 3-hr., 45-min. test more than once, so fewer had an opportunity to improve their performance. Scores were bound to slide.

But tucked into the reams of data the College Board included with the new scores was some wonderful news: I was wrong. In 2003 I spent six months tracking the development of the new SAT. I sat through hours of test-development sessions and long debates — sometimes fiery, sometimes soul-crushingly boring — over new questions. I even learned how to grade SAT essays. TIME ran my resulting story on its cover that October.

The story did make some predictions that turned out right. For instance, the new test favors girls more than the old test did. It is a long-standing tenet of testmaking that girls outperform boys on writing exams. For reasons I am not foolish enough to speculate about in print, girls are better than boys at fixing grammar and constructing essays, so the addition of a third SAT section, on writing, was almost certain to shrink the male-female score gap. It did. Girls trounced boys on the new writing section, 502 to 491. Boys still outscored girls overall, thanks largely to boys’ 536 average on the math section, compared with girls’ 502. But boys now lead on the reading section by just three points, 505 to 502; that gap was eight points last year. What changed? The new test has no analogies (“bird is to nest” as “dog is to doghouse”), and boys usually clobbered girls on analogies.

My story also predicted that the addition of the writing section would damage the SAT’s reliability. Reliability is a measure of how similar a test’s results are from one sitting to the next. Theoretically, if a test had a standard error of measurement of 0 points, you would score exactly the same each time you took it. But no test is that good. The pre-2005 SAT had a standard error of measurement of about 30 points for each section. In other words, if you got a 500 on the math section, your “true” score was anywhere between 470 and 530. But the new writing section, which includes not only a multiple-choice grammar segment but the subjective essay, has a standard error of measurement of 40 points, meaning a kid who gets a 760 on the section may actually be a perfect 800 — or a clever-but-no-genius 720. In short, the College Board sacrificed some testing reliability in order to include writing.

Finally, I was right about one other thing: that the grading of the essay would reward formulaic, colorless writing over sharp young voices. The average essay score for kids who wrote in first person was 6.9, compared with 7.2 for the half who didn’t write in first person. (A 1-12 scale is used to grade essays. That grade is then combined with the score on a multiple-choice grammar segment of the writing test and translated into the familiar 200-to-800 points.) As my editors know well, first-person writing isn’t always great. But along with this year’s test scores, the College Board distributed a guide called “20 Outstanding SAT Essays” — all of them perfect scores — and many are unbearably mechanical and cliché-ridden (“smooth sailing always comes after the storm”; “they say that history repeats itself”). One of the 20 kids did write a funny, sweet little story about a child’s birthday party, but this is a typical thought from another, who wrote about Vietnam: “As victory elluded (sic) the ever increasing forces more and more, the situation drastically worsened.” Remember: this is one of the 20 best writers the College Board could find.

But there is good news. The central contention of my 2003 story was that the SAT’s shift from an abstract-reasoning test to a test of classroom material like Algebra II would hurt kids from failing schools. I worried that the most vulnerable students would suffer when faced with the new SAT. I was wrong. In fact, the very poorest children — those from families earning less than $20,000 a year — improved their SAT performance this year. It was a modest improvement — just three points — but significant given the overall slump in scores. And noncitizen residents and refugees saw their scores rise by an impressive 13 points. It was middle-class and rich kids who account for the much-reported decline.

What explains those wonderfully unpredictable findings? The College Board has no firm answers. But its top researcher, Wayne Camara, suggests a theory that, while self-serving for the College Board, fits the data: the new SAT is less coachable. When designing the new test, the Board banned analogies and “quantitative comparisons” (flummoxing math questions that asked you to compare two complex quantities). “I think those items disadvantaged students who did not have the resources, the motivation, the awareness, to figure out how to approach them,” says Camara. “By eliminating those, the test becomes much less about strategy.” By focusing more on what high schools teach and less on tricky reasoning questions, the SAT became more egalitarian.

I’ve never been so happy to be wrong.

• JOHN CLOUD
• A slight decline in scores made headlines, but the real story is that the new test is closing the gender and income gap