But how much is eight points: On December 29, the Washington Post asked a pair of semi-reasonable questions about Arne Duncan, Obama’s education secretary. Those questions:

How well did Chicago’s children fare when Duncan led that city’s schools? More specifically, how well were they scoring on national tests at the time Duncan left his post? And how much did their test scores improve in the years of Duncan’s tenure?

Duncan was “CEO” (groan) of Chicago’s schools from June 2001 through January 2009. Using new test results (in math alone) from the National Assessment of Educational Progress (the NAEP), Nick Anderson tried to answer these questions. His report appeared on the Post’s front page. In our view, it suggests some ways education reporting could improve in this, the spanking new year.

For our money, Anderson’s report was a bit misleading right from its first two paragraphs. How well did Chicago’s kids score in 2009, compared to kids in other cities? For our money, this is oddly misleading:

ANDERSON (12/29/09): Soon after Arne Duncan left his job as schools chief here to become one of the most powerful U.S. education secretaries ever, his former students sat for federal achievement tests. This month, the mathematics report card was delivered: Chicago trailed several cities in performance and progress made over six years.

Miami, Houston and New York had higher scores than Chicago on the National Assessment of Educational Progress. Boston, San Diego and Atlanta had bigger gains. Even fourth-graders in the much-maligned D.C. schools improved nearly twice as much since 2003.

From the highlighted passage, the reader might think that three urban systems (Miami, Houston, New York) “had higher scores than Chicago,” and that several other urban systems (Boston, San Diego, Atlanta) recorded “bigger gains” than Chicago. From this passage, would readers guess that Boston and San Diego also recorded higher math scores than Chicago—substantially higher scores, in fact? In his third paragraph, Anderson says that Chicago “is nowhere near the head of the pack in urban school improvement.” But for our money, he has already produced an oddly misleading account of some basic facts.

In 2009, how well did Chicago’s kids score in math, as compared to kids in other urban systems? Anderson never quite explains—and his first attempt at an overview struck us as oddly misleading. This leads to our second question—how much improvement did Chicago kids show from 2003 to 2009, during the bulk of Duncan’s tenure? This is an important type of question. We think it’s handled poorly too—in a way which can be improved.

What sorts of gains did Chicago record during Duncan’s tenure? Inside the paper, on page A6, readers get a basic idea—though only from a graphic which shows NAEP score gains (from 2003 through 2009) for ten major urban school systems (click here). When it comes to size of improvement, Chicago finishes seventh (out of ten) at the fourth grade level; it finishes sixth (out of ten) among eighth-graders. But an obvious problem arises in the title, and explanatory text, of the Post’s chart:

Ranking city math gains
Here are mathematics test score gains on a 500-point scale for public schools in selected cities from 2003 to 2009.

At the fourth-grade level, Chicago is shown making an 8-point gain; the gain at eighth grade is nine points. But here’s the problem: Are those large or small score gains? Do they represent a significant gain in academic improvement, or just a trivial bump in the numbers? Anderson makes no attempt to say. Since we’re dealing with a “500-point scale,”readers may assume that these score gains are absurdly small, to the point of virtual insignificance.

In fact, these are quite significant gains, if one applies a rough rule of thumb often applied to scores on the NAEP. In this very rough rule of thumb, ten points on the NAEP scale is said to represent roughly one year of academic achievement. If one applies this rough rule of thumb, Chicago kids were scoring almost one year higher in math at the end of Duncan’s tenure. If we apply this rough rule of thumb, Chicago’s kids achieved a significant gain under Duncan—though some other cities did better.

Would that be a sensible way to evaluate these score gains? We don’t know, because Anderson, like many other reporters, made no attempt to place these score gains in any sort of statistical context. Increasingly, big newspapers are using NAEP scores as a way of measuring achievement and improvement. But such reports have little meaning, unless reporters make some attempt to place NAEP scores, and NAEP score gains, in some recognizable context.

Chicago’s fourth-graders scored eight points higher. But people! How much is eight points? For our money, Anderson should have called some experts and asked.

NAEP scores are increasingly being reported in newspapers like the Post. Unless such score are placed in some wider context, readers are left to wonder what these score gains might mean or suggest.

Absolute scores: When it comes to math scores in 2009, Chicago’s fourth-graders tied for seventh among those ten cities. Chicago’s eighth-graders finished sixth. Warning! Some big cities are bigger than others! For various reasons, challenges may differ. Among the nation’s biggest cities, Chicago scored a fair chunk below New York and Houston, somewhat ahead of L.A.

For full NAEP results, click here.

A brand-new possible problem: Until recent years, NAEP results had rarely been used in the way they’re used here—to evaluate the work of educators, in this case a big-city superintendent. In recent years, the NAEP has been used to evaluate such officials much more frequently. As a result, there is now a reason for superintendents—on the city or state level—to put their thumbs on the scale, if possible, in administering the NAEP.

Sorry. In the course of the past fifty years, flat-out cheating has always followed when testing programs have been used for “accountability” purposes. This has happened on the classroom level, and right on up the scale. Question: Since the NAEP only tests a sample of a state’s students, could a state superintendent see that an above-average sample of kids is used, thus exaggerating her state’s achievement and progress? Could any other irregularities occur? We don’t know the answer to those questions. In our jaundiced, world-weary view, reporters should start finding out.

Jay’s view: Jay Mathews offered a very different view of Anderson’s report. Remember, though—Jay went to Hillsdale High. A lucky duck, we went to Aragon.

Special report: After reform!

PART ONE—GROUNDHOG PROBLEM: Two days after Christmas, Ezra Klein wrote a perfectly sensible piece in the Washington Post. “After health care, we need Senate reform,” its sensible headline said.

In particular, we need to reform the Senate filibuster, Klein argued.

While accepting the good sense of that proposition, we had a somewhat jaundiced reaction to Ezra’s sensible piece. After health care reform, we need health care reform, our analysts sadly cried, as if with one voice. The health care reform which is likely to pass will extend coverage in substantial ways. But will it address the ludicrous level of spending which characterizes our health care “system?” In recent weeks, many experts have praised the Senate bill’s approach to costs. Sadly, we remain skeptics.

Some assessments have seemed reassuring, coming from our most trusted voices. On December 4, Paul Krugman said the Senate bill includes “the first really serious attempt to control health care costs,” to cite one leading example. But just last week, this was the gloomy, glass-somewhat-full assessment of the Times editorial board:

NEW YORK TIMES EDITORIAL (12/30/09): The inexorably rising cost of hospital and medical care is the underlying factor that drives up premiums, deductibles and co-payments. No one yet has an answer to the problem. But the bills would launch an array of pilot projects to test new payment and health care delivery systems within Medicare. These include, for example, incentives to coordinate hospital and post-hospital care to head off needless readmissions, better coordination of care for the chronically ill, and incentives for doctors to provide a patient's total care for a flat fee instead of charging for each test or service provided.

The Senate bill would set up an independent board to spur the use of programs that save money or improve care—subject to Congressional veto. Optimists believe the savings might come quickly but this could still take many years. Without passing a reform bill, there is little chance of success.

“No one yet has an answer to the problem” of medical costs, the editors said. A cynical thought passed through our heads: Could that be because no one yet has made an attempt to explain this gigantic problem?

All year long, we’ve screamed and howled about this matter, sending our thoughts off into the void. American’s spending on health care is so vast—so far off the charts—that it would simply be comical, if it didn’t represent such a massive societal problem. On a per capita basis, we spend two to three times what other developed nations spend. And yet, our health outcomes are roughly the same.

We have a word for that: “Looting.”

Few others have wasted much time on this remarkable problem. It has been stunning to see how hard all major sectors have worked to avoid discussing this problem this year. The mainstream press has aggressively avoided this problem—but the liberal press has avoided it too. We find it hard to believe that a problem we’ve all avoided so grimly will somehow turn out to be solved.

We think the reform bill will be worth passing. But our assessment differs from Ezra’s. Here’s our assessment: After health care reform is passed, we’ll have to prepare for health care reform! In the next three days, we’ll use a briefly famous book to consider what kind of approach might serve us in the future.

The book was written by T. R. Reid; it was called The Healing of America. It had its ten minutes of fame—not a second more—when it appeared last summer. Rereading it over the Christmas vacation, we were struck again by its maddening blend of strengths and weaknesses. Reid does some things very well; we think he does other things very poorly. For that reason, we thought the book provides a good framework for discussing a question: Where do we go from here?

Your country will pass this way again; presumably, we’ll be discussing health care reform again at some point in the future. As progressives, do we want to win the fight when this topic emerges again, perhaps in the year 2020? Do we want to produce winning reform at that time, in a way we couldn’t accomplish this year? Do we want to have the public with us? If so, the planning has to start right now; we can’t wait until late 2019, then start to demand real reform.

Knowing full well that we’re wasting our time, we’ll spend the next three days discussing what liberals ought to start doing—if we want to produce a better reform the next time we meet at this place.

TOMORROW—PART 2: Reid limns costs