But how much is eight points: On December 29, the Washington Post asked a pair of semi-reasonable questions about Arne Duncan, Obamas education secretary. Those questions:
How well did Chicagos children fare when Duncan led that citys schools? More specifically, how well were they scoring on national tests at the time Duncan left his post? And how much did their test scores improve in the years of Duncans tenure?
Duncan was CEO (groan) of Chicagos schools from June 2001 through January 2009. Using new test results (in math alone) from the National Assessment of Educational Progress (the NAEP), Nick Anderson tried to answer these questions. His report appeared on the Posts front page. In our view, it suggests some ways education reporting could improve in this, the spanking new year.
For our money, Andersons report was a bit misleading right from its first two paragraphs. How well did Chicagos kids score in 2009, compared to kids in other cities? For our money, this is oddly misleading:
ANDERSON (12/29/09): Soon after Arne Duncan left his job as schools chief here to become one of the most powerful U.S. education secretaries ever, his former students sat for federal achievement tests. This month, the mathematics report card was delivered: Chicago trailed several cities in performance and progress made over six years.
Miami, Houston and New York had higher scores than Chicago on the National Assessment of Educational Progress. Boston, San Diego and Atlanta had bigger gains. Even fourth-graders in the much-maligned D.C. schools improved nearly twice as much since 2003.
From the highlighted passage, the reader might think that three urban systems (Miami, Houston, New York) had higher scores than Chicago, and that several other urban systems (Boston, San Diego, Atlanta) recorded bigger gains than Chicago. From this passage, would readers guess that Boston and San Diego also recorded higher math scores than Chicagosubstantially higher scores, in fact? In his third paragraph, Anderson says that Chicago is nowhere near the head of the pack in urban school improvement. But for our money, he has already produced an oddly misleading account of some basic facts.
In 2009, how well did Chicagos kids score in math, as compared to kids in other urban systems? Anderson never quite explainsand his first attempt at an overview struck us as oddly misleading. This leads to our second questionhow much improvement did Chicago kids show from 2003 to 2009, during the bulk of Duncans tenure? This is an important type of question. We think its handled poorly tooin a way which can be improved.
What sorts of gains did Chicago record during Duncans tenure? Inside the paper, on page A6, readers get a basic ideathough only from a graphic which shows NAEP score gains (from 2003 through 2009) for ten major urban school systems (click here). When it comes to size of improvement, Chicago finishes seventh (out of ten) at the fourth grade level; it finishes sixth (out of ten) among eighth-graders. But an obvious problem arises in the title, and explanatory text, of the Posts chart:
Ranking city math gains
Here are mathematics test score gains on a 500-point scale for public schools in selected cities from 2003 to 2009.
At the fourth-grade level, Chicago is shown making an 8-point gain; the gain at eighth grade is nine points. But heres the problem: Are those large or small score gains? Do they represent a significant gain in academic improvement, or just a trivial bump in the numbers? Anderson makes no attempt to say. Since were dealing with a 500-point scale,readers may assume that these score gains are absurdly small, to the point of virtual insignificance.
In fact, these are quite significant gains, if one applies a rough rule of thumb often applied to scores on the NAEP. In this very rough rule of thumb, ten points on the NAEP scale is said to represent roughly one year of academic achievement. If one applies this rough rule of thumb, Chicago kids were scoring almost one year higher in math at the end of Duncans tenure. If we apply this rough rule of thumb, Chicagos kids achieved a significant gain under Duncanthough some other cities did better.
Would that be a sensible way to evaluate these score gains? We dont know, because Anderson, like many other reporters, made no attempt to place these score gains in any sort of statistical context. Increasingly, big newspapers are using NAEP scores as a way of measuring achievement and improvement. But such reports have little meaning, unless reporters make some attempt to place NAEP scores, and NAEP score gains, in some recognizable context.
Chicagos fourth-graders scored eight points higher. But people! How much is eight points? For our money, Anderson should have called some experts and asked.
NAEP scores are increasingly being reported in newspapers like the Post. Unless such score are placed in some wider context, readers are left to wonder what these score gains might mean or suggest.
Absolute scores: When it comes to math scores in 2009, Chicagos fourth-graders tied for seventh among those ten cities. Chicagos eighth-graders finished sixth. Warning! Some big cities are bigger than others! For various reasons, challenges may differ. Among the nations biggest cities, Chicago scored a fair chunk below New York and Houston, somewhat ahead of L.A.
For full NAEP results, click here.
A brand-new possible problem: Until recent years, NAEP results had rarely been used in the way theyre used hereto evaluate the work of educators, in this case a big-city superintendent. In recent years, the NAEP has been used to evaluate such officials much more frequently. As a result, there is now a reason for superintendentson the city or state levelto put their thumbs on the scale, if possible, in administering the NAEP.
Sorry. In the course of the past fifty years, flat-out cheating has always followed when testing programs have been used for accountability purposes. This has happened on the classroom level, and right on up the scale. Question: Since the NAEP only tests a sample of a states students, could a state superintendent see that an above-average sample of kids is used, thus exaggerating her states achievement and progress? Could any other irregularities occur? We dont know the answer to those questions. In our jaundiced, world-weary view, reporters should start finding out.
PART ONEGROUNDHOG PROBLEM: Two days after Christmas, Ezra Klein wrote a perfectly sensible piece in the Washington Post. After health care, we need Senate reform, its sensible headline said.
In particular, we need to reform the Senate filibuster, Klein argued.
While accepting the good sense of that proposition, we had a somewhat jaundiced reaction to Ezras sensible piece. After health care reform, we need health care reform, our analysts sadly cried, as if with one voice. The health care reform which is likely to pass will extend coverage in substantial ways. But will it address the ludicrous level of spending which characterizes our health care system? In recent weeks, many experts have praised the Senate bills approach to costs. Sadly, we remain skeptics.
Some assessments have seemed reassuring, coming from our most trusted voices. On December 4, Paul Krugman said the Senate bill includes the first really serious attempt to control health care costs, to cite one leading example. But just last week, this was the gloomy, glass-somewhat-full assessment of the Times editorial board:
NEW YORK TIMES EDITORIAL (12/30/09): The inexorably rising cost of hospital and medical care is the underlying factor that drives up premiums, deductibles and co-payments. No one yet has an answer to the problem. But the bills would launch an array of pilot projects to test new payment and health care delivery systems within Medicare. These include, for example, incentives to coordinate hospital and post-hospital care to head off needless readmissions, better coordination of care for the chronically ill, and incentives for doctors to provide a patient's total care for a flat fee instead of charging for each test or service provided.
The Senate bill would set up an independent board to spur the use of programs that save money or improve caresubject to Congressional veto. Optimists believe the savings might come quickly but this could still take many years. Without passing a reform bill, there is little chance of success.
No one yet has an answer to the problem of medical costs, the editors said. A cynical thought passed through our heads: Could that be because no one yet has made an attempt to explain this gigantic problem?
All year long, weve screamed and howled about this matter, sending our thoughts off into the void. Americans spending on health care is so vastso far off the chartsthat it would simply be comical, if it didnt represent such a massive societal problem. On a per capita basis, we spend two to three times what other developed nations spend. And yet, our health outcomes are roughly the same.
We have a word for that: Looting.
Few others have wasted much time on this remarkable problem. It has been stunning to see how hard all major sectors have worked to avoid discussing this problem this year. The mainstream press has aggressively avoided this problembut the liberal press has avoided it too. We find it hard to believe that a problem weve all avoided so grimly will somehow turn out to be solved.
We think the reform bill will be worth passing. But our assessment differs from Ezras. Heres our assessment: After health care reform is passed, well have to prepare for health care reform! In the next three days, well use a briefly famous book to consider what kind of approach might serve us in the future.
The book was written by T. R. Reid; it was called The Healing of America. It had its ten minutes of famenot a second morewhen it appeared last summer. Rereading it over the Christmas vacation, we were struck again by its maddening blend of strengths and weaknesses. Reid does some things very well; we think he does other things very poorly. For that reason, we thought the book provides a good framework for discussing a question: Where do we go from here?
Your country will pass this way again; presumably, well be discussing health care reform again at some point in the future. As progressives, do we want to win the fight when this topic emerges again, perhaps in the year 2020? Do we want to produce winning reform at that time, in a way we couldnt accomplish this year? Do we want to have the public with us? If so, the planning has to start right now; we cant wait until late 2019, then start to demand real reform.
Knowing full well that were wasting our time, well spend the next three days discussing what liberals ought to start doingif we want to produce a better reform the next time we meet at this place.
TOMORROWPART 2: Reid limns costs