WILL THE REAL MAURY SCHOOL PLEASE STAND UP! Jay misstates Virginias procedure—a procedure which adds up to fraud: // link // print // previous // next //
WEDNESDAY, MARCH 8, 2006
OUR SERIES ON MAURY: On series on Virginia test scores continues. For Part 2, scroll down a tad.
MORE NOTES ON WHO CARES ABOUT ED: Over the weekend, we enjoyed an e-mail exchange with Michelle Pilecki, who disagrees with one of last weeks posts in ways well let her express for herself. Check out her piece at the Huffington Post, Who Cares About Education for Low-Income Kids?
We admire Michelle, because she lives in Pittsburgh (which apparently makes her a mensch and a prole), while we live in Baltimore, which apparently makes us an Agent To Power. Mordant jesting to the side, we had a perfectly good exchange, and we recommend her post—a post which does in fact wander a bit away from our original topic.
Michelle says she found it odd when we restated our point in an e-mail. (But here's what I said: Liberal journals and liberal bloggers don't discuss low-income education. As a general matter, I think that's completely obvious. These topics are almost never discussed.) Indeed, we do think that statement is perfectly obvious—although we told Michelle, as we said last week, that we dont mean that as an insult or slander, just as a statement of fact and a challenge. Do liberals discuss low-income ed in the journal and blogs where we help shape the discourse? Once again, as a general matter, we think that the answer is clear.
Mr. Somerby has a somewhat valid point in criticizing the blogosphere for the lack of attention sparked by an excellent investigative Los Angeles Times series...on the large number of dropouts in LA's Class of 2005, Michelle writes. She may have said that because we asked her a question. The Huffington Post is based in L.A., as are many of its bloggers. But when the Los Angeles Times did that superlative series, how many times was it mentioned in the Huffington Post? We didnt bother to check before asking—but well guess the answer is small or non-existent, and we note that Michelle doesnt speak to this point. This wasnt meant as a slam at the HuffPo or at its bloggers, just as a reflection of basic reality. For most liberal journals and opinion-makers, low-income ed has been off the play-list for these past many years.
We think Michelle is dreaming a bit—and were inclined to think that shes settling. At one point, she claims that preschool for low-income kids has been a hot topic:
PILECKI: Disclosure time: In my off-web life as a freelance writer, I recently wrapped up a magazine story, to be published this spring, on one aspect of the pre-K movement, so I've been awash in research. Despite Mr. Somerby's comment, pre-K for at-risk kids has been a hot topic, e.g. in the next day's Washington Post, E. J. Dionne wrote about California's Proposition 82, which would raise taxes to guarantee preschool for every 4-year-old in the state. Further disclosure: The proposition is the brainchild of fellow HuffPoster Rob Reiner, but I know neither him nor the particulars of this pre-K proposal.Is it true? Has pre-K for at-risk kids been a hot topic? Note the way Michelle closes this passage. She herself has been writing an article about some part of the pre-K movement, and even she doesnt know the particulars of Rob Reiners pre-K proposal! Why is even Michelle uninformed? Because Robs plan is almost never discussed! For ourselves, we greatly admire Reiner for his long-standing commitment and effort. (Robs a big dude—he does not have to do this.) But preschool just isnt a very hot topic in the venues we were discussing. In last years The Shame of the Nation, for example, Jonathan Kozol discussed the declining number of low-income kids who have access to Head Start, the nations best-known (and largest) pre-K program. In spite of the generally high level of approval Head Start has received over the years...40 percent of three- and four-year-olds were denied this opportunity in 2001, a percentage of exclusion that has risen steeply in the subsequent four years, he writes (page 52). After citing some places where preschool is common, he writes this: More commonly in urban neighborhoods, large numbers of children have received no preschool education and they come into their kindergarten year without the minimal social skills that children need in order to participate in class activities and without even such very modest early-learning skills as knowing how to hold a pencil, identify perhaps a couple of shapes or colors, or recognize that printed pages go from left to right. We dont know enough about this topic to evaluate everything Jonathan said. But what was the public reaction to this part of his book? Of course! There was no reaction at all! This book was not discussed in liberal journals or blogs—including at the Huffington Post. This isnt meant as an insult to the worlds tribal souls. But the general situation is perfectly clear.
There's a growing national movement for universal access to pre-K, whether liberal bloggers and journals know it or not, Michelle writes. That may or may not be true, but that misses the point we were making. In this fifth installment of a six-day series (five parts of which were numbered), we were asking why the nations low-income education policies are often framed by a vacuous discourse—by the kind of vacuous discourse which turned a kid like Gabriela Ocampo (Los Angeles, California) into a high school drop-out. In part, we said, the discussion is empty because liberal journals and liberal bloggers almost never take part (along with many other groups, by the way). Result? The discussion is left to second-tier entities, with outcomes which are often unfortunate. As we told Michelle yesterday, our basic reaction to her post is this: Man, are you ever willing to settle! At present, theres little discussion of low-income ed; low-income children, like Gabriela, just dont seem to count for much. Sir, may we have just a tiny bit more? Michelles post seems to say.
PART 2—IN A WORD, FRAUD: Will the real Maury School please stand up? The Posts Jay Mathews—and we at THE HOWLER—have been trying to sort out the puzzling data which appeared on the schools state report card last fall (see THE DAILY HOWLER, 3/7/06; Maury is a low-income school in Alexandria, Virginia). According to one chart—a chart Jay used for a front-page report in the Post last month—92 percent of Maurys kids passed Virginias Reading/Language Arts tests in the spring of 2005. Needless to say, that passing rate sounded extremely good. (Third and fifth grades were the only grades tested.) But uh-oh! A second set of charts on the Maury report card showed some very different data; in these charts, only 27 percent of the schools third-graders passed this same state test. (Statewide, 77 percent of third-graders passed.) So which one is the real Maury School? The school where 92 percent passed the state reading test? Or the school whose third-graders performed so poorly? Which school is the real Maury School? Will the real Maury School please stand up?
Sadly, were still forced to think that the real Maury School is the one with those low third-grade reading scores. (Instead of saying Reading/Language Arts, well shorten it down from here.) We base that judgment on the explanation we got from Alexandrias director of testing, Monte Dawson. It was a cosmically bad explanation—one which Mathews clearly misstated in the follow-up report he posted on-line just last week.
What is the state of Virginias explanation for those dueling data? How did a school in which 27 percent of third-graders passed the state test get credited with an overall passing rate of 92 percent? Last Wednesday, Jay tried to answer that question. Heres his account of the way that miserable 27 became an impressive 92:
MATHEWS (2/28/06): I reconstructed what happened with the help of Virginia state officials. When Maury's 19 third-graders took the English test the first time last spring, five passed and 14 did not. Of the 24 fifth-graders who took the English test, 22 passed. The school worked with the third-graders who did not pass it and gave them a retest, and 12 passed on that second try.Quick note: Jays fifth-grade data seem to be slightly wrong. Tedious details below.
Back to the basics: In this passage, Jay describes a testing procedure which seems to be perfectly sensible. The third-grade kids were given the state reading test last spring (the spring of 2005). Five out of 19 passed the first time; the other 14 kids were retested, and 12 of them passed on that second attempt. Hence, 17 of the 19 third-graders passed the reading test last spring. For the record, there may be a retest bias problem with such a procedure; if kids take the very same test two times in short order, it may skew results. But leaving aside that possible problem, Jay describes a testing procedure which does makes perfect good sense. Unfortunately, this isnt the procedure Dawson described when we inquired about those Maury test scores. And it isnt the procedure described by the state in a passage quoted by Jay himself in his report last week.
Jay has described a sensible process. But it seems fairly clear that this is not the process which produced Maurys high passing rate—the passing rate which put the school atop the Posts front page, described as a study in progress.
What actually produced Maurys high passing rate? In Jays own report last week, he quotes some of the language sent to us by Dawson, Alexandrias director of testing. In the following passage, Jay misstates what Dawson told us, then quotes Dawsons actual language. As you can see from reading the Dawson quotation, Dawson doesnt describe the sensible procedure which Jay describes in the passage above. He describes a totally different process—a process which adds up to fraud:
MATHEWS (2/28/06): Somerby dug further, and soon got an explanation from Alexandria schools testing and assessment director Monte Dawson, my prime source on the story. The big jump, Dawson explained, came mostly from the fact that 12 of those third-graders retook the test and passed it, upping the passing rate considerably, but only because of a odd mathematical rule approved by the state school board in 2000. Somerby quoted Dawson's explanation:Dawson clearly didnt tell us that 12 of those third-graders retook the test. Indeed, that clearly is not what his quotation says. That quoted passage doesnt describe the sensible process from Jays report. It doesnt say that third grade students were allowed to take the test a second time last spring. In fact, it says something vastly different; it says that fourth grade students who failed the third grade test in 2004, got to retake the third grade test in 2005. Jay describes a sensible process; that statement (and others—see below) describes something vastly different.
What actually happened at Maury last year? What actually got the schools passing rate up? Believe it or not, this is what happened, according to everything we have been told: Only 5 of Maurys third-grade kids passed the third-grade reading test. But an unspecified number of Maurys fourth-grade kids also were given the third-grade test; when 12 of them passed the third-grade test, they were added to the total of third-graders (five) who had passed. No, none of this makes any sense—but it did get Maurys passing rate up. Only 5 of 19 actually passed—till the state added in those 12 ringers.
Can this possibly be the way this state is computing its passing rates? We know, we know—it sounds impossible. Its hard to believe that any state would engage in such total nonsense. But that is clearly what Dawson told us when we corresponded last month. Here, for example, is the actual language he sent when we e-mailed him the first time. We posted this language in real time (see THE DAILY HOWLER, 2/8/06). Hang on for a wild ride:
DAWSONS FIRST E-MAIL: Remediation Recovery, which started with the 2001 [state tests], is a third reason for apparent score disparities. Students in grades 4, 6, or 9 may retake failed English: Reading/Literature and Research or mathematics tests for grades 3, 5, or 8, respectively, following a Remediation Recovery program. Additionally, students who failed Algebra I, Geometry, or Algebra II and who are enrolled in a Remediation Recovery program may retake a given EOC mathematics test. Tables 6, 7, 15, 17, and 20 display the number of students who retook the failed SOL, the percentage who passed, the number who passed (Bonus number), and the potential benefit to the school (Recovery Bonus or Unadjusted + Recovery score). In the State's calculations to determine accreditation, the number of students who pass the targeted test following a Remediation Recovery program will be added to the number of students who passed the SOLs in the same content area. For example, a fourth graders passing grade 3 mathematics score will be added to that schools grade 3 mathematics passing scores. At other grade levels, the passing mathematics score will be added to the schools collapsed SOL mathematics scores (for accreditation calculations, all mathematics scores are collapsed or averaged together to create one passing percentage). Remediation Recovery students will be included in the unadjusted number of students who passed, but not in the number of students tested, hence the term Recovery Bonus. Said another way, passing Remediation Recovery students are added to the numerator, but not to the denominator. What this means is that a passing percentage exceeding 100 percent is possible (Note: while this overview reports percentages more than 100 percent, the State caps pass rates at 100 percent).Yes, its true. Believe it or not, thats the language Dawson sent in response to our first queries. (Well post his two complete e-mails below). And no, that passage doesnt describe the sensible process from Jays report. According to that passage from Dawson, fourth-graders who failed the third-grade test in the previous year take the third-grade test again, although they now are in the fourth grade—and if they pass, they get added onto the total number of third-graders who passed. This is an utterly ludicrous process, but Dawson described it again, quite clearly, when we asked for a clarification. We posted the following language last month; see THE DAILY HOWLER, 2/9/06:
DAWSONS SECOND E-MAIL: You are correct that the adjusted passing percentage for third grade English: Reading and Writing was 27 person (33 percent unadjusted). Please note that Maury was more effective (71 percent) than the division (61 percent) in remediating fourth grade students. Thus, in the end, the State added in 12 passing fourth grade students at Maury when making their accreditation calculations.Were still not sure we understand all of that, for reasons which must be obvious. But in this second e-mail, Dawson made part of this process quite clear; in the end, the State added in 12 passing fourth grade students at Maury when making their accreditation calculations. Fourth-grade kids took the third-grade test; when they passed, they were added onto the third-grade totals. This is not the sensible process Jay described in last weeks report. Indeed, this is something quite different—an apparent fraud on the public conducted by Virginias state Department of Ed.
Amazing, isnt it? In third grade, only 5 of 19 Maury students actually passed the state reading test. This was a miserable passing rate; it was far below the 77 percent statewide passing rate. So how did Maury end up on the Posts page one, described as a study in pride, progress? Simple! The state added twelve fourth graders onto that total and pretended that 17 third-graders passed! This is, plain and simple, a fraud—an act of deception against the states citizens and against the Maury Schools parents.
Drink it in: Five out of 19 third-graders passed. The state found a way to change 5 to 17. And the Post put the story at the top of page one, where readers were told about Maurys great progress. So yes, well say it one last time: Unless theres something here which is still unexplained, this is an act of outright fraud—the latest in the endless drive to gimmick up public school test scores.
TOMORROW—PART 3: Why did we check Maurys scores?
MORE INDICATIONS FROM JAYS REPORT: We know, we know—the rational mind prefers not to think that any entity, let alone a state government, would execute so absurd a procedure. And lets be fair; this may be why Jay misdescribed the plain language of Dawsons two e-mails. In each of his e-mails, Dawson said it; the Maury School had remediated some fourth-grade kids and added them to the number of third-grade kids who passed last years reading test. Yes—when these fourth-graders passed the third-grade test, they were added to the puny number of third-graders who actually passed. Five out of 19 third-graders passed; by the time the state got through, we were told it was really 17.
In his report, Jay says he was told something different, something more sensible—but part of his report suggests that he may simply have misunderstood the absurd procedure the state described. Well now post a chunk of Jays report in which he describes a confusing conversation with a state Department of Education spokesman. In this passage, you see Jay struggling to explain the odd mathematics of what is allegedly a sensible process—a process which wouldnt be hard to explain if it happened as Jay thinks it did:
MATHEWS: I reconstructed what happened with the help of Virginia state officials. When Maury's 19 third-graders took the English test the first time last spring, five passed and 14 did not. Of the 24 fifth-graders who took the English test, 22 passed. The school worked with the third-graders who did not pass it and gave them a retest, and 12 passed on that second try.Jay describes an absurdly complex mathematical procedure, and tries to explain its rationale. But if last springs failing third-grade kids were simply retested a few weeks later, that wouldnt be hard to explain at all. If you reported results in the usual way, the explanation would be very simple. Youd simply give the public the facts: Third-graders could take the test two times, and 17 out of 19 eventually passed. You wouldnt have to jump through mathematical hoops—unless the real procedure is the one Dawson described in his e-mails, the one which seems to involve the state in an act of ridiculous fraud.
Did Jay misunderstand what he was told? Frankly, we wouldnt be all that surprised; the rational mind does tend to recoil against the ludicrous thing this state did. Then theres this: If Charles Pyle is just a press spokesman (and not a test specialist), he may have misunderstood this matter, assuming that the states procedures couldnt possibly make so little sense. He may have mis-explained the matter to Jay, assuming the states conduct made sense. But at any rate, in last weeks report, Jay does quote Dawsons clear language (see above): Remediation Recovery, which has been around since 2001, means that fourth grade students who failed the third grade test in 2004, got to retake the third grade test in 2005. Fourth grade kids took the third grade test—and when some of the fourth-graders passed, they were added to the tiny number of third-graders who actually passed. But then, scams like this have gone on for decades, since the dawn of accountability. As we proceed, well discuss this history in a bit more details—and well tell you why a scam like this matters.
DAWSONS COMPLETE E-MAILS: For those who want to cross every T, here are Dawsons two complete e-mails. By the way, Dawson was very prompt and straightforward. This policy comes from the state, not from the straightforward Dawson:
DAWSONS FIRST E-MAIL (2/6/06):
Remediation Recovery, which started with the 2001 SOLs, is a third reason for apparent score disparities. Students in grades 4, 6, or 9 may retake failed English: Reading/Literature and Research or mathematics tests for grades 3, 5, or 8, respectively, following a Remediation Recovery program. Additionally, students who failed Algebra I, Geometry, or Algebra II and who are enrolled in a Remediation Recovery program may retake a given EOC mathematics test. Tables 6, 7, 15, 17, and 20 display the number of students who retook the failed SOL, the percentage who passed, the number who passed (Bonus number), and the potential benefit to the school (Recovery Bonus or Unadjusted + Recovery score). In the State's calculations to determine accreditation, the number of students who pass the targeted test following a Remediation Recovery program will be added to the number of students who passed the SOLs in the same content area. For example, a fourth graders passing grade 3 mathematics score will be added to that schools grade 3 mathematics passing scores. At other grade levels, the passing mathematics score will be added to the schools collapsed SOL mathematics scores (for accreditation calculations, all mathematics scores are collapsed or averaged together to create one passing percentage). Remediation Recovery students will be included in the unadjusted number of students who passed, but not in the number of students tested, hence the term Recovery Bonus. Said another way, passing Remediation Recovery students are added to the numerator, but not to the denominator. What this means is that a passing percentage exceeding 100% is possible (Note: while this overview reports percentages more than 100%, the State caps pass rates at 100 percent).
DAWSONS SECOND E-MAIL (2/7/06):
FOR FIGURE FILBERTS ONLY: We mentioned that Jays fifth-grade data seem to be slightly wrong too. When Maury's 19 third-graders took the English test the first time last spring, five passed and 14 did not, he wrote. Of the 24 fifth-graders who took the English test, 22 passed. But according to the states grade-level charts, 83 percent of fifth-graders passed; presumably, that would be 20 out of 24 (83.3 percent, rounded down). At this grade level, it would seem that two ringers (sorry: remediated students) were added to the actual N. When a state is busy misleading the public, every little bit helps.
One last point: Virginia officials employ a vast array of names for the test in question. Sometimes they call it the English test (thats the name Jay tends to use). Sometimes, they call it Reading/Language Arts. In his e-mails, Dawson calls it English: Reading and Writing. The confusion which stemmed from this Babel helped make those Virginia school report cards—now taken down—an utterly incomprehensible mess. (Yes, as weve noted, there was much more confusion.) But now, school report cards have been taken down—for the entire state of Virginia. At present, if you live in Virginia, theres no way to check your schools scores.