PILOT PROGRAM! Good news! The Virginian-Pilot is trying to learn what its state’s passing rates really meant: // link // print // previous // next //
TIERNEY TOMORROW: Tomorrow, we’ll review John Tierney’s recent column on charter schools. Today, though, we’re sticking to that continuing story. And good news! The Richmond Times-Dispatch and the (Norfolk) Virginian-Pilot have both been on the case. Good news! Virginians deserve to be fully informed about this important matter. For basic links to past work, see below.
Continuing story: Yes, Virginia!
TIMES-DISPATCH REPORTS: Good news! On Tuesday, the Richmond Times-Dispatch filed a short report about Virginia’s missing report cards. “State report cards inaccessible on site,” the headline said. Here’s the complete dispatch:
RICHMOND TIMES-DISPATCH (3/14/06): The 2005 Virginia School Report Cards cannot be accessed on the Virginia Department of Education's Web site because of technical problems, a department spokesman said yesterday.The cards can’t be accessed “because of technical problems?” It’s hard to see how that could be accurate. As we noted yesterday, the Virginia report cards can be accessed—if you know what link to click. (Just click here, for example.) They just can’t be accessed from the state’s web site, where people would normally go to find them. At the web site, the state continues to say they’re “temporarily unavailable.” We don’t know why the link to this material has been disabled. But it’s rather hard to believe that the answer is “technical problems.”That part of the site has been down since March 3.
The report cards include information on test scores, accreditation and adequate yearly progress (AYP) rates for the state, school systems and individual schools.
The department hopes to have the problem fixed in the next day or so, according to a department spokesman.
As we’ve said, we don’t know why this basic information has been removed from public view. But the explanation received by the Times-Dispatch does seem a bit hard to credit.
PILOT PROGRAM: In the past two days, we’ve heard from personnel at both the Times-Dispatch and the (Norfolk) Virginian-Pilot; each newspaper has a reporter or two trying to sort out this ongoing story (basic links below). For what it’s worth, each paper also seemed to know, in the past, that the “passing rates” on the state’s “school report cards” included some kids who had been retested. The papers had apparently decided not to report on this matter—and we haven’t found anyone yet who can really explain the state’s procedure in a definite way.
Example: What produced Maury Elementary’s high passing rate in Reading/Language Arts last year? (From now on, we’ll just call it “reading.”) Did 5 out of 19 Maury third-graders pass the state’s reading test last spring? Or was it really 17 out of 19? The school’s “report card” presents the first passing rate at one point (27 percent); but at another, much more prominent place, it says the overall passing rate for grades 3 and 5 was 92 percent. According to Jay Matthews’ February 28 report, this high passing rate derived from these figures: 17 out of 19 third-graders passed, along with 22 out of 24 fifth-graders.
So what exactly was the procedure which turned that 5 out of 19 third-graders into that much higher score?
Yesterday, we spent a fair chunk of time on the phone with a reporter for the Virginian-Pilot. Her initial interpretation was this: The 12 third-graders who failed the reading test last spring were retested at some point this year, as fourth-graders. (Note: She wasn’t sure about this.) That was not what we took Alexandria testing director Monte Dawson to have told us in his e-mails. Here, for example, is the language Jay quoted in that February 28 report:
DAWSON E-MAIL: Remediation Recovery, which has been around since 2001, means that fourth grade students who failed the third grade test in 2004, got to retake the third grade test in 2005. Up until this year (2005), if they passed the third grade test, then they were included in the numerator only of the calculation to determine the third grade passing score. As illustration, if 4 out of 5 third grade students passed and 1 out of 5 fourth grade Remediation Recovery students passed, the passing percentage would be 100 percent.In this e-mail, Dawson clearly refers to “fourth grade students who failed the third grade test in 2004.” We took this to mean that, when twelve of these students passed the test as fourth-graders in 2005, that number was added to the total of “third grade students” who passed the test that year. (As we’ve endlessly noted, this procedure would be completely bizarre.) The reporter from the Virginian-Pilot suggested something different. Her suggestion: When third-graders failed the test last year, they were restested this year as fourth-graders. That isn’t the meaning we took from Dawson’s e-mail, and we don’t know if that’s what actually happened. But we’ll guess that this may turn out to be right, in part because so many schools have boosted their passing rates through “Remediation Recovery.” It’s hard to believe that so many schools could engage in such an irrational process without someone complaining about it.
Is this what happened? When third-graders failed a test last spring (in the spring of 2005), were they retested at some point this year, as fourth-graders? If so, let’s note the following about this procedure:
This procedure would make (somewhat) more sense than the one we thought Dawson described. But, alas, the same problem obtains. Do Virginia’s parents know what they’re getting when they check a school’s passing rate? For example, when they see that 92 percent of Maury’s kids passed last year’s “Grade 3 and 5" reading tests, do they understand that many of these kids only passed the tests this year—a year late? Do they understand that the vast majority of students who passed the third-grade test only passed it this year, as fourth-graders?
Overview: Around 1970, public scrutiny began to put pressure on public school testing programs. Ever since, schools and school systems have searched for ways to inflate average scores and, now, passing rates. If this writer’s suggestion turns out to be right, Virginia found a way to do this through “Remediation Recovery.” Let’s state the obvious—every school in America could improve “passing rates” if they kept retesting their third-grade students on into the fourth-grade year. Is there some educational reason to retest a child the next year? We’ll mark ourselves as skeptics on that. But one thing is certain; engaging in next-year retesting will drive up every school’s passing rates. The school will look better to the public—and no, the public will not understand where those passing rates come from, or what they actually mean. (As we’ve said, we haven’t spoken to a single Virginia education reporter who seemed clear about the way this four-year-old process actually works.)
Yes, Virginia: Every school will improve “passing rates” if fourth-grade kids take the third-grade tests. And it can lead to absurd situations, as it did with the Post and Maury. Remember what happened: When these tests were given last spring, Maury recorded the second-lowest score in the state of Virginia on third-grade reading. Only one school in the entire state scored lower during this statewide test session. But thanks to the still-murky process known as Remediation Recovery, Maury was being hailed just nine months later, at the top of the Post’s front page—and no, the Post didn’t mention the school’s remarkably low passing rates from the previous spring. What is really happening at Maury? We don’t have the slightest idea. But it’s just strange when a school can score this low and still be hailed as “a study in progress”—without anyone being told about those very low scores.
Did third-grade students who failed last spring get tested again this year as fourth-graders? We’re still not sure (for some doubts, see below). But can we make a simple suggestion? Here it is, and it’s really quite radical: Virginia should give third-graders the third-grade test! After that, the state should tell the public how many kids passed! Yes, this will have one big disadvantage—this will lower the state’s passing rates. But it will have one major advantage—it won’t be a big giant scam.
REASON TO DOUBT: Is the writer from the V-Pilot right? Did some of last year’s third-grade students get retested this year as fourth-graders? (She wasn’t sure.) That may be the case, but it doesn’t quite jibe with the mathematical mumbo-jumbo included in the two e-mails we were sent by Monte Dawson, Alexandria’s director of testing. (Link to the e-mails below.)
Example: Dawson sent us technical language explaining what the state would do if a school’s passing rate ended up exceeding 100 percent! (Answer: They’d “cap” the passing rate at 100!) It’s hard to see how this could happen under the V-Pilot writer’s scenario. At Maury, for example, only 19 students would have been given the third-grade test; there would be no way that more than 19 students could end up passing it. This could only happen if the state retested fourth-graders who flunked as third-graders in 2004, then added their number onto the total of third-graders who passed in 2005. (In that situation, a given school could easily produce a passing rate which exceeded 100 percent.) This is one of the reasons why we interpreted Dawson’s e-mails as we did.
Second problem: Note the part of Dawson’s second e-mail which we highlight below:
DAWSON E-MAIL: Remediation Recovery, which has been around since 2001, means that fourth grade students who failed the third grade test in 2004, got to retake the third grade test in 2005. Up until this year (2005), if they passed the third grade test, then they were included in the numerator only of the calculation to determine the third grade passing score. As illustration, if 4 out 5 third grade students passed and 1 out of 5 fourth grade Remediation Recovery students passed, the passing percentage would be 100 percent. However, beginning July 12, 2005, passing Remediation Recovery students were added to the numerator and the denominator.Despite that e-mail’s semi-clear language, we assumed this meant that (forgive us) the number of passing students would be added to the denominator in school years which started after July 2005. We say that because what Dawson describes doesn’t seem to be what happened with last year’s third-grade class at Maury. In that instance, 5 out of 19 third-graders passed the reading test—and, according to Jay Mathews, a “12" was subsequently added to the numerator only, transforming “5 out of 19" into “17 out of 19" and helping produce an overall passing rate of 92 percent. According to Dawson, this procedure will change—the “12" will be added to the denominator too. But this procedure will be absurd if the same kids are being retested the next year, as fourth-graders. Here’s what will happen in a Maury-of-the-future: Nineteen kids will take the third-grade test in the spring of their third-grade year. Five of them will pass the test, creating this fraction: 5 out of 19. The next year, when they’re fourth-graders, the 14 failing students will be retested, and 12 will pass. Then, that “12" will be added to the numerator and the denominator, producing this new passing rate—17 out of 31 (55 percent). But to what would that fraction/passing rate correspond? Only 19 students will have been tested! And 17 of them will have passed. This new “passing rate” would be virtually impossible to explain. It would correspond, pretty much, to nothing. As we’ve noted before, this mumbo-jumbo doesn’t seem to make sense if the same kids are just being retested. It would make a kind of sense if an entirely different group of kids were being added into the stew.
(By the way: What will happen to the two kids who fail the retest? Apparently, that “2" will just be thrown away. Under this bizarre statistical scenario, that number—“2"—won’t figure into the new passing rate.)
Yes, of course, it’s sadly true—the state of Virginia would be capable of a procedure this cock-eyed. And we’re still not sure what really produced that 92 percent passing rate at Maury. But here’s what the state of Virginia should so, and once again, it’s extremely radical: The state should give third-graders the third-grade test, then tell the public how many kids passed! Yes, this would reduce the “rate inflation” which has made the state DOE look so good. But it would also produce the following boon—citizens would get to see real information about their children’s actual schools. They wouldn’t be handed incomprehensible data which no one can quite explain.
Let’s say it again: The state should test its third-grade kids. It should then report how many kids passed! This would produce straightforward data—and lead to straightforward public discussions. And oh yes—if Maury really does improve with its new principal, that will show up in future years’ testing! In future years, we hope this school will rock our socks—with impressive passing rates which are both straightforward and real.
FINAL NOTE: All hail the Times-Dispatch and the Virginian-Pilot! Let’s hope they report this all the way out, informing us all in the process.
BASIC LINKS: On February 2, Maury hit the top of the Post’s front page. You know what to do—just click here.
We questioned this story the following week. See THE DAILY HOWLER, 2/6/06, then click forward from there.
Post reporter Jay Mathews followed up on February 28. Click here and you can read every word.
We responded all last week—and the state of Virginia has removed its school report cards from view. See THE DAILY HOWLER, 3/7/06, for our first installment.
For the full text of Monte Dawson’s two e-mails, see THE DAILY HOWLER, 3/8/06.
For a list of Alexandria schools, just click here. Keep clicking to see each school’s “report card.” All schools in Virginia can be accessed from this basic link.