WIENER (1/2/06): One thing put in stark relief is the low level of state standards. Students who demonstrate proficiency on their own state's tests often perform far below that level on NAEP, suggesting that the states have set standards too low to indicate adequate academic preparation.As we said on Friday, weve never understood the logic of Wieners position as it applies to low-achieving kids. That is, weve never understood why raising standards would help a kid who cant come close to meeting the bar which already exists. But in Saturdays Post, educational researcher Gerald Bracey challenged Wiener in a different way. Are some states setting their standards too low? Actually, NAEP is setting its standards too high, Bracey says in a letter:
BRACEY (1/7/06): Ross Wiener writes that kids do well on state tests but not on the National Assessment of Educational Progress (NAEP). Therefore, he concludes, states have low standards. Actually, NAEP standards are too high. The Government Accountability Office, the National Academy of Sciences, the National Academy of Education and the National Center for Research on Evaluation, Standards and Student Testing have all studied NAEP achievement levels and found them wanting.As he continues, Bracey quotes the National Academy of Sciences: "NAEP's current achievement level-setting procedures remain fundamentally flawed, appropriate validity evidence for the cut scores is lacking, and the process has produced unreasonable results.
We suggest that you read the entire letter. (Below, well link to a fuller presentation by Bracey.) But heres an example of what Bracey means when he says that NAEPs standards are too high:
BRACEY: What does the National Academy of Sciences consider an "unreasonable result"? Just this: In the Third International Mathematics and Science Study, American fourth-graders ranked third in science among the 26 participating nations. Yet only 30 percent scored proficient on the NAEP's science assessment. Only 30 percent of our world-beating kids are proficient? Unreasonable. Ludicrous.For the record, the study to which Bracey refers took place in 1996.
Its easy to see why Bracey would say that the NAEP standard was too high in this instance. We wouldnt use that language ourselves; wed be inclined to say that the standard was artificially high. (Or: Misleadingly high.) To explain why, well now toss in a point of general logic.
When we set the standard for passing a test of this type, it always involves subjective judgments. What sorts of things must fifth-graders do to be judged proficient in reading (or math)? We can try to tie our answer to traditional notions of grade level work. But those traditional notions of grade level work werent taken from tablets which Moses delivered. Presumably, conventional notions of fifth grade work are based on observations of past fifth grade kids. But there has never been a time when fifth graders all functioned at the same level. To what level should a fifth grader be held? Inevitably, thats a matter of judgment. And oh yes! Wherever you end up setting the bar, it will be far below where some kids are working—and far above the functioning level of other fifth-grade kids.
Remember, kids are different! Consider reading, for example. In America today, many fifth graders can handle books written on traditional sixth or seventh grade level (or beyond). And many fifth graders are functioning years below grade level—or are barely able to read at all. What must fifth graders do before we say that they are proficient? Inevitably, thats a subjective judgment. We can set the bar as high (or as low) as we want. In the end, theres no right answer.
In 1996, was the NAEP standard right? To that, there can be no ultimate answer. But its easy to see why someone might say that the standard was artificially high. More specifically, its easy to see a problem which may have resulted; citizens may well have been misled by the gloomy results of that years NAEP test. Today, are we being misled, in some similar way, by the standards set by the NAEP? Bracey raises an excellent point—a point which should be explored further.
BRACEYS FULLER PRESENTATION: In his letter, Bracey cites a September article in which he questioned those high NAEP standards. To read that full piece, just click here.
BUT HOW DO WE MEET HIGHER STANDARDS: Will raising standards help struggling kids—kids who are reading far below level? As a general matter, weve never quite understood the logic of that supposition. When we taught fifth-grade kids reading on second-grade level (who might be sixth or seventh grade by age), we understood that they werent meeting standards. What we didnt necessarily understand was how we could help them do better.
On Friday, we posted Wieners prompt response to a question we sent him about this matter. In retrospect, we thought his reply deserved more review. Lets recall what he said:
WIENER E-MAIL: Expectations play a critical role in determining what students will be taught, and what they will learn. If the standards are pegged to low expectations, teachers (and the parents and students themselves) may be lulled into thinking they're doing well enough—or very close to it, which inevitably will be seen as good enough for some—even when skills/knowledge are not close to adequate. Ideally, systems for teacher training, curriculum design and evaluation, formative assessments, etc., should be built around the standards—that is, what students are expected to learn. If we start out aiming for mediocrity or worse, we might be giving ourselves a pat on the back when we reach abysmally low levels of achievement.Wieners specific example involves high school students. We wouldnt doubt that the students described might be better served by that college prep class—as opposed to some remedial course which may be a glorified storage facility. But as a general matter, we dont see why low-scoring fifth graders (or their teachers) will be helped by raising the bar on state tests. In our teaching days, we knew that we werent making the grade. We certainly didnt think we were doing well enough. We just didnt know how to do better.
Research documents that students learn more when they are put into more academically rigorous courses; this is true even for students who had been at the lowest levels of achievement in the past. In fact, students who previously were in the lowest quartile for achievement failed less often when they were put in college prep courses, compared to previously similarly performing students who were put into remedial courses. Standards can help push teachers to teach to higher levels (or not).
Of course, Wieners example isnt necessarily exhaustive; there may be other research which suggests that raising the bar helps low-achieving fifth graders. But heres a shortcoming we often observe with theorists of the standards movement. Such analysts suggest this: If we put more pressure on teachers (through higher standards and public accountability), they will respond by teaching better. But what should these teachers actually do to help their students meet higher standards? On that, these theorists often have no suggestions. Quite often, they demand that schools produce better results, without ever telling them how.
Its amazingly easy to raise the bar without explaining how to get over it. Mordantly grumbling, we sometimes say this: Nice work—if you can get it.