On NAEP scores, “proficient” does not mean performing at grade level

Using NAEP as a indicator of grade level performance is a mistake, according to Tom Loveless.

610

The recently released NAEP scores, which measure student performance in terms like “basic,” “proficient,” and “advanced,” do not indicate whether or not students are performing at grade level and should be used as one tool among many in determining how well a state is performing, according to education expert Tom Loveless. 

New Hampshire outperformed almost every other state in 2017, but a significant achievement gap remains. Read more about the results here

From the Brookings Institution:

NAEP does not report the percentage of students performing at grade level. NAEP reports the percentage of students reaching a “proficient” level of performance. Here’s the problem. That’s not grade level.

In this post, I hope to convince readers of two things:

1. Proficient on NAEP does not mean grade level performance. It’s significantly above that.

2. Using NAEP’s proficient level as a basis for education policy is a bad idea.

DOES NAEP PROFICIENT MEAN GRADE LEVEL?

The National Center for Education Statistics (NCES) states emphatically, “Proficient is not synonymous with grade level performance.” The National Assessment Governing Board has a brochure with information on NAEP, including a section devoted to myths and facts. There, you will find this:

Myth: The NAEP Proficient level is like being on grade level.

 

Fact: Proficient on NAEP means competency over challenging subject matter. This is not the same thing as being “on grade level,” which refers to performance on local curriculum and standards. NAEP is a general assessment of knowledge and skills in a particular subject.

Equating NAEP proficiency with grade level is bogus. Indeed, the validity of the achievement levels themselves is questionable. They immediately came under fire in reviews by the U.S. Government Accountability Office, the National Academy of Sciences, and the National Academy of Education. The National Academy of Sciences report was particularly scathing, labeling NAEP’s achievement levels as “fundamentally flawed.”

 

HOW UNREALISTIC IS NAEP PROFICIENT?

Shortly after NCLB was signed into law, Robert Linn, one of the most prominent psychometricians of the past several decades, called ”the target of 100% proficient or above according to the NAEP standards more like wishful thinking than a realistic possibility.” History is on the side of that argument. When the first main NAEP in mathematics was given in 1990, only 13% of eighth graders scored proficient and 2% scored advanced. Imagine using “proficient” as synonymous with grade level—85% scored below grade level!

The 1990 national average in eighth grade scale scores was 263. In 2015, the average was 282, a gain of 19 scale score points.

That’s an impressive gain. Analysts who study NAEP often use 10 points on the NAEP scale as a back of the envelope estimate of one year’s worth of learning. Eighth graders have gained almost two years. The percentage of students scoring below basic has dropped from 48% in 1990 to 29% in 2015. The percentage of students scoring proficient or above has more than doubled, from 15% to 33%. That’s not bad news; it’s good news.

But the cut point for NAEP proficient is 299. By that standard, two-thirds of eighth graders are still falling short. Even students in private schools, despite hailing from more socioeconomically advantaged homes and in some cases being selectively admitted by schools, fail miserably at attaining NAEP proficiency. More than half (53 percent) are below proficient.

Today’s eighth graders have made it about half-way to NAEP proficient in 25 years, but they still need to gain almost two more years of math learning (17 points) to reach that level. And, don’t forget, that’s just the national average, so even when that lofty goal is achieved, half of the nation’s students will still fall short of proficient. Advocates of the NAEP proficient standard want it to be for all students. That is ridiculous. Another way to think about it: proficient for today’s eighth graders reflects approximately what the average twelfth grader knew in mathematics in 1990. Someday the average eighth grader may be able to do that level of mathematics. But it won’t be soon, and it won’t be every student.

In the 2007 Brown Center Report on American Education, I questioned whether NAEP proficient is a reasonable achievement standard. That year, a study by Gary Phillips of American Institutes for Research was published that projected the 2007 TIMSS scores on the NAEP scale. Phillips posed the question: based on TIMSS, how many students in other countries would score proficient or better on NAEP? The study’s methodology only produces approximations, but they are eye-popping.

Singapore was the top scoring nation on TIMSS that year, but even there, more than a quarter of students fail to reach NAEP proficient. Japan is not usually considered a slouch on international math assessments, but 43% of its eighth graders fall short. The U.S. looks weak, with only 26% of students proficient. But England, Israel, and Italy are even weaker. Norway, a wealthy nation with per capita GDP almost twice that of the U.S., can only get 9 out of 100 eighth graders to NAEP proficient.

Finland isn’t shown in the table because it didn’t participate in the 2007 TIMSS. But it did in 2011, with Finland and the U.S. scoring about the same in eighth grade math. Had Finland’s eighth graders taken NAEP in 2011, it’s a good bet that the proportion scoring below NAEP proficient would have been similar to that in the U.S. And yet articles such as “Why Finland Has the Best Schools,” appear regularly in the U.S. press.

WHY IT MATTERS

The National Center for Education Statistics warns that federal law requires that NAEP achievement levels be used on a trial basis until the Commissioner of Education Statistics determines that the achievement levels are “reasonable, valid, and informative to the public.” As the NCES website states, “So far, no Commissioner has made such a determination, and the achievement levels remain in a trial status. The achievement levels should continue to be interpreted and used with caution.”

Confounding NAEP proficient with grade-level is uninformed. Designating NAEP proficient as the achievement benchmark for accountability systems is certainly not cautious use. If high school students are required to meet NAEP proficient to graduate from high school, large numbers will fail. If middle and elementary school students are forced to repeat grades because they fall short of a standard anchored to NAEP proficient, vast numbers will repeat grades.

On NAEP, students are asked the highest level math course they’ve taken. On the 2015 twelfth grade NAEP, 19% of students said they either were taking or had taken calculus. These are the nation’s best and the brightest, the crème-de la crème of math students. Only one in five students work their way that high up the hierarchy of American math courses. If you are over 45 years old and reading this, the proportion who took calculus in high school is less than one out of ten. In the graduating class of 1990, for instance, only 7% of students had taken calculus.

Unsurprisingly, calculus students are also typically taught by the nation’s most knowledgeable math teachers. The nation’s elite math students paired with the nation’s elite math teachers: if any group can prove NAEP proficient a reasonable goal and succeed in getting all students over the NAEP proficiency bar, this is the group.

But they don’t. A whopping 30% score below proficient on NAEP. For black and Hispanic calculus students, the figures are staggering. Two-thirds of black calculus students score below NAEP proficient. For Hispanics, the figure is 52%. The nation’s pre-calculus students also fair poorly (69% below proficient). Then the success rate falls off a cliff. In the class of 2015, more than nine out of ten students whose highest math course was Trigonometry or Algebra II fail to meet the NAEP proficient standard.

These data defy reason; they also refute common sense. For years, educators have urged students to take the toughest courses they can possibly take. Taken at face value, the data in Table 3 rip the heart out of that advice. These are the toughest courses, and yet huge numbers of the nation’s star students, by any standard aligned with NAEP proficient, would be told that they have failed. Some parents, misled by the confounding of proficient with grade level, might even mistakenly believe that their kids don’t know grade level math.

This report originally appeared in the Brookings Institution’s Brown Center Chalkboard on June 13, 2016. Read the full report here.