By Lillian Mongeau

To get a better understanding of how well students can solve complex problems and apply science to real-life scenarios, the National Assessment for Education Progress recently used hands-on experiments as a way to test 4th, 8th, and 12th grade students, and found that this kind of assessment gives a much more accurate reflection of student comprehension.

Results from a 2009 round of testing called The Nation’s Report Card Science in Action: Hands-On and Interactive Computer Task, examined 6,000 students—2,000 at each grade level—from across the country. Students performed tasks like testing water samples (12th grade) and assembling electric circuits (4th grade). They also participated in interactive computer tasks that simulated longer term experiments, like observing plant growth. In both scenarios, students were evaluated on their ability to perform the tasks, observe the results and draw conclusions.

“The bottom line is, we learned so much more that we couldn’t have learned from those paper and pencil tests,” said Jack Buckley, commissioner at the National Center for Education Statistics, which creates the annual “Nation’s Report Card” based on the results of tests like this one administered by the National Assessment for Educational Progress (NAEP).

But what they learned was a mixed bag.

A majority of students at all grade levels (76 percent) were able to perform the simpler experiments correctly and accurately observe the results. However, when experiments involved more complicated data sets, students’ ability to execute and observe fell sharply — only 36 percent of students tested across grade levels were able to complete the tasks under these conditions.

The test also revealed a disconnect between observation and explanation. Even though a majority of students (71 percent) were able to draw the correct conclusions from the results of their experiments, less than a third (30 percent) were able to explain their results.

For example, one of the hands-on tasks for 12th grade students was to determine the best location for a new town based on water quality. The students were expected to test various water samples for specific pollutants and then compare those levels to a chart put out by the Environmental Protection Agency. A whopping 75 percent of students were able to do this accurately. But when it came time to make a recommendation for where the new town should be built, only 11 percent of students were able to explain their recommendation using the data they’d collected.

The conclusion? “[Students] can conduct science investigations using limited data sets, but many students lack the ability to explain results. The report shows that students were challenged by parts of investigations requiring more variables to manipulate, strategic decision-making in collecting data, and the explanation of why a certain result was the correct conclusion,” the report states.

For the most part, student performance broke down as it usually does along ethnic and economic lines. Low-income students performed worse than their wealthier peers and black and Latino students performed worse than their white and Asian counterparts. However, there were a few notable exceptions.

On some parts of some tests, black and Latino students did as well or nearly as well as white students. For example, on a computer task that required 4th grade students to observe plant growth, 80 percent of students came to the correct conclusion. Eighty-one percent of white students got the right answer, 79 percent of black students did and 74 percent of Hispanic students did. (Eighty-six percent of Asian and Pacific Islander students got that one right.)

What’s more, though male students generally outperform female students on the national science assessment, female students beat male students on the hands-on tasks.

Alan Friedman, a physicist and the chair of the committee in charge of developing national assessments, said that as a scientist he was relieved that students did well on the first section of the test. “There’s no way for them to memorize for this test. You really had to think on your feet,” he said.

Still, Friedman said, he wasn’t shocked that students struggled to explain their results. “Unfortunately, that’s not surprising,” he said.

Though hands-on standardized tests aren’t brand new, they have historically been too expensive and complicated to use on a wide scale. And the technology needed for interactive computer tasks has not been up to snuff until recent years.

Officials at NAEP said tests like these are more accurate and provide far more detailed results.  Buckley said they must become the norm to keep up with new curriculum standards meant to keep pace with the changing world of science and technology.

“We’re in a really good position to provide models for assessment,” Buckley said, that can “provide information on what students can know and do that’s called for in the new standards.”

Flickr: t3rmin4t0r

By showing the percentage of students who pass state standardized tests in math and language arts, in, for example, Oakland, Calif., compared to Oakland Unified School District, to the state of California, and to the NAEP exam, parents can assess for themselves how each school stacks up to other schools in the country.

Here’s why that’s important, as explained on the site:

State assessments are different from the NAEP. Each state determines its own definition of proficiency, whereas the NAEP’s definition is the same across the country. The national definition of proficiency is more rigorous than most states’ definitions.

The site, which was announced at Education Nation conference Monday, also shows whether the state is one of 35 states that has complied with Common Core Standards, which calls for uniform standards across the country; high school graduation requirements; and a comprehensive link to how to use the data.

Once they’re armed with the information, parents are prompted to ask questions — lots of them — on a range of topics, from what’s the gap from national to state results, to what districts are doing to make sure teachers are effective.

What’s the objective? I asked Bill Jackson, founder and CEO of GreatSchools.

“We want to provide parents and the public with a high-level picture of how schools, states, districts and the country are doing,” he said. “It’s a tool for parents to be informed. The goal is to make sure kids are on track to be able to go to college and to compete for better jobs. That’s the kind of realism we need. It’s setting our expectations on a higher level.”

Jackson sited the state of Tennessee as an example of one that might benefit from Scorecard. “Until recently, it had a highly inflated view of their achievements,” he said.Their “inflated” state scores were not a realistic reflection of school performance when placed in the context of federal test scores.

“We hope that other states have a really good discussion about that, too” Jackson said. “I don’t think it serves anybody if there’s too much wishful thinking.”

In the coming weeks, I’ll check in with a variety of sources — parents, administrators, and teachers — to see how the site will affect their decision-making process.

Other partners in the Education Nation Scorecard project include Achieve, the Alliance for Excellent Education, Education Week, the National Parent Teacher Association (PTA), and other education data thought leaders.