Beyond Grades: Do Games Have a Future As Assessment Tools?

| April 28, 2014 | 13 Comments
  • Email Post


Most tests represent a snapshot of one moment in the trajectory of a student’s academic journey, extrapolating what the student has learned overall. There are plenty of ways educators are trying to supplement those tests with more nuanced, formative assessments. With the advent of game-based learning, educators have been investigating how data collected from video game play could provide insight into the way students think as they explore new concepts.

A report from the game developers, learning specialists and psychometricians involved with GlassLab’s project SimCityEDU finds that there’s great potential for games and assessment, but a lot of work to be done before games are used as primary assessment tools.

SimCityEDU is a game created to introduce environmental science to middle school students. Through game play, students are asked to conduct interviews, review research, interpret information they’ve gathered, draw conclusions, graph data, and make decisions about how to best protect the environment in a prototype city, and with each step, the game assesses every choice.

The goal is to move beyond tests that give little meaningful information about how or what students have learned, instead using data gathered from actions in a game to paint a picture of growth of knowledge. The report finds that this game-based assessment is a promising new field, but the work is still in its initial phases. The report finds that the close collaboration of all parties is crucial. Psychometricians, game designers, and learning specialists all need to work together to understand how to create the best learning experience. Just as game creators should work closely with teachers when creating the games, rather than just handing off their finished products, psychometricians need to be part of the designing and creating phases, too.

“The tests we have don’t do a good job of measuring what’s important to us and they don’t measure what they do measure very well.”

“The tests that we use today in school don’t test for things that are that useful,” said Anya Kamenetz, author of forthcoming book on future of assessment called The Test and author of the GlassLab report’s executive summary, (and a contributing writer to MindShift). “Accountability tests [measure] reading and math,” she said, but students need to know so much more than those subjects in the world outside of school. Games more accurately test those less tangible qualities by building higher order thinking skills into the game play.

“The state of mind that game designers are chasing is similar to what teachers are trying to do, which is get students in the zone of proximal development,” Kamenetz said. That means finding the “sweet spot” where the material isn’t too easy or too hard. It should provide a challenging and thus engaging experience, but not be such a frustrating experience that students quit. Games reach this state naturally by preventing players from moving on until they’ve mastered the skills needed at lower levels.

That’s why psychometricians are so excited about game play as a lens into learning. Psychometrics is an old discipline that tries to quantify intellectual achievement. Its practitioners use statistics to try and figure out what people know and how they think. Psychometricians are often responsible for creating tests.

“Psychometrics has something to bring to the table,” Kamenetz said. But at the same time, game-based learning is challenging the psychometric field to move into the 21st century. The data analytics available to educators now are changing the landscape for psychometricians, who need to evolve a new set of tools to measure the principles important to the discipline — reliability, generalizability, comparability and validity — within a new context.

Games are proving to be a good place to start the delicate partnership between the diverse working styles of game developers, educators and psychometricians.

“There’s reading, there’s science involved, there’s strategy and systems thinking and there’s perseverance,” Kamenetz said about SimCityEDU. “You can’t artificially tease those apart, you have to look at them all together.” It’s that holistic view that is so novel. Current tests isolate subjects and skills, rather than testing how a student uses everything he knows to explore, test out ideas, and gain experience.

“Clearly this isn’t the kind of test you can drop from the sky in a high stakes context and expect people to perform,” Kamenetz said. Players respond differently to games for many reasons including level of experience and interest in game play, not to mention whether they understand the academic concepts. But, Kamenetz argues that bubble tests take a certain amount of familiarity with the format as well.

“The tests we have don’t do a good job of measuring what’s important to us and they don’t measure what they do measure very well,” Kamenetz said. “So we need a better test.” The game-based assessment work is promising, but still a long way off, and alternatives need to be developed in tandem.

“One thing that might stop this in its tracks is the ability to extract data from naturalistic interactions,” Kamenetz said. Tools are already being tested that eavesdrop on students in unstructured situations that can fairly accurately predict knowledge based on the words used and complexity of thought. If this technology catches on it would allow educators to removing the artificial game world from the assessment, making it a very real test.

Ultimately, the GlassLab report is a call to psychometricians to continue the work on game-based assessments. It questions how tests are currently built and the role psychometricians can have on changing them more than on the impact of game-based assessment in the classroom. Predictive technologies and complex data analytics already exist in the commercial world, so why shouldn’t those techniques be applied to the important work of teaching?


Explore: , ,

  • Email Post
  • Cynthia Duncalfe

    I agree that “the tests that we use today in school don’t test for things that are that useful,” and am very encouraged to see progress in ways we can extract meaningful assessment from students’ actions in game-based learning. In my mind, the best educational games mimic real-world decision making, and engage the student in critical thinking – there is no need to think of these kinds of games as a second rate substitute for any other method of learning.

  • Pingback: Beyond Grades: Do Games Have a Future As Assessment Tools? | NLG Consulting

  • Pingback: Beyond Grades: Do Games Have a Future As Assessment Tools?

  • Delia Quellman

    Cynthia, I totally agree with you. Tests in schools are not encompassing things that are useful to students. It’s important to implement educational games that do mimic real-world decision making, and engage students in higher level thinking skills (Bloom’s taxonomy). That constant feedback loop is essential to students in order for them to self-assess and immediately hit their targets.

    • Cynthia Duncalfe

      I agree; thanks, Delia.

  • Rene

    Yes, GBAs with computer adaptive learning provide fine grained assessments! PK-2nd try Native Numbers for number sense (iPad app) for this type of GBA!!!

  • Pingback: How Games Lead Kids to the Good Stuff: Understanding Context | MindShift

  • Maryann Countryman

    I think some of the best, engaging games that encourage critical
    thinking are games where there isn’t necessarily a right or wrong
    decision-making structure, but a game structure where the decisions of
    the player change the storyline or next “level.” Critical thinking ties
    in with predicting cause and effect relationships, which games absolutely emphasize!

  • Pingback: Beyond Grades: Do Games Have a Future As Assessment Tools? | My Educational Technology Blog: A Place of Resources and Tools for Educators

  • Pingback: Column: Using video games instead of grades at school? | Gabriele Ferri's research blog

  • Pingback: the game-player’s hat | 500 Hats

  • Melissa Schaefers

    I think testing through real-world problem solving would be very beneficial to students. Traditional testing requires students to memorize subject matter so that they can regurgitate it onto paper. Most students do not remember the testing material within a week of testing. To use something that actually measures the way that they use the learned information would be so much more helpful and accurate. I can’t wait until the technology is developed enough to use this type of testing in my classroom.

  • Pingback: Can Games Make High-Stakes Tests Obsolete? | MindShift