Zombie Division : a methodological case study for the evaluation of game-based learning

This paper discusses the methodological designs and technologies used to evaluate an educational videogame in order to support researchers in the design of their own evaluative research in the field of game-based learning. The Zombie Division videogame has been used to empirically evaluate the effectiveness of a more intrinsically integrated approach to creating educational games. It was specifically designed to deliver interventions as part of research studies examining differences in learning outcomes and motivation predicted by theoretical contrasts in educational design. The game was used in a series of evaluative studies, which employed experimental methodologies based around one or more treatment groups and a control. Multiple choice questions were used to measure knowledge and understanding before and after interventions (pre, post and delayed) and time-on-task was used as a measure of motivation and preference during interventions. Qualitative interview data was also collected and analysed as part of many of the studies in order to help support and explain the findings in more detail. The experimental methodologies applied in these studies were augmented by a range of bespoke technology systems. This included an automated testing system which could randomly assign participants to treatment groups so that pre-test statistics were closely matched between groups. Large quantities of process data were recorded about players’ interactions with the game in the form of time-stamped log files, and a stream of compressed controller data was saved allowing an entire playing session to be replayed in a video-like form. This rich set of process data was mined as part of a post-hoc analysis in order to identify evidence to help to enrich the understanding of users’ interactions with the game. This paper details the methodological design of both published and unpublished studies, as well as reflecting upon some of the potential pitfalls of classroom-based evaluations in order to illustrate successful and unsuccessful approaches for evaluating game-based learning.