Cognitive Task Analysis - Analyzing the Cognition of Portal 2 Gameplay and Design

As we wrote about in our last blog post, ADL Initiative's Next-Generation Learner Team has been performing a cognitive task analysis (CTA) on the gameplay and design of Portal 2, a puzzle/logic game developed by Valve Software. Portal 2 had previously been utilized by the team in a study to test the effects of five game design features thought to improve cognitive adaptability (implicit rules, implicit shifting of rule sets, dynamic changing environments, implicit reinforcement, and open-ended gameplay). An informal review had been conducted to determine which commercial off-the-shelf game might contain these five design characteristics but a formal, empirical analysis of the game's design had not been conducted at the time of the study. Additionally, a review of relevant literature revealed that CTA had never been used to identify specific design tenets within a video game. Further, very little had been done to map either the cognitive and behavioral performances of expert gamers or the cognitive requirements of an entire game.

With these gaps in knowledge in mind, we set out to develop a protocol for performing cognitive task analysis on Portal 2 to accomplish two things: 1) to map the cognition of gameplay within Portal 2 and 2) to pinpoint the presence or absence of the five design tenets for adaptability. Since then, from the game that is divided into nine "chapters" each consisting of several different levels, we have collected CTA data for all 18 levels found in Chapters 1 and 2 and also sampled one level from each of the subsequent seven chapters. We noted the required cognitive and mechanical steps the player had to go through to complete each level, as well as what sub-goals and smaller puzzles comprised each level, what affordances and tools were present, what requisite and pre-requisite knowledge was required, and what audio and visual cues the player takes note of in each level. Analysis of the data led to the determination if and how much of the five design characteristics were present.

Results to Date

Initial analysis of the qualitative data collected from Portal 2 CTA has been quite promising so far. While this is only the beginning of the total set of information we hope to draw from the data, analysis so far has confirmed the modulating—but persistent—presence of all five design characteristics throughout the game.

Features 1 & 2: Implicit Rules and Implicit Shifting of Rule Sets

The composition of the rule sets of each level was examined, in terms of how many of the level's rules are old or new, and how many are implicitly or explicitly explained to the player. Implicit rules turned out to make up the majority of those present in each level, and some number of new rules—either implicit or explicit—emerged in each level surveyed.

Feature 3: Dynamic Shifting Environments

The number of changes to the environment—either in the form of altered environmental features or new ones—in each level in relation to previous levels in the game was also examined. In 18 out of 23 levels sampled, new environmental features or changes to environmental features were noted, indicating an overall dynamism and tendency towards change in the environments in the game.

Feature 4: Implicit Reinforcement for Actions/Choices to Accomplish a Final Goal

The majority of reinforcement and feedback for a player's choices and actions in the game was shown to be implicit, i.e., the player has to infer from what goes on in the game whether or not his or her particular action or choice was correct (as opposed to being given direct confirmation that he or she made a choice that would help them achieve the end goal of the level).

Feature 5: Open-Endedness

In addition to the findings above, analysis of the CTA data showed a total of 88 instances of open-endedness within the levels sampled. Examples of open-endedness include conditions such as unconstrained time, multiple ways of accomplishing a task, freedom of movement, control of portal placement, and choice of tools (not all of which are relevant). Though no game will ever be without rules—as rules are part of the definition of what makes a game a game—and thus will always have some constraints to gameplay, relative degrees of open-endedness are important to consider when looking for features that might foster cognitive adaptability.

Audio and Visual Cue Processing

We also recorded and coded the audio and visual cues noted by the player in each level by relevance: those that are critical to the player accomplishing the task of completing the level, those that are helpful in completing the level but the task can be accomplished if the player does not note the cue, and those that are irrelevant or distracting to the player.

This information about the fluctuation of different types of audio and visual cues throughout the game can inform us as to how much information processing and information filtering the player has to do in order to parse the irrelevant cues from the critical and helpful cues. We'll also be able to glean the needed information from the right cues. An example of the audio and visual cue analysis is captured in the chart above.

Next Steps

Though initial results of the analysis of CTA data are exciting, there is still more work to be done. For one, it would be ideal to examine all levels of the game. A complete analysis would make the data set more complete and would allow us to trace the transformative nature of the game chronologically—how requisite knowledge in one level becomes pre-requisite knowledge for completing the next, or how what begins as a puzzle in one level becomes a piece of knowledge or tool the player uses in subsequent levels. In conjunction with this, the team intends to analyze the progression of a player's cognitive load (the taxation on one's working memory capacity) throughout the game. We will keep you informed through future blog posts as we continue on our journey to understand how game design influences or enhances cognitive capabilities such as cognitive adaptability.