Sero! Closed

The Challenge

Although training and development programs within corporate environments have seen a significant expansion in recent years, there was little research into workplace learning assessments. Learning assessments can be used to gauge employee knowledge in specific domain areas, which can be useful in identifying strengths and weaknesses, and determining where improvements are needed.

About the Project

Instead of using concept maps as a learning aid which has been its traditional usage, the Sero! project set out to use concept maps as a knowledge assessment tool to gauge learning progress over the course of education and training curricula. Concept maps are a diagramming technique of knowledge organization and representation. Relationships between concepts are made by visually placing concepts in circles or boxes, and then lines are drawn to link concepts to each other to demonstrate relationships along with linking words to specify the relationship. Propositions are then created from two or more linked concepts to form a meaningful statement reflecting the knowledge of the learner.

Although the actual set-up and analysis for concept maps has been traditionally challenging, the Sero! platform was designed to realize efficiencies in their use to enable deeper learning assessment.

• For assessors, Sero! enables semi-automated assessment authoring, proctoring, analysis, and feedback to learners.

• For learners, Sero! distributes assessments and provides an intuitive introduction to, and completion of, concept map-based assessments.

This project also facilitated a summative assessment capability, guided by user-centered design. Learners interact with concept maps using Sero! in different stages of completion, for example, by manipulating the position of the concepts, linking phrases, or changing the words inside the nodes to create different propositions, making connections between nodes, and identifying errors within the map. Sero! leverages Experience API (xAPI) to support learner/assessor feedback with reporting capabilities and provides both assessors and learners with access to analytics to enable adaptive learning.

To create the assessment, an assessor uses Sero! to enter a set of propositions that form the concept map. Then the assessor identifies which concepts, linking phrases, and/or connectors they want to assess. Sero! converts the items into one of five assessment types: multiple choice, generate and fill-in, select and fill-in, connectors, and error recognition/revision. The use of multiple item types within a concept map is an innovation made possible through the Sero! project since prior research only focused on one type per concept map. Learners are provided with each of the concept map types to complete and/or correct to conduct the assessment.

Sero! was piloted in an adult learning environment to capture user experience and feedback. Usability tests were conducted which employed maturing versions of Sero! and increasingly complex tasks. The United States Marine Corps participated in the usability test to help determine design inconsistencies and usability problem areas within the graphical user interface, and help establish baseline user performance, including time on task, and user-satisfaction levels of the user interface for future usability evaluations. The findings from these sessions were used to inform ongoing development of the software.

The outcome of the Sero! project was a prototype web-based, xAPI-compliant open-source software package, along with system requirements, use cases and design documents. Sero! v.1 made the short list for a Reimagination Education award in the Learning Assessment Category in 2018, and this tool has been further developed to expand its assessment capabilities. In 2019, the Air Force Research Laboratory funded an initiative to include formative assessment and mental model comparison capabilities in Sero! v.2, and in 2020, the Army Research Institute awarded a contract to explore and develop automated authoring algorithms for concept mapping-based assessments in Sero! v.3.

Publications