The ADL Initiative Successfully Completes FY18 TLA Developmental Testing with SWEG(A)

The Special Warfare Education Group (Airborne) (SWEG(A)) hosted the ADL Initiative’s 2018 Total Learning Architecture (TLA) Test and Demonstration event at their facility in Fort Bragg, NC, 14–17 August 2018. This event marked the second empirical trial of a TLA prototype, and it provided ADL Initiative researchers with the opportunity to observe its operation under semi-realistic, ecological conditions.

This TLA event involved 60 Soldiers from Fort Bragg, along with SWEG(A) and ADL Initiative staff and sponsored performers from Eduworks, Institute for Defense Analyses, Perigean Technologies, and SoarTech. The Soldiers served as TLA testing participants, each completing roughly 12 contact hours with the system and interacting with training on Combat Awareness, Tracking, and Profiling.

TLA and the 2018 TLA Prototype

Soldiers’ xAPI-based performance was aggregated across all learning activities and visualized in real-time.

The TLA is a collection of specifications for accessing and making use of learning-related data, or in other words, a set of documents that define how learning and development technologies “talk” to one another. Those specifications documents define, for instance, Application Programing Interfaces (APIs), data elements, metadata schemas, open architecture designs, and middleware services. Ultimately, once matured and implemented, the TLA will enable personalized, data-driven, and lifelong technology-enabled learning across the DoD and potentially beyond.

To test the viability of the 2018 TLA draft specifications, the researchers needed to implement a reference implementation model, or an example prototype of a possible TLA system. For the 2018 trial; the prototype included the following learning activity providers with which the participants interacted:

• iPad-based Personal E-books for Learning (PEBL)
• Web-based Moodle, a traditional, off-the-shelf open-source LMS
• Web-based Perceptive Adaptive Learning Modules (PALMs)
• Web-based Sero! a web-based concept map assessment tool
• Web-based Static content viewer for PDFs and static webpages
• Web-based Unity 3D “games” (with an xAPI wrapper added)
• Web-based YouTube video player
• Live training activity, with the “scores” inserted manually via a “grade book”

The 2018 prototype also included services for cross-system learning recommendations, managing and asserting competencies, managing learning evidence, aggregating and visualization of learner data, and managing learner profiles. These back-end capabilities were designed to support and interact with the holistic system (i.e., across all the learning applications).

These middleware and back-end services and data components included:

• Competency and Skill System (CASS), an xAPI evidence-to-competency mapper
• Fast Learning from Unlabeled Episodes for Next-Generation Tailoring (FLUENT) recommender
• An Activity Index of the various user-facing learning activities and their metadata
• A Learning Record Store to manage xAPI data, using off-the-shelf software

A summary of the interoperability specifications, data elements, and middleware considered during the 2018 TLA development spiral is included in the upcoming I/ITSEC 2018 publication, “Total Learning Architecture Moving into the Future.”

Soldiers' xAPI-based performance was aggregated
across all learning activities and visualized in real-time.

Testing Outcomes and Way Ahead

The TLA prototype generated more than 400,000 xAPI activity statements, just over 50,000 created from students’ interactions and around 250,000 from system logs. The participants also completed pre- and post-tests, reactions surveys, and focus groups. Similarly, the developers provided their observations of the experience and recorded formal observations of participants interacting with the prototype system. Initial data collected from the testing event show positive outcomes—but also clear areas for improvement.

The 2018 testing event illustrated the need for additional research in areas, such as: discovery services to automate the collection of metadata and alignment to competency objects, additional metadata fields to describe content and competencies, concept and practical interactions among competencies, and learner profile representations. Also, there is a need for a learner service that pushes information about learners that different components need (e.g., mastery estimates, preferences).

For 2019, the ADL Initiative intends to focus on updating the architecture, honing the metadata data elements, and improving the maturity of associated xAPI Profiles. The 2018 demonstration revealed the limitations of a point-to-point architecture design, in areas such as scalability and service modularity. For the next spiral of development, the TLA prototype will use a streaming data architecture, with the data sources available as streams, allowing any service to subscribe to their information. The success of 2019 reference implementation prototype will rest largely on the creation and documentation of a microservices architecture that allows each TLA component to publish and subscribe to different data streams.

The 2019 prototype will also incorporate a more robust xAPI Profile, to more strictly control the terms used and to define a broader, more meaningful set of data elements that can inform more useful learning analytics. Similarly, in 2019, the ADL Initiative will focus on improving the data and metadata elements associated with the content activity index and learner profiles.

Additional information about the 2018 testing outcomes and 2019 engineering plans is forthcoming. Reports on the progress of this effort will be posted on the ADL Initiative website, as it becomes available. The ADL Initiative also plans to start an open working group in 2019, to encourage broader collaboration across the public sector and professional learning specifications community.