TLA CMM Level 2 - Rollup of Learner Performance across Learning Activities

Once an xAPI instrumented activity is connected to the TLA Reference Implementation (via the Learning Record Store (LRS)), other connected systems have the ability to glean insights from the learner performance data generated by that activity. The raw data collected in the LRS provides valuable insights into the learner pathways and decisions made within the context of each learning activity. These data can be used to provide automated feedback, remediation, instructor support, and can provide an understanding of how and why a learner performed the way they did within a specific learning activity.

Figure 4 - TLA Master Object Model (MOM) State Models

The raw learner data from each learning activity can be connected to build learner insights beyond what a single learning activity is capable of. The TLA uses the concept of Control Loops, as shown in Figure 4, to delineate how this 'lifelong' learner data might be used by different systems and at different levels of granularity across different time horizons. In other words, the same data collected from a learning activity may be used in different ways depending on the purpose for which the data is being used. The five control loops in order of ascending time horizons address:

• Control Loop 1: Using learner performance data to optimize the transfer of learning within the current learning activity (e.g., Intelligent Tutoring, Instructor Support).
• Control Loop 2: Using learner performance data across numerous learning activities to optimize a learner's progress toward a credential.
• Control Loop 3: Using longitudinal learner data to prioritize the pursuit of credentials or activities to meet requirements for a potential job.
• Control Loop 4: Lifelong learning data to support career field management and the planning of education and training goals for an overall career trajectory.
• Control Loop 5: Lifelong learning data to support the establishment of a new career.

Additional information can be found in the 2019 TLA Report (opens new window) and in the updated TLA Functional Requirements (opens new window).

Given the diversity of different learning activities, occupational domains, and the xAPI Profiles that govern the xAPI Implementation within those systems, the TLA needs a way to normalize the data coming out of each activity. The TLA CMM Level 2 incorporates the TLA Master Object Model (MOM) to 'roll up' the raw learner data into meaningful information that other connected systems can use. The TLA MOM includes xAPI statements that describe key learner milestones for tracking and managing learner progression across all learning activities a learner encounters. The TLA MOM statements are used to support the different control loops defined above.

Figure 5 - TLA Master Object Model (MOM) State Models

As shown in Figure 5, learning activities store raw learner data inside an LRS. Raw learner data in the form of xAPI statements are governed by xAPI Profiles and can include extremely detailed information about learner interactions inside the activity. These data support Control Loop 1. The TLA MOM normalizes the xAPI statements coming out of a learning activity, as well as other learning systems such as schedulers, competency management systems, and career field planning tools. The TLA MOM is divided into different states that describe the different interactions a learner will perform.

The TLA MOM's Learning Activity State defines the key learner interactions a user will perform in a learning activity from initialization to completion or termination. TLA MOM verbs in this state conform to the cmi5 Specification (opens new window), which follows the lifecycle of the legacy SCORM cmi.core run-time data model. The use of cmi5 normalizes performance data and allows edge systems to perform their own adjudication so there are no conflicts within the TLA core data of what “correct performance” looks like.

The Learning Event State is associated with the activities that occur before a learner has interacted with a learning activity (e.g., requested, approved, scheduled, recommended, among others). It describes the context under which the learner pursued their learning (associated with Control Loops 2 and 3). TLA verbs for this state are also generated after learner performance evidence has been generated or when an activity has been completed. These verbs are used to contextualize learner performance against a related set of competencies (e.g., validated, qualified, conferred, inferred, among others).

Career States are the verbs associated with moving to or progressing toward jobs on a career arc. These MOM statements would be generated by career field management tools, human resource systems, or other systems associated with Control Loops 4 and 5. Career States change slowly over time as a learner moves from job to job and meets different career milestones (e.g., promoted, detailed, selected, among others).

The TLA MOM is implemented as an xAPI Profile that generates xAPI statements at key points in the learner lifecycle and includes pointers to the raw learner data for systems that need it. The latest version of the TLA MOM in JSON format can be found on the TLA MOM GitHub (opens new window) page. The 'md' version of the MOM provides an overview of the MOM in human-readable format. Each TLA MOM Statement is comprised of the following:

• Actor: This field is populated by each learner's Unique User ID (UUID). Within the TLA Sandbox, this is generated by KeyCloak.
• Verb: This field is populated in accordance with the TLA MOM's Learner State Models (described in Figure 5 above).
• Object: This field depends on the type of system that is generating the TLA MOM statement. For example, when generated by a learning activity, this field will be populated using the Unique Activity ID associated with each connected Learning Activity within the TLA enclave (e.g., sandbox). The Unique Activity ID points to additional information from the course catalog that describes that learning activity.

Learning Event Management (LEM) Services

The sandbox supports learning activities that natively support xAPI through the implementation of xAPI Profiles, but also supports legacy systems through a suite of LEM services that use the data generated by a learning activity (e.g., system, platform, app) to trigger the creation of xAPI statements. The TLA Sandbox uses a federated LRS structure that features three, logically separated LRSs.

• The Noisy LRS collects raw xAPI statements from each learning activity. Learning activities can either use the Sandbox LRS or their own xAPI conformant LRS to collect these statements.
• The Transactional LRS collects TLA MOM statements that are sent from learning activities, schedulers, recommendation systems, career field management tools, and other connected learning systems.
• The Authoritative LRS collects TLA MOM statements that have been processed and verified by the CaSS Competency Management System.

The TLA Reference Implementation takes all learning events in the form of cmi5 and MOM statements and sends them to a single LRS. MOM statements refer to xAPI statements defined by the TLA’s MOM profile, which is intended to create a “Learner Centric” stateless system. The MOM is still being revised so has not yet been published, but the beta draft of this profile is available to share upon request. Access to the Transactional and Authoritative LRS is restricted. For the endpoint and credentials required to send and query statements from the the LRSs, please contact TLA Technical Support. It is important to note that the LEM services and LRS structure used in the TLA Sandbox are specific to the ADL Initiative’s Reference Implementation. Other TLA compliant enclaves will likely meet these TLA requirements using different tools, technologies, or methods. Additional information on the services used in the Reference Implementation can be found on the TLA LEM GitHub (opens new window) page.

Kafka

Apache Kafka (opens new window) is used in the TLA Reference Implementation to route xAPI statements to the appropriate LRS partitions (i.e., noisy, transactional, and authoritative). The TLA Kafka GitHub project (opens new window) includes the container definitions and scripts for standing up a 3/3 SASL-authenticated Kafka cluster with Docker. This configuration of Kafka includes 3 Kafka brokers and 3 Zookeeper nodes. Zookeeper is used to manage the cluster topology. This configuration runs within a t3.xlarge EC2 instance, but more robust systems will likely require additional brokerage services across machines / availability zones.

3 scripts are used to maintain the Kafka cluster:

• tla-topics.txt: Test file that lists the Kafka topics.
• export-topics.sh: Creates the Kafka topics. This is required because Kafka's auto-topics are disabled in the TLA.
• list-topics.sh: Lists all topics available in the cluster.

Current topics include:

• learner-xapi: All connected learning activities send their xAPI statements to this topic.
• system-xapi: This topic is used to emit logs sent as xAPI statements. This topic is not currently used in the Reference Implementation.
• resolve-pending: Topic reserved for statments that are deemed MOM relevant but require additional resolution. This topic may not be required in other TLA implementations. It is specific to the way we filter and sort xAPI statements across the different LRS partitions (i.e., noisy, transactional, and authoritative).
• resolved-xapi: Transactional LRS statements that have been resolved or confirmed as MOM statements.
• authority-xapi: Authoritative LRS statements that have been validated or confirmed. Also called assertions of competency.

The TLA Kafka cluster includes 3 topics that can be used to test messages between producers and consumers: test-1, test-2, test-3.

Kafka Statement Viewer

Figure 6 - Kafka Statement Viewer

The Kafka statement viewer shown in Figure 6 is a NodeJS Express (opens new window) application that allows users to specify the Kafka brokers to use and the topics they want to monitor. Out of the box, this viewer allows users to monitor and filter xAPI data traffic across all Kafka topics. In this case, each topic is also color coded so they can visually see the Noisy LRS data, the Transactional LRS data, and the Authoritative LRS data as it is being sent to each LRS. Each statement can be expanded to review the specific contents of each xAPI statement. The Kafka statement viewer GitHub (opens new window) site provides additional information on how this might be used.

xAPI Processing

The MOM does not define how statements are interpreted by other systems. The general workflow is that a Learner Record Provider (LRP) should be able to send xAPI statements with zero enclave-specific modifications. To accomplish this, all xAPI statements are routed through an Apache Kafka cluster in real-time, where Kafka-consuming services would process them.

Statements are divided into four topics (i.e., data streams):

• learner-xapi (Noisy, contains every statement in our setup)
• resolve-pending (candidates for resolution, see below)
• resolved-xapi (Transactional)
• authority-xapi (Authoritative)

Figure 7 - Federated LRS structure in the TLA Sandbox

As shown in Figure 7, the TLA Reference Implementation uses a federated set of LRS solutions within the TLA Sandbox. This approach stores all raw learner data in the noisy LRS and maintains data ownership with the owners of the learning activity. The transactional LRS collects TLA MOM statements that 'roll up' and normalize the learner performance data coming out of each activity. The authoritative LRS only includes verified competency assertions which will be discussed in the next chapter. To reduce costs, a single LRS was instantiated that uses virtual partitions to store the different types of xAPI statements (i.e., noisy, transactional, authoritative). Several TLA LEM services are used to filter and sort the xAPI messages coming into the LRS. These include:

• Mail Sorter - This is the first service in the chain and the only service that actually subscribes to learner-xapi.

• Resolver - This service checks to see if the statement's object.id is relevant to a learning resource's Activity ID within the TLA Experience Index. This is the only service subscribed to resolve-pending.

• Scheduler - This service reads from resolved-xapi to interpret the schedule-relevant statements within the MOM profile and updates that part of the learner profile.

• Assertion Generator - This services reads from resolved-xapi and writes to authority-xapi. It interprets the completion-relevant statements and generates competency assertions based on that activity's definition in the Experience Index. There is no logic incorporated into this service and it is not included in the docker-compose.yml file because this function is handled in the TLA sanbox by CaSS.

• Goal Service - This services reads from both resolved-xapi and authority-xapi to check for goal-relevant MOM statements. These statements either create or update learner goals that are listed in the learner profile (described in the next section). In TLA terms, a learner goal is generated by a number of different systems that sit outside of an actual learning activity (e.g., scheduler, recommender, career field management tools).

The ADL Initiative is interested in instrumenting these types of systems to generate TLA MOM statements. If you have tools or systems that have potential to connect, please contact TLA Technical Support to discuss.

Learner Profile System

The TLA Sandbox has a rudimentary learner profile system (opens new window) that reads from the Authoritative LRS and provides a historical ledger of all learner interactions with any of the connected systems. The learner profile is tied to the Identity and User Management service as well as a Competency Management System (CMS). It subscribes to the authority-xapi kafka stream that is populated by the CMS. This stream contains authoritative assertions of competence and awarded credentials (e.g, badges, degrees, certifications). The CMS reads learner records from the resolved-xapi kafka stream (i.e., the Transactional LRS).

The learner profile also tracks a user’s roles and permissions within the TLA and uses the Learner’s OpenID as generated by Keycloak. Currently, access to the learner profile system is limited to the learner and their OpenID Connect (OIDC) session. The ADL Initiative is currently working to mature this capability by enabling role-based permissions for viewing, filtering, and visualizing learner data.

Querying from the Learner Profile

While access to the learner profile is not granted as a default to the TLA Sandbox, users can connect the learner profile on an 'as-needed' basis. Please contact TLA Technical Support to discuss your requirements.

To query the Learner Profile make a GET request to the Learner Profile endpoint: <profileURL>/me?secret=<secret>&user=<uuid>. Headers are not required. Upon a successful query, the Learner Profile will return a large JSON object with all Learner Profile data:

   "id": 2,
   "uuid": "UserIRI",
   "handle": "HandleURI",
   "uxRoles": ["Administrator" and/or "Learner" and/or "Observer" and/or "Instructor" ],
   "learnerState": "asserted",
   "goals": [
      {
         "user": " UserIRI",
         "goal": "GoalURI",
         "type": "competency",
         "status": "satisfied"
      }
   ],
   "tasks": [
      {
         "id": 3,
         "uri": "TaskURI",
         "user": "UserIRI",
         "status": "new | completed"
      }
   ],
   "competencies": [
      {
         "user": "UserURI",
         "competencyID": "CompetencyURI",
         "confidence": "-1 to 1"
      }
   ]
}

CMM Level 2 - Pulling it all together

Conformance to the TLA CMM Level 2 results in a single transactional LRS capability that aggregates and stores learner data from all connected learning activities which use their own LRS to collect the raw learner data. TLA MOM statements include each learner's UUID and the unique Activity ID assigned to the learning activity that generates the MOM statement. This provides traceability of learner data for all learning activities that a learner encounters within an organization. Using this information, other connected systems can identify what education and training activities a learner has completed, use catalog descriptions to learn more about each activity, and evaluate learner performance in those activities.

Within CMM Level 2, each learning activity's unique Activity ID may be located in different organizational course catalogs. For example some learning activities may be listed in a catalog attached to an LMS while other learning activities might be part of a Learning Experience Provider (LXP). The TLA Reference Imlementation includes an Experience Index and a rudimentary course catalog that can be used to support CMM Level 2 testing and evaluation. However, the TLA Reference Implementation's course catalog capability is working toward CMM Level 3 to enable a network of course catalogs that federate into an enterprise-wide catalog capability. The Experience Index and course catalog are discussed further in the CMM Level 3 section of this document.