Skip to main content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Publications

The ADL Initiative is an active participant in research and academia. Here you will find a full library of conference papers, conference abstracts, conference PPTs, books, journal articles, and technical reports composed by partners, collaborators, and ADL Initiative team members. Check back often for new entries.

Publication Results

Data Analytics and Visualization Environment for xAPI and the Total Learning Architecture: DAVE Learning Analytics Algorithms

2019; Yet Analytics, Inc.
This report introduces a language for defining the functionality of learning analytics algorithms in terms of Operations, Primitives and Algorithms which will be used to define Algorithms corresponding to an initial set of learning analytics questions.
This report introduces a language for defining the functionality of learning analytics algorithms in terms of Operations, Primitives and Algorithms which will be used to define Algorithms corresponding to an initial set of learning analytics questions.
DAVE

Competency-Based Learning

2019; Smith, Brent; Gordon, Jerry; Hernandez, Mike
Military education and training encompass many different schools, universities, and training programs designed to foster technical, professional, and leadership skills in military service members. Historically, there has been a separation between the education and training communities across the services. Education occurs incrementally and...
Military education and training encompass many different schools, universities, and training programs designed to foster technical, professional, and leadership skills in military service members. Historically, there has been a separation between the education and training communities across the services. Education occurs incrementally and involves grappling with ambiguity while thinking and reflecting about the concepts being learned. Training is linked to readiness and offers opportunities to apply an individual's knowledge, skills, and abilities in a manner that provides immediate feedback and progress measurement. Within the current context, training and education also have different reporting structures, motivations, and logistical requirements such as fuel, personnel, and the access to the appropriate environments or equipment. Future warfighters must rapidly prepare and adapt to function in an increasingly volatile and complex environment. Where current assessment methods judge knowledge retention for education and skill proficiency for training, there is a significant gap between those metrics and operational readiness and performance. Future force management and readiness metrics must directly link preparations to operations. The class of metrics used to do this is referred to as competencies – a measure of how well someone can use their acquired skills and knowledge to complete task(s). To enable the future learning ecosystems, all the ways to assess the ability of a person will need a digital representation. The on-going research on Experience API (xAPI) provides the capability to capture any transaction of learning. The Advanced Distributed Learning Initiative (ADL) is continuing to explore how those xAPI transactions can be trusted as evidence from an authoritative source.
W900KK-17-D-0004

The NATO Advanced Distributed Learning Handbook

2019; NATO Training Group
This handbook provides practical guidance to organizations for the creation and/or the procurement, implementation, and evaluation of Advanced Distributed Learning (ADL). This handbook is not meant to be all-encompassing. It is an overarching look at ADL concepts and some of the tools used to develop ADL. Nations are intended to use this handbook...
This handbook provides practical guidance to organizations for the creation and/or the procurement, implementation, and evaluation of Advanced Distributed Learning (ADL). This handbook is not meant to be all-encompassing. It is an overarching look at ADL concepts and some of the tools used to develop ADL. Nations are intended to use this handbook as a starting point and supplement it with additional guidance that is applicable to their own policies and procedures.

Talent Development Toolkit Requirements and Architecture Study

2019; Gordon, Jerry
This report defines detailed technical requirements, an initial architecture, a phased implementation strategy, and evaluation metrics for the Talent Development Toolkit (TDT), i.e., the operational learning ecosystem for the Intelligence Community (IC). The target audience for this report includes technical managers and technology...
This report defines detailed technical requirements, an initial architecture, a phased implementation strategy, and evaluation metrics for the Talent Development Toolkit (TDT), i.e., the operational learning ecosystem for the Intelligence Community (IC). The target audience for this report includes technical managers and technology subject-matter experts. Managers should focus on reading the introduction, summary findings, and recommendations. Technical performers can review the detailed findings, including the architectural diagrams, requirements, and recommendations to develop the detailed specifications they need for acquiring, modifying, installing, and configuring TDT components.

2018 Total Learning Architecture Final Report

2019; Smith, Brent; Gordon, Jerry
The Total Learning Architecture (TLA) program sponsored by the ADL Initiative seeks to develop a set of policy and standards defining the process for developing a learning ecology, where multiple services and learning opportunities (of various modalities and points of delivery)can be managed in an integrated environment.
The Total Learning Architecture (TLA) program sponsored by the ADL Initiative seeks to develop a set of policy and standards defining the process for developing a learning ecology, where multiple services and learning opportunities (of various modalities and points of delivery) can be managed in an integrated environment. The TLA serves to: 1) shift from a course-centric focus to a learner-centric focus, where competencies managed more efficiently (for savings in cost and schedule) and effectively for the learner’s professional development path; 2) provide the opportunities for data interoperability between implementations to aid in enterprise-wide decision support analysis and in reuse of competency and content between agencies, and; 3) capture all experiences, including on-the-job training (OJT), use of performance support and job aids, simulation, distributed learning, and traditional classroom learning to provide a more rounded portrait of personnel capability and aptitude to aid in the team building and detailing process.
TLA

Modernizing Learning: Building the Future Learning Ecosystem

2019; Schatz, S.; Vogel-Walcutt, J.
Modernizing Learning: Building the Future Learning Ecosystem is an implementation blueprint for connecting learning experiences across time and space. This co-created plan represents an advancement of how and where learning will occur in the future. Extensive learning and technological research has been conducted across the myriad disciplines and communities needed to develop this holistic maturation of the learning continuum...
The future learning ecosystem–a holistic, lifelong, personalized learning paradigm–represents a contrast to the Industrial Age model of time-focused, one-size-fits-all learning. It promises to substantively change the way we learn, moving away from old models of disconnected experiences to a connected continuum of lifelong learning, personalized, driven by data, and delivered across diverse locations, media, and periods of time. Modernizing Learning: Building the Future Learning Ecosystem examines the pivot our systems and society need to make to enable these lifelong, experiential, interconnected learning journeys, and it outlines a blueprint across technology, learning science, policy, and organizational factors for achieving this vision. This blueprint responds to external demands, such as the pace of global change and increased need for technical expertise, while simultaneously considering the new opportunities enabled by emerging technologies and our growing understanding of neuroscience and human cognition. The book is the outcome of a multi-year study recently completed by the ADL Initiative. More than 85 stakeholders from across communities contributed, bringing perspectives from K–12 educators to military trainers and from technology specialists to government leaders. This publication provides a snapshot of the progress made to date, incorporating real-world case studies and published research to provide a basis for the blueprint. Then it outlines what we need to build for tomorrow as well as what this near-future system will enable our children, workforce, society, and military personnel to achieve. Book chapters address topics such as lifelong learning, learning systems interoperability, analytics and visualizations, personalization, learning engineers, distributed learning instructional theory, self-regulated learning, competency-based learning, culture change, and strategic planning.

Online Access, Distribution and Further Collaborative Development of the NATO ADL Handbook

2018; Jefferson Institute
Several options were reviewed for the online access, interactive distribution and further collaborative development of the NATO ADL handbook. Mapping the immediate, adjacent and theoretical stakeholders, firm recommendations can be made for LRS options, server set up requirements, and a list of recommended verbs to be monitored in the collaborative environment.
Several options were reviewed for the online access, interactive distribution and further collaborative development of the NATO ADL handbook, using the following criteria: Level of interactivity, xAPI compatibility, Set up convenience, and User friendly. Mapping the immediate, adjacent and theoretical stakeholders, firm recommendations can be made for LRS options, server set up requirements, and a list of recommended verbs to be monitored in the collaborative environment.

ADL AS A TOOL OF SECURITY SECTOR REFORM: Observations and the Case of the Balkan RADLI project

2018; Presnal, Aaron, Ph.D
In a modern democratically controlled military, everyone must continually improve throughout her career, not only during periods of regular education. On the other hand, a small military needs all its personnel at work. Courses offered requiring months away from military posts are no longer acceptable. Downward budget pressure on partners also means...
In a modern democratically controlled military, everyone must continually improve throughout her career, not only during periods of regular education. On the other hand, a small military needs all its personnel at work. Courses offered requiring months away from military posts are no longer acceptable. Downward budget pressure on partners also means fewer teaching staff, requiring more efficient use of each faculty member and subject matter expert.

ADL in Exercises

2018; Jefferson Institute
The success of a training event depends, in part, upon its return on investment, that is, how well it prepares the participants, given the time and available resources. In many exercises, the planning team must allocate significant time to get the training audience up to speed – time that could otherwise be spent on activities that would enhance the...
The success of a training event depends, in part, upon its return on investment, that is, how well it prepares the participants, given the time and available resources. In many exercises, the planning team must allocate significant time to get the training audience up to speed – time that could otherwise be spent on activities that would enhance the learning experience and development of the exercise. This is particularly true in multinational events, given the diversity of participants, their wide ranges of background knowledge and prior experiences.

Advanced Distributed Learning Task 6 Acquisition for New Distributed Learning Report

2018; Goodson, Kathy, Ph.D.
This report documents our research on successful acquisition related processes associated to distributed learning modernization and acquisition templates/tools related to standards and enablers. It identifies and provides common language, practices, and procedures that enable successful acquisition of DL technology for the DoD. This...
This report documents our research on successful acquisition related processes associated to distributed learning modernization and acquisition templates/tools related to standards and enablers. It identifies and provides common language, practices, and procedures that enable successful acquisition of DL technology for the DoD. This research provided insight into the investigation, validation, and documentation of distributed learning acquisitions approaches, and provides resources on how to approach acquisition for the following areas:(1) Stakeholders desire to have more options for contract vehicles and alternate approaches to acquire distributed learning technologies and services. (2) Understanding, reconciliation, and acquisition of distributed learning capabilities as they apply to the developer, contracting personnel, and information technology (IT) managers.
W911QY-16-C-0008

Total Learning Architecture: Moving Into the Future

2018; IITSEC; Smith; Gallagher; Schatz; Vogel-Walcutt
Increasingly, the defense community requires a continuous, adaptive learning enterprise that delivers the right training, education, and just-in-time support, in the right ways and at the right time. The Total Learning Architecture (TLA), now in its second iteration of development, is intended to help meet that vision. The TLA is a set of internet and...
Increasingly, the defense community requires a continuous, adaptive learning enterprise that delivers the right training, education, and just-in-time support, in the right ways and at the right time. The Total Learning Architecture (TLA), now in its second iteration of development, is intended to help meet that vision. The TLA is a set of internet and software specifications being developed to create the interoperability backbone of this future learning ecosystem. The products derived from this project include technical implementation guidelines, Application Programming Interfaces (APIs), middleware, and data model descriptions that help training, education, and personnel management technologies seamlessly communicate through integrated coherent systems.
TLA

Leveraging Science and Technology to Launch Innovation in Learning

2018; IITSEC; Vierling, Kendy; et. al.
In remarks about the National Defense Strategy, the U.S. Secretary of Defense, Jim Mattis, observed that in a time of rapid technological change and an over-stretched military, "Success does not go to the country that develops a new technology first, but rather, to the one that better integrates it and more swiftly adapts its way of fighting" (2018,...
In remarks about the National Defense Strategy, the U.S. Secretary of Defense, Jim Mattis, observed that in a time of rapid technological change and an over-stretched military, "Success does not go to the country that develops a new technology first, but rather, to the one that better integrates it and more swiftly adapts its way of fighting" (2018, emphasis is ours). Services within the Department of Defense (DoD) and agencies across the national security community are transitioning from an antiquated learning model to a more agile one that better leverages innovation and emerging learning science and technology (S&T). Service members and the civilian workforce seek to integrate these new learning capabilities and expect to obtain more personalized and accelerated learning as a result.

Team Orlando: Community of Progress

2018; IITSEC; Vogel-Walcutt, J.; et. al.
Team Orlando has become an international icon for the modeling, simulation, and training community. Countless examples of cooperation demonstrate the exceptional advantage of having co-located representatives from every military branch, academia, and industry. The benefit of this community exceeds the sum contributions of the players but...
Team Orlando has become an international icon for the modeling, simulation, and training community. Countless examples of cooperation demonstrate the exceptional advantage of having co-located representatives from every military branch, academia, and industry. The benefit of this community exceeds the sum contributions of the players but measuring its impact has not been straightforward. Thus to date, a clear picture of the return on investment (ROI) with associated data has not yet been accomplished. Nonetheless, the stakeholders of this community require tangible measures of impact. While those of us intimately involved with Team Orlando can easily appreciate the impact of this synergy, making it transparent and understandable to others can be a challenge.

The Truth About Blockchains and How They Apply to Training

2018; IITSEC; Robson, R.; Hernandez, M.
This presentation covers how blockchains work, real world implementations, blockchains and training, and resources and Q&A's.
This presentation covers how blockchains work, real world implementations, blockchains and training, and resources and Q&A's.

Learning Analytics with xAPI in a Multinational Military Exercise

2018; IITSEC; Presnall, A.; Radivojevic, V.
As the truism goes, "You can't manage what you don't measure." However, assessing performance in training exercises has classically presented a measurement challenge, made more complex by the paucity of timely, relevant, comparable data on the training audience's performance. Even as the field of learning analytics becomes increasingly...
As the truism goes, "You can't manage what you don't measure." However, assessing performance in training exercises has classically presented a measurement challenge, made more complex by the paucity of timely, relevant, comparable data on the training audience's performance. Even as the field of learning analytics becomes increasingly sophisticated, military training exercises continue to be assessed in largely subjective and superficial ways. In short, while we may know if the training was completed, it is difficult to objectively answer the basic question: did the exercise do any good? xAPI is an emerging capability to support learning analytics, but until recently has remained largely untested as a solution for delivering comparable results across complex multiplatform asynchronous learning and performance data feeds at scale. Viking 18, a large multinational civilmilitary exercise, aspires toward full operational integration of Advanced Distributed Learning (ADL) as an integral part of the exercise experience, including the associated learning analytics supported by xAPI. This paper presents a case study and lessons learned from the implementation of xAPI in the Viking 18 exercise. It also delivers a summary of the resulting Viking 18 learning analytics, including data frome-learning courses matched against quantitative observation data from the exercise management tool, with the aim of gaining insight on the relationships between training and performance against exercise objectives. As such, we crack open the door to aggregation of exercise performance data in support of operational and strategic planning. Analysis clearly suggests a pattern of enhanced training outcomes by units with higher rates of Introduction to Viking(pre-training e-learning) course completion.
W900KK-17-D-0004
MADLx

Integrating Advanced Distributed Learning into Multinational Exercises

2018; IITSEC; Ljung, N.; Ax, T.; Presnall, A.; Schatz, S.
Armed forces around the world face similar challenges regarding exercises and live training. The context is increasingly multinational, and operations and supporting systems are more complex, all requiring more extensive training. To thrive in these volatile, complex and evolving security environments, military personnel also require an...
Armed forces around the world face similar challenges regarding exercises and live training. The context is increasingly multinational, and operations and supporting systems are more complex, all requiring more extensive training. To thrive in these volatile, complex and evolving security environments, military personnel also require an expanding range of competencies and at higher levels of proficiency, and the quick acquisition of new knowledge and skills to confront novel multi-domain challenges. In resource constrained systems, this outcome must be achieved without significantly increasing training and education time or costs. Operational integration of Advanced Distributed Learning (ADL) into multinational exercises is one step toward this goal. As computer-aided exercises increase in number and scope and educational systems go online, the ability to produce ADL training and education becomes a critical capability. ADL has been a supportive learning concept in the Viking series of computer-aided exercises since 2003. The ADL-concept for Viking 2018 expanded the scope and scale of learning resources to include assets of both pre-training and operational value, and highlighted analytics on a common dashboard with data from both the Learning Management System and the Command and Control system. Thus, it was possible to compare pre-training data to data from the execution phase of the exercise, including data from the designated evaluation team and exit interviews with participants. Based on previous lessons identified and data from Viking 18, this paper shows that ADL, blended into a computer-aided exercise, produces more effective and efficient outcomes. To bolster the effort of integrating ADL into future exercises, the paper also describes an outline for a multi-year design-based strategic plan to achieve operational integration of ADL across a suite of joint and national exercises to support and enable readiness.
MADLx

At the Tipping Point: Learning Science and Technology as Key Strategic Enablers for the Future of Defense and Security

2018; I/ITSEC; Raybourn, E.M.; Schatz, S.; Vogel-Walcutt, J.; Vierling, K.
According to former U.S. Secretary of Defense, Ash Carter, today's national security environment is "dramatically different - and more diverse and complex in the scope of its challenges - than the one we've been engaged with for the last 25 years, and it requires new ways of thinking and new ways of acting" (2016, emphasis is ours). These new ways cannot be...
According to former U.S. Secretary of Defense, Ash Carter, today's national security environment is "dramatically different - and more diverse and complex in the scope of its challenges - than the one we've been engaged with for the last 25 years, and it requires new ways of thinking and new ways of acting" (2016, emphasis is ours). These new ways cannot be achieved without significant changes to lifelong (or at least career-long) personnel development. This paper focuses on one aspect of that (r)evolution, i.e., specifically examining the challenges, goals, projects, and recommended actions related to the transformation of training and education in the defense and security sectors.

Advanced Distributed Learning Task 5(b) Cybersecurity Barriers to Distributed Learning Modernization Report

2018; Goodson, Kathy, Ph.D.
The Advanced Distributed Advanced Distributed Learning (ADL) Initiative asked the Potomac Institute for Policy Studies (PIPS) to conduct a study to identify cybersecurity distributed learning (DL) modernization barriers to understand the potential implications of promulgating the DoDI 1322.cm on costs, implementation, consequences, and...
The Advanced Distributed Advanced Distributed Learning (ADL) Initiative asked the Potomac Institute for Policy Studies (PIPS) to conduct a study to identify cybersecurity distributed learning (DL) modernization barriers to understand the potential implications of promulgating the DoDI 1322.cm on costs, implementation, consequences, and gaps, and to develop implementation recommendations. The Potomac Institute conducted research to identify, determine the implications of, and recommend solution approaches associated with related policies that may affect implementation of DL in accordance with the updated DoDI 1322 policies. As part of this task, the Potomac Institute interviewed government stakeholders to identify DL roadblocks created by cybersecurity policies and processes within different Services (Army, Navy, Marine Corps, Air Force) and Organizations (Joint organizations/Joint Knowledge Online) are facing, and their approaches to addressing them.
W911QY-16-C-0008

Financial Readiness for Learning (FR-PAL) Final Report

2018; Murphy, Jennifer
Sen$e is a mobile application designed to support service members and their families with financial readiness information using micro-learning, gamification, and other motivational techniques to cultivate life-long learning. Sen$e supports service member financial literacy and preparedness through engaging "bite-sized," personally...
Sen$e is a mobile application designed to support service members and their families with financial readiness information using micro-learning, gamification, and other motivational techniques to cultivate life-long learning. Sen$e supports service member financial literacy and preparedness through engaging "bite-sized," personally relevant, multimedia content. To encourage continued use, Sen$e features motivational elements such as push notifications, gamification, and visual progress indicators. In addition to educational content, validated financial tools and resources are incorporated into this user-centered application. Sen$e was developed using human-centered design principles and Agile software development processes. This iterative approach allowed for significant front-end analyses, engagement with end-users, derivation of user needs into requirements, development, and testing throughout each phase of the project to ensure a holistic and user-friendly system. Project performance progressed through a three-phased development process. Deliveries made throughout each phase of the project underwent an extensive and thorough review process. The process of meeting the project goals for developing a useful, usable, and user-centered financial application, for service members and their families, is described, in detail, in the following final report.
W911QY-16-C-0163
Sen$e

Privacy Support for the Total Learning Architecture: Summit Report

2018; Knijnenburg, B.P.; et al
How can we reconcile the need for extensive customizability with users' apparent lack of skills and motivation to manage their own privacy settings? In this report we investigate User-Tailored Privacy as means to support users' privacy decision-making. With User-Tailored Privacy (UTP), a system would first measure users' privacy-related...
How can we reconcile the need for extensive customizability with users' apparent lack of skills and motivation to manage their own privacy settings? In this report we investigate User-Tailored Privacy as means to support users' privacy decision-making. With User-Tailored Privacy (UTP), a system would first measure users' privacy-related characteristics and behaviors, use this as input to model their privacy preferences, and then adapt the system's privacy settings to these preferences (Figure 1).This adaptation could take the form of a default setting or a recommendation, either with or without an accompanying justification.
PS4TLA

US DoD xAPI Profile Server Recommendations

2018; Bowe, M.; Silvers, A.E.
In order for a federated network of xAPI Profile servers to serve their purpose, there needs to be a way for an organization or group to create an xAPI Profile and share the information they authored without risk of losing data or outside groups changing the given xAPI Profile. This requires a governance model where a central authority informs profile...
In order for a federated network of xAPI Profile servers to serve their purpose, there needs to be a way for an organization or group to create an xAPI Profile and share the information they authored without risk of losing data or outside groups changing the given xAPI Profile. This requires a governance model where a central authority informs profile servers of who should have access to edit which profiles. Such an authoritative body also needs to identify which profiles are the most appropriate in a given context, verifying the quality and substance of an xAPI Profile. This is required for tools reading profiles to know which profiles are most high priority to ingest and use.
#W911QY-16-C-0109
DISC

Financial Readiness for Learning (FR-PAL) Summative Usability Test Results Report

2018; Murphy, Jennifer
The Financial Readiness-Personal Assistant for Learning (FR-PAL), Sen$e, is a native mobile application designed to support service members and their families with financial readiness information using micro-learning, gamification, and other motivational techniques to cultivate life-long learning. The usability and effectiveness of Sen$e...
The Financial Readiness-Personal Assistant for Learning (FR-PAL), Sen$e, is a native mobile application designed to support service members and their families with financial readiness information using micro-learning, gamification, and other motivational techniques to cultivate life-long learning. The usability and effectiveness of Sen$e was evaluated using active military service members. Eight current service members completed usability testing to determine the extent to which user expectations of the application structure, content, interfaces, and functionality were met. This usability evaluation identified benefits of the interface and overall application as well as potential design concerns and areas for enhancement. Benefits included the clean, intuitive, and simple design and display of information. There were some concerns with respect to navigation and content organization within the system. Twenty-seven service members completed the effectiveness evaluation where knowledge gained from the application was assessed. Participant feedback was collected through semi-structured interviews. Results from this summative evaluation are presented for usability, user experience, user satisfaction, and application effectiveness. Recommendations for future design and development of the Sen$e application are provided with the intent of improving the ser experience and usability of the application.
W911QY-16-C-0163
Sen$e

PS4TLA: Privacy Support for the Total Learning Architecture: Modeling Factors

2018; Knijnenburg, B.P.; et al
The purpose of this document is to make recommendations for implementing User-Tailored Privacy (UTP) into Total Learning Architecture (TLA)-based systems and to inform ADL and other TLA performers about the Modeling Factors that need to be considered in the context of this implementation. The set of recommendations put forth in this document will...
The purpose of this document is to make recommendations for implementing User-Tailored Privacy (UTP) into Total Learning Architecture (TLA)-based systems and to inform ADL and other TLA performers about the Modeling Factors that need to be considered in the context of this implementation. The set of recommendations put forth in this document will allow ADL and other TLA performers to build a user-tailored privacy decision-support system that supports users in making better privacy decisions.
PS4TLA

Recommendation across Many Learning Systems to Optimize Teaching and Training

2018; AHFE, Applied Human Factors and Ergonomics; Neville, K.J.; Folsom-Kovarik, J.T.
To help learners navigate the multitude of learning resources soon to become available in the Total Learning Architecture (TLA) ecosystem, a Recommender algorithm will give learners learning resource recommendations. Recommendations will support immediate training needs and provide guidance throughout one's career. This paper describes...
To help learners navigate the multitude of learning resources soon to become available in the Total Learning Architecture (TLA) ecosystem, a Recommender algorithm will give learners learning resource recommendations. Recommendations will support immediate training needs and provide guidance throughout one's career. This paper describes initial work to define the logic that will be used by the Recommender. It describes our use of (1) expertise acquisition theory and (2) research on the learning effects of learner state and characteristics. The descriptions are accompanied by examples of relevant research and theory, the learner-support guidelines they suggest, and ways to translate the guidelines into Recommender logic. The TLA, together with the Recommender, have significant potential to aid professionals across a range of complex work domains, such as cy ber operations, with their career development and growth and the acceleration of their expertise attainment.
W911QY-16-C-0019
TLA

Learning Record Provider Professional Certification Recommendations

2018; Bowe, M.; Silvers, A.E.
Professionals who work with the Experience API (xAPI) need a formal body of knowledge to generate high quality data with xAPI. Some of this knowledge can be sourced directly from xAPI's specifications and the multitude articles, documents, reports and books devoted to applied research and development with xAPI. Practice leaders, even in these early...
Professionals who work with the Experience API (xAPI) need a formal body of knowledge to generate high quality data with xAPI. Some of this knowledge can be sourced directly from xAPI's specifications and the multitude articles, documents, reports and books devoted to applied research and development with xAPI. Practice leaders, even in these early days for xAPI, can identify hard-won lessons from working with xAPI despite a gap in tools and maturity in commercial, off-the-shelf implementations of xAPI within authoring tools and other learning technologies.
#W911QY-16-C-0109
DISC

Mobile Learning Decision Path

2018; Adayana Government Group
Government agencies interested in implementing mobile learning (mLearning) solutions are doing so through pilots and trial and error. There are many important decisions to make prior to and during the design, development, and implementation of an mLearning solution. Best practices for reusing and redesigning existing learning content for the mobile platform do not currently exist. The many variables to consider would be more readily addressed by an mLearning decision support resource. This project was an effort completed for ADL (Advanced Distributed Learning) in coordination with the CTTSO (Combating Terrorism Technical Support Office)
Government agencies interested in implementing mobile learning (mLearning) solutions are doing so through pilots and trial and error. There are many important decisions to make prior to and during the design, development, and implementation of an mLearning solution. Best practices for reusing and redesigning existing learning content for the mobile platform do not currently exist. The many variables to consider would be more readily addressed by an mLearning decision support resource. This project was an effort completed for ADL (Advanced Distributed Learning) in coordination with the CTTSO (Combating Terrorism Technical Support Office)
MLDP

Sero!: A Learning Assessment Platform for Adult Learning Environments

2018; Moon, Brian; Rizvi, Sana
Training and development programs within corporate environments have seen a significant expansion in recent years. Yet there is little research regarding workplace learning assessments. Learning assessments could be used as a tool to gauge employee knowledge in a specific domain area, which can be useful in identifying areas of strengths and weaknesses and indicate where improvement is needed. One way to effectively and efficiently assess workplace learning could be with Concept Maps, a diagramming technique of knowledge representation that has been used to capture the mental models of domain experts. Concept Maps can in turn be used to assess the progress of learners through training and learning curricula. Although the actual set-up and analysis for Concept Maps tend to be rather time consuming and laborious, Sero!, a developing learning assessment platform, holds promise for realizing efficiencies in their use to enable deeper learning assessment. This paper explores the literature about workplace learning assessment, reviews the applicability of Concept Maps for such assessment, summarizes the user-centered design and development of Sero!, and reflects on the boundary conditions for its use.
Training and development programs within corporate environments have seen a significant expansion in recent years. Yet there is little research regarding workplace learning assessments. Learning assessments could be used as a tool to gauge employee knowledge in a specific domain area, which can be useful in identifying areas of strengths and weaknesses and indicate where improvement is needed. One way to effectively and efficiently assess workplace learning could be with Concept Maps, a diagramming technique of knowledge representation that has been used to capture the mental models of domain experts. Concept Maps can in turn be used to assess the progress of learners through training and learning curricula. Although the actual set-up and analysis for Concept Maps tend to be rather time consuming and laborious, Sero!, a developing learning assessment platform, holds promise for realizing efficiencies in their use to enable deeper learning assessment. This paper explores the literature about workplace learning assessment, reviews the applicability of Concept Maps for such assessment, summarizes the user-centered design and development of Sero!, and reflects on the boundary conditions for its use.
CMAPS

Using Competencies to Map Performance across Multiple Activities

2017; IITSEC; Robson, R.; Poltrack, J.
When a single training system accumulates data on learner performance, the data are stored in a way determined by the system's designers. This enables the system to access these data and to apply them to its interactions with learners.In environments such as live-virtual-constructive federations, each component may store performance data in its own...
When a single training system accumulates data on learner performance, the data are stored in a way determined by the system's designers. This enables the system to access these data and to apply them to its interactions with learners.In environments such as live-virtual-constructive federations, each component may store performance data in its own way, making it difficult for one component to access and use data produced by another. To enable cross-component sharing of performance data,it is necessary to establish shared definitions of skills and outcomes; create a common language for expressing performance data;interpret data produced at wildly differing levels of granularity; and (in some cases) satisfy a large array of security and privacy requirements. This paper is based on work done by the US Advanced Distributed Learning (ADL) Initiative, the Credential Engine foundation, and several standards bodies. It starts by discussing the above challenges and their manifestations in use cases ranging from federations of sophisticated adaptive training and simulation systems to more traditional online learning environments. The paper then describes a potential solution for collecting and processing assertions of competency, skills, and performance from multiple sources. Each assertion is of the form "Learner X has (or has not) achieved competency Y at level Z with confidence p based on evidence E." "Competencies" are drawn from shared, machine-readable frameworks that can represent knowledge, skills, ability, and objectives. Assertions can be collected directly or generated by ingesting granular performance data and correlating it to competencies, enabling algorithms that use explicit rules and relationships to draw further inferences. We have recently tested one such system with Army special operators (n = 79) at the JFK Special Warfare Center and School as part of the ADL's Total Learning Architecture demonstration. The paper discusses our approach to establishing trusted networks and complying with privacy and security requirements; how we used the Experience API (xAPI) and industry standards; and lessons learned from that event.
CaSS

Teaching and Learning Differently: Personalized E-Books for Learning (PEBL)

2017; IITSEC; Robson, E.; Berking, P.
Although current eBooks are little more than print shown in an e-reader, as a mobile, connected platform, eBooks have the unrealized potential to support more pedagogical approaches than traditional books, including experiential, problem-based, dynamic, and social learning. To realize this potential, the Personalized eBooks for Learning (PEBL)...
Although current eBooks are little more than print shown in an e-reader, as a mobile, connected platform, eBooks have the unrealized potential to support more pedagogical approaches than traditional books, including experiential, problem-based, dynamic, and social learning. To realize this potential, the Personalized eBooks for Learning (PEBL) project has developed a specification that enables new capabilities in eBooks while maintaining the advantages of the "book" format. The PEBL project, funded by the US Advanced Distributed Learning (ADL) Initiative, has extended the standard EPUB3 format to enable eBooks to communicate with other systems in live, virtual, simulated, and constructive environments; to embed and exchange data with simulations, games, and intelligent tutoring systems; and to serve as competency-based training environments. This paper describes new pedagogical approaches that are enabled by PEBL that go beyond the affordances of current eBooks, and outlines the modular and expandable structure of the PEBL specification. New capabilities include: book content triggered by the learner skills and competency levels; instructor-driven content brokering; dynamic individual and team progress dashboards and native analytics; and exercises in which learners communicate and collaborate with each other. These can each be separately provided by different learning technologies, but not all together in the convenient, media-rich, portable eBooks. To demonstrate these capabilities, this paper reports on a next-generation cyber security eBook that contains examples of many of the new features and functionality available through PEBL-compliant eBooks and eReaders. We discuss both the pedagogic approaches and technical implementations used in the Cybersecurity demonstration PEBL eBook, and how this eBook has integrated with other technologies using the ADL's Total Learning Architecture (TLA). Finally, this paper outlines the possibilities for using PEBL-enabled eBooks in conjunction with virtual and mixed reality training, and we outline other implementations and specifications related to PEBL that have resulted from the Institute of Electrical and Electronics Engineers (IEEE) Actionable Data Book Industry Connections activity.
PEBL

Performance Measurement in LVC Distributed Simulations: Lessons from OBW

2017; IITSEC; Hoke, J.; Townsend, L.; Giambarberee, S.; Schatz, S.
Operation Blended Warrior (OBW) 2016 marked the second year of a three-year effort to document lessons learned and understand barriers to implementing Live, Virtual, Constructive (LVC) distributed training. In the first year of the event, LVC focus areas included connectivity, interoperability, data standards, after-action review, and cyber...
Operation Blended Warrior (OBW) 2016 marked the second year of a three-year effort to document lessons learned and understand barriers to implementing Live, Virtual, Constructive (LVC) distributed training. In the first year of the event, LVC focus areas included connectivity, interoperability, data standards, after-action review, and cyber security. Year two introduced additional focus areas: multi-level security, cross domain solutions, long-haul feeds, and performance measurement. This paper focuses on this latter area - defining and collecting performance measures. Performance measurement in simulation-based training faces formidable obstacles, including the identification of individual and collective performance dimensions, how these dimensions relate to training goals, and how training transfers to operational readiness. Blending of LVC elements introduces additional complexity, not only for human performance assessment but also for evaluating the effectiveness and efficiency of the technical system. In this paper, we present the measures defined and collected during OBW in four primary areas: 1) cost analysis, 2) network performance, 3) trainee performance, and 4) whether OBW met the expectations of participating organizations. We also discuss three categories of Measures of Effectiveness (MoEs) and Measures of Performance (MoPs) established by the OBW Strategic Integrated Product Team: Programmatic, Technological, and Learning.These MoEs and MoPs will facilitate annual comparisons of performance measurement at OBW and encourage use of the event as a sandbox to design and validate LVC performance measurement tools. Finally, we present the goals and measures established for Performance Measurement during OBW 2017 and recommendations for future events
OBW

Developing a Financial Readiness Mobile Personal Assistant for Learning

2017; IITSEC; Hannigan, F.; Murphy, J.; Daly, T.; Udell, C.; Pfieffer, D.; Schatz, S.; Birtwhistle, M.
While Service members and their families receive unique financial benefits as a result of their service, few are well educated in the best ways to manage their finances. Consequently, they may fall prey to predatory lending practices, high pressure sales pitches, and other financial pitfalls. Adding to the complexity of this problem is the fact that...
While Service members and their families receive unique financial benefits as a result of their service, few are well educated in the best ways to manage their finances. Consequently, they may fall prey to predatory lending practices, high pressure sales pitches, and other financial pitfalls. Adding to the complexity of this problem is the fact that Service members' financial priorities change throughout their lives, both during service and after separation. 10 U.S. Code § 992 requires financial literacy training to be provided at specific points in Service members' careers, including significant life events, at promotion, and in concert with deployments. However, there is no guidance for what specific training content should be provided or when. Further, there is no solution to provide Service members complete access to financial literacy information. Due to the breadth of the financial domain, an effective solution must tailor content to the individual Service member, providing the right information in a timely way, and in an engaging and effective way. Although adaptive training is not a new concept, it is rarely designed to provide lifelong support to learners. Even less frequently are these solutions designed for mobile devices. In this paper, we describe research that designs and develops a mobile Personal Assistant for Learning to support Service member financial literacy. First, we describe the front-end analysis conducted to determine the scope of the domain and identification of learning objectives. Second, we discuss the process to develop a framework mapping the content to specific points in a Service member's career to enable an adaptive learning experience. We also present research findings to support our approach. Our goal with this paper is to share the process used to design adaptive content to support lifelong learning and discuss lessons learned through our research.
Sen$e

Total Learning Architecture Development: A Design-Based Research Approach

2017; IITSEC; Gallagher, P.S.; Folsom-Kovarik, J.T.; Schatz, S.; Barr, A.; Turkaly, S.
Organizations that use learning technology to educate and train are facing a new set of interoperability problems. Many new products - including adaptive systems, intelligent digital tutors, real-time data analytics, and interactive e-books - offer dramatic learning benefits. However, these products primarily "stand alone" and work outside of...
Organizations that use learning technology to educate and train are facing a new set of interoperability problems. Many new products - including adaptive systems, intelligent digital tutors, real-time data analytics, and interactive e-books - offer dramatic learning benefits. However, these products primarily "stand alone" and work outside of typical browser-based delivery environments controlled by traditional learning management systems. Furthermore, the effectiveness of these "intelligent systems" often depends on their access to data generated by and stored in other systems. The U.S. DoD Advanced Distributed Learning (ADL) Initiative is designing a framework of specifications, called the Total Learning Architecture (TLA), to ultimately enable "plug-and-play" interoperability of learning technologies. That is, the TLA will allow these new products to interoperate with each other, with other existing learning systems, and even human capital management technologies. Because of the rapid rate of innovation in such distributed technologies, we adopted a multiyear design-based research approach. During the project's first year, an initial set of specifications have been developed and evaluated for technical and functional adequacy using a multi-round Delphi approach with a panel of international participants (n = 54). Also, in partnership with the U.S. Army JFK Special Warfare Center and School's Special Warfare Education Group, we conducted a live prototype test and demonstration with Special Operations Soldiers (n = 73). This yielded data on the nascent system's functionality, performance, user experience, and learning potential. Analysis of these data will lead to recommendations, which in turn will inform the second cycle of TLA development process.This paper summarizes the TLA concept, development process, first-year analysis efforts and outcomes, and lessons learned leading to design improvements for the second year of TLA development.
BE-02-4039, W911QY-16-C-0019-P00005
TLA

A Mobile Strategy for Self-Directed Learning in the Workplace

2017; IITSEC; Freed, M.; Yarnall, L.; Spaulding, A.; Gervasio, M.
Traditional approaches to workplace training often treat learners as equally prepared, drive them through too much content in too short a time, and conclude before ensuring retention. These departures from ideal instructional practice have a common cause –the need to fit learning activities into constrained episodes such as classroom...
Traditional approaches to workplace training often treat learners as equally prepared, drive them through too much content in too short a time, and conclude before ensuring retention. These departures from ideal instructional practice have a common cause –the need to fit learning activities into constrained episodes such as classroom presentations and e-learning courses. Fortunately, advances in mobile technology, learning science, and artificial intelligence are making it possible to deliver learning experiences in less constrained conditions, with reduced risk of overload, and better alignment with an individual's mental and situational readiness to learn. We developed a mobile strategy that leverages these advances to support adult learning, and implemented this strategy in PERLS, a mobile application that recommends bite-sized learning materials - or microcontent - through a deck of electronic cards. An intelligent algorithm tracks progress and recommends content based on principles of self-regulated learning, goal-setting, and adult learning motivation. Essentially, PERLS aims to engage users in becoming better self-regulated learners on the job.In this paper, we describe the PERLS mobile learning strategy and results of a study of its use in support of training ofDefense Support for Civil Authorities (DSCA). By drawing from observations, online usage data, learning outcome measures, and surveys of learner characteristics and attitudes, this paper provides evidence of the feasibility of using this approach to enhance self-directed learning activity among military personnel.
W911QY-12-C-0171
PERLS

Approximate Graph Matching for Mistake-tolerant Skill Assessment

2017; ACS, Advances in Cognitive Systems; Gerasio, M.; Jones, C.; Myers, K.
This paper presents an approach to automated assessment for online training based on approximate graph matching. The algorithm lies at the core of two prototype training systems that we have built in accord with U.S. Army training materials: one for the use of a collaborative visualization and planning tool, the other for rifle maintenance. The...
This paper presents an approach to automated assessment for online training based on approximate graph matching. The algorithm lies at the core of two prototype training systems that we have built in accord with U.S. Army training materials: one for the use of a collaborative visualization and planning tool, the other for rifle maintenance. The algorithm uses approximate graph-matching techniques to align a representation of a student response for a training exercise with a predefined solution model for the exercise. The approximate matching enables tolerance to learner mistakes, with deviations in the alignment providing the basis for feedback that is presented to the student. Given that graph matching is NP-complete, the algorithm uses a heuristic approach to balance computational performance with alignment quality. A comprehensive experimental evaluation shows that our technique scales well while retaining the ability to identify correct alignments for responses containing realistic types and numbers of learner mistakes.
W911QY-14-C-0023

Maturation, Evaluation, Alignment and Testing of Sero!

2017; ADL Publication; Moon, Brian
This Technical Report Study/Services Ad Hoc Reports and Works is intended to deliver 3.2.4 a Base POP Final Report by summarizing the work performed during the Base POP, assess progress, identify solved and unsolved technical problems, and set forth specific work details for the Optional POP. An itemized assessment of progress is included in Appendix...
This Technical Report Study/Services; Ad Hoc Reports and Works is intended to deliver 3.2.4 a Base POP Final Report by summarizing the work performed during the Base POP, assess progress, identify solved and unsolved technical problems, and set forth specific work details for the Optional POP. An itemized assessment of progress is included in Appendix A. Such sections follow the format specified in DI-MGMT-80227. This report assumes familiarity with the acronyms and terminology in use under the Total Learning Architecture (TLA) program and by ADL.
W911QY-16-C-0081
CMAPS

Exploring Assessment Mechanisms in the Total Learning Architecture (TLA)

2017; Chapter in Book - GIFT; Goodwin, G; Folsom-Kovarik, J.T.; Johsnon, A.; Schatz, S.; Sottilare, R.
The focus of this chapter is on the challenges and potential solutions to conducting realtime and long-term assessments of performance, learning, and domain competency in the Total Learning Architecture (TLA). TLA, a distributed learning ecosystem, is being developed by the US Office of the Secretary of Defense to support capabilities for...
The focus of this chapter is on the challenges and potential solutions to conducting realtime and long-term assessments of performance, learning, and domain competency in the Total Learning Architecture (TLA). TLA, a distributed learning ecosystem, is being developed by the US Office of the Secretary of Defense to support capabilities for instruction anytime and anywhere. TLA is an evolving set of standardized specifications that enable responsible sharing of essential learning data between applications using common interfaces and data models. The applications that could be part of the TLA ecosystem range from simple desktop applications to immersive simulations to mobile apps, and would serve as either service providers or consumers. Expected services include applications like intelligent tutoring systems (ITSs; e.g., Auto Tutor, Cognitive Tutor, or Generalized Intelligent Framework for Tutoring [GIFT]-based tutor), which provide information to other services and consume information from other services. The TLA is expected to provide services including experience tracking, competency assessment, learner modeling, and content brokering. All of these fundamentally involve learner assessments. Experience tracking (via the experience application programming interface [xAPI]) provides a standard for encoding and storing data about learners'interactions with learning experiences and applications, providing fine grained evidence that can make assessment precise and timely. TLA will also establish a common way for systems to reference and represent competencies and competency relationships, supporting assessment sharing. Learner models will contain data about assessed mastery of competencies as well as traits, preferences, individual differences, and demographic data. Learner models that are broadly accessible to learning applications will support uptodate and accurate competency assessment. Content brokering (i.e., recommending future experiences and training) also depends on learner assessments. Content brokering will support just in time learning and sequencing of learning events. Competency models will enable content to be tailored to the individual learner's needs. Training applications like GIFT operating in the TLA environment will both consume learner data available through TLA and provide learner data as they complete training. It will be challenging to insure that all training applications will be able to both obtain necessary learner data from TLA as well as insure that they all output learner measures that can be used by other applications within TLA. This chapter explores some of the challenges of integrating a training application like GIFT into the TLA. This includes discussion of the discovery and development of methods to assess competency based on xAPI statements and recommendations for augmenting xAPI statements to facilitate interoperability among training applications and the TLA through methods such as semantic analysis.
TLA

Humans as the Strong Link in Securing the Total Learning Architecture

2017; Applied Human Factors and Ergonomics, AHFE; Maymí, F.; Woods, A.; Folsom-Kovarik, J.
This paper describes a proposed approach, centered on human factors, for securing the Total Learning Architecture (TLA). The TLA, which is being developed for the United States Department of Defense, will rely on large stores of personal data that could be targeted by sophisticated adversaries. We describe the TLA and its envisioned users at a fairly...
This paper describes a proposed approach, centered on human factors, for securing the Total Learning Architecture (TLA). The TLA, which is being developed for the United States Department of Defense, will rely on large stores of personal data that could be targeted by sophisticated adversaries. We describe the TLA and its envisioned users at a fairly high level before describing expected classes of attacks against it. We then examine existing and proposed controls that, if properly managed, should allow users and service providers to significantly reduce the risks to the system.
W911QY-16-C-0019
TLA

Intelligent Instructional Hand Offs

2017; EDM; Fanscal, S.E.; Yudelson, M.V.; Berman, S.R.
Learners in various contemporary settings (e.g., K-12 classrooms, online courses, professional/vocational training) find themselves in situations in which they have access to multiple technology-based learning platforms and often one or more non-technological resources (e.g., human instructors or on-demand human tutors). Instructors,...
Learners in various contemporary settings (e.g., K-12 classrooms, online courses, professional/vocational training) find themselves in situations in which they have access to multiple technology-based learning platforms and often one or more non-technological resources (e.g., human instructors or on-demand human tutors). Instructors, similarly, find themselves in situations in which they can provide learners with a variety of options for instruction, practice, homework, and other activities. We seek data-driven guidance to help facilitate intelligent instructional "hand offs" between learning resources. To begin this work,we focus on an important element of self-regulated learning, namely help seeking.We build classifier models based on proxies for learner prior knowledge and data-driven inferences about learners'disengaged behavior(e.g., gaming the system)and affective states(e.g., confusion)to determine the extent to which (and when) learners tended to seek out help via human tutoring while using an intelligent tutoring system for mathematics. Insights into cognitive, behavioral, and affective factors associated with help seeking outside of a system will drive future work into providing automated, intelligent guidance to both learners and instructors.We close with discussion of the limitations of the present analysis and avenues for future work on intelligently guiding instructional hand offs
W911QY-14-C-0019
TLA - GIFT

Demonstration-based Solution Authoring for Skill Assessment

2017; ACS, Advances in Cognitive Systems; Gervasio, M.; Wessel, M.; Myers, K.
The high cost of developing content has been a major impediment to the widespread deployment of intelligent training systems. To enable automated skill assessment, traditional approaches have required significant time investment by highly trained individuals to encode first-principles domain models for the training task. In contrast,...
The high cost of developing content has been a major impediment to the widespread deployment of intelligent training systems. To enable automated skill assessment, traditional approaches have required significant time investment by highly trained individuals to encode first-principles domain models for the training task. In contrast, approaches grounded in example-based methods have been shown to significantly reduce authoring time. This paper reports on an approach to creating solution models for automated skill assessment using an example-based methodology, specifically targeting domains for which solution models must support robustness to learner mistakes. With this approach, a content author creates a baseline solution model by demonstrating a solution instance and then specifies a set of annotations to generalize from that instance to a comprehensive solution model. Results from a user study show that domain experts are comfortable with the approach and capable of applying it to generate quality solution models.
W911QY-14-C-0023
ITS

More Than the Sum of Their Parts: Case Study and General Approach for Integrating Learning Applications

2017; MODSIM; Freed, M.; Folsom-Kovarik, J.T.; Schatz, S.
Learning system interoperability standards let applications share or connect infrastructure, reducing development costs, administrative burden, and user friction. We propose a more comprehensive form of interoperability that supports shared userdata, user interfaces, and session management services. This will let applications leverage...
Learning system interoperability standards let applications share or connect infrastructure, reducing development costs, administrative burden, and user friction. We propose a more comprehensive form of interoperability that supports shared userdata, user interfaces, and session management services. This will let applications leverage differences in how and when they are used to achieve better learning outcomes than is possible with only native capabilities. To demonstrate this concept, we integrated two complementary learning applications. PERLS is a Personal Assistant for Learning phone app that supports adult self-regulated learning. It recommends learning activities across a range of applications and devices, and provides a coherent user experience for learners as they progress through parallel trajectories varying in topic, timespan,intensity, and (in)formality. PALMs is a web service that uses flashcard-like interactions to train visual knowledge. It accelerates learning by adaptively optimizing the content sequence based on a model of learning rate and retention. The applications were integrated to let PERLS recommend and launch PALMs content, then seamlessly hand off interaction control so that users are unaware of switching applications. PALMs was made able to schedule user interactions in future PERLS sessions, making it possible to space practice on a longer timescale to increase retention of PALMs content.Without PALMs, PERLS has no principled way to accelerate visual knowledge learning. Without PERLS, PALMs has no way to extend its retention enhancing model to long timescales. Integration required substantial time and cooperation to define objectives, co-designAPIs, and synchronize engineering across organizations. The Total Learning Architecture (TLA), currently in development, will reduce integration requirements with the goal of making cross-application synergy more common. In this paper, we describe our technical approach to combining PERLS and PALMS, then describe howTLA, treating this effort as a use case,could reduce barriers for comparable future integration efforts.
W911QY-12-C0171, W911QY-16-C-0019
PERLS

The Changing Face of Military Learning

2017; Army Learning Journal; Schatz, S. Ph.D.; Fautua, D.T, Ph.D.; Stodd, J.; Reitz, E.
Globalization, social media, ever-increasing computing power, and the proliferation of low cost advanced technologies have created a level of worldwide complexity and rapid change never before seen. To remain competitive in this environment, the Department of Defense and our coalition allies must identify new ways to empower our forces. In this...
Globalization, social media, ever-increasing computing power, and the proliferation of low cost advanced technologies have created a level of worldwide complexity and rapid change never before seen. To remain competitive in this environment, the Department of Defense and our coalition allies must identify new ways to empower our forces. In this article, we assert that part of that solution includes increased investments in our Human Dimension. Specifically, we argue that military personnel require an expanded set of competencies, higher levels of nuanced skills such as critical thinking and emotional intelligence, and more efficient and agile pathways to expertise, and that achieving these outcomes depends, at least in part, on revising the military learning enterprise. Toward this end, we outline a vision for the future of military learning, painting a picture of the "art of the possible" and proposing a road map that outlines five enabling conditions needed to achieve this future vision: (1) cultivate ubiquitous learner centric, technology enabled instruction; (2) build upon the foundations of data driven learning; (3) foster a learning culture at the organizational level; (4) encourage and empower social learning; and (5) draw upon deliberate practices and the evidence based body of knowledge from learning science. Enacting any one of these conditions will pose significant challenges, and particular science or technology gaps associated with each condition create additional hurdles. Nonetheless, we argue that the time is right, in terms of understanding and demand, to take action. One major step in that direction is to agree upon a shared grand strategy, that is a vision for our Human Dimension and the military learning system that empowers it. That is the professional dialog this article attempts to help inform and encourage.

Distributed Learning Gap Report

2017; ADL Publication; Johnson, Haag
The DoD faces a growing challenge to meet the breadth, depth, and tempo of its expanding education and training needs. While budgets are shrinking, the complexity of missions is increasing and demands on personnel are growing. Technologies - such as distributed learning capabilities - can help address these challenges, but the training, education,...
The DoD faces a growing challenge to meet the breadth, depth, and tempo of its expanding education and training needs. While budgets are shrinking, the complexity of missions is increasing and demands on personnel are growing. Technologies - such as distributed learning capabilities - can help address these challenges, but the training, education, and operations communities must make informed decisions about which technologies to pursue and how to best implement them. The Advanced Distributed Learning (ADL) Initiative was established to explore how Federal training and education programs and policies can better support flexible, lifelong learning through the use of technology. In 2016, the ADL Initiative conducted a requirements campaign to examine the status quo of distributed learning across the DoD and related Federal security agencies, pinpoint shared distributed learning gaps among stakeholders, and help guide future research and development investments (from ADL Initiative as well as other Federal agencies) to mitigate those gaps.

PS4TLA: Privacy Support for the Total Learning Architecture Operational Characteristics

2017; Knijnenburg, B.P.; et al
The Total Learning Architecture (TLA) is a set of specifications to enable the development of next-generation learning systems. As the TLA specifications are being developed, there exists an opportunity to implement Privacy by Design(PbD), where privacy is treated as a fundamental part of the system, and taken into account throughout the entire...
The Total Learning Architecture (TLA) is a set of specifications to enable the development of next-generation learning systems. As the TLA specifications are being developed, there exists an opportunity to implement Privacy by Design(PbD), where privacy is treated as a fundamental part of the system, and taken into account throughout the entire development lifecycle of the system, starting at the early stages of design and development [51, 207, 308, 314, 335]. This document therefore describes the potential impact of the Operational Characteristics (OCs) of TLA-based systems on users' privacy concerns. The OCs are aspects of TLA-based systems that can be implemented in various ways. The purpose of this document is to allow ADL and other TLA performers to select the operational variants that best alleviate users' privacy concerns.
PS4TLA

Death To The Privacy Calculus?

2017; SSRN Journal, Social Science Research Network; Knijnenburg, B.P.; Cherry, D.; Wilkinson, D.; Sivakumar, S.; Raybourn, E.; Sloan, H.
The "privacy calculus" has been used extensively to describe how people make privacy-related decisions. At the same time, many researchers have found that such decisions are often anything but calculated. More recently, the privacy calculus has been used in service of machine learning approaches to privacy. This position paper discusses the...
The "privacy calculus" has been used extensively to describe how people make privacy-related decisions. At the same time, many researchers have found that such decisions are often anything but calculated. More recently, the privacy calculus has been used in service of machine learning approaches to privacy. This position paper discusses the practical and ethical questions that arise from this use of the privacy calculus.
W911QY-16-C-0105
PS4TLA

User-Tailored Privacy by Design

2017; USEC, Usable Security Conference; Sivakumar, S.; Wilkinson, D.; Cherry, D.; Knijnenburg, B.P.
The "privacy by design" philosophy addresses privacy aspects early in the design and development of an information system. While privacy by design solutions often provide considerable advantages over "post hoc" privacy solutions, they are usually not customized to the needs of individual users. Further, research shows that users...
The "privacy by design" philosophy addresses privacy aspects early in the design and development of an information system. While privacy by design solutions often provide considerable advantages over "post hoc" privacy solutions, they are usually not customized to the needs of individual users. Further, research shows that users differ substantially in their privacy management strategies. Thus, how can we support such broad privacy needs in a comprehensive and user-centered way? This paper presents the idea of user-tailored privacy by design, a design methodology that combines multiple privacy features into a single intelligent user interface. We discuss how this methodology moves beyond the "one-size-fits-all" approach of existing privacy by design solutions and the narrow focus on information disclosure of existing user-tailored privacy solutions. We illustrate our approach through an implementation of user-tailored privacy by design within Facebook based on six privacy management profiles that were discovered in recent work, and subsequently extend this idea to the context of the Total Learning Architecture (TLA), which is a next generation learning platform that uses pervasive user monitoring to provide highly adaptive learning recommendations.
W911QY-16-C-0105

Advancements in Distributed Learning (ADL) Environment in Support of Transformation

2017; North Atlantic Treaty Organization Neuilly-sur-Seine Cedex France
RTG HFM-212 successfully addressed its stated objectives as listed in the TAD/TAP. This success was due to a substantial NATO/PDP organizational and policy base as well as it developed a strong working relationship with the NATO Allied Command Transformation (ACT) in Norfolk, VA, USA. With the funding support provided by the CWP (Coalition Warfare...
RTG HFM-212 successfully addressed its stated objectives as listed in the TAD/TAP. This success was due to a substantial NATO/PDP organizational and policy base as well as it developed a strong working relationship with the NATO Allied Command Transformation (ACT) in Norfolk, VA, USA. With the funding support provided by the CWP (Coalition Warfare Program), this excellent collaboration and data sharing of the MoLE, and joint funding from the US Army Telemedicine and Advanced Technology Research Center (TATRC), RTG HFM-212 is considered an outstanding success. It utilized an existing NATO infrastructure, actively collaborated with other NATO assets (e.g., the NATO Training Group and the PfP Consortium), and built a transition path toward the delivery of its work to NATO school houses. By accomplishing the RTG goals of creating and strengthening partnerships among 24 key Partner Nations, RTG HFM-212 has contributed sizable technology awareness as well as real capabilities to the NATO education and training communities.

Toward Intelligent Instructional Handoffs Between Humans and Machines

2016; NIPS, Neural Information Processing Systems; Ritter, S.; Fancsali, S.E.; Yudelson, M.; Rus, V.; Berman, S.
We describe preliminary results of the Integrating Human and Automated Tutoring Systems (IHATS) Project, the goal of which is to leverage a unique dataset containing information about student usage of both an automated tutoring system for algebra as well as transcripts of chat sessions between these same students and human tutors. We seek answers to...
We describe preliminary results of the Integrating Human and Automated Tutoring Systems (IHATS) Project, the goal of which is to leverage a unique dataset containing information about student usage of both an automated tutoring system for algebra as well as transcripts of chat sessions between these same students and human tutors. We seek answers to questions about what affective, behavioral, and cognitive factors predict that students will seek human assistance (and/or factors that drive them to human assistance) while using an automated tutoring system and what characterizes when such tutor tuttee interactions will be enhance learning, using data from the automated tutoring system to measure such learning. The project leverages a variety of statistical and machine learning techniques to answer these questions. Longer term, answers to these questions will be vital to developing systems that can intelligently guide students (i.e., make instructional handoffs) between automated and human sources of assistance as and when necessary.
W911QY-15-C-0070
IHATS

Joint Inference for Mode Identification in Tutorial Dialogues

2016; COLING, Conference on Computational Linguistics; Deepak, V.; Vasile, R.
Identifying dialogue acts and dialogue modes during tutorial interactions is an extremely crucial sub-step in understanding patterns of effective tutor tutee interactions. In this work, we devel opa novel joint inference method that labels each utterance in a tutoring dialogue session with a dialogue act and a specific mode from a set of pre-defined...
Identifying dialogue acts and dialogue modes during tutorial interactions is an extremely crucial sub-step in understanding patterns of effective tutor tutee interactions. In this work, we devel opa novel joint inference method that labels each utterance in a tutoring dialogue session with a dialogue act and a specific mode from a set of pre-defined dialogue acts and modes, respectively. Specifically, we develop our joint model using Markov Logic Networks (MLNs), a framework that combines first-order logic with probabilities, and is thus capable of representing complex, uncertain knowledge. We define first-order formulas in our MLN that encode the interdependencies between dialogue modes and more fine-grained dialogue actions. We then use a joint inference to jointly label the modes as well as the dialogue acts in an utterance. We compare our system against a pipeline system based on SVMs on a real-world dataset with tutoring sessions of over 500 students. Our results show that the joint inference system is far more effective than the pipeline system in mode detection, and improves over the performance of the pipeline system by about 6 points in F1 score. The joint inference system also performs much better than the pipeline system in the context of labeling modes that highlight important pedagogical steps in tutoring.
W911QY-15-C-0070

Black Swans and the Limits of Hierarchy

2016; IITSEC; Stodd, J.; Reitz, E.A.; Schatz, S., Ph.D.; Fautua, D., Ph.D.
Black Swans are unknowable events, by their nature unpredictable, asymmetric and easier to recognize in retrospect:nothing is clear at the outset, there is only an emergent sense of understanding, and traditional diagnose/act/review cycles may trap us into mismatched known responses. The unknowable cannot be mitigated or fully understood through...
Black Swans are unknowable events, by their nature unpredictable, asymmetric and easier to recognize in retrospect:nothing is clear at the outset, there is only an emergent sense of understanding, and traditional diagnose/act/review cycles may trap us into mismatched known responses. The unknowable cannot be mitigated or fully understood through established system, process or formal hierarchy. Instead the unknowable requires diagnosis (emergent under-standing), creation of a coherent narrative (our momentary understanding),sense making(crowd sourcing, interplay between formal and tacit/group knowledge) and reflection (to capture and share). Proceeding through all four steps during all events (including events that only in retrospect are regarded as black swans) requires organizational dyna-mism. Traditional organizations can become exhausted by tackling successive compounding changes. They take hard charging approaches, solve first-order problems, but are tired at the end. Do this constantly, and they become exhausted. They are not adapted to this new space. Dynamic organizations have adaptability in their DNA: They don'tget breathless; they thrive in change. Typically they have a devolved, distributed strength, based not in systematized,formalized, codified practice, but strong sense making and distributed action.This paper presents a case for building more efficient and agile pathways to cognitive expertise, as described our 2015I/ITSEC paper,"The Changing Face of Military Learning" (Schatz, Fautua, Reitz, & Stodd, 2015). How do we create learning environments that encourage dynamic responses, sense making, creating diverse teams of different mindsets?How can this be done within the limitations of organizations, like the military, which are by their nature formal hierarchies using formal decision making processes? How do we build learning systems that create inherently curious organizations? To answer these questions, we will present a model for developing adaptable military organizations with the necessary culture and readiness to more successfully confront black swans.

Advanced Distributed Learning: A Global Perspective

2016; IITSEC; Schatz, S, Ph.D.; et. al.
Coalition military forces have become the norm, and to be ready for multinational operations, militaries must prepare using relevant training, education, and exercises. Many nations and security coalitions use advanced distributed learning technologies to, at least partially, meet this demand. Towards that end, this paper examines the...
Coalition military forces have become the norm, and to be ready for multinational operations, militaries must prepare using relevant training, education, and exercises. Many nations and security coalitions use advanced distributed learning technologies to, at least partially, meet this demand. Towards that end, this paper examines the intersection of multinational military training and education and innovations in learning science and technology, specifically those involving distributed learning. To write this piece, authors from 12 nations/NATO collaborated. Content contributors serve as military officers or civil servants in their defense ministries, and each works in the technology-enabled learning domain. The paper begins by highlighting national strategies in support of multinational collaboration. It then discusses collaborative efforts involving technology-based learning. Next, it describes a survey of the authors' organizations, which examined their priorities, challenges, and uses of distributed learning. Finally, the paper closes with our survey findings and recommended next steps for consideration by coalition military training and education stakeholders.Some key findings from the survey include the following: Use of distributed learning is expanding across the board. On average across surveyed organizations, enhancing the instructional quality (pedagogy/andragogy) of distributed learning ranks as the number-one focus area. Mobile learning and, possibly, game-based distributed learning may see an uptake in use, but there are unmet needs in the areas of learning analytics and, correspondingly, xAPI and learner/teacher performance dashboards. Finally, international partnerships - specifically in the area of distributed learning - yield tangible benefits.

Strategic Compression and the Military's Pursuit of Cognitive Readiness

2016; IITSEC; Reist, J.W.; Fautua, D. Ph.D.; Reitz, E.; Schatz, S., Ph.D.; Stodd, J.
The Joint Force of today is facing an inflection point with respect to the way it believes it should learn, train, and educate. The generation that fought America's longest war is more savvy and intuitive but also sober on what really "works." The authors have observed, for instance, that "Jointness" is being redefined, and correspondingly preparation...
The Joint Force of today is facing an inflection point with respect to the way it believes it should learn, train, and educate. The generation that fought America's longest war is more savvy and intuitive but also sober on what really "works." The authors have observed, for instance, that "Jointness" is being redefined, and correspondingly preparation for it is shifting, focusing less on systems and more on developing higher-order cognitive skills, such as problem-solving, adaptiveness, and anticipation. However, trainers are unclear on how to create interventions that simultaneously un-tap the cognitive potential of their staffs, enable accurate and actionable performance measurement, and support affordable scalability to geographically dispersed training audiences. We argue that part of the solution requires exploiting web-based technologies that can create semi-immersive experiences in "strategic compression," which we use as a short-hand reference for the stresses of operational/strategic crisis action planning and decision-making under time constraints, contextual complexity, and organizational ambiguity. If those conditions are rendered in a distributed team simulation (part-task trainer) and paired with appropriate instructional interventions, this creates opportunities for trainers to facilitate quality interventions in support of a group's decision-making processes (i.e., "strategic reflection"). The authors build upon previously published research related to technology-enabled joint training at the lower-end of the cognitive learning taxonomy (i.e., remember, understand, and apply; see Fautua, Schatz, et. al., I/ITSEC 2014) to provide preliminary findings in establishing similar interventions for the taxonomy's upper end: analyze, evaluate, and create.

Education and Training Technology Requirements for DoD Distributed Learning

2016; IITSEC; Johnson, Andy
The Department of Defense faces a growing challenge to meet the breadth, depth, and tempo of its expanding education and training needs. While budgets are shrinking, the complexity of missions is increasing and demands on personnel are growing. Learning technologies can help address this, but the research and acquisition communities must make...
The Department of Defense faces a growing challenge to meet the breadth, depth, and tempo of its expanding education and training needs. While budgets are shrinking, the complexity of missions is increasing and demands on personnel are growing. Learning technologies can help address this, but the research and acquisition communities must make informed decisions about which technologies to pursue. Various military offices publish guidance on this question. For instance, some Defense agencies publish "Science and Technology Objectives," including direction on next-generation learning requirements and associated technologies. However, these publications generally lack linkages to one another - particularly those authored by different Services - which leads to duplications of effort and missed opportunities for coordination. Further, because these publications typically derive from high-level strategy guidance (i.e., "top-down" direction), they may inadvertently omit some lower-echelon needs (i.e., "bottom-up" inputs). To address these gaps, the Advanced Distributed Learning (ADL) Initiative executed an interagency requirements campaign to collate existing publications, crosswalk Defense agency needs, and search for yet-undiscovered requirements associated with training, education, and related performance support. As part of this effort, we also conducted interviews with widespread stakeholders across the Department. The results are being assembled into a data visualization, which viewers will be able to access via a government website. Although our Requirements Campaign is still underway, this paper describes our requirements engineering process as well as initial results from it. In particular, we highlight the "top ten" requirements in terms of frequency and apparent priority (for the stakeholders), and we discuss opportunities for developing or acquiring new learning technologies to address the priority objectives.

Total Learning Architecture (TLA) Enables Next-generation Learning via Meta-adaptation

2016; IITSEC; Folsom-Kovarik, J.T.; Raybourn, E.M.
Technology is becoming ever more central to teaching and training. In classrooms, students use intelligent tutors and adaptive tests instead of textbooks and worksheets. In daily life, mobile devices enable blended, on-demand and ubiquitous life-long learning applications. Connected, pervasive media enables compelling transmedia learning...
Technology is becoming ever more central to teaching and training. In classrooms, students use intelligent tutors and adaptive tests instead of textbooks and worksheets. In daily life, mobile devices enable blended, on-demand and ubiquitous life-long learning applications. Connected, pervasive media enables compelling transmedia learning (Raybourn, 2014) experiences. However, learning opportunities are still often implemented in "walled gardens" or stand-alone technology systems that must be manually curated and coordinated with all the others through costly, one-off efforts of developers or individual instructors.
W911QY-16-C-0019
TLA

Review and Assessment of Personnel Competencies and Job Description Models and Methods

2016; Institute for Defense Analysis, IDA; Belanich, J.; Franklin, M.; Lall, P.
Organizations are always looking for new ways to acquire, manage, and retain the talent needed to reach their goals. To achieve this goal, some government and private organizations emphasize the need to better understand personnel and occupational information and data for improved competency management. Competencies and detailed job descriptions...
Organizations are always looking for new ways to acquire, manage, and retain the talent needed to reach their goals. To achieve this goal, some government and private organizations emphasize the need to better understand personnel and occupational information and data for improved competency management. Competencies and detailed job descriptions generally are recognized for their value to enhance organizational management. The Department of Defense (DoD) defines a competency as "an (observable) measurable pattern of knowledge, abilities, skills, and other characteristics that individuals need in order to successfully perform their work" (Department of Defense 2008, 8). The terms in DoD's definition are comparable to many others' definitions. However, there is variability in how organizations use competencies for human capital management. What this means and how these terms are applied in practice are inconsistent within job fields and across organizations.
HQ0034-14-D-0001
MOSAIC

Intelligent Coaching Systems in Higher-Order Applications: Lessons from Automated Content Creation Bottlenecks

2016; IAC; Greuel, C.; Yarnall, L.; Ziker, C.; Kernbaum, A.; Murray, J.
Procedural skills are an increasingly pervasive requirement in today's world, in areas ranging from IT system administration to complex data analyses, from automotive equipment repair to intricate medical diagnosis. The acquisition of procedural skills requires learning by doing learners gain knowledge by trying to solve challenge problems,...
Procedural skills are an increasingly pervasive requirement in today's world, in areas ranging from IT system administration to complex data analyses, from automotive equipment repair to intricate medical diagnosis. The acquisition of procedural skills requires learning by doing learners gain knowledge by trying to solve challenge problems, exploring the usage and limitations of tools and techniques, getting feedback on oversights and mistakes, and requesting assistance in the face of impasses and confusion. Intelligent virtual environments (VEs) hold promise for improving learner directed instruction in these contexts. Such systems trace the progress of learners as they perform training tasks, and can insert immediate coaching or provide performance evaluation to focus learner attention, link knowledge to activity, and accelerate the shifts between abstract and concrete learning. VE technology is widely used to improve self directed learning of hands on manual procedures, but it also shows appreciable promise for the use of modeling tools in a diverse range of higher order applied fields, such as design engineering, policy analytics, and econometrics. To realize this vision, research must address the formidable bottlenecks around content creation and explore the types of reusable content libraries relevant to the subject domains. In this paper, we describe two interactive training projects that developed prototypes for automated content creation. A third project illustrates a suite of learning object libraries to support engineering instruction. The first project, Semantically enabled Automated Assessment in Virtual Environments (SAVE), uses a 3D browser based simulation environment for hands-on training in equipment maintenance, supplemented by automated generation of instructional exercise solutions. SAVE allows a subject matter expert (SME) to use interactive simulations for modeling the correct steps applied to given procedural tasks and provide a rapid way to extract their knowledge. The system logs an SME's activity, which becomes the reference model against which learner activity is compared in automated assessment. The second project, ARMentor, delivers augmented reality (AR) overlays in headmounted displays worn by student technicians while they learn vehicle maintenance. An automated speech system interacts with the learners as they perform equipment adjustments and troubleshoot electrical faults. To deliver audible step-by-step guidance, a prototype text-to-speech translator was developed to convert steps as written in the technical manual into the voice of a virtual coach. The third project, Simulation for Manufacturing and Prototyping with a Learning Environment (SiMPLE), developed tools to allow learners to construct electromechanical simulations, providing an intelligent coaching system to enable them to iteratively refine their design specifications. These tools include object libraries with embedded engineering computations and suites of scripts for design coaching, design testing, and physical prototyping once a working simulation is achieved.
W911QY-14-C-0023
SAVE

A Comparative Study of Visual Cues for Annotation-Based Navigation Support in Adaptive Educational Hypermedia

2016; UMAP; Hosseini, R.; Brusilovsky, P.
Adaptive link annotation is one of the most well-known adaptive navigation support technologies that aims to guide hypermedia users to the most relevant information by personalizing the appearance of hyperlinks. Past work assumed no difference between different interface implementations of personalization approaches that are conceptually the...
Adaptive link annotation is one of the most well-known adaptive navigation support technologies that aims to guide hypermedia users to the most relevant information by personalizing the appearance of hyperlinks. Past work assumed no difference between different interface implementations of personalization approaches that are conceptually the same. The goal of the current study was to determine whether the choice of visual cues does matter by conducting a user study with several alternative designs for link annotation in interactive code examples.
W911QY13C0032
ANS

Tensor Factorization for Student Modeling and Performance Prediction in Unstructured Domain

2016; EDM; Sahebi, S.; Lin, Y.; Brusilovsky, P.
We propose a novel tensor factorization approach, Feedback-Driven Tensor Factorization (FDTF), for modeling student learning process and predicting student performance. This approach decomposes a tensor that is built upon students' attempt sequence, while considering the quizzes students select to work with as its feedback. FDTF does not...
We propose a novel tensor factorization approach, Feedback-Driven Tensor Factorization (FDTF), for modeling student learning process and predicting student performance. This approach decomposes a tensor that is built upon students' attempt sequence, while considering the quizzes students select to work with as its feedback. FDTF does not require any prior domain knowledge, such as learning resource skills, concept maps, or Qmatrices. The proposed approach differs significantly from other tensor factorization approaches, a sit explicitly models the learning progress of students while interacting with the learning resources. We compare our approach to other state-of-the-art approaches in the task of Predicting Student Performance (PSP). Our experiments show that FDTF performs significantly better compared to baseline methods, including Bayesian Knowledge Tracing and a state-of-the-art tensor factorization approach.
W911QY-13-C-0032

Preliminary Results on Dialogue Act Classification in Chat-based Online Tutorial Dialogues

2016; EDM; Vasile, R.; Banjade, R.; Maharjan, N.; Morrison, D.; Ritter, S.; Yudelson, M.
We present in this paper preliminary results with dialogue act classification in human-to-human tutorial dialogues. Dialogue acts are ways to characterize the intentions and actions of the speakers in dialogues based on the language-as-action theory. This work serves our larger goal of identifying patterns of tutors' actions, in the form of...
We present in this paper preliminary results with dialogue act classification in human-to-human tutorial dialogues. Dialogue acts are ways to characterize the intentions and actions of the speakers in dialogues based on the language-as-action theory. This work serves our larger goal of identifying patterns of tutors' actions, in the form of dialogue act and subact sequences, that relate to various aspects of learning. The preliminary results we obtained for dialogue act classification using a supervised machine learning approach are promising.
W911QY-15-C-0070

Tailored Cybersecurity Training in LVC Environments

2016; MODSIM; Nicholson, D., Ph.D.; Massey, L.; O'Grady, R.; Ortiz, E.
Cyber vulnerabilities are continually emerging as a threat to our national and economic security and stability. Reports indicate a tremendous gap in skilled personnel capable of filling our growing need for a Cyber Security workforce to operate, analyze, protect, and defend our critical infrastructure systems. In response, the Department of...
Cyber vulnerabilities are continually emerging as a threat to our national and economic security and stability. Reports indicate a tremendous gap in skilled personnel capable of filling our growing need for a Cyber Security workforce to operate, analyze, protect, and defend our critical infrastructure systems. In response, the Department of Homeland Security has developed a national strategic program geared toward education, the National Initiative for Cybersecurity Careers and Studies (NICCS). This program has developed the National Cybersecurity Workforce Framework which "provides a blueprint to categorize, organize, and describe cybersecurity work into Specialty Areas, tasks, and knowledge, skills and abilities (KSAs)" (NICCS, 2015). There is a logical progression to turn to modeling and simulation-based training systems to provide experiential learning to augment the knowledge and skills being developed in classroom and e-learning cyber security certification and degree programs. By using a scenario-based approach in Live, Virtual and Constructive (LVC) simulation, trainees can practice higher order skills and have an opportunity to experience realistic stressors in dynamic situations. We will present concepts for use of on-going research into three different interactive cybersecurity training activities 1) a 3D gaming environment for Insider Threat training, 2) a virtual Cyber Security Instruction Environment (CYSTINE)for penetration testing with cognitive agent defenders and 3) the use of red-team verse blue-team,live simulation,exercises as realistic, challenging experiences for computer network defense. We will discuss these cyber learning experiences within a use case of a trainee progressing through a sequence of training tailored to his or her personal needs and objectives,such as envisioned within our early research on a project entitled Fast Learning from Unlabeled Episodes for Next-generation Tailoring(FLUENT) as part of Advanced Distributed Learning's(ADL)future Training and Learning Architecture (TLA).
W911QY-16-C-0019
FLUENT

Semantic and Episodic Learning to Integrate Diverse Opportunities for Life-Long Learning

2016; MODSIM; Folsom-Kovarik, J.T.; Jones, R.M.; Schmorrow, D.
The Advanced Distributed Learning (ADL) Initiative has developed the Training and Learning Architecture (TLA) with the goal of using information technology to change the paradigm of education from occasional classroom study and training to pervasive and lifelong activity. TLA views education content providers as services that produce educational content relevant to particular learning needs and contexts. A content brokering service assesses an individual learner’s current learning needs, and recommends content that is suitable to those needs but also appropriate to the situation the learner is in (e.g., recommending listening to a particular podcast if the learner is driving somewhere). Key technologies for the TLA vision are a Learning Record Store (LRS), which stores a continuously updated record of learning activity and outcomes, as well as a content meta-tagging language that enables the mapping of particular educational tools and content to specific situations. One challenge associated with content meta-tagging is that it requires significant manual effort, especially as content and technologies change. TLA also does not yet have a capability to identify new meta-tags or relationships between existing tags, implying that it may miss some opportunities for effective instruction. We describe a new research effort called FLUENT (Fast Learning from Unlabeled Episodes for Next-generation Tailoring), which will learn new tags and relationships to improve the overall coverage and effectiveness of content delivery in TLA. FLUENT will use a hybrid machinelearning approach that includes episodic learning, heuristic search based on analogical mapping, and an explanationbased learning capability that uses a background knowledge base of causation in instruction to discover relationships from examples. The knowledge-based learning approach will allow effective learning in a domain where statistical learning methods would suffer from the sparse data available.
The Advanced Distributed Learning (ADL) Initiative has developed the Training and Learning Architecture (TLA) with the goal of using information technology to change the paradigm of education from occasional classroom study and training to pervasive and lifelong activity. TLA views education content providers as services that produce educational content relevant to particular learning needs and contexts. A content brokering service assesses an individual learner’s current learning needs, and recommends content that is suitable to those needs but also appropriate to the situation the learner is in (e.g., recommending listening to a particular podcast if the learner is driving somewhere). Key technologies for the TLA vision are a Learning Record Store (LRS), which stores a continuously updated record of learning activity and outcomes, as well as a content meta-tagging language that enables the mapping of particular educational tools and content to specific situations. One challenge associated with content meta-tagging is that it requires significant manual effort, especially as content and technologies change. TLA also does not yet have a capability to identify new meta-tags or relationships between existing tags, implying that it may miss some opportunities for effective instruction. We describe a new research effort called FLUENT (Fast Learning from Unlabeled Episodes for Next-generation Tailoring), which will learn new tags and relationships to improve the overall coverage and effectiveness of content delivery in TLA. FLUENT will use a hybrid machinelearning approach that includes episodic learning, heuristic search based on analogical mapping, and an explanationbased learning capability that uses a background knowledge base of causation in instruction to discover relationships from examples. The knowledge-based learning approach will allow effective learning in a domain where statistical learning methods would suffer from the sparse data available.
W911QY-16-C-0019
FLUENT

Tailored Cybersecurity Training in LVC Environments (Slides)

2016; MODSIM; Folsom-Kovarik, J., Ph.D.; Nicholson, D., Ph.D.; Massey, L.; O'Grady, R.; Ortiz, E.
This presentation in tailored cybersecurity training addresses the cybersecurity workforce need, describes stakeholders, reviews what actions are being done today and what is new.
This presentation in tailored cybersecurity training addresses the cybersecurity workforce need, describes stakeholders, reviews what actions are being done today and what is new.
FLUENT

An Intelligent Interface for Learning Content: Combining an Open Learner Model and Social Comparison to Support Self-Regulated Learning and Engagement

2016; IUI, Intelligent User Interfaces; Guerra, J.; Somyurek, S.; Hossein, R.; Brusilovsky, P.
We present the Mastery Grids system, an intelligent interface for online learning content that combines open learner modeling (OLM) and social comparison features. We grounded the design of Mastery Grids in self-regulated learning and learning motivation theories, as well as in our past work in social comparison, OLM, and adaptive navigation...
We present the Mastery Grids system, an intelligent interface for online learning content that combines open learner modeling (OLM) and social comparison features. We grounded the design of Mastery Grids in self-regulated learning and learning motivation theories, as well as in our past work in social comparison, OLM, and adaptive navigation support. The force behind the interface is the combination of adaptive navigation functionality with the mastery-oriented aspects of OLM and the performance-oriented aspects of social comparison. We examined different configurations of Mastery Grid sin two classroom studies and report the results of analysis of log data and survey responses. The results show how Mastery Grids interacts with different factors, like gender and achievement-goal orientation, and ultimately, its impact on student engagement, performance, and motivation.
W911QY-13-C-0032
OLM

Animated Examples as Practice Content in a Java Programming Course

2016; Special Interest Group on Computer Science Education, SIGCSE; Hosseini, R.; Sirkiä, T.; Guerra, J.; Brusilovsky, P.; Malmi, L.
Code examples are commonly used learning resources that help students grasp various programming structures and concepts. However, example code usually requires explanations about what each line or part of the code does. Otherwise, students may find it difficult to follow an example. In this paper, we compare two types of code examples that use...
Code examples are commonly used learning resources that help students grasp various programming structures and concepts. However, example code usually requires explanations about what each line or part of the code does. Otherwise, students may find it difficult to follow an example. In this paper, we compare two types of code examples that use different techniques to describe important concepts in the code annotated and animated examples. The former displays an explanation for a subset of lines in plain text, whereas the latter visualizes code execution. We studied the use and impact of these enhanced examples, provided as nonmandatory practice content, in three introductory Java courses. Our results suggest that animated examples are more engaging and have a positive impact on students' learning. As compared to annotated examples, students spent more time with animated examples and more likely completed them. Also, a positive relationship was found between the number of explored animated examples and the overall course grade.

Open Social Student Modeling for Personalized Learning

2015; IEEE TRANSACTIONS; Brusilovsky, P.; Somyurek, S.; Guerra, J.; Hosseini, R.; Zadorozhny, V.; Durlach, J.
Open student modeling (OSM) is an approach to technology-based learning, which makes student models available to the learners for exploration. OSM is known for its ability to increase student engagement, motivation, and knowledge reflection. A recent extension of OSM known as open social student modeling (OSSM) complements cognitive aspects of OSM...
Open student modeling (OSM) is an approach to technology-based learning, which makes student models available to the learners for exploration. OSM is known for its ability to increase student engagement, motivation, and knowledge reflection. A recent extension of OSM known as open social student modeling (OSSM) complements cognitive aspects of OSM with social aspects by allowing students to explore models of peer students and/or an aggregated class model. In this paper, we introduce an OSSM interface, MasteryGrids, and report the results of a large-scale classroom study, which explored the impact of the social dimension of OSSM. Students in a database management course accessed non-required learning materials(examples and problems) via the MasteryGrids interface using either OSM or OSSM. The results revealed that OSSM-enhanced learning, especially for students with lower prior knowledge, compared with OSM. It also enhanced user attitude and engagement. Amount of student usage, efficiency of student usage, and student attitude varied depending on the combination of interface condition (OSM/OSSM), gender, and student social comparison orientation.
W911QY13C0032
OSSM

Impact of Open Social Student Modeling on Self-Assessment of Performance

2015; E-Learn, AACE; Somyürek, S.; Brusilovsky, P.
This study examines the impact of Open Student Modeling (OSM) and its extension known as Open Social Student Modeling (OSSM) on students' self-assessment of their SQL programming performance. It also explores the relationship between self-assessment and normalized gain. The study was performed with graduate students of University of Pittsburgh...
This study examines the impact of Open Student Modeling (OSM) and its extension known as Open Social Student Modeling (OSSM) on students' self-assessment of their SQL programming performance. It also explores the relationship between self-assessment and normalized gain. The study was performed with graduate students of University of Pittsburgh taking Database Course for the first semester 2014/2015 and ran for 11 weeks. The results demonstrated that both OSM and OSSM positively affect students' regular self-assessment ability. However, only OSSM was able to positively impact students' relative self-assessment ability (i.e., compare their SQL knowledge to others). The results also indicated that, students' self-assessments are positively and highly correlated to their normalized gain.
W911QY13C0032
OSSM

Value of Social: Comparing Open Student Modeling and Open Social Student Modeling

2015; UMAP; Brusilovsky, P.; Somyurek, S.; Guerra, J.; Hosseini, R.; Zadorozhny, V.
Open Student Modeling (OSM) is a popular technology that makes traditionally hidden student models available to the learners for exploration. OSM is known for its ability to increase student engagement, motivation, and knowledge reflection. A recent extension of OSM known as Open Social Student Modeling (OSSM) attempts to enhance cognitive aspects...
Open Student Modeling (OSM) is a popular technology that makes traditionally hidden student models available to the learners for exploration. OSM is known for its ability to increase student engagement, motivation, and knowledge reflection. A recent extension of OSM known as Open Social Student Modeling (OSSM) attempts to enhance cognitive aspects of OSM with social aspects by allowing students to explore models of peer students or the whole class. In this paper, we introduce MasteryGrids, a scalable OSSM interface and report the results of a large-scale classroom study that explored the value of adding social dimension to OSM. The results of the study reveal a remarkable engaging potential of OSSM as well as its impact on learning effectiveness and user attitude
OSSM

The Value of Social: Comparing Open Student Modeling and Open Social Student Modeling (Slides)

2015; UMAP; Brusilovsky, P.; Somyurek, S.; Guerra, J.; Hosseini, R.; Zadorozhny, V.
This presentation describes Open Social Student Modeling and its evaluation. The presentation goes into why they studied the comparison and what they have done since submitting their paper.
This presentation describes Open Social Student Modeling and its evaluation. The presentation goes into why they studied the comparison and what they have done since submitting their paper. The paper can be found at https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=7359179.
OSSM

A Framework for Multifaceted Evaluation of Student Models

2015; EDM; Huang, Y.; Gonźalez-Brenes, J.P.; Kumhar, R.; Brusilovsky, P.
Latent variable models, such as the popular Knowledge Tracing method, are often used to enable adaptive tutoring systems to personalize education. However, finding optimal model parameters is usually a difficult non-convex optimization problem when considering latent variable models. Prior work has reported that latent variable models obtained...
Latent variable models, such as the popular Knowledge Tracing method, are often used to enable adaptive tutoring systems to personalize education. However, finding optimal model parameters is usually a difficult non-convex optimization problem when considering latent variable models. Prior work has reported that latent variable models obtained from educational data vary in their predictive performance, plausibility, and consistency. Unfortunately, there are still nounified quantitative measurements of these properties. This paper suggests a general unified framework (that we call Polygon) for multifaceted evaluation of student models. The framework takes all three dimensions mentioned above into consideration and offers novel metrics for the quantitative comparison of different student models. These properties affect the effectiveness of the tutoring experience in a way that traditional predictive performance metrics fall short. The present work demonstrates our methodology of comparing Knowledge Tracing with a recent model called Feature-Aware Student Knowledge Tracing (FAST) on datasets from different tutoring systems. Our analysis suggests that FAST generally improves on Knowledge Tracing along all dimensions studied.
N00014-12-C-0535
FAST

A Framework for Multifaceted Evaluation of Student Models (Slides)

2015; EDM; Huang, Y.; Gonźalez-Brenes, J.P.; Kumhar, R.; Brusilovsky, P.
These slides are related to the paper called A Framework for Multifaceted Evaluation of Student Models that can be found at http://www.educationaldatamining.org/EDM2015/uploads/papers/paper_164.pdf.
These slides are related to the paper called A Framework for Multifaceted Evaluation of Student Models that can be found at http://www.educationaldatamining.org/EDM2015/uploads/papers/paper_164.pdf.
FAST

Nextgen Navy eLearning Tracking

2014; Miller, William E
The Navy's eLearning (NeL) computer-based learning system relies on a Learning Management System (LMS) for content delivery and tracking learning information. One major obstacle with NeL's current LMS implementation is that tracking of learning can only be done while a user is on a desktop computer using an Internet browser to connect to the LMS...
The Navy's eLearning (NeL) computer-based learning system relies on a Learning Management System (LMS) for content delivery and tracking learning information. One major obstacle with NeL's current LMS implementation is that tracking of learning can only be done while a user is on a desktop computer using an Internet browser to connect to the LMS software. However, not all learning takes place within an Internet browser on a desktop computer. The Experience-API (xAPI), also known as Tin Can API and SCORM 2.0, is a standard maintained by Advanced Distributed Learning (ADL) that decouples the tracking of learning information from the content delivery. Any piece of software implementing the xAPI standard running on any networked device can track learning activity and store that data inside of a Learning Record Store (LRS). A prototype system was developed in a virtual environment to showcase the use of the xAPI/LRS to track quiz data, and the quiz data could then be synced from the LRS to the LMS. The prototype showed that xAPI, along with its LRS, can overcome the NeL's AtlasPro LMS limitation of only tracking learning from a user's desktop computer using an Internet browser.

Institutionalizing Blended Learning into Joint Training: A Case Study and Ten Recommendations

2014; Fautua, D.; Schatz, S.; Reitz, E.; Bockelman, P.
Teachers and trainers have relied upon blended learning for at least two decades, and scholars have authored (and continue to write!) thousands of papers about it. Yet, despite the wealth of individual experience and reams of published theory available, institutionalizing high-quality blended learning processes in large, dispersed organizations...
Teachers and trainers have relied upon blended learning for at least two decades, and scholars have authored (and continue to write!) thousands of papers about it. Yet, despite the wealth of individual experience and reams of published theory available, institutionalizing high-quality blended learning processes in large, dispersed organizations still poses significant challenges. The Joint Staff J7, Deputy Director for Joint Training initiated the Continuum of eLearning (CoL) project in order to overcome barriers to institutionalization of integrating blended learning into the joint training enterprise. This enterprise affects thousands of personnel each year, and diverse service and program office stakeholders support it. Hence the CoL project involved identifying ways to systematize instructional best practices, build a common vision for blended learning among the various stakeholders, formalize organizational processes for it, and empirically demonstrate the system s feasibility and value. As of 2014, Joint Staff J7 (Joint Training) has implemented the resulting Blended Learning Training System (BLTS), comprising the unified blended learning concept, corresponding set of processes, and growing repository of blended learning materials. We have demonstrated its viability at eight combatant command training events and collected data on its effectiveness at PANAMAX 2012, Savannah Shield 2013, and Turbo Challenge 2014. This paper summarizes the three-year research project, data collection outcomes, and resulting BLTS; it also includes generalizable lessons learned about institutionalizing blended learning in complex organizations.

Journal of Advanced Distributed Learning Technology

2014; Journal of Advanced Distributed Learning Technology, JADLeT; Durlach, Paula, P.h.D; et. al.
The year 2014 marks the fifteenth anniversary of the Advanced Distributed Learning (ADL) Initiative that was established in the United States (U.S.) under executive order in 1999 by then President William Clinton. As a result of ExecutiveOrder 13111, the U.S. Department of Defense (DoD) was directed to lead federal government participation in...
The year 2014 marks the fifteenth anniversary of the Advanced Distributed Learning (ADL) Initiative that was established in the United States (U.S.) under executive order in 1999 by then President William Clinton. As a result of ExecutiveOrder 13111, the U.S. Department of Defense (DoD) was directed to lead federal government participation in incorporating emerging technologies to develop standards for training software and associated services. The order further directed DoD to provide guidance to Defense agencies and advise the civilian agencies, as appropriate, on how best to use these standards for large-scale development and implementation of efficient and effective distributed learning technologies. Since the inception of the ADL Initiative in 1999, and the introduction of the Sharable Content Object Reference Model (SCORM®), ADL programs gained worldwide recognition. As a result, the "ADL Partnership Network" was established. The ADL Partnership Network currently includes fourteen Partnership Co-Labs, with common interests to support cooperative development of ADL capabilities. Current government partners include: Canada, Korea, Latin America and Caribbean Regions, NATO Allied Command Transformation (ACT), New Zealand, Norway, Poland, Romania, Serbia, United Kingdom and the United States of America. Current non-government associates include the Academic Co-Lab (Madison, WI) and the Center for Intelligent Tutoring Systems Research and Development (Memphis, TN). The ADL Partnership Network vision is to provide access to the highest quality education and training, tailored to individual needs, delivered cost effectively, anytime, anywhere. In support of this vision the ADL Partnership Network harnesses the power of distributed learning technologies to include online courseware, training games, virtual worlds, mobile technology and other learning technologies to provide high-quality, easily accessible, adaptable, 6 JADLET Journal of Advanced DistributedLearning Technology and cost-effective education and training. Working with international military and government agencies, industry, academia, and various professional organizations, the ADL Partnership Network conducts research, evaluation, and validation of specifications and standards with the goal of advancing education and training. The Partnership Network and their respective Ministries of Defense (MOD) recognize that successfully implementing the ADL vision can best be achieved through collaboration. To be most effective in today's environment, this collaboration should occur in both a national and international context with participation from military, government, industry, and academia. The ADL Partnership Network provides a collaborative synergy that leverages the best practices from industry, academia, government and our international partners to define common specifications and standards for distributed learning and training content. The Partnership Network research effort is focused for the purpose of developing and assessing common tools, standards, content, and guidelines for ADL. Through this collaborative effort the ADL Partnership Co-Labs serve as the focal points for coordinating ADL activities throughout the world.provided in classrooms with a teacher present

Goal Orientation, Self-Efficacy, and "Online Measures" in Intelligent Tutoring Systems

2014; Cognitive Science Society, COGSCI; Fanscal, S.E.; Bernacki, M.L.; Nokes-Malach T.J.; Yudelson, M.; Ritter, S.
While goal orientation and related factors like learner self-efficacy are of great interest to learning science researchers, some voice concerns regarding the measurement of such factors using self-report questionnaires. To address these concerns, recent work has explored the use of behavioral indicators like hint-seeking and glossary use in...
While goal orientation and related factors like learner self-efficacy are of great interest to learning science researchers, some voice concerns regarding the measurement of such factors using self-report questionnaires. To address these concerns, recent work has explored the use of behavioral indicators like hint-seeking and glossary use in intelligent tutoring systems like Carnegie Learning's Cognitive Tutor®(CT) as alternative, "online" measures of goal orientation. We re-examined this approach by measuring 273 CT users' achievement goals and self-efficacy judgments via embedded questionnaires and their hint-seeking and glossary use via log data. Using graphical causal models and linear structural equation models to observe structural relationships among goal orientations, self-efficacy, behaviors, and learning outcomes, we found that tracing orientations via "online measures" is more nuanced than perhaps previously appreciated. We describe complex relations observed in the model among motivations, behaviors,and outcomes and discuss the implications for the online measurement of motivation.
W911QY-13-C-0026

Where in the Data Stream Are We?: Analyzing the Flow of Text in Dialogue-Based Systems for Learning (ADL Chapter 19 pg 237-245) 1

2014; (Army Research Lab Book); Morrison, D.M.; Nye, B.; Hu, X
This chapter continues a discussion we began in the first volume in this series (Hu, Morrison&Cai, 2013) concerning the use of "learner micromodels" in dialogue-based ITSs. As we originally defined it, a learner micromodel in an ITS is an estimate of a learner's cognitive and/or affective state at a given time in an ITS session, based entirely on the...
This chapter continues a discussion we began in the first volume in this series (Hu, Morrison&Cai, 2013) concerning the use of "learner micromodels" in dialogue-based ITSs. As we originally defined it, a learner micromodel in an ITS is an estimate of a learner's cognitive and/or affective state at a given time in an ITS session, based entirely on the real-time dynamics of that particular session. An example of such a micromodel, which we call a Learner's Characteristic Curve (LCC), tracks two features of a learner's recent dialogue history–novelty and relevance–where novelty is a measure of the degree to which the learner's contributions to the dialogue add something new, and relevance is a measure of the degree to which his or her contributions conform to an expected answer. This model is called a "characteristic curve"because the learner's trajectory on these measures over time can be evaluated against certain archetypal curves or boundaries (e.g., thresholds). In this chapter,we identify an interesting complication with this design, then expand the discussion to include some general principles concerning the analysis of the data sets and live streams of data that are beginning to flow in vast quantities from Internet-based learning environments, including those with human tutors, artificially intelligent tutors, and, perhaps most interestingly, hybrid systems of the future.
W911QY-14-C-0019
LCC

Reinforcing Math Knowledge by Immersing Students in a Simulated Learning-By-Teaching Experience

2014; International Artificial Intelligence in Education Society, IAIIES; Lenat, D.B.; Durlach, P.J.
We often understand something only after we've had to teach or explain it to someone else. Learning by teaching (LBT) systems exploit this phenomenon by playing the role of tutee. BELLA, our sixth-grade mathematics LBT systems, departs from other LBT systems in several ways: (1) It was built not from scratch but by very slightly extending the ontology and...
We often understand something only after we've had to teach or explain it to someone else. Learning by teaching (LBT) systems exploit this phenomenon by playing the role of tutee. BELLA, our sixth-grade mathematics LBT systems, departs from other LBT systems in several ways: (1) It was built not from scratch but by very slightly extending the ontology and knowledge base of an existing large AI system, Cyc. (2) The"teachable agent" Elle begins not with a tabula ras abut rather with an understanding of the domain content which is close to the human student's. (3) Most importantly, Elle never actually learns anything directly from the human tutor! Instead, there is a superagent (Cyc) which already knows the domain content extremely well. BELLA builds up a mental model of the human student by observing them interact with Elle. It uses that Socratically to decide what Elle's current mental model should be(what concepts and skills Elle should already know, and what sorts of mistakes it should make) so as to best help the user to overcome their current confusions. All changes to the Elle model are made by BELLA, not by the user the only learning going on is BELLA learning more about the user but from the user's point of view it often appear sas though Elle were attending to them and learning from them. Our main hypothesis is that this may prove to be a particularly powerful and effective illusion to maintain.
W911QY-12-C-0106
LBT / BELLA

Videogame Design for Cognitive Enhancement through Micro-Puzzle Cognitive Profiling

2013; IITSEC; Gallagher, P.S.; Prestwich, S.
The Advanced Distributed Learning (ADL) Initiative's Next Generation Learner researchers previously investigated whether five video game design features hypothesized to be contained within Portal 2 might increase cognitive adaptability (CA). Their results highlighted a lack of understanding of the cognitive elements of video games within the...
The Advanced Distributed Learning (ADL) Initiative's Next Generation Learner researchers previously investigated whether five video game design features hypothesized to be contained within Portal 2 might increase cognitive adaptability (CA). Their results highlighted a lack of understanding of the cognitive elements of video games within the literature. Subsequently, a protocol for applying cognitive task analysis (CTA) to video games was developed and a CTA was performed on Portal 2 to understand the cognitive components, decisions, and knowledge needed for successful gameplay, as well as to gain a detailed understanding of its design. As a result of the CTA, a compendium of within-level tasks and puzzles the player must complete, referred to as "micro-puzzles," was compiled, and mapped to the five design features for CA.Results from the initial study showed that certain measures of CA were increased in those playing Portal 2; however, the design of Portal 2 was treated as a "black box." Through performing a CTA, the presence of the five design characteristics for adaptability was validated by location and by micro-puzzle. Although precisely identified and mapped by game location, there were no specific alignments identified between cognitive measures and micro-puzzle attributes, or between micro-puzzle typology and design feature support. For this reason, the researchers are cognitively codifying micro-puzzles in Portal 2 by type according to their measurable cognitive attributes. This involves defining the micro-puzzles and mapping them to cognitive skills, measurable by the CANTAB battery of tests for CA, followed by empirical testing in the game environment. This paper details this codification and mapping, as well as efforts to build levels in Portal 2 based upon this information in order to cultivate specific cognitive skills, empirically validate the correlation of puzzle type in-game to cognitive gains, and further validate hypothesized game design features to improve cognitive functioning.

PERLS: An Approach to Pervasive Personal Assistance in Adult Learning

2013; IITSEC; Freed, M.; Yarnall, L.; Dinger, J.; Gervasio, M.; Overholtzer, A.; Pérez-Sanagustin M.; Roschelle, J.; Spaulding, A.
Adult learners in both military and civilian settings increasingly use mobile devices for "Pervasive Learning" (Banavar et al., 2000 Thomas, 2007), which occurs without classrooms, instructors, and training facilities. By expanding options for what, when, and how we learn, Pervasive Learning has the potential to remedy stubborn deficiencies of...
Adult learners in both military and civilian settings increasingly use mobile devices for "Pervasive Learning" (Banavar et al., 2000; Thomas, 2007), which occurs without classrooms, instructors, and training facilities. By expanding options for what, when, and how we learn, Pervasive Learning has the potential to remedy stubborn deficiencies of traditional instruction. The central feature of PERLS is a virtual personal assistant that supports self-learning by recommending specific content, general topics, and various learning actions based on learners' interests, available time, and location. PERLS is intended to guide learners to resources located in both formal (closed corpus) and informal (open corpus) repositories. In this paper, we present the pedagogical design, user interface, system architecture,initial concept validation results, and field test goals for PERLS, a prototype PERvasive Learning System. The concept validation and field-testing take place in one civilian corporate context. The concept validation indicated that adult learners in the corporate setting favored limited use of "push" reminders to engage in learning and broader use of adaptive lists of content that have been intelligently informed by contextual data about their interests and available time for learning. Planned field tests will examine system functionality, usability, and impacts on self-learning habits around corporate onboarding content for new hires.
W911QY-12-C0171
PERLS

Cognitive Task Analysis: Analyzing the Cognition of Gameplay & Game Design

2013; IITSEC; Gallagher P.S.; Prestwich, S.
A prior study performed by ADL, which measured the effects of certain video game design features thought to increase cognitive adaptability, brought to light how little is understood of the cognitive elements of video games, even by those who design them, let alone those who wish to study them or utilize them for learning or improving cognitive...
A prior study performed by ADL, which measured the effects of certain video game design features thought to increase cognitive adaptability, brought to light how little is understood of the cognitive elements of video games, even by those who design them, let alone those who wish to study them or utilize them for learning or improving cognitive functioning. It has long been standard practice by instructional designers and those in industrial/organizational psychology to utilize a process known as cognitive task analysis (CTA) in order to analyze the cognitive and behavioral requirements of an expert-level performance of a certain task. They are broadly recognized as an effective tool for cognitive mapping. ADL researchers scoured the existing base of research in industrial/organizational psychology, cognitive psychology, and gaming, as well as consulted with experts in the video game industry, and found that no efforts to apply CTA methods to or cognitively map a video game - identifying and enumerating on features such as implicit and explicit rules and reinforcement, rule and environment shifts, audio and visual cues, behavioral and cognitive requirements of players, and goals, sub-goals, and micro-puzzles - in order to analyze the cognitive effects of its design could be found. It was towards the end of filling these gaps in knowledge about the cognitive makeup of Portal 2, as well as developing a methodology for applying CTA techniques to video games for future research and utilization for learning, that the researchers at the ADL Initiative are undertaking an effort to perform a cognitive task analysis on Portal 2 play. This paper details the novel methods the team has developed, both by adapting traditional CTA methods to analyze the cognitive requirements of gameplay as well as creating techniques to capture the cognitive effects of unique video game design principles, as well as initial findings.

Advanced Distributed Learning: Enabling Enhanced Learning Experiences (Slides)

2013; e-Learning Software for Education, eLSE; Regan, D.
Abstract not available.
MoTIF

Drill Evaluation for Training Procedural Skills

2013; Artificial Intelligence in Education; Myers; Gervasio, K.; Jones, M.; McIntyre, C.; Keifer, K.
The acquisition of procedural skills requires learning by doing. Ideally, a student would receive real-time assessment and feedback as he attempts practice problems designed to exercise the targeted skills. This paper describes an automated assessment and feedback capability that has been applied to training for a complex software system in...
The acquisition of procedural skills requires learning by doing. Ideally, a student would receive real-time assessment and feedback as he attempts practice problems designed to exercise the targeted skills. This paper describes an automated assessment and feedback capability that has been applied to training for a complex software system in widespread use throughout the U.S. Army. The automated assessment capability uses soft graph matching to align a trace of student actions to a predefined gold standard of allowed solutions, providing a flexible basis to evaluate student performance, identify problems, give hints, and suggest pointers to relevant tutorial documentation. Collectively, these capabilities facilitate self-directed learning of the training curriculum. ecision aiding) available on-demand, anytime and anywhere.
FA8750-09-D-0183
DEFT

Sharing Learning Content: Beyond the Technology

2012; IITSEC; Regan; Twitchell, D.; Archibald, D.; Marvin, T.
This presentation covers goals, DITIS, CORDRA and the ADL registry, content repositories, human perspective, and growing the sharing community.
This presentation covers goals, DITIS, CORDRA and the ADL registry, content repositories, human perspective, and growing the sharing community.

The Next Generation of SCORM: Innovation for the Global Force

2012; IITSEC; Poltrack, J.; Haag J.; Johsnon, A.; Hruska, N.
This presentation covers the background and requirements of SCORM. It also introduces the next learning specification that expands on the capabilities of SCORM, called Total Learning Architecture (TLA).
This presentation covers the background and requirements of SCORM. It also introduces the next learning specification that expands on the capabilities of SCORM, called Total Learning Architecture (TLA).
SCORM

Mobile Learning: Not Just Another Delivery Method

2012; IITSEC; Haag, J; Berking, P
This presentation explores the effectiveness of mobile course delivery. It covers the background, mobile learning definitions, ISD and ID Models, research goals, literature review learning theories and instructional strategies. The presentation also introduces the proposed ADL Initiative framework, along with lessons learned and future research work.
This presentation explores the effectiveness of mobile course delivery. It covers the background, mobile learning definitions, ISD and ID Models, research goals, literature review learning theories and instructional strategies. The presentation also introduces the proposed ADL Initiative framework, along with lessons learned and future research work.

Continuum of eLearning: 2012 Project Summary Report

2012; Fautua, D.; Schatz, S.; Taylor, A.; Reitz, E.; Bartlett, K.; Killilea, J.
This report outlines a variety of challenges associated with joint military training and education. It begins by detailing gaps in collective (live) training and discussing limitations associated with online learning. These limitations were uncovered through reviews of the literature, as well as interviews with military stakeholders and...
This report outlines a variety of challenges associated with joint military training and education. It begins by detailing gaps in collective (live) training and discussing limitations associated with online learning. These limitations were uncovered through reviews of the literature, as well as interviews with military stakeholders and reactions surveys from military online learners. Next, it outlines a phased approach to address the gaps and limitations of individual, team, and collective training through a Joint Event Life Cycle (JELC) optimized blended learning approach. This approach, called the Continuum of eLearning (CoL), represents both a training capability and a systematic methodology for enhancing military online learning. The report describes the CoL approach and features, highlighting those functions that were implemented in 2012 as part of Version 1.0, as well as describing future plans for V2.0 and V3.0. Finally, the report also documents the results of the beta test of CoL V1.0, which was tested during the PANAMAX 2012 joint and multinational training event. The beta test revealed that demographic variables, such as rank and service, affected overall knowledge levels; however, when these were controlled for, the CoL courses had a positive impact on personnel's knowledge and joint attitudes. The surveys that asked about participants' past and present experiences with online learning also provide a set of requirements for future versions of the CoL .

Implementation Fest: The Last Decade

2010; Murray, Kristy
ADL Vision: Provide access to the highest quality education and training, tailored to individual needs, delivered cost effectively, anywhere and anytime. Web-based Learning * Content Repositories * Distributed Simulation * Job Aiding * Immersive Learning Technologies.
ADL Vision: Provide access to the highest quality education and training, tailored to individual needs, delivered cost effectively, anywhere and anytime. Web-based Learning * Content Repositories * Distributed Simulation * Job Aiding * Immersive Learning Technologies.

Research Foundations for the Advanced Distributed Learning Initiative

2010; Institute for Defense Analysis, IDA; Fletcher, J.D.
The Advanced Distributed Learning (ADL) Initiative was established by the Office of the Undersecretary of Defense for Personnel and Readiness (OUSD(P&R)) in 1997. Its purpose was to assist the military Services in making "learning" (education, training, and performance/decision aiding) available on-demand, anytime and anywhere. "Learning" in...
The Advanced Distributed Learning (ADL) Initiative was established by the Office of the Undersecretary of Defense for Personnel and Readiness (OUSD(P&R)) in 1997. Its purpose was to assist the military Services in making "learning" (education, training, and performance/decision aiding) available on-demand, anytime and anywhere. "Learning" in ADL is used as a catch-all designator for education, training, and performance/decision aiding. "Distributed" in ADL signifies learning that can be provided in classrooms with a teacher present, in the field linking together widely dispersed instructors and students, or standing alone with no instructor other than the computer itself. "Advanced" in ADL implies affordable, interactive, adaptive, on-demand instruction delivered using computer technology so that it is available anytime, anywhere. Empirical research, available as early as the 1960s, suggested the feasibility of ADL and its goals. It has shown that: - Individualized, tutorial 'learning' (including individualized performance/decision aiding) can be provided affordably by technology-based learning. - Technology-based learning can be more effective and can produce greater return on investment than conventional instructional approaches across many instructional objectives and subject matters. - Technology-base learning allows education, training, performance aiding, and decision aiding to be delivered on platforms ranging from hand-held devices, to desk-top computers, to capabilities embedded in operational equipment. Statistical findings from this research may be summarized by a "Rule of Thirds." It states that application of technology-based learning can reduce the cost of instruction by about one-third. Additionally it can either reduce instructional time to reach instructional goals by about one-third, or increase the skills and knowledge acquired by about one-third while holding instructional time constant. The long-term vision for ADL is an extrapolation from such developments as portable, increasingly accessible computing, the global information infrastructure, modular object-oriented architectures, and natural language processing. The march toward devices that might be described as personal learning associates seems inevitable. These devices will act as personal accessories. They will respond on demand to requests for education, training, and performance aiding by assembling relevant objects from the global infrastructure and engaging the user in guided conversations to enhance the knowledge and skills and/or problem solving capabilities of individuals and/or groups of dispersed individuals whose devices are wirelessly linked together.
DASW01-04-C-0003, Task BE-2-1624
Advanced Distributed Learning Common Framework

Cost-Benefit Study of a Project to Lower Cost and Improve Fleet Readiness through Integrating the Management of Technical Information

2010; Levine, Dan
This paper describes a cost-benefit analysis by the Institute for Defense Analyses of the Bridge Project that ADL (Advanced Distributed Learning) is conducting for the Office of Secretary of Defense for Acquisition, Technology and Logistics (OSD(AT&L)) to improve the management of Integrated Logistics Support (ILS). The Project is part of the OSD...
This paper describes a cost-benefit analysis by the Institute for Defense Analyses of the Bridge Project that ADL (Advanced Distributed Learning) is conducting for the Office of Secretary of Defense for Acquisition, Technology and Logistics (OSD(AT&L)) to improve the management of Integrated Logistics Support (ILS). The Project is part of the OSD RTOC program (Reduction in Total Ownership Cost). The Bridge Project focuses on integrating (Bridging) the management and production of technical manuals and training courses. The benefits would be lower cost to produce these manuals and courses in the future, and improved readiness through insuring the delivery of consistent and up-to-date logistics support to the Fleet. Manuals and courses are currently produced by entirely separate processes. Tech writers and course developers obtain contractor data on systems and equipments in parallel they express the information in different formats, they organize the data in different structures, and they store the data in different repositories. Cost is therefore higher because of duplication of resources and the difficulties in re-using data. The lack of integration can also reduce readiness, since it opens up the possibility that the tech manuals and training courses present disparate information, thus depriving ship operators and maintainers of the most effective support. The Bridge Project seeks to relieve these problems by designing new software technical and business processes to integrate the production of technical manuals and training courses. All technical and learning content would be expressed by the same digital specification.

Immersive Environments in ADL

2009; Smith, Peter
This ADL Initiative presentation covers the potential of immersive environments by describing current beliefs, issues in the market and targeting a wide array of audiences.
POTENTIAL OF IMMERSIVE ENVIRONMENTS: Current beliefs - Better than conventional CBT - Faster than classroom - Cheaper than simulation * Issues -Currently not enough research - No standards/metrics for doing research - No agreed upon definitions. THE MARKET IS QUICKLY GROWING: Offer new ways to do old things * Learn - Experience - Motivate - Interact/Immerse - Observe. TARGET A WIDE ARRAY OF AUDIENCES: Business - Training - Education - Entertainment - Community.

ADL State of the Union. Where We Are and Where We Are Going

2009; Jesukiewicz, Paul
ADL VISION: Provide access to the highest quality education and training, tailored to individual needs, delivered cost effectively, anywhere and anytime: Web-based Learning - Content Repositories - Distributed Simulation - Job Aiding - Immersive Learning Technologies. NEW DIRECTION: Priorities * New Strategic Plan (Sep 09) - R&D (BAA) - Technical...
ADL VISION: Provide access to the highest quality education and training, tailored to individual needs, delivered cost effectively, anywhere and anytime: Web-based Learning - Content Repositories - Distributed Simulation - Job Aiding - Immersive Learning Technologies. NEW DIRECTION: Priorities * New Strategic Plan (Sep 09) - R&D (BAA) - Technical - Harmonize with global community - Extended and Innovate - Evaluation * DoD ROI (efficiency and effectiveness) data gathering and reporting - New Partnerships - Restructuring - New PM structure - Realignment of Co-Labs & Partnership Labs.

The Prospects for Increasing the Reuse of Digital Training Content

2009; Shanley, Michael; et. al.
Distributed learning (DL) offers the promise of self-paced learning and training at any time and in any place, as well as new technologies for developing and delivering content and tracking student performance. Although demand for DL is increasing, DL still represents a small percentage of all learning and training, in part because of the high cost of...
Distributed learning (DL) offers the promise of self-paced learning and training at any time and in any place, as well as new technologies for developing and delivering content and tracking student performance. Although demand for DL is increasing, DL still represents a small percentage of all learning and training, in part because of the high cost of developing and maintaining electronic-learning (e-Learning) materials. Development costs for DL might be reduced if digital content could be reused on a large scale? i.e., if existing digital content could be used to produce new content or applied to a new context or setting. One option for encouraging widespread reuse is to create and link learning object repositories? i.e., searchable databases in which digital content is stored in the form of learning objects and accessed by others to create new course content. In 2006, RAND was asked to examine how the Advanced Distributed Learning (ADL) Initiative and the Department of Defense (DoD) more broadly might encourage reuse through the use of learning object repositories and the eventual emergence of a learning object economy. The study's primary focus was on the extent to which incentives and other enablers currently are and might be used to encourage training development (TD) organizations to develop a reuse mechanism (especially the supply side of it) supported by repositories. Four key questions guided the research: To what extent are TD organizations currently engaged in reuse at this stage of technological development? 2. To what extent do organizations find reuse a worthwhile investment? 3. To what extent do disincentives to wider sharing of learning objects impede reuse? 4. To what extent do organizations know how to implement a reuse strategy?.

The Advanced Distributed Learning (ADL) Vision and Getting From Here To There

2005; Fletcher, J.D.
The Advanced Distributed Learning (ADL) initiative was undertaken by the Department of Defense (DoD) at the request of the White House Office of Science and Technology Policy (OSTP) and in cooperation with the other Federal Agencies. Its goal is to make education, training, and performance aiding accessible anytime and anywhere. This goal is being...
The Advanced Distributed Learning (ADL) initiative was undertaken by the Department of Defense (DoD) at the request of the White House Office of Science and Technology Policy (OSTP) and in cooperation with the other Federal Agencies. Its goal is to make education, training, and performance aiding accessible anytime and anywhere. This goal is being pursued through intense and frequent interactions among industry, government, and academic participants. ADL specifications are now being adopted across Europe, Asia, the Pacific Rim, and the Americas.

Utility and Applicability of the Sharable Content Object Reference Model (SCORM) Within Navy Higher Education

2004; Kohistany, M. B.; Zacharopoulos, I.Z.
This thesis critically analyzes the Sharable Content Object Reference Model (SCORM) within higher education and examines SCORM's limitations within a realistic application environment versus within a theoretical/conceptual platform. The thesis also examines environments better suited for implementation of SCORM technology. In addressing...
This thesis critically analyzes the Sharable Content Object Reference Model (SCORM) within higher education and examines SCORM's limitations within a realistic application environment versus within a theoretical/conceptual platform. The thesis also examines environments better suited for implementation of SCORM technology. In addressing the research questions, it was discovered that from the current standards set forth by Advanced Distributed Learning (ADL), SCORM is not well suited for higher education. SCORM technology will prove of greater utility within the Navy Training environment than in higher education. In their effort to share information, higher education institutions would benefit more from a Content Management System in conjunction with a Learning Management System. Subsequent chapters addressed the limitations of SCORM, provided a comparison of the applicability of SCORM within the separate domains of naval Education and Training, and provided a prototype of a Content Management System for institutions of higher learning.

Military Transformation: Progress and Challenges for DOD's Advanced Distributed Learning Programs

2003; Ensign, J; Akaka, D.K.; Hefley, J.; Ortiz, S.P.; Curtin, N.P.
The Department of Defense (DOD) spends more than $17 billion annually for military schools that offer nearly 30,000 military training courses to almost 3 million military personnel and DOD civilians, much of it to maintain readiness. To better meet the diverse defense challenges of the future, DOD is transforming its forces, including its training,...
The Department of Defense (DOD) spends more than $17 billion annually for military schools that offer nearly 30,000 military training courses to almost 3 million military personnel and DOD civilians, much of it to maintain readiness. To better meet the diverse defense challenges of the future, DOD is transforming its forces, including its training, for a post-Cold War environment that favors more rapid deployment and responsiveness. DOD's Training Transformation Strategy emphasizes the use of advanced distributed learning (ADL) programs such as Internet-based training as critical to achieving the department's training and overarching transformation goals and to deliver the highest quality training cost effectively anytime, anywhere, whether active duty, reserve, or civilian personnel. ADL is instruction that does not require an instructor's presence; can use more than one media; and emphasizes the use of reusable content, networks, and learning management systems. The authors initiated this review of DOD's ADL programs, pursuant to their basic legislative responsibilities, because of the importance DOD has placed on them as a key to achieving the department's transformation efforts. Specifically, they addressed the following questions: (1) What are DOD's expectations for the programs? (2) How is DOD managing ADL and what progress is being made in implementing the programs? and (3) What major challenges are affecting the programs' implementation? The authors did not assess the effectiveness of the programs at this time because most are in the early stages of implementation; thus, their objective was to provide a baseline document concerning the focus, status, and magnitude of DOD's ADL programs. In late August and early September 2002, GAO briefed Congress on the results of their work. This report summarizes and updates the major observations provided at those briefings. Army, Navy, and Marine Corps, Air Force, and Joint Staff ADL plans are included.

Gateway Newsletter. Volume 14. Number 1, 2003

2003; Johnston, Judith; et. al.
This newsletter contains the following articles: Transitioning Perspectives to Optimize Advanced Training Designs Future Combat Systems Manpower, Personnel, and Training (MPT): The Challenges and Paths Forward Calendar Training in the 21st Century: A Human Systems Integration Perspective and The Advanced Distributed Learning (ADL)...
This newsletter contains the following articles: Transitioning Perspectives to Optimize Advanced Training Designs; Future Combat Systems Manpower, Personnel, and Training (MPT): The Challenges and Paths Forward; Calendar; Training in the 21st Century: A Human Systems Integration Perspective; and The Advanced Distributed Learning (ADL) Initiative; Products. The Human Systems Information Analysis Center (HSIAC) is the gateway to worldwide sources of up-to-date human systems information for designers, engineers, researchers, and number factors specialists. HSIAC's primary objective is to acquire, analyze and disseminate timely information about human systems/ergonomics.

What do Sharable Instructional Objects Have to do With Intelligent Tutoring Systems, and Vice Versa?

2001; Fletcher, J D
By reviewing data on classroom and tutorial instruction, this document presents a perspective on the value of technology-based instruction in general and intelligent tutoring systems (ITSs) in particular. It tinds the mixed-initiative dialogue and on-demand, real-time generative capabilities to be defining functionalities of ITSs. These...
By reviewing data on classroom and tutorial instruction, this document presents a perspective on the value of technology-based instruction in general and intelligent tutoring systems (ITSs) in particular. It tinds the mixed-initiative dialogue and on-demand, real-time generative capabilities to be defining functionalities of ITSs. These functionalities are motivated by basic research into human learning and cognition. From this perspective, development ot the Advanced Distributed Learning (ADL) initiative and ITSs are viewed as parallel but presently independent activities. This document describes and discusses their common interests in the development and availability of (accessible, interoperable, durable, and reusable) instructional objects on the World Wide Web. These sharable instructional objects can act as either instructional content or algorithmic agents. Sharable instructional objects are likely to reduce the costs and increase the effectiveness of ADL technologies and ITSs. Development of these objects should be cooperatively promoted and pursued by the ADL and ITS communities. Doing so may lead to the development and wide use of personal learning associates, which would allow high- quality instruction and decision-aiding to become ubiquitous and attordable.

Theoretical Foundation for Advanced Distributed Learning Research

2001; Hays, Robert T
Advanced Distributed Learning (ADL) has been embraced by the Department of Defense, other Government agencies, private industry, and academia as a tool that has the potential to improve the efficiency and effectiveness of modem instructional systems. As with any other instructional approach, ADL cannot achieve its full potential if the individuals...
Advanced Distributed Learning (ADL) has been embraced by the Department of Defense, other Government agencies, private industry, and academia as a tool that has the potential to improve the efficiency and effectiveness of modem instructional systems. As with any other instructional approach, ADL cannot achieve its full potential if the individuals and organizations involved in its development fail to understand that ADL systems are controlled and constrained by system principles. The goal of this paper is to sensitize individuals working in all aspects of ADL systems to the power of a system view and to provide several examples of system methods and tools. The paper presents a theoretical foundation for an ADL research agenda based upon an overview of system concepts and discussion of how they apply to the development and implementation of ADL systems. It argues that the adoption of a system perspective will improve the communication and understanding among the diverse groups engaged in ADL system development and can help ADL mature into an effective instructional alternative. A number of ADL research issues are discussed and several system-oriented methodologies are suggested to address these issues.

The Impact of the DoD Mobile Code Policy on Advanced Distributed Learning, Web-Based Distance Learning and Other Educational Missions

2001; Halloran, Margaret; et. al.
Mobile code is software that downloads via the Internet and runs on users' workstations without the users' knowledge. Mobile code can be both potentially beneficial and harmful to systems and networks in the Department of Defense (DoD). The DoD Memorandum, dated 7 November 2000, Policy Guidance for Use of Mobile Code Technologies in Department of...
Mobile code is software that downloads via the Internet and runs on users' workstations without the users' knowledge. Mobile code can be both potentially beneficial and harmful to systems and networks in the Department of Defense (DoD). The DoD Memorandum, dated 7 November 2000, Policy Guidance for Use of Mobile Code Technologies in Department of Defense (DoD) Information Systems establishes DoD wide policy on the use of mobile code in DoD information systems and computers. The memorandum, also referred to as the DoD Mobile Code Policy or just the Policy, defines mobile code and mobile code technologies as follows: Mobile code is defined (for the purposes of the Policy) as software obtained from systems outside the enclave boundary, transferred across a network, downloaded and executed on a local system (e.g., a computer with a Web browser) without explicit installation or execution by the recipient. Mobile code technologies are software technologies that provide the mechanisms for the production and use of mobile code (e.g., Sun Microsystems' Java and JavaScript; Microsoft Corporation's VBScript and ActiveX). Many of the interactive components of web-based distance learning and Advanced Distributed Learning (ADL) are programmed using mobile code and mobile code technologies. In addition, distance learning and ADL often requires access to materials outside of the enclave as defined by the DoD Mobile Code Policy, as the course content or assignments of this instruction requires users to access web sites that are not developed and/or maintained by personnel within the DoD. For these reasons, ADL and web-based distance learning programs may be more sensitive to the DoD Mobile Code Policy than other base operations and a thorough test of the effect of the implementation of the DoD Mobile Code Policy is warranted.

Sharable Courseware Object Reference Model (SCORM), Version 1.0

2000; Ball, R; Burke, R; Fletcher, D; Hoberney, A; Jesukiewicz, P
The Department of Defense (DoD), in cooperation with the Office of Science and Technology, established the Advanced Distributed Learning (ADL) Initiative to develop a strategy and process for using learning and information technologies together to modernize education and training in the DoD and across federal agencies. The ADL initiative has...
The Department of Defense (DoD), in cooperation with the Office of Science and Technology, established the Advanced Distributed Learning (ADL) Initiative to develop a strategy and process for using learning and information technologies together to modernize education and training in the DoD and across federal agencies. The ADL initiative has defined high-level requirements for learning content or learning content objects. These requirements include accessibility, reusability, durability, and interoperability. To meet these requirements, content objects must be developed in accord with specifications that are widely agreed upon, accepted and used. The Sharable Courseware Object Reference Model (SCORM) was developed to articulate clear and implementable specifications for the development of Web-based learning content objects. Version 1.0 addresses three major issues. Course Structure Format, Runtime Environment, and Metadata. Later versions will address these and other issues needed to develop the specification.
SCORM

Department of Defense Strategic Plan for Advanced Distributed Learning

1999; ODUSD(R), Readiness and Training
Section 378 of Public Law 105-261, the Strom Thurmond National Defense Authorization Act for Fiscal Year 1999 (hereafter, The Act), requires the Secretary of Defense to develop a strategic plan for guiding and expanding distance learning initiatives within the Department of Defense, to include a provision for the expansion of such initiatives over...
Section 378 of Public Law 105-261, the Strom Thurmond National Defense Authorization Act for Fiscal Year 1999 (hereafter, The Act), requires the Secretary of Defense to develop a strategic plan for guiding and expanding distance learning initiatives within the Department of Defense, to include a provision for the expansion of such initiatives over five consecutive years, beginning in FY 2000 (Appendix A). Section 378 states that the Secretary may take into account the ongoing collaborative effort, between the Department of Defense, other Federal agencies, and private industry, that is known as the Advanced Distribut[ed] Learning Initiative, but that the strategic plan must specifically focus on the education and training goals and objectives of the Department of Defense.


Results per page
10
25
50