Learning ecosystems frequently blend human and digital forms of instruction and assessment. However, teachers and tutors, while effective, are costly and scarce relative to automated forms of instruction, creating a need to make most efficient use of the teacher’s time. We describe results from the ADL-funded Integrating Human and Automated Tutoring Systems (IHATS) project, which used learning analytics and machine learning techniques to analyze a large learning dataset, including tens of millions of student actions in an AI-driven tutoring system for math called MATHia along with chat transcripts between MATHia users and human tutors to whom they could turn for assistance. IHATS identifies what drives learners to seek out human tutoring while using an automated system as well as what characterizes successful interactions with human tutors. Our analysis includes measures of both cognitive and non-cognitive factors, relying on measures of errors, hint seeking, and skill mastery within MATHia as well as data-driven “detector” models of behaviors like gaming the system and affective states like boredom, confusion, and frustration. These constructs are analyzed in tandem with machine annotations of chat transcripts for various dialogue acts and modes. We emphasize the idea that learners seek the tutor’s help for both cognitive (i.e., informational) and non- cognitive (i.e., affective and affirmational) reasons, provide analytics to illustrate this point, and describe how this rich, large dataset can be used by researchers to develop and demonstrate xAPI and other facets of the Total Learning Architecture (TLA) to better support blended learning.
Stephen E. Fancsali, Ph.D., is a Research Scientist at Carnegie Learning, Inc., where he focuses on learning analytics and educational data mining projects involving Carnegie Learning’s intelligent tutoring systems and blended learning solutions for mathematics, including MATHia in the K-12 market and Mika for higher education. His research interests include algorithmic causal inference and discovery from observational data sets, especially applications in domains like education, developing interpretable predictive analytics for end users like teachers and students, and data-driven improvement of cognitive skill models used in intelligent tutoring systems. He received a Ph.D. in Logic, Computation, and Methodology from Carnegie Mellon University.