Understanding effective math learning strategies can help Intelligent Tutoring Systems (ITSs) adapt to a student’s problem-solving strategy. This can lead to improved learning gains, engagement, and motivation. However, while ITSs may be designed to teach several alternative math learning strategies, it is often difficult to analyze whether learners can internalize these strategies and execute them effectively in varied contexts. In particular, when scaling up such analyses across large groups of learners, the task becomes even more complex.
The growth of Artificial Intelligence (AI) methods and tools offers unique opportunities to analyze math learning strategies at a large scale. In this work, we present an approach we call ASTRA (AI-based Strategy Analysis), in which we use state-of-the-art methods in AI representation learning to discover hidden structure in ITS data and analyze math strategies. In particular, we show that we can adapt methods that have revolutionized language understanding, such as BERT (Bidirectional Encoder Representations from Transformers) models, to learn representations for math learning strategies.
We train the AI models using large-scale data involving several thousand learners working on 6th- and 7th-grade math, collected from Carnegie Learning’s MATHia platform. To do this, we identify specific topics, also called workspaces, within MATHia where the design allows students to execute multiple strategies to solve a problem. While “math strategies” is indeed a broad term, in this work, we define it more precisely based on sequences of actions performed by learners.
Thus, by observing sequences of actions performed by students, we develop an approach to pre-train the BERT model in an unsupervised manner, without requiring external labels. The model learns representations, also called embeddings, of the strategies students follow when solving problems in that workspace. The objective of pre-training is to understand hidden structure within the sequences of actions, which correspond to a strategy, by looking for patterns across a large amount of data.
Using the pre-trained embeddings, we explore several downstream tasks that can help improve the design of the ITS. Specifically, the ITS can use predictions or insights from the AI model to help address misconceptions and nudge students toward more effective strategies. To do this, we develop learning methods that fine-tune the embeddings for different tasks, such as: i) identifying correct strategies, ii) analyzing the effectiveness of strategies, and iii) understanding how strategies are learned over time.
We present quantitative and qualitative results from our studies to evaluate the efficacy of our approach. Finally, we conclude with what we learned about the feasibility and limitations of AI methods in understanding math strategies at scale.