Identifying Gaps in Students’ Explanations of Code Using LLMs
Abstract
This study investigates methods based on Large Language Models (LLMs) to identify gaps or missing parts in learners’ self-explanations. This work is part of our broader effort to automate the evaluation of students’ freely generated responses, which in this work are learners’ self-explanations of code examples during code comprehension activities. We experimented with two methods and four distinct LLMs in two distinct settings. One method prompts LLMs to identify gaps in learners’ self-explanations, whereas the other method relies on LLMs performing a sentence-level semantic similarity task to identify gaps. We evaluated these methods in two settings: (i) simulated data generated using LLMs and (ii) actual student data. Results revealed the semantic similarity method significantly improves task performance over the zero-shot prompting for gap identification (the holistic method), i.e., over the standard method of prompting LLMs to directly address the gap identification task.
Publication Title
Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Recommended Citation
Banjade, R., Oli, P., Sajib, M., & Rus, V. (2024). Identifying Gaps in Students’ Explanations of Code Using LLMs. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 14830 LNAI, 268-275. https://doi.org/10.1007/978-3-031-64299-9_21
