CS Events

PhD Defense

AUTOMATED FEEDBACK GENERATION FOR PROGRAMMING ASSIGNMENTS

 

Download as iCal file

Monday, August 23, 2021, 10:00am - 12:00pm

 

Speaker: Georgiana Haldeman

Location : Remote via Zoom

Committee

Thu D. Nguyen (thesis chair/director)

He Zhu (co-advisor)

Ulrich Kremer

Craig Zilles (external member)

Event Type: PhD Defense

Abstract: Autograding systems are being increasingly deployed to meet the challenges of teaching programming at scale. Studies show that formative feedback can greatly help novices learn programming. This work extends an autograder, enabling it to provide corrective and formative feedback on programming assignment submissions using a mixed-approach. First methodology starts with the design of a knowledge map, which is the set of concepts and skills that are necessary to complete an assignment, followed by the design of the assignment and that of a comprehensive test suite for identifying logical errors in the submitted code. Test cases are used to test the student submissions and learn classes of common errors. For each assignment, I train a classifier that automatically categorizes errors in a submission based on the outcome of the test suite. The instructor maps the errors to corresponding concepts and skills and writes hints to help students find their misconceptions and mistakes. I apply this methodology to two assignments in the Introduction to Computer Science course and find that the automatic error categorization has a 90% average accuracy. I report and compare data from two semesters, one semester when hints are given for the two assignments and one when hints are not given. Results show that the percentage of students who successfully complete the assignments after an initial erroneous submission is three times greater when hints are given compared to when hints are not given. However, on average, even when hints are provided, almost half of the students fail to correct their code so that it passes all the test cases. Using insights about student errors gained through applying the first approach, the second approach was designed to repair student programs and provide feedback on how to correct the program and on associated misconceptions. I start by compiling a set of programming repairs that are commonly needed to repair student programs across assignments. The repair set consists of modifications to both the statements and the program structure. These repairs fix errors that point to misconceptions about program structuring and other programming concepts. Each repair is used to modify the behavior of the student program until it matches the expected behavior, measured using a similar test suite as the first approach. Additionally, I provide a search procedure for finding a set of repairs that correct the behavior of a student program. I apply this approach to multiple assignments from the Introduction to Computer Science course and report a higher success rate than with previous approaches, as high as 20% of the attempted student programs for specific assignments. Manual analysis of the repaired programs reveals that the repairs do not introduce stylistic issues such as dead code. However, for some assignments this approach is successful for less than half of the attempted student programs. More research is needed to understand how to provide accurate automated feedback to all student programs and for all types of assignments, and how this type of feedback specifically impacts learning and teaching.

 

Join Zoom Meeting
https://rutgers.zoom.us/j/93558091324?pwd=V044UW9TSHoxMTNEZzVnc1l3WllqQT09

Join by SIP


Meeting ID: 935 5809 1324
Password: 282390
One tap mobile
+16465588656,,93558091324# US (New York)
+13017158592,,93558091324# US (Washington DC)

Join By Phone
+1 646 558 8656 US (New York)
+1 301 715 8592 US (Washington DC)
+1 312 626 6799 US (Chicago)
+1 669 900 9128 US (San Jose)
+1 253 215 8782 US (Tacoma)
+1 346 248 7799 US (Houston)
Meeting ID: 935 5809 1324
Find your local number: https://rutgers.zoom.us/u/adsFknXH5

Join by Skype for Business
https://rutgers.zoom.us/skype/93558091324