Iterative Repair of Social Robot Programs from Implicit User Feedback via Bayesian Inference

Creating natural and autonomous interactions with social robots requires rich, multi-modal sensory input from the user. Writing interactive robot programs that make use of this input can demand tedious and error-prone tuning of program parameters, such as tuning thresholds on noisy sensory streams for detecting whether the robot's user is engaged or not. This tuning process dealing with low-level streams and parameters makes programming of social robots time-consuming and inaccessible for people who could benefit the most from unique use cases of social robots. To address this challenge, we propose the use of iterative program repair, where programmers create an initial program sketch in our new Social Robot Program Transition Sketch Language (SoRTSketch), a domain-specific language that supports expressing uncertainties related to thresholds in transition functions. The program is then iteratively repaired using Bayesian inference based on corrections of interaction traces that are either provided by the programmer or derived from implicit feedback given by the user during the interaction. Based on experiments with a human simulator and with 10 human users, we demonstrate the ease and effectiveness of this approach in improving social robot programming and program outputs that represent three common human-robot interaction patterns.

Virtually presented at Robotics: Science and Systems 2020 (https://roboticsconference.org/)

For details, see https://roboticsconference.org/program/papers/28/

Transcript

1. Robotics: Science and Systems, July 12-16, 2020 Iterative Repair of Social Robot Programs from Implicit User Feedback via Bayesian Inference Michael Jae-Yoon Chung and Maya Cakmak
 University of Washington, Computer Science & Engineering
2. Photo: USC Robotics Research Lab Photo: MIT Media Lab/Personal Robots Group Photo: Yale Social Robotics Lab
3. Photo: USC Robotics Research Lab Photo: Yale Social Robotics Lab Photo: MIT Media Lab/Personal Robots Group Photo: MIT Media Lab/Personal Robots Group Photo: MIT Media Lab/Personal Robots Group Photo: MIT Media Lab/Personal Robots Group
4. Research Goal Facilitate fluent and autonomous social robot behavior authoring with an expressive yet maintainable approach
6. program sketch 1 lookingAtRobot = debounce(abs(faceYawAngle < ??minDisengagedAngle), ??disengagedTimeout); ... if (curState == "read" && sayFinished && humanAvailable && !allDone) { return "read"; } else if (curState == "read" && (sayFinished && humanBusy && !allDone || goBackTapped)) { return "wait"; } else if (curState == "read" && sayFinished && allDone) { return "stop"; } else if (curState == "wait" && (humanAvailable || nextTapped && !hold))) { return "read"; } Hole variable name Distribution ??minDisengagedAngle Categorical ??disengagedTimeout Categorical ... ...
7. Interaction trace program sketch 1 interaction0 2 Note: the robot in the video is not running the example program
8. program sketch 1 interaction0 2 interaction trace 3 correction0 4 600 700 800 900 1000 1100 1200 1300 1400 1500 1600 1700 1800 1900 500 say wait read wait goBackTapped Time (ms) inputs outputs state trace wait corrected state trace lookingAtRobot Error Correction
9. 600 700 800 900 1000 1100 1200 1300 1400 1500 1600 1700 1800 1900 500 say wait wait read wait goBackTapped Time (ms) inputs outputs state trace corrected state trace lookingAtRobot Error Correction User feedback program sketch 1 interaction0 2 interaction trace 3 correction0 4
10. program sketch 1 interaction0 2 interaction trace 3 correction0 4 (repaired) Interaction0 5 ... ... ... ... ... ... At each iteration, do 1. Run the program and collect an interaction trace 2. Obtain corrections from the programmer or the user 3. Re-estimate hole distributions using the input and corrected state traces (posterior) 4. Fill-in holes using maximum a posteriori estimation (MAP) for the next iteration
11. • Human simulators for sampling input and ground truth state trace • Baseline: search-based repair • 3 scenarios, 5 repair iterations • Measures • % overlap (interaction quality) • Repair time in ms (speed) A top-down human-simulator. Samples from states, substates (intent), and then inputs Simulation Experiment: Setup
12. 0 1 2 3 4 5 repair iterations 0.0 0.2 0.4 0.6 0.8 1.0 1.2 overlap % Storytelling 0 1 2 3 4 5 repair iterations 0.0 0.2 0.4 0.6 0.8 1.0 1.2 Neck Exercise 0 1 2 3 4 5 repair iterations 0.0 0.2 0.4 0.6 0.8 1.0 1.2 Open-ended Q&A bayesian search method Simulation Experiment: % Overlap Results
13. 0 1 2 3 4 5 repair iterations 0.0 0.2 0.4 0.6 0.8 1.0 1.2 overlap % Storytelling 0 1 2 3 4 5 repair iterations 0.0 0.2 0.4 0.6 0.8 1.0 1.2 Neck Exercise 0 1 2 3 4 5 repair iterations 0.0 0.2 0.4 0.6 0.8 1.0 1.2 Open-ended Q&A bayesian search method Simulation Experiment: Speed Results 1 2 3 4 5 repair iterations 0 2,000 4,000 6,000 8,000 10,000 12,000 time (ms) Storytelling 1 2 3 4 5 repair iterations 0 1,000 2,000 3,000 4,000 Neck Exercise 1 2 3 4 5 repair iterations 0 20,000 40,000 60,000 80,000 100,000 120,000 Open-ended Q&A bayesian search method
14. • 10 participants • 7 males, 23.2 yrs old on average • A custom-built social robot • 2 scenarios, 4 iterations Human experiment setup. Human Experiment: Setup
15. Qualitative Results: Before Repair
16. Qualitative Results: After Repair
17. More Results and Discussions • Simulation experiments • Noisy corrections study • Human experiments • Quantitative results • Findings from data analysis • Robot behavior adaptation experiment • Discussions and future direction Thank you!