Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Enabling End-Users to Create Interactive Real-World Robot Applications through Novel Interfaces and Automation

Enabling End-Users to Create Interactive Real-World Robot Applications through Novel Interfaces and Automation

Interactive robots in human environments have the potential to change our daily lives by providing education, eldercare, entertainment services, and more. However, creating new applications and behaviors for such robots is complex for many reasons, including the challenges of prototyping and testing. This thesis presents our research on designing and evaluating an information-gathering application for a mobile robot in a university venue and exploring the use of robots for gathering customer feedback in the hospitality industry. Our goal is to enable non-experts, such as salespeople and designers, to create and explore new robot applications without requiring additional technical expertise. To that end, we share our experiences of iteratively adapting a robot programming system for creating socially interactive robots in commercial spaces. Based on these experiences, we identified the ability to \textit{express concurrency} and \textit{modify low-level behaviors with ease }as two important requirements for interactive robot programming systems. We categorize common concurrency patterns and present a framework for identifying programming constructs required to express concurrency; we then implemented and investigated three representative concurrency interfaces for a block-based visual robot programming system and assessed them with a systematic evaluation and a user study. To modify low-level behaviors, we present an iterative robot program repair approach that enables programmers or the robot's users to easily change program details that govern a robot's low-level behaviors to improve interaction quality; we evaluated our approach with both simulation and human experiments. Throughout the document, we discuss our findings, results, insights, and the limitations of our approaches. We conclude the thesis by envisioning directions for future work.

Talk video: https://youtu.be/pTml6yEIjcw
Talk narrative: https://docs.google.com/document/d/14xNVnrfuxXqaBYvQ9KOEsc8S7fl9ScsQ1XLLBJiKlgU
Thesis paper: https://mjyc.github.io/assets/pdfs/PhD%20Thesis%20-%20Michael%20Jae-Yoon%20Chung.pdf

Mike Chung

July 30, 2020
Tweet

More Decks by Mike Chung

Other Decks in Research

Transcript

  1. Michael Jae-Yoon Chung, 07/30/2020 Enabling End-Users to Create Interactive Real-World

    Robot Applications through Novel Interfaces and Automation University of Washington, Computer Science & Engineering Ph.D. Final Examination 1
  2. Thank You! • HCR Lab • Savioke • RSE Lab

    • Neural Systems Lab • CSE Staff • Family and Friends 2
  3. 3

  4. Challenges • Challenge 1: Need for interactive behaviors and content,

    involving programmers is not scalable ➡ End-User Programming (EUP) • Challenge 2: Need for adaptation to unique users and environment, but EUP not sufficiently expressive ➡ Automatic Program Repair 4
  5. Research Goal To facilitate creating applications and behaviors for interactive

    robots with human-centered design and programming systems for non-experts. 5
  6. Outline 1. Information Gathering Robots in Commercial Spaces
 (IROS2015&2016, ROMAN2018)

    2. Programming System for Interactive Service Robots
 (ICSR2016) 3. Concurrency in Programming Interactive Robots
 (ROMAN2020) 4. Iterative Program Repair with Implicit User Feedback
 (RSS2020) 6
  7. Outline 1. Information Gathering Robots in Commercial Spaces
 (IROS2015&2016, ROMAN2018)

    2. Programming System for Interactive Service Robots
 (ICSR2016) 3. Concurrency in Programming Interactive Robots
 (ROMAN2020) 4. Iterative Program Repair with Implicit User Feedback
 (RSS2020) share our experiences 7
  8. Outline 1. Information Gathering Robots in Commercial Spaces
 (IROS2015&2016, ROMAN2018)

    2. Programming System for Interactive Service Robots
 (ICSR2016) 3. Concurrency in Programming Interactive Robots
 (ROMAN2020) 4. Iterative Program Repair with Implicit User Feedback
 (RSS2020) address the two main problems 8
  9. Outline 1. Information Gathering Robots in Commercial Spaces
 (IROS2015&2016, ROMAN2018)

    2. Programming System for Interactive Service Robots
 (ICSR2016) 3. Concurrency in Programming Interactive Robots
 (ROMAN2020) 4. Iterative Program Repair with Implicit User Feedback
 (RSS2020) [IROS 2015] Michael Jae-Yoon Chung, Andrzej Pronobis, Maya Cakmak, Dieter Fox, Rajesh P.N. Rao. Designing Information Gathering Robots for Human- Populated Environments. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2015. [IROS 2016] Michael Jae-Yoon Chung, Andrzej Pronobis, Maya Cakmak, Dieter Fox, Rajesh P.N. Rao. Autonomous Question Answering with Mobile Robots in Human-Populated Environments. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2016. [RO-MAN 2018] Michael Jae-Yoon Chung, Maya Cakmak. How was your stay?: Exploring the Use of Robots for Gathering Customer Feedback in the Hospitality Industry. IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), 2018. 9
  10. Formative Studies • To Identify & Gather Requirements • Need

    Finding Interviews,
 Online User Surveys Deployment Evaluations • To Test Feasibility in Context • To Extract Design Implications • Long-term (max. 129days) Image source: https://www.ihg.com/crowneplaza/hotels/gb/en/milpitas/sfomp/hoteldetail/photos-tours 13
  11. Findings: Requirements Hotels care about • catching unhappy 
 guests

    on-site • spreading good reviews Guests care about • convenience • privacy 14
  12. Findings: Design Implications • keep interactions short • use movement

    judiciously • tailor interactions to target group 15
  13. Outline 1. Information Gathering Robot Applications in Commercial Spaces (IROS2015&2016,

    ROMAN2018) 2. A Programming System for Interactive Service Robots (ICSR2016) 3. Concurrency Interfaces for Programming Interactive Robots (ROMAN2020) 4. Iterative Repair of Robot Programs with Implicit User Feedback (RSS2020) [ICSR 2016] Michael Jae-Yoon Chung, Justin Huang, Leila Takayama, Tessa Lau, Maya Cakmak. Iterative Design of a System for Programming Socially Interactive Service Robots. International Conference on Social Robotics (ICSR), 2016. 16
  14. Motivation Information Gathering • Long-tailed Question Types Customer Feedback •

    Needs for Customized Contents • for each guest group, e.g., • tourists • business travelers 17
  15. Research Question 
 “How should we design a programming system


    to support creating interactive behaviors for service robots?” • What are the programming requirements? • What are the challenges for users? 18
  16. Descriptions of 2/5 locations involved in the formative study 1.

    Understand/Prepare The Key Finding:
 Most desired use cases involved interacting with visitors 20
  17. 1. Understand/Prepare CustomPrograms (Huang et al., HRI2016) Extended CustomPrograms •

    Block-based Visual Programming • Battletested userability 
 for non-experts 21
  18. Prototype/Evaluate/Refine #1 A picture from the deployment • Prototyped 2

    apps with airport staff members • Deployed at an airport
 4 times (3-4hrs each) over 2 weeks • Collected systems logs and observation notes 22
  19. Prototype/Evaluate/Refine #2 • Airport staff members led programming • Deployed

    at an airport
 on 2 weekends
 (Fri/Sat/Sun, 9am-5pm) • Focused observing
 the non-experts,
 e.g., staff members • Made small updates 23
  20. Key Extentions Name Returns getDetectedPeople() List enable more proactive behaviors

    Name Returns goToUntil(string location, string eventName) bool enable more reactive behaviors enable more expressive behaviors Perception primitives Interruptible primitives Asynchonous primitives Name Returns startSound(string sound) void 24
  21. Findings: Requirements • Expressing Concurrency • for creating rich interactions

    (with ease) • Modifying Low-level Behaviors • to create natural and robust interactions • We’ll revisit these soon… 25
  22. Findings: Challenges for Non-Experts • Programming from 
 Scratch and

    Debugging • Writing Readable and 
 Re-usable Code • Creating Engaging Behaviors A program used in the field 26
  23. Outline 1. Information Gathering Robots in Commercial Spaces
 (IROS2015&2016, ROMAN2018)

    2. Programming System for Interactive Service Robots
 (ICSR2016) 3. Concurrency in Programming Interactive Robots
 (ROMAN2020) 4. Iterative Program Repair with Implicit User Feedback
 (RSS2020) [RO-MAN 2020] Michael Jae-Yoon Chung, Maya Cakmak. IConCodeIt! A Comparison of Concurrency Interfaces in Block-based Visual Robot Programming. IEEE International Symposium on Robot and Human Interactive Communication (RO- MAN), 2020. 27
  24. Reviewing the Extentsions “Go to ‘Lobby 1’ WHILE waiting for

    a screen touch event” start “Start sound AND shimmy in PARALLEL” 28
  25. Research Question 
 “How should we design a concurrency interface

    for block-based visual programming for interactive robots?” 29
  26. Block-Based Visual Robot Programming • Why? • End-users’ familiarity •

    Widely-used • Assumptions • Visual Programming Interface + Robot Actions • Wraps a procedural scripting language, e.g., python ABB's Wizard Easy Programming
 is based off Blockly. | Credit: ABB 30
  27. 31

  28. Robot APIs: Actions Name say(string text) gesture(string expression) displayText(string text,

    number duration) displayText(string[] choices, number duration) sleep(number duration) Name startSaying(string text) startGesture(string expression) startDisplayText(string text, number duration) startDisplayText(string[] choices, number duration) startSleep(number duration) Blocking Non-Blocking 32
  29. Robot APIs: Events Event Name Value Type speechDetected string buttonPressed

    string Event Name Value Type sayDone boolean gestureDone boolean displayTextDone boolean displayButtonDone boolean sleepDone boolean Externally
 Triggered Action-
 Triggered 33
  30. Robot APIs: Events Event Name Last Event as State Name

    Value Type speechDetected lastDetectedSpeech string buttonPressed lastPressedButton string Event Name Action Status State Name Value Type sayDone isSaying boolean gestureDone isGesturing boolean displayTextDone isDisplayingText boolean displayButtonDone isDisplayingButton boolean sleepDone isSleeping boolean Externally
 Triggered Action-
 Triggered States 35
  31. Robot APIs: Events // register a callback when("speechDetected", function() {...})

    // block until event waitFor("speechDetected") // read a current state value getLastEventValue("speechDetected") 36
  32. Three Interfaces Async • asynchronous procedure calls • imperative
 paradigm

    • used in e.g., socket programming Callback • callback • event-driven paradigm • used in, e.g., UI programming Waitfor • promise-based event synchronization • imperative 
 paradigm • e.g., async/await in event-loop 37
  33. Three Interfaces Name Blocking actions Non-
 Blocking actions Blocking event

    (wait) callback
 (when) State
 accessors async ❌ ✅ ❌ ❌ ✅ callback ❌ ✅ ❌ ✅ ✅ waitfor ✅ ❌ ✅ ❌ ✅ 38
  34. Example Robot Behavior Step 1: The robot should 
 say

    “Hello there, my name is Meebo. Goodbye now!” 
 and wait for the human to say something. 
 Step 2: When the robot finished speaking 
 or the human says something,
 the robot should display “On standby.” 39
  35. Concurrency Patterns Event Combinations • Action & Action • Action

    & Events • Events & Events Event Compositions • Wait-for-all • Wait-for-one 43
  36. Example Robot Behavior Step 1: The robot should 
 say

    “Hello there, my name is Meebo. Goodbye now!” 
 and wait for the human to say something. 
 Step 2: When the robot finished speaking 
 or the human says something,
 the robot should display “On standby.” Action Wait-for-One Event: speechDetected Combination: Action & Event
 Composition: Wait-for-One 44
  37. User Study: Procedure • Conducted Online • Between-group design •

    Participants • (N = 21, 14 females) • with some 
 programming experiences • Study duration: max. 90 min 45
  38. User Study: Measures • Measure: Ease-of-Use • Scores using a

    rubric
 for Correctness • Number of blocks
 for Conciseness 46
  39. Task Description • Step 1: On start, the robot should


    display “Press or say ‘start’ to begin interaction”
 and display a button with “start” text. • Step 2: 
 If the user presses the button or says “start,” 
 then the robot should introduce itself by 
 saying “Hello, my name is Meebo”
 and make a happy gesture. • Step 3: … • Step 4: … Action Event Wait-for-One Action Action Wait-for-All 48
  40. Results: Correctness 0 5 10 15 Average Score async callback

    waitfor (most correct) waifor > async > callback (least correct) 49
  41. (most concise) waifor < async < callback (least concise) 0

    20 40 60 80 100 120 Average Number of Blocks async callback waitfor async callback -2 0 Ave async callback waitfor async callback Results: Conciseness 50
  42. Summary 1. Robot API Categorization • with concurrency in mind

    2. Three Representative Interfaces • async, callback, waitfor 3. Comparison Study • systematic evaluation (not presented) • user study 52
  43. Outline 1. Information Gathering Robot Applications in Commercial Spaces (IROS2015&2016,

    ROMAN2018) 2. A Programming System for Interactive Service Robots (ICSR2016) 3. Concurrency Interfaces for Programming Interactive Robots (ROMAN2020) 4. Iterative Repair of Robot Programs with Implicit User Feedback (RSS2020) [RSS 2020] Michael Jae-Yoon Chung, Maya Cakmak. Iterative Repair of Social Robot Programs from Implicit User Feedback via Bayesian Inference. Robotics: Science and Systems (RSS), 2020. 53
  44. Motivation • Non-programmers need to modify low-level robot behaviors •

    Critical for the quality of interactions • But exposing low-level APIs is not an option • Parameter tuning is hard even for programmers • Uncertain limits, scale, sensitivity, consequences, … • Requires human users to test 54
  45. 56

  46. 57

  47. program sketch, K, Θ 1 lookingAtRobot = debounce(
 abs(inputFaceYawAngle <

    ??minDisengagedAngle),
 ??disengagedTimeout); speaking = ... if (state == "read" && inputSayFinished 
 && (lookingAtRobot || speaking)) { return "wait"; } ... Hole variable name Distribution ??minDisengagedAngle Categorical ??disengagedTimeout Categorical ... ... 58
  48. Interaction trace 1 interaction0
 I0, O0 2 Note: the robot

    in the video is not running the example program program sketch, K, Θ 59
  49. 1 2 600 700 800 900 1000 1100 1200 1300

    1400 1500 1600 1700 1800 1900 500 say wait read wait goBackTapped Time (ms) inputs outputs state trace wait corrected state trace lookingAtRobot Error Correction program sketch, K, Θ interaction0
 I0, O0 corrected0
 b Oi <latexit sha1_base64="yZtHgQaDiOpztI9WGGMk2koDamQ=">AAAB9HicbVBNS8NAEJ34WetX1aOXxSJ4KokIeix68WYF+wFtKJvNtl262cTdSaWE/g4vHhTx6o/x5r9x2+agrQ8GHu/NMDMvSKQw6Lrfzsrq2vrGZmGruL2zu7dfOjhsmDjVjNdZLGPdCqjhUiheR4GStxLNaRRI3gyGN1O/OeLaiFg94DjhfkT7SvQEo2glv/MkQj6gmN1NuqJbKrsVdwayTLyclCFHrVv66oQxSyOukElqTNtzE/QzqlEwySfFTmp4QtmQ9nnbUkUjbvxsdvSEnFolJL1Y21JIZurviYxGxoyjwHZGFAdm0ZuK/3ntFHtXfiZUkiJXbL6ol0qCMZkmQEKhOUM5toQyLeythA2opgxtTkUbgrf48jJpnFc8t+LdX5Sr13kcBTiGEzgDDy6hCrdQgzoweIRneIU3Z+S8OO/Ox7x1xclnjuAPnM8fGq6STQ==</latexit> <latexit sha1_base64="yZtHgQaDiOpztI9WGGMk2koDamQ=">AAAB9HicbVBNS8NAEJ34WetX1aOXxSJ4KokIeix68WYF+wFtKJvNtl262cTdSaWE/g4vHhTx6o/x5r9x2+agrQ8GHu/NMDMvSKQw6Lrfzsrq2vrGZmGruL2zu7dfOjhsmDjVjNdZLGPdCqjhUiheR4GStxLNaRRI3gyGN1O/OeLaiFg94DjhfkT7SvQEo2glv/MkQj6gmN1NuqJbKrsVdwayTLyclCFHrVv66oQxSyOukElqTNtzE/QzqlEwySfFTmp4QtmQ9nnbUkUjbvxsdvSEnFolJL1Y21JIZurviYxGxoyjwHZGFAdm0ZuK/3ntFHtXfiZUkiJXbL6ol0qCMZkmQEKhOUM5toQyLeythA2opgxtTkUbgrf48jJpnFc8t+LdX5Sr13kcBTiGEzgDDy6hCrdQgzoweIRneIU3Z+S8OO/Ox7x1xclnjuAPnM8fGq6STQ==</latexit> <latexit sha1_base64="yZtHgQaDiOpztI9WGGMk2koDamQ=">AAAB9HicbVBNS8NAEJ34WetX1aOXxSJ4KokIeix68WYF+wFtKJvNtl262cTdSaWE/g4vHhTx6o/x5r9x2+agrQ8GHu/NMDMvSKQw6Lrfzsrq2vrGZmGruL2zu7dfOjhsmDjVjNdZLGPdCqjhUiheR4GStxLNaRRI3gyGN1O/OeLaiFg94DjhfkT7SvQEo2glv/MkQj6gmN1NuqJbKrsVdwayTLyclCFHrVv66oQxSyOukElqTNtzE/QzqlEwySfFTmp4QtmQ9nnbUkUjbvxsdvSEnFolJL1Y21JIZurviYxGxoyjwHZGFAdm0ZuK/3ntFHtXfiZUkiJXbL6ol0qCMZkmQEKhOUM5toQyLeythA2opgxtTkUbgrf48jJpnFc8t+LdX5Sr13kcBTiGEzgDDy6hCrdQgzoweIRneIU3Z+S8OO/Ox7x1xclnjuAPnM8fGq6STQ==</latexit> <latexit sha1_base64="hP+6LrUf2d3tZaldqaQQvEKMXyw=">AAAB2XicbZDNSgMxFIXv1L86Vq1rN8EiuCozbnQpuHFZwbZCO5RM5k4bmskMyR2hDH0BF25EfC93vo3pz0JbDwQ+zknIvSculLQUBN9ebWd3b/+gfugfNfzjk9Nmo2fz0gjsilzl5jnmFpXU2CVJCp8LgzyLFfbj6f0i77+gsTLXTzQrMMr4WMtUCk7O6oyaraAdLMW2IVxDC9YaNb+GSS7KDDUJxa0dhEFBUcUNSaFw7g9LiwUXUz7GgUPNM7RRtRxzzi6dk7A0N+5oYkv394uKZ9bOstjdzDhN7Ga2MP/LBiWlt1EldVESarH6KC0Vo5wtdmaJNChIzRxwYaSblYkJN1yQa8Z3HYSbG29D77odBu3wMYA6nMMFXEEIN3AHD9CBLghI4BXevYn35n2suqp569LO4I+8zx84xIo4</latexit> <latexit sha1_base64="VNMucIJiVZIQJrak5ikYkaAixzg=">AAAB6XicbZBLS8NAFIVvfNZatbp1EyyCq5K40aXgxp0V7APaUCaTm3boZBJnbiol9He4caGIf8id/8bpY6GtBwY+zpnh3jlhJoUhz/t2Nja3tnd2S3vl/crB4VH1uNIyaa45NnkqU90JmUEpFDZJkMROppElocR2OLqd5e0xaiNS9UiTDIOEDZSIBWdkraD3LCIcMirup33Rr9a8ujeXuw7+EmqwVKNf/epFKc8TVMQlM6brexkFBdMkuMRpuZcbzBgfsQF2LSqWoAmK+dJT99w6kRun2h5F7tz9/aJgiTGTJLQ3E0ZDs5rNzP+ybk7xdVAIleWEii8Gxbl0KXVnDbiR0MhJTiwwroXd1eVDphkn21PZluCvfnkdWpd136v7Dx6U4BTO4AJ8uIIbuIMGNIHDE7zAG7w7Y+fV+VjUteEsezuBP3I+fwDNppDu</latexit> <latexit sha1_base64="VNMucIJiVZIQJrak5ikYkaAixzg=">AAAB6XicbZBLS8NAFIVvfNZatbp1EyyCq5K40aXgxp0V7APaUCaTm3boZBJnbiol9He4caGIf8id/8bpY6GtBwY+zpnh3jlhJoUhz/t2Nja3tnd2S3vl/crB4VH1uNIyaa45NnkqU90JmUEpFDZJkMROppElocR2OLqd5e0xaiNS9UiTDIOEDZSIBWdkraD3LCIcMirup33Rr9a8ujeXuw7+EmqwVKNf/epFKc8TVMQlM6brexkFBdMkuMRpuZcbzBgfsQF2LSqWoAmK+dJT99w6kRun2h5F7tz9/aJgiTGTJLQ3E0ZDs5rNzP+ybk7xdVAIleWEii8Gxbl0KXVnDbiR0MhJTiwwroXd1eVDphkn21PZluCvfnkdWpd136v7Dx6U4BTO4AJ8uIIbuIMGNIHDE7zAG7w7Y+fV+VjUteEsezuBP3I+fwDNppDu</latexit> <latexit sha1_base64="mrgQmcMOginzwk/WPKN1HKHnrFk=">AAAB9HicbVA9SwNBEN2LXzF+RS1tFoNgFe5stAza2BnBfEByhL29SbJkb+/cnYuEI7/DxkIRW3+Mnf/GTXKFJj4YeLw3w8y8IJHCoOt+O4W19Y3NreJ2aWd3b/+gfHjUNHGqOTR4LGPdDpgBKRQ0UKCEdqKBRYGEVjC6mfmtMWgjYvWAkwT8iA2U6AvO0Ep+90mEMGSY3U17oleuuFV3DrpKvJxUSI56r/zVDWOeRqCQS2ZMx3MT9DOmUXAJ01I3NZAwPmID6FiqWATGz+ZHT+mZVULaj7UthXSu/p7IWGTMJApsZ8RwaJa9mfif10mxf+VnQiUpguKLRf1UUozpLAEaCg0c5cQSxrWwt1I+ZJpxtDmVbAje8surpHlR9dyqd+9Watd5HEVyQk7JOfHIJamRW1InDcLJI3kmr+TNGTsvzrvzsWgtOPnMMfkD5/MHGW6SSQ==</latexit> <latexit sha1_base64="yZtHgQaDiOpztI9WGGMk2koDamQ=">AAAB9HicbVBNS8NAEJ34WetX1aOXxSJ4KokIeix68WYF+wFtKJvNtl262cTdSaWE/g4vHhTx6o/x5r9x2+agrQ8GHu/NMDMvSKQw6Lrfzsrq2vrGZmGruL2zu7dfOjhsmDjVjNdZLGPdCqjhUiheR4GStxLNaRRI3gyGN1O/OeLaiFg94DjhfkT7SvQEo2glv/MkQj6gmN1NuqJbKrsVdwayTLyclCFHrVv66oQxSyOukElqTNtzE/QzqlEwySfFTmp4QtmQ9nnbUkUjbvxsdvSEnFolJL1Y21JIZurviYxGxoyjwHZGFAdm0ZuK/3ntFHtXfiZUkiJXbL6ol0qCMZkmQEKhOUM5toQyLeythA2opgxtTkUbgrf48jJpnFc8t+LdX5Sr13kcBTiGEzgDDy6hCrdQgzoweIRneIU3Z+S8OO/Ox7x1xclnjuAPnM8fGq6STQ==</latexit> <latexit sha1_base64="yZtHgQaDiOpztI9WGGMk2koDamQ=">AAAB9HicbVBNS8NAEJ34WetX1aOXxSJ4KokIeix68WYF+wFtKJvNtl262cTdSaWE/g4vHhTx6o/x5r9x2+agrQ8GHu/NMDMvSKQw6Lrfzsrq2vrGZmGruL2zu7dfOjhsmDjVjNdZLGPdCqjhUiheR4GStxLNaRRI3gyGN1O/OeLaiFg94DjhfkT7SvQEo2glv/MkQj6gmN1NuqJbKrsVdwayTLyclCFHrVv66oQxSyOukElqTNtzE/QzqlEwySfFTmp4QtmQ9nnbUkUjbvxsdvSEnFolJL1Y21JIZurviYxGxoyjwHZGFAdm0ZuK/3ntFHtXfiZUkiJXbL6ol0qCMZkmQEKhOUM5toQyLeythA2opgxtTkUbgrf48jJpnFc8t+LdX5Sr13kcBTiGEzgDDy6hCrdQgzoweIRneIU3Z+S8OO/Ox7x1xclnjuAPnM8fGq6STQ==</latexit> <latexit sha1_base64="yZtHgQaDiOpztI9WGGMk2koDamQ=">AAAB9HicbVBNS8NAEJ34WetX1aOXxSJ4KokIeix68WYF+wFtKJvNtl262cTdSaWE/g4vHhTx6o/x5r9x2+agrQ8GHu/NMDMvSKQw6Lrfzsrq2vrGZmGruL2zu7dfOjhsmDjVjNdZLGPdCqjhUiheR4GStxLNaRRI3gyGN1O/OeLaiFg94DjhfkT7SvQEo2glv/MkQj6gmN1NuqJbKrsVdwayTLyclCFHrVv66oQxSyOukElqTNtzE/QzqlEwySfFTmp4QtmQ9nnbUkUjbvxsdvSEnFolJL1Y21JIZurviYxGxoyjwHZGFAdm0ZuK/3ntFHtXfiZUkiJXbL6ol0qCMZkmQEKhOUM5toQyLeythA2opgxtTkUbgrf48jJpnFc8t+LdX5Sr13kcBTiGEzgDDy6hCrdQgzoweIRneIU3Z+S8OO/Ox7x1xclnjuAPnM8fGq6STQ==</latexit> <latexit sha1_base64="yZtHgQaDiOpztI9WGGMk2koDamQ=">AAAB9HicbVBNS8NAEJ34WetX1aOXxSJ4KokIeix68WYF+wFtKJvNtl262cTdSaWE/g4vHhTx6o/x5r9x2+agrQ8GHu/NMDMvSKQw6Lrfzsrq2vrGZmGruL2zu7dfOjhsmDjVjNdZLGPdCqjhUiheR4GStxLNaRRI3gyGN1O/OeLaiFg94DjhfkT7SvQEo2glv/MkQj6gmN1NuqJbKrsVdwayTLyclCFHrVv66oQxSyOukElqTNtzE/QzqlEwySfFTmp4QtmQ9nnbUkUjbvxsdvSEnFolJL1Y21JIZurviYxGxoyjwHZGFAdm0ZuK/3ntFHtXfiZUkiJXbL6ol0qCMZkmQEKhOUM5toQyLeythA2opgxtTkUbgrf48jJpnFc8t+LdX5Sr13kcBTiGEzgDDy6hCrdQgzoweIRneIU3Z+S8OO/Ox7x1xclnjuAPnM8fGq6STQ==</latexit> <latexit sha1_base64="yZtHgQaDiOpztI9WGGMk2koDamQ=">AAAB9HicbVBNS8NAEJ34WetX1aOXxSJ4KokIeix68WYF+wFtKJvNtl262cTdSaWE/g4vHhTx6o/x5r9x2+agrQ8GHu/NMDMvSKQw6Lrfzsrq2vrGZmGruL2zu7dfOjhsmDjVjNdZLGPdCqjhUiheR4GStxLNaRRI3gyGN1O/OeLaiFg94DjhfkT7SvQEo2glv/MkQj6gmN1NuqJbKrsVdwayTLyclCFHrVv66oQxSyOukElqTNtzE/QzqlEwySfFTmp4QtmQ9nnbUkUjbvxsdvSEnFolJL1Y21JIZurviYxGxoyjwHZGFAdm0ZuK/3ntFHtXfiZUkiJXbL6ol0qCMZkmQEKhOUM5toQyLeythA2opgxtTkUbgrf48jJpnFc8t+LdX5Sr13kcBTiGEzgDDy6hCrdQgzoweIRneIU3Z+S8OO/Ox7x1xclnjuAPnM8fGq6STQ==</latexit> <latexit sha1_base64="yZtHgQaDiOpztI9WGGMk2koDamQ=">AAAB9HicbVBNS8NAEJ34WetX1aOXxSJ4KokIeix68WYF+wFtKJvNtl262cTdSaWE/g4vHhTx6o/x5r9x2+agrQ8GHu/NMDMvSKQw6Lrfzsrq2vrGZmGruL2zu7dfOjhsmDjVjNdZLGPdCqjhUiheR4GStxLNaRRI3gyGN1O/OeLaiFg94DjhfkT7SvQEo2glv/MkQj6gmN1NuqJbKrsVdwayTLyclCFHrVv66oQxSyOukElqTNtzE/QzqlEwySfFTmp4QtmQ9nnbUkUjbvxsdvSEnFolJL1Y21JIZurviYxGxoyjwHZGFAdm0ZuK/3ntFHtXfiZUkiJXbL6ol0qCMZkmQEKhOUM5toQyLeythA2opgxtTkUbgrf48jJpnFc8t+LdX5Sr13kcBTiGEzgDDy6hCrdQgzoweIRneIU3Z+S8OO/Ox7x1xclnjuAPnM8fGq6STQ==</latexit> 3 60
  50. 600 700 800 900 1000 1100 1200 1300 1400 1500

    1600 1700 1800 1900 500 say wait wait read wait goBackTapped Time (ms) inputs outputs state trace corrected state trace lookingAtRobot Error Correction User feedback 1 2 corrected0
 b Oi <latexit sha1_base64="yZtHgQaDiOpztI9WGGMk2koDamQ=">AAAB9HicbVBNS8NAEJ34WetX1aOXxSJ4KokIeix68WYF+wFtKJvNtl262cTdSaWE/g4vHhTx6o/x5r9x2+agrQ8GHu/NMDMvSKQw6Lrfzsrq2vrGZmGruL2zu7dfOjhsmDjVjNdZLGPdCqjhUiheR4GStxLNaRRI3gyGN1O/OeLaiFg94DjhfkT7SvQEo2glv/MkQj6gmN1NuqJbKrsVdwayTLyclCFHrVv66oQxSyOukElqTNtzE/QzqlEwySfFTmp4QtmQ9nnbUkUjbvxsdvSEnFolJL1Y21JIZurviYxGxoyjwHZGFAdm0ZuK/3ntFHtXfiZUkiJXbL6ol0qCMZkmQEKhOUM5toQyLeythA2opgxtTkUbgrf48jJpnFc8t+LdX5Sr13kcBTiGEzgDDy6hCrdQgzoweIRneIU3Z+S8OO/Ox7x1xclnjuAPnM8fGq6STQ==</latexit> <latexit sha1_base64="yZtHgQaDiOpztI9WGGMk2koDamQ=">AAAB9HicbVBNS8NAEJ34WetX1aOXxSJ4KokIeix68WYF+wFtKJvNtl262cTdSaWE/g4vHhTx6o/x5r9x2+agrQ8GHu/NMDMvSKQw6Lrfzsrq2vrGZmGruL2zu7dfOjhsmDjVjNdZLGPdCqjhUiheR4GStxLNaRRI3gyGN1O/OeLaiFg94DjhfkT7SvQEo2glv/MkQj6gmN1NuqJbKrsVdwayTLyclCFHrVv66oQxSyOukElqTNtzE/QzqlEwySfFTmp4QtmQ9nnbUkUjbvxsdvSEnFolJL1Y21JIZurviYxGxoyjwHZGFAdm0ZuK/3ntFHtXfiZUkiJXbL6ol0qCMZkmQEKhOUM5toQyLeythA2opgxtTkUbgrf48jJpnFc8t+LdX5Sr13kcBTiGEzgDDy6hCrdQgzoweIRneIU3Z+S8OO/Ox7x1xclnjuAPnM8fGq6STQ==</latexit> <latexit sha1_base64="yZtHgQaDiOpztI9WGGMk2koDamQ=">AAAB9HicbVBNS8NAEJ34WetX1aOXxSJ4KokIeix68WYF+wFtKJvNtl262cTdSaWE/g4vHhTx6o/x5r9x2+agrQ8GHu/NMDMvSKQw6Lrfzsrq2vrGZmGruL2zu7dfOjhsmDjVjNdZLGPdCqjhUiheR4GStxLNaRRI3gyGN1O/OeLaiFg94DjhfkT7SvQEo2glv/MkQj6gmN1NuqJbKrsVdwayTLyclCFHrVv66oQxSyOukElqTNtzE/QzqlEwySfFTmp4QtmQ9nnbUkUjbvxsdvSEnFolJL1Y21JIZurviYxGxoyjwHZGFAdm0ZuK/3ntFHtXfiZUkiJXbL6ol0qCMZkmQEKhOUM5toQyLeythA2opgxtTkUbgrf48jJpnFc8t+LdX5Sr13kcBTiGEzgDDy6hCrdQgzoweIRneIU3Z+S8OO/Ox7x1xclnjuAPnM8fGq6STQ==</latexit> <latexit sha1_base64="hP+6LrUf2d3tZaldqaQQvEKMXyw=">AAAB2XicbZDNSgMxFIXv1L86Vq1rN8EiuCozbnQpuHFZwbZCO5RM5k4bmskMyR2hDH0BF25EfC93vo3pz0JbDwQ+zknIvSculLQUBN9ebWd3b/+gfugfNfzjk9Nmo2fz0gjsilzl5jnmFpXU2CVJCp8LgzyLFfbj6f0i77+gsTLXTzQrMMr4WMtUCk7O6oyaraAdLMW2IVxDC9YaNb+GSS7KDDUJxa0dhEFBUcUNSaFw7g9LiwUXUz7GgUPNM7RRtRxzzi6dk7A0N+5oYkv394uKZ9bOstjdzDhN7Ga2MP/LBiWlt1EldVESarH6KC0Vo5wtdmaJNChIzRxwYaSblYkJN1yQa8Z3HYSbG29D77odBu3wMYA6nMMFXEEIN3AHD9CBLghI4BXevYn35n2suqp569LO4I+8zx84xIo4</latexit> <latexit sha1_base64="VNMucIJiVZIQJrak5ikYkaAixzg=">AAAB6XicbZBLS8NAFIVvfNZatbp1EyyCq5K40aXgxp0V7APaUCaTm3boZBJnbiol9He4caGIf8id/8bpY6GtBwY+zpnh3jlhJoUhz/t2Nja3tnd2S3vl/crB4VH1uNIyaa45NnkqU90JmUEpFDZJkMROppElocR2OLqd5e0xaiNS9UiTDIOEDZSIBWdkraD3LCIcMirup33Rr9a8ujeXuw7+EmqwVKNf/epFKc8TVMQlM6brexkFBdMkuMRpuZcbzBgfsQF2LSqWoAmK+dJT99w6kRun2h5F7tz9/aJgiTGTJLQ3E0ZDs5rNzP+ybk7xdVAIleWEii8Gxbl0KXVnDbiR0MhJTiwwroXd1eVDphkn21PZluCvfnkdWpd136v7Dx6U4BTO4AJ8uIIbuIMGNIHDE7zAG7w7Y+fV+VjUteEsezuBP3I+fwDNppDu</latexit> <latexit sha1_base64="VNMucIJiVZIQJrak5ikYkaAixzg=">AAAB6XicbZBLS8NAFIVvfNZatbp1EyyCq5K40aXgxp0V7APaUCaTm3boZBJnbiol9He4caGIf8id/8bpY6GtBwY+zpnh3jlhJoUhz/t2Nja3tnd2S3vl/crB4VH1uNIyaa45NnkqU90JmUEpFDZJkMROppElocR2OLqd5e0xaiNS9UiTDIOEDZSIBWdkraD3LCIcMirup33Rr9a8ujeXuw7+EmqwVKNf/epFKc8TVMQlM6brexkFBdMkuMRpuZcbzBgfsQF2LSqWoAmK+dJT99w6kRun2h5F7tz9/aJgiTGTJLQ3E0ZDs5rNzP+ybk7xdVAIleWEii8Gxbl0KXVnDbiR0MhJTiwwroXd1eVDphkn21PZluCvfnkdWpd136v7Dx6U4BTO4AJ8uIIbuIMGNIHDE7zAG7w7Y+fV+VjUteEsezuBP3I+fwDNppDu</latexit> <latexit sha1_base64="mrgQmcMOginzwk/WPKN1HKHnrFk=">AAAB9HicbVA9SwNBEN2LXzF+RS1tFoNgFe5stAza2BnBfEByhL29SbJkb+/cnYuEI7/DxkIRW3+Mnf/GTXKFJj4YeLw3w8y8IJHCoOt+O4W19Y3NreJ2aWd3b/+gfHjUNHGqOTR4LGPdDpgBKRQ0UKCEdqKBRYGEVjC6mfmtMWgjYvWAkwT8iA2U6AvO0Ep+90mEMGSY3U17oleuuFV3DrpKvJxUSI56r/zVDWOeRqCQS2ZMx3MT9DOmUXAJ01I3NZAwPmID6FiqWATGz+ZHT+mZVULaj7UthXSu/p7IWGTMJApsZ8RwaJa9mfif10mxf+VnQiUpguKLRf1UUozpLAEaCg0c5cQSxrWwt1I+ZJpxtDmVbAje8surpHlR9dyqd+9Watd5HEVyQk7JOfHIJamRW1InDcLJI3kmr+TNGTsvzrvzsWgtOPnMMfkD5/MHGW6SSQ==</latexit> <latexit sha1_base64="yZtHgQaDiOpztI9WGGMk2koDamQ=">AAAB9HicbVBNS8NAEJ34WetX1aOXxSJ4KokIeix68WYF+wFtKJvNtl262cTdSaWE/g4vHhTx6o/x5r9x2+agrQ8GHu/NMDMvSKQw6Lrfzsrq2vrGZmGruL2zu7dfOjhsmDjVjNdZLGPdCqjhUiheR4GStxLNaRRI3gyGN1O/OeLaiFg94DjhfkT7SvQEo2glv/MkQj6gmN1NuqJbKrsVdwayTLyclCFHrVv66oQxSyOukElqTNtzE/QzqlEwySfFTmp4QtmQ9nnbUkUjbvxsdvSEnFolJL1Y21JIZurviYxGxoyjwHZGFAdm0ZuK/3ntFHtXfiZUkiJXbL6ol0qCMZkmQEKhOUM5toQyLeythA2opgxtTkUbgrf48jJpnFc8t+LdX5Sr13kcBTiGEzgDDy6hCrdQgzoweIRneIU3Z+S8OO/Ox7x1xclnjuAPnM8fGq6STQ==</latexit> <latexit sha1_base64="yZtHgQaDiOpztI9WGGMk2koDamQ=">AAAB9HicbVBNS8NAEJ34WetX1aOXxSJ4KokIeix68WYF+wFtKJvNtl262cTdSaWE/g4vHhTx6o/x5r9x2+agrQ8GHu/NMDMvSKQw6Lrfzsrq2vrGZmGruL2zu7dfOjhsmDjVjNdZLGPdCqjhUiheR4GStxLNaRRI3gyGN1O/OeLaiFg94DjhfkT7SvQEo2glv/MkQj6gmN1NuqJbKrsVdwayTLyclCFHrVv66oQxSyOukElqTNtzE/QzqlEwySfFTmp4QtmQ9nnbUkUjbvxsdvSEnFolJL1Y21JIZurviYxGxoyjwHZGFAdm0ZuK/3ntFHtXfiZUkiJXbL6ol0qCMZkmQEKhOUM5toQyLeythA2opgxtTkUbgrf48jJpnFc8t+LdX5Sr13kcBTiGEzgDDy6hCrdQgzoweIRneIU3Z+S8OO/Ox7x1xclnjuAPnM8fGq6STQ==</latexit> <latexit sha1_base64="yZtHgQaDiOpztI9WGGMk2koDamQ=">AAAB9HicbVBNS8NAEJ34WetX1aOXxSJ4KokIeix68WYF+wFtKJvNtl262cTdSaWE/g4vHhTx6o/x5r9x2+agrQ8GHu/NMDMvSKQw6Lrfzsrq2vrGZmGruL2zu7dfOjhsmDjVjNdZLGPdCqjhUiheR4GStxLNaRRI3gyGN1O/OeLaiFg94DjhfkT7SvQEo2glv/MkQj6gmN1NuqJbKrsVdwayTLyclCFHrVv66oQxSyOukElqTNtzE/QzqlEwySfFTmp4QtmQ9nnbUkUjbvxsdvSEnFolJL1Y21JIZurviYxGxoyjwHZGFAdm0ZuK/3ntFHtXfiZUkiJXbL6ol0qCMZkmQEKhOUM5toQyLeythA2opgxtTkUbgrf48jJpnFc8t+LdX5Sr13kcBTiGEzgDDy6hCrdQgzoweIRneIU3Z+S8OO/Ox7x1xclnjuAPnM8fGq6STQ==</latexit> <latexit sha1_base64="yZtHgQaDiOpztI9WGGMk2koDamQ=">AAAB9HicbVBNS8NAEJ34WetX1aOXxSJ4KokIeix68WYF+wFtKJvNtl262cTdSaWE/g4vHhTx6o/x5r9x2+agrQ8GHu/NMDMvSKQw6Lrfzsrq2vrGZmGruL2zu7dfOjhsmDjVjNdZLGPdCqjhUiheR4GStxLNaRRI3gyGN1O/OeLaiFg94DjhfkT7SvQEo2glv/MkQj6gmN1NuqJbKrsVdwayTLyclCFHrVv66oQxSyOukElqTNtzE/QzqlEwySfFTmp4QtmQ9nnbUkUjbvxsdvSEnFolJL1Y21JIZurviYxGxoyjwHZGFAdm0ZuK/3ntFHtXfiZUkiJXbL6ol0qCMZkmQEKhOUM5toQyLeythA2opgxtTkUbgrf48jJpnFc8t+LdX5Sr13kcBTiGEzgDDy6hCrdQgzoweIRneIU3Z+S8OO/Ox7x1xclnjuAPnM8fGq6STQ==</latexit> <latexit sha1_base64="yZtHgQaDiOpztI9WGGMk2koDamQ=">AAAB9HicbVBNS8NAEJ34WetX1aOXxSJ4KokIeix68WYF+wFtKJvNtl262cTdSaWE/g4vHhTx6o/x5r9x2+agrQ8GHu/NMDMvSKQw6Lrfzsrq2vrGZmGruL2zu7dfOjhsmDjVjNdZLGPdCqjhUiheR4GStxLNaRRI3gyGN1O/OeLaiFg94DjhfkT7SvQEo2glv/MkQj6gmN1NuqJbKrsVdwayTLyclCFHrVv66oQxSyOukElqTNtzE/QzqlEwySfFTmp4QtmQ9nnbUkUjbvxsdvSEnFolJL1Y21JIZurviYxGxoyjwHZGFAdm0ZuK/3ntFHtXfiZUkiJXbL6ol0qCMZkmQEKhOUM5toQyLeythA2opgxtTkUbgrf48jJpnFc8t+LdX5Sr13kcBTiGEzgDDy6hCrdQgzoweIRneIU3Z+S8OO/Ox7x1xclnjuAPnM8fGq6STQ==</latexit> <latexit sha1_base64="yZtHgQaDiOpztI9WGGMk2koDamQ=">AAAB9HicbVBNS8NAEJ34WetX1aOXxSJ4KokIeix68WYF+wFtKJvNtl262cTdSaWE/g4vHhTx6o/x5r9x2+agrQ8GHu/NMDMvSKQw6Lrfzsrq2vrGZmGruL2zu7dfOjhsmDjVjNdZLGPdCqjhUiheR4GStxLNaRRI3gyGN1O/OeLaiFg94DjhfkT7SvQEo2glv/MkQj6gmN1NuqJbKrsVdwayTLyclCFHrVv66oQxSyOukElqTNtzE/QzqlEwySfFTmp4QtmQ9nnbUkUjbvxsdvSEnFolJL1Y21JIZurviYxGxoyjwHZGFAdm0ZuK/3ntFHtXfiZUkiJXbL6ol0qCMZkmQEKhOUM5toQyLeythA2opgxtTkUbgrf48jJpnFc8t+LdX5Sr13kcBTiGEzgDDy6hCrdQgzoweIRneIU3Z+S8OO/Ox7x1xclnjuAPnM8fGq6STQ==</latexit> program sketch, K, Θ interaction0
 I0, O0 3 61
  51. (repaired) Interaction1
 I1, O1 ... ... ... ... ... ...

    At each iteration, do 1. Run the program and collect an interaction trace 2. Obtain corrections from the programmer or the user 3. Re-estimate hole distributions using the input and corrected state traces (posterior) 4. Fill-in holes using maximum a posteriori estimation (MAP) for the next iteration corrected0
 b Oi <latexit sha1_base64="yZtHgQaDiOpztI9WGGMk2koDamQ=">AAAB9HicbVBNS8NAEJ34WetX1aOXxSJ4KokIeix68WYF+wFtKJvNtl262cTdSaWE/g4vHhTx6o/x5r9x2+agrQ8GHu/NMDMvSKQw6Lrfzsrq2vrGZmGruL2zu7dfOjhsmDjVjNdZLGPdCqjhUiheR4GStxLNaRRI3gyGN1O/OeLaiFg94DjhfkT7SvQEo2glv/MkQj6gmN1NuqJbKrsVdwayTLyclCFHrVv66oQxSyOukElqTNtzE/QzqlEwySfFTmp4QtmQ9nnbUkUjbvxsdvSEnFolJL1Y21JIZurviYxGxoyjwHZGFAdm0ZuK/3ntFHtXfiZUkiJXbL6ol0qCMZkmQEKhOUM5toQyLeythA2opgxtTkUbgrf48jJpnFc8t+LdX5Sr13kcBTiGEzgDDy6hCrdQgzoweIRneIU3Z+S8OO/Ox7x1xclnjuAPnM8fGq6STQ==</latexit> <latexit sha1_base64="yZtHgQaDiOpztI9WGGMk2koDamQ=">AAAB9HicbVBNS8NAEJ34WetX1aOXxSJ4KokIeix68WYF+wFtKJvNtl262cTdSaWE/g4vHhTx6o/x5r9x2+agrQ8GHu/NMDMvSKQw6Lrfzsrq2vrGZmGruL2zu7dfOjhsmDjVjNdZLGPdCqjhUiheR4GStxLNaRRI3gyGN1O/OeLaiFg94DjhfkT7SvQEo2glv/MkQj6gmN1NuqJbKrsVdwayTLyclCFHrVv66oQxSyOukElqTNtzE/QzqlEwySfFTmp4QtmQ9nnbUkUjbvxsdvSEnFolJL1Y21JIZurviYxGxoyjwHZGFAdm0ZuK/3ntFHtXfiZUkiJXbL6ol0qCMZkmQEKhOUM5toQyLeythA2opgxtTkUbgrf48jJpnFc8t+LdX5Sr13kcBTiGEzgDDy6hCrdQgzoweIRneIU3Z+S8OO/Ox7x1xclnjuAPnM8fGq6STQ==</latexit> <latexit sha1_base64="yZtHgQaDiOpztI9WGGMk2koDamQ=">AAAB9HicbVBNS8NAEJ34WetX1aOXxSJ4KokIeix68WYF+wFtKJvNtl262cTdSaWE/g4vHhTx6o/x5r9x2+agrQ8GHu/NMDMvSKQw6Lrfzsrq2vrGZmGruL2zu7dfOjhsmDjVjNdZLGPdCqjhUiheR4GStxLNaRRI3gyGN1O/OeLaiFg94DjhfkT7SvQEo2glv/MkQj6gmN1NuqJbKrsVdwayTLyclCFHrVv66oQxSyOukElqTNtzE/QzqlEwySfFTmp4QtmQ9nnbUkUjbvxsdvSEnFolJL1Y21JIZurviYxGxoyjwHZGFAdm0ZuK/3ntFHtXfiZUkiJXbL6ol0qCMZkmQEKhOUM5toQyLeythA2opgxtTkUbgrf48jJpnFc8t+LdX5Sr13kcBTiGEzgDDy6hCrdQgzoweIRneIU3Z+S8OO/Ox7x1xclnjuAPnM8fGq6STQ==</latexit> <latexit sha1_base64="hP+6LrUf2d3tZaldqaQQvEKMXyw=">AAAB2XicbZDNSgMxFIXv1L86Vq1rN8EiuCozbnQpuHFZwbZCO5RM5k4bmskMyR2hDH0BF25EfC93vo3pz0JbDwQ+zknIvSculLQUBN9ebWd3b/+gfugfNfzjk9Nmo2fz0gjsilzl5jnmFpXU2CVJCp8LgzyLFfbj6f0i77+gsTLXTzQrMMr4WMtUCk7O6oyaraAdLMW2IVxDC9YaNb+GSS7KDDUJxa0dhEFBUcUNSaFw7g9LiwUXUz7GgUPNM7RRtRxzzi6dk7A0N+5oYkv394uKZ9bOstjdzDhN7Ga2MP/LBiWlt1EldVESarH6KC0Vo5wtdmaJNChIzRxwYaSblYkJN1yQa8Z3HYSbG29D77odBu3wMYA6nMMFXEEIN3AHD9CBLghI4BXevYn35n2suqp569LO4I+8zx84xIo4</latexit> <latexit sha1_base64="VNMucIJiVZIQJrak5ikYkaAixzg=">AAAB6XicbZBLS8NAFIVvfNZatbp1EyyCq5K40aXgxp0V7APaUCaTm3boZBJnbiol9He4caGIf8id/8bpY6GtBwY+zpnh3jlhJoUhz/t2Nja3tnd2S3vl/crB4VH1uNIyaa45NnkqU90JmUEpFDZJkMROppElocR2OLqd5e0xaiNS9UiTDIOEDZSIBWdkraD3LCIcMirup33Rr9a8ujeXuw7+EmqwVKNf/epFKc8TVMQlM6brexkFBdMkuMRpuZcbzBgfsQF2LSqWoAmK+dJT99w6kRun2h5F7tz9/aJgiTGTJLQ3E0ZDs5rNzP+ybk7xdVAIleWEii8Gxbl0KXVnDbiR0MhJTiwwroXd1eVDphkn21PZluCvfnkdWpd136v7Dx6U4BTO4AJ8uIIbuIMGNIHDE7zAG7w7Y+fV+VjUteEsezuBP3I+fwDNppDu</latexit> <latexit sha1_base64="VNMucIJiVZIQJrak5ikYkaAixzg=">AAAB6XicbZBLS8NAFIVvfNZatbp1EyyCq5K40aXgxp0V7APaUCaTm3boZBJnbiol9He4caGIf8id/8bpY6GtBwY+zpnh3jlhJoUhz/t2Nja3tnd2S3vl/crB4VH1uNIyaa45NnkqU90JmUEpFDZJkMROppElocR2OLqd5e0xaiNS9UiTDIOEDZSIBWdkraD3LCIcMirup33Rr9a8ujeXuw7+EmqwVKNf/epFKc8TVMQlM6brexkFBdMkuMRpuZcbzBgfsQF2LSqWoAmK+dJT99w6kRun2h5F7tz9/aJgiTGTJLQ3E0ZDs5rNzP+ybk7xdVAIleWEii8Gxbl0KXVnDbiR0MhJTiwwroXd1eVDphkn21PZluCvfnkdWpd136v7Dx6U4BTO4AJ8uIIbuIMGNIHDE7zAG7w7Y+fV+VjUteEsezuBP3I+fwDNppDu</latexit> <latexit sha1_base64="mrgQmcMOginzwk/WPKN1HKHnrFk=">AAAB9HicbVA9SwNBEN2LXzF+RS1tFoNgFe5stAza2BnBfEByhL29SbJkb+/cnYuEI7/DxkIRW3+Mnf/GTXKFJj4YeLw3w8y8IJHCoOt+O4W19Y3NreJ2aWd3b/+gfHjUNHGqOTR4LGPdDpgBKRQ0UKCEdqKBRYGEVjC6mfmtMWgjYvWAkwT8iA2U6AvO0Ep+90mEMGSY3U17oleuuFV3DrpKvJxUSI56r/zVDWOeRqCQS2ZMx3MT9DOmUXAJ01I3NZAwPmID6FiqWATGz+ZHT+mZVULaj7UthXSu/p7IWGTMJApsZ8RwaJa9mfif10mxf+VnQiUpguKLRf1UUozpLAEaCg0c5cQSxrWwt1I+ZJpxtDmVbAje8surpHlR9dyqd+9Watd5HEVyQk7JOfHIJamRW1InDcLJI3kmr+TNGTsvzrvzsWgtOPnMMfkD5/MHGW6SSQ==</latexit> <latexit sha1_base64="yZtHgQaDiOpztI9WGGMk2koDamQ=">AAAB9HicbVBNS8NAEJ34WetX1aOXxSJ4KokIeix68WYF+wFtKJvNtl262cTdSaWE/g4vHhTx6o/x5r9x2+agrQ8GHu/NMDMvSKQw6Lrfzsrq2vrGZmGruL2zu7dfOjhsmDjVjNdZLGPdCqjhUiheR4GStxLNaRRI3gyGN1O/OeLaiFg94DjhfkT7SvQEo2glv/MkQj6gmN1NuqJbKrsVdwayTLyclCFHrVv66oQxSyOukElqTNtzE/QzqlEwySfFTmp4QtmQ9nnbUkUjbvxsdvSEnFolJL1Y21JIZurviYxGxoyjwHZGFAdm0ZuK/3ntFHtXfiZUkiJXbL6ol0qCMZkmQEKhOUM5toQyLeythA2opgxtTkUbgrf48jJpnFc8t+LdX5Sr13kcBTiGEzgDDy6hCrdQgzoweIRneIU3Z+S8OO/Ox7x1xclnjuAPnM8fGq6STQ==</latexit> <latexit sha1_base64="yZtHgQaDiOpztI9WGGMk2koDamQ=">AAAB9HicbVBNS8NAEJ34WetX1aOXxSJ4KokIeix68WYF+wFtKJvNtl262cTdSaWE/g4vHhTx6o/x5r9x2+agrQ8GHu/NMDMvSKQw6Lrfzsrq2vrGZmGruL2zu7dfOjhsmDjVjNdZLGPdCqjhUiheR4GStxLNaRRI3gyGN1O/OeLaiFg94DjhfkT7SvQEo2glv/MkQj6gmN1NuqJbKrsVdwayTLyclCFHrVv66oQxSyOukElqTNtzE/QzqlEwySfFTmp4QtmQ9nnbUkUjbvxsdvSEnFolJL1Y21JIZurviYxGxoyjwHZGFAdm0ZuK/3ntFHtXfiZUkiJXbL6ol0qCMZkmQEKhOUM5toQyLeythA2opgxtTkUbgrf48jJpnFc8t+LdX5Sr13kcBTiGEzgDDy6hCrdQgzoweIRneIU3Z+S8OO/Ox7x1xclnjuAPnM8fGq6STQ==</latexit> <latexit sha1_base64="yZtHgQaDiOpztI9WGGMk2koDamQ=">AAAB9HicbVBNS8NAEJ34WetX1aOXxSJ4KokIeix68WYF+wFtKJvNtl262cTdSaWE/g4vHhTx6o/x5r9x2+agrQ8GHu/NMDMvSKQw6Lrfzsrq2vrGZmGruL2zu7dfOjhsmDjVjNdZLGPdCqjhUiheR4GStxLNaRRI3gyGN1O/OeLaiFg94DjhfkT7SvQEo2glv/MkQj6gmN1NuqJbKrsVdwayTLyclCFHrVv66oQxSyOukElqTNtzE/QzqlEwySfFTmp4QtmQ9nnbUkUjbvxsdvSEnFolJL1Y21JIZurviYxGxoyjwHZGFAdm0ZuK/3ntFHtXfiZUkiJXbL6ol0qCMZkmQEKhOUM5toQyLeythA2opgxtTkUbgrf48jJpnFc8t+LdX5Sr13kcBTiGEzgDDy6hCrdQgzoweIRneIU3Z+S8OO/Ox7x1xclnjuAPnM8fGq6STQ==</latexit> <latexit sha1_base64="yZtHgQaDiOpztI9WGGMk2koDamQ=">AAAB9HicbVBNS8NAEJ34WetX1aOXxSJ4KokIeix68WYF+wFtKJvNtl262cTdSaWE/g4vHhTx6o/x5r9x2+agrQ8GHu/NMDMvSKQw6Lrfzsrq2vrGZmGruL2zu7dfOjhsmDjVjNdZLGPdCqjhUiheR4GStxLNaRRI3gyGN1O/OeLaiFg94DjhfkT7SvQEo2glv/MkQj6gmN1NuqJbKrsVdwayTLyclCFHrVv66oQxSyOukElqTNtzE/QzqlEwySfFTmp4QtmQ9nnbUkUjbvxsdvSEnFolJL1Y21JIZurviYxGxoyjwHZGFAdm0ZuK/3ntFHtXfiZUkiJXbL6ol0qCMZkmQEKhOUM5toQyLeythA2opgxtTkUbgrf48jJpnFc8t+LdX5Sr13kcBTiGEzgDDy6hCrdQgzoweIRneIU3Z+S8OO/Ox7x1xclnjuAPnM8fGq6STQ==</latexit> <latexit sha1_base64="yZtHgQaDiOpztI9WGGMk2koDamQ=">AAAB9HicbVBNS8NAEJ34WetX1aOXxSJ4KokIeix68WYF+wFtKJvNtl262cTdSaWE/g4vHhTx6o/x5r9x2+agrQ8GHu/NMDMvSKQw6Lrfzsrq2vrGZmGruL2zu7dfOjhsmDjVjNdZLGPdCqjhUiheR4GStxLNaRRI3gyGN1O/OeLaiFg94DjhfkT7SvQEo2glv/MkQj6gmN1NuqJbKrsVdwayTLyclCFHrVv66oQxSyOukElqTNtzE/QzqlEwySfFTmp4QtmQ9nnbUkUjbvxsdvSEnFolJL1Y21JIZurviYxGxoyjwHZGFAdm0ZuK/3ntFHtXfiZUkiJXbL6ol0qCMZkmQEKhOUM5toQyLeythA2opgxtTkUbgrf48jJpnFc8t+LdX5Sr13kcBTiGEzgDDy6hCrdQgzoweIRneIU3Z+S8OO/Ox7x1xclnjuAPnM8fGq6STQ==</latexit> <latexit sha1_base64="yZtHgQaDiOpztI9WGGMk2koDamQ=">AAAB9HicbVBNS8NAEJ34WetX1aOXxSJ4KokIeix68WYF+wFtKJvNtl262cTdSaWE/g4vHhTx6o/x5r9x2+agrQ8GHu/NMDMvSKQw6Lrfzsrq2vrGZmGruL2zu7dfOjhsmDjVjNdZLGPdCqjhUiheR4GStxLNaRRI3gyGN1O/OeLaiFg94DjhfkT7SvQEo2glv/MkQj6gmN1NuqJbKrsVdwayTLyclCFHrVv66oQxSyOukElqTNtzE/QzqlEwySfFTmp4QtmQ9nnbUkUjbvxsdvSEnFolJL1Y21JIZurviYxGxoyjwHZGFAdm0ZuK/3ntFHtXfiZUkiJXbL6ol0qCMZkmQEKhOUM5toQyLeythA2opgxtTkUbgrf48jJpnFc8t+LdX5Sr13kcBTiGEzgDDy6hCrdQgzoweIRneIU3Z+S8OO/Ox7x1xclnjuAPnM8fGq6STQ==</latexit> 3 1 2 program sketch, K, Θ interaction0
 I0, O0 4 62
  52. • Human simulators for sampling input and ground truth state

    trace • Baseline: search-based repair • 3 scenarios, 5 repair iterations • Measures • % overlap (interaction quality) • Repair time in ms (speed) A top-down human-simulator. Samples from states, substates (intent), and then inputs Simulation Experiment: Setup 66
  53. 0 1 2 3 4 5 repair iterations 0.0 0.2

    0.4 0.6 0.8 1.0 1.2 overlap % Storytelling 0 1 2 3 4 5 repair iterations 0.0 0.2 0.4 0.6 0.8 1.0 1.2 Neck Exercise 0 1 2 3 4 5 repair iterations 0.0 0.2 0.4 0.6 0.8 1.0 1.2 Open-ended Q&A bayesian search method Simulation Experiment: % Overlap Results 67
  54. 0 1 2 3 4 5 repair iterations 0.0 0.2

    0.4 0.6 0.8 1.0 1.2 overlap % Storytelling 0 1 2 3 4 5 repair iterations 0.0 0.2 0.4 0.6 0.8 1.0 1.2 Neck Exercise 0 1 2 3 4 5 repair iterations 0.0 0.2 0.4 0.6 0.8 1.0 1.2 Open-ended Q&A bayesian search method Simulation Experiment: Speed Results 1 2 3 4 5 repair iterations 0 2,000 4,000 6,000 8,000 10,000 12,000 time (ms) Storytelling 1 2 3 4 5 repair iterations 0 1,000 2,000 3,000 4,000 Neck Exercise 1 2 3 4 5 repair iterations 0 20,000 40,000 60,000 80,000 100,000 120,000 Open-ended Q&A bayesian search method 68
  55. • 10 participants • 3 females, avg 23.2 yrs •

    A custom-built social robot • 2 scenarios, 4 iterations Human experiment setup. Human Experiment: Setup 69
  56. • Objective • Before/After repair overlap (%) • # of

    interaction breakdowns • total duration (sec) • Subjective • 3 HRI fluency questionnaire (Hoffman 2019) Human experiment setup. Human Experiment: Measures 70
  57. Human Experiment: Objective Measures 0 1 2 3 repair iter.

    0 50 100 150 duration (in sec) Neck Ex. 0 1 2 3 repair iter. 0 50 100 150 Open. Q&A 0 1 2 3 repair iter. -1 0 1 2 3 4 5 6 7 number of corrections Neck Ex. 0 1 2 3 repair iter. -1 0 1 2 3 4 5 6 7 Open. Q&A 71
  58. Human Experiment: Subjective Measures One of subjective questions asked:
 


    Please rate your agreement with the following statements: • “I trusted the robot to do the right thing at the right time." 1: strongly disagree
 5: strongly agree 0 1 2 3 repair iter. 0 1 2 3 4 5 6 rating Neck Ex. 0 1 2 3 repair iter. 0 1 2 3 4 5 6 Open. Q&A 72
  59. Directions for Future Research • Starting with Programming Language for

    HRI • HRI Authoring as Programming Synthesis • Intuitive UIs for Non-Experts 75
  60. Summary • Interactive Robot Applications in Commercial Spaces • User-Requirements

    and Design Implications • Concurrency in Interactive Robots Programming • Parameter Tuning via Iterative Program Repair 76
  61. Thank You! • HCR Lab • Savioke • RSE Lab

    • Neural Systems Lab • CSE Staff • Family and Friends 78