Tweet
Tweet

Slide 1

Slide 1 text

Michael Jae-Yoon Chung, 07/30/2020 Enabling End-Users to Create Interactive Real-World Robot Applications through Novel Interfaces and Automation University of Washington, Computer Science & Engineering Ph.D. Final Examination 1

Slide 2

Slide 2 text

Thank You! • HCR Lab • Savioke • RSE Lab • Neural Systems Lab • CSE Staff • Family and Friends 2

Slide 3

Slide 3 text

3

Slide 4

Slide 4 text

Challenges • Challenge 1: Need for interactive behaviors and content, involving programmers is not scalable ➡ End-User Programming (EUP) • Challenge 2: Need for adaptation to unique users and environment, but EUP not sufficiently expressive ➡ Automatic Program Repair 4

Slide 5

Slide 5 text

Research Goal To facilitate creating applications and behaviors for interactive robots with human-centered design and programming systems for non-experts. 5

Slide 6

Slide 6 text

Outline 1. Information Gathering Robots in Commercial Spaces
 (IROS2015&2016, ROMAN2018) 2. Programming System for Interactive Service Robots
 (ICSR2016) 3. Concurrency in Programming Interactive Robots
 (ROMAN2020) 4. Iterative Program Repair with Implicit User Feedback
 (RSS2020) 6

Slide 7

Slide 7 text

Outline 1. Information Gathering Robots in Commercial Spaces
 (IROS2015&2016, ROMAN2018) 2. Programming System for Interactive Service Robots
 (ICSR2016) 3. Concurrency in Programming Interactive Robots
 (ROMAN2020) 4. Iterative Program Repair with Implicit User Feedback
 (RSS2020) share our experiences 7

Slide 8

Slide 8 text

Outline 1. Information Gathering Robots in Commercial Spaces
 (IROS2015&2016, ROMAN2018) 2. Programming System for Interactive Service Robots
 (ICSR2016) 3. Concurrency in Programming Interactive Robots
 (ROMAN2020) 4. Iterative Program Repair with Implicit User Feedback
 (RSS2020) address the two main problems 8

Slide 9

Slide 9 text

Outline 1. Information Gathering Robots in Commercial Spaces
 (IROS2015&2016, ROMAN2018) 2. Programming System for Interactive Service Robots
 (ICSR2016) 3. Concurrency in Programming Interactive Robots
 (ROMAN2020) 4. Iterative Program Repair with Implicit User Feedback
 (RSS2020) [IROS 2015] Michael Jae-Yoon Chung, Andrzej Pronobis, Maya Cakmak, Dieter Fox, Rajesh P.N. Rao. Designing Information Gathering Robots for Human- Populated Environments. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2015. [IROS 2016] Michael Jae-Yoon Chung, Andrzej Pronobis, Maya Cakmak, Dieter Fox, Rajesh P.N. Rao. Autonomous Question Answering with Mobile Robots in Human-Populated Environments. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2016. [RO-MAN 2018] Michael Jae-Yoon Chung, Maya Cakmak. How was your stay?: Exploring the Use of Robots for Gathering Customer Feedback in the Hospitality Industry. IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), 2018. 9

Slide 10

Slide 10 text

Information Checking Robots
 (Chung et al., IROS2015&2016) Is Mike in the robotics lab? 10

Slide 11

Slide 11 text

Source: The Robot Report, 6/25/2019 Source: CNBC, 6/21/2019 11

Slide 12

Slide 12 text

Custom Feedback Collection Robots
 (Chung et al., ROMAN2018) 12

Slide 13

Slide 13 text

Formative Studies • To Identify & Gather Requirements • Need Finding Interviews,
 Online User Surveys Deployment Evaluations • To Test Feasibility in Context • To Extract Design Implications • Long-term (max. 129days) Image source: https://www.ihg.com/crowneplaza/hotels/gb/en/milpitas/sfomp/hoteldetail/photos-tours 13

Slide 14

Slide 14 text

Findings: Requirements Hotels care about • catching unhappy 
 guests on-site • spreading good reviews Guests care about • convenience • privacy 14

Slide 15

Slide 15 text

Findings: Design Implications • keep interactions short • use movement judiciously • tailor interactions to target group 15

Slide 16

Slide 16 text

Outline 1. Information Gathering Robot Applications in Commercial Spaces (IROS2015&2016, ROMAN2018) 2. A Programming System for Interactive Service Robots (ICSR2016) 3. Concurrency Interfaces for Programming Interactive Robots (ROMAN2020) 4. Iterative Repair of Robot Programs with Implicit User Feedback (RSS2020) [ICSR 2016] Michael Jae-Yoon Chung, Justin Huang, Leila Takayama, Tessa Lau, Maya Cakmak. Iterative Design of a System for Programming Socially Interactive Service Robots. International Conference on Social Robotics (ICSR), 2016. 16

Slide 17

Slide 17 text

Motivation Information Gathering • Long-tailed Question Types Customer Feedback • Needs for Customized Contents • for each guest group, e.g., • tourists • business travelers 17

Slide 18

Slide 18 text

Research Question 
 “How should we design a programming system
 to support creating interactive behaviors for service robots?” • What are the programming requirements? • What are the challenges for users? 18

Slide 19

Slide 19 text

Overview of the Approach 1. Understand/Implement 2. Prototype/Evaluate/Refine 3. Prototype/Evaluate Hello! I’m Relay 19

Slide 20

Slide 20 text

Descriptions of 2/5 locations involved in the formative study 1. Understand/Prepare The Key Finding:
 Most desired use cases involved interacting with visitors 20

Slide 21

Slide 21 text

1. Understand/Prepare CustomPrograms (Huang et al., HRI2016) Extended CustomPrograms • Block-based Visual Programming • Battletested userability 
 for non-experts 21

Slide 22

Slide 22 text

Prototype/Evaluate/Refine #1 A picture from the deployment • Prototyped 2 apps with airport staff members • Deployed at an airport
 4 times (3-4hrs each) over 2 weeks • Collected systems logs and observation notes 22

Slide 23

Slide 23 text

Prototype/Evaluate/Refine #2 • Airport staff members led programming • Deployed at an airport
 on 2 weekends
 (Fri/Sat/Sun, 9am-5pm) • Focused observing
 the non-experts,
 e.g., staff members • Made small updates 23

Slide 24

Slide 24 text

Key Extentions Name Returns getDetectedPeople() List enable more proactive behaviors Name Returns goToUntil(string location, string eventName) bool enable more reactive behaviors enable more expressive behaviors Perception primitives Interruptible primitives Asynchonous primitives Name Returns startSound(string sound) void 24

Slide 25

Slide 25 text

Findings: Requirements • Expressing Concurrency • for creating rich interactions (with ease) • Modifying Low-level Behaviors • to create natural and robust interactions • We’ll revisit these soon… 25

Slide 26

Slide 26 text

Findings: Challenges for Non-Experts • Programming from 
 Scratch and Debugging • Writing Readable and 
 Re-usable Code • Creating Engaging Behaviors A program used in the field 26

Slide 27

Slide 27 text

Outline 1. Information Gathering Robots in Commercial Spaces
 (IROS2015&2016, ROMAN2018) 2. Programming System for Interactive Service Robots
 (ICSR2016) 3. Concurrency in Programming Interactive Robots
 (ROMAN2020) 4. Iterative Program Repair with Implicit User Feedback
 (RSS2020) [RO-MAN 2020] Michael Jae-Yoon Chung, Maya Cakmak. IConCodeIt! A Comparison of Concurrency Interfaces in Block-based Visual Robot Programming. IEEE International Symposium on Robot and Human Interactive Communication (RO- MAN), 2020. 27

Slide 28

Slide 28 text

Reviewing the Extentsions “Go to ‘Lobby 1’ WHILE waiting for a screen touch event” start “Start sound AND shimmy in PARALLEL” 28

Slide 29

Slide 29 text

Research Question 
 “How should we design a concurrency interface for block-based visual programming for interactive robots?” 29

Slide 30

Slide 30 text

Block-Based Visual Robot Programming • Why? • End-users’ familiarity • Widely-used • Assumptions • Visual Programming Interface + Robot Actions • Wraps a procedural scripting language, e.g., python ABB's Wizard Easy Programming
 is based off Blockly. | Credit: ABB 30

Slide 31

Slide 31 text

31

Slide 32

Slide 32 text

Robot APIs: Actions Name say(string text) gesture(string expression) displayText(string text, number duration) displayText(string[] choices, number duration) sleep(number duration) Name startSaying(string text) startGesture(string expression) startDisplayText(string text, number duration) startDisplayText(string[] choices, number duration) startSleep(number duration) Blocking Non-Blocking 32

Slide 33

Slide 33 text

Robot APIs: Events Event Name Value Type speechDetected string buttonPressed string Event Name Value Type sayDone boolean gestureDone boolean displayTextDone boolean displayButtonDone boolean sleepDone boolean Externally
 Triggered Action-
 Triggered 33

Slide 34

Slide 34 text

Robot APIs: Events // register a callback when("speechDetected", function() {...}) // block until event waitFor("speechDetected") 34

Slide 35

Slide 35 text

Robot APIs: Events Event Name Last Event as State Name Value Type speechDetected lastDetectedSpeech string buttonPressed lastPressedButton string Event Name Action Status State Name Value Type sayDone isSaying boolean gestureDone isGesturing boolean displayTextDone isDisplayingText boolean displayButtonDone isDisplayingButton boolean sleepDone isSleeping boolean Externally
 Triggered Action-
 Triggered States 35

Slide 36

Slide 36 text

Robot APIs: Events // register a callback when("speechDetected", function() {...}) // block until event waitFor("speechDetected") // read a current state value getLastEventValue("speechDetected") 36

Slide 37

Slide 37 text

Three Interfaces Async • asynchronous procedure calls • imperative
 paradigm • used in e.g., socket programming Callback • callback • event-driven paradigm • used in, e.g., UI programming Waitfor • promise-based event synchronization • imperative 
 paradigm • e.g., async/await in event-loop 37

Slide 38

Slide 38 text

Three Interfaces Name Blocking actions Non-
 Blocking actions Blocking event (wait) callback
 (when) State
 accessors async ❌ ✅ ❌ ❌ ✅ callback ❌ ✅ ❌ ✅ ✅ waitfor ✅ ❌ ✅ ❌ ✅ 38

Slide 39

Slide 39 text

Example Robot Behavior Step 1: The robot should 
 say “Hello there, my name is Meebo. Goodbye now!” 
 and wait for the human to say something. 
 Step 2: When the robot finished speaking 
 or the human says something,
 the robot should display “On standby.” 39

Slide 40

Slide 40 text

Async • uses a loop and state accessors to “wait” 40

Slide 41

Slide 41 text

Callback • “finish” callback function 41

Slide 42

Slide 42 text

WaitFor • wait_for_one • wait_for(
 "speechDetected") 42

Slide 43

Slide 43 text

Concurrency Patterns Event Combinations • Action & Action • Action & Events • Events & Events Event Compositions • Wait-for-all • Wait-for-one 43

Slide 44

Slide 44 text

Example Robot Behavior Step 1: The robot should 
 say “Hello there, my name is Meebo. Goodbye now!” 
 and wait for the human to say something. 
 Step 2: When the robot finished speaking 
 or the human says something,
 the robot should display “On standby.” Action Wait-for-One Event: speechDetected Combination: Action & Event
 Composition: Wait-for-One 44

Slide 45

Slide 45 text

User Study: Procedure • Conducted Online • Between-group design • Participants • (N = 21, 14 females) • with some 
 programming experiences • Study duration: max. 90 min 45

Slide 46

Slide 46 text

User Study: Measures • Measure: Ease-of-Use • Scores using a rubric
 for Correctness • Number of blocks
 for Conciseness 46

Slide 47

Slide 47 text

“hello” “Bye” Online Study UI 47

Slide 48

Slide 48 text

Task Description • Step 1: On start, the robot should
 display “Press or say ‘start’ to begin interaction”
 and display a button with “start” text. • Step 2: 
 If the user presses the button or says “start,” 
 then the robot should introduce itself by 
 saying “Hello, my name is Meebo”
 and make a happy gesture. • Step 3: … • Step 4: … Action Event Wait-for-One Action Action Wait-for-All 48

Slide 49

Slide 49 text

Results: Correctness 0 5 10 15 Average Score async callback waitfor (most correct) waifor > async > callback (least correct) 49

Slide 50

Slide 50 text

(most concise) waifor < async < callback (least concise) 0 20 40 60 80 100 120 Average Number of Blocks async callback waitfor async callback -2 0 Ave async callback waitfor async callback Results: Conciseness 50

Slide 51

Slide 51 text

Results A loop for checking a change change induced here 51

Slide 52

Slide 52 text

Summary 1. Robot API Categorization • with concurrency in mind 2. Three Representative Interfaces • async, callback, waitfor 3. Comparison Study • systematic evaluation (not presented) • user study 52

Slide 53

Slide 53 text

Outline 1. Information Gathering Robot Applications in Commercial Spaces (IROS2015&2016, ROMAN2018) 2. A Programming System for Interactive Service Robots (ICSR2016) 3. Concurrency Interfaces for Programming Interactive Robots (ROMAN2020) 4. Iterative Repair of Robot Programs with Implicit User Feedback (RSS2020) [RSS 2020] Michael Jae-Yoon Chung, Maya Cakmak. Iterative Repair of Social Robot Programs from Implicit User Feedback via Bayesian Inference. Robotics: Science and Systems (RSS), 2020. 53

Slide 54

Slide 54 text

Motivation • Non-programmers need to modify low-level robot behaviors • Critical for the quality of interactions • But exposing low-level APIs is not an option • Parameter tuning is hard even for programmers • Uncertain limits, scale, sensitivity, consequences, … • Requires human users to test 54

Slide 55

Slide 55 text

Choreographe (Pot et al., ROMAN2009) a code editor 55

Slide 56

Slide 56 text

56

Slide 57

Slide 57 text

57

Slide 58

Slide 58 text

program sketch, K, Θ 1 lookingAtRobot = debounce(
 abs(inputFaceYawAngle < ??minDisengagedAngle),
 ??disengagedTimeout); speaking = ... if (state == "read" && inputSayFinished 
 && (lookingAtRobot || speaking)) { return "wait"; } ... Hole variable name Distribution ??minDisengagedAngle Categorical ??disengagedTimeout Categorical ... ... 58

Slide 59

Slide 59 text

Interaction trace 1 interaction0
 I0, O0 2 Note: the robot in the video is not running the example program program sketch, K, Θ 59

Slide 60

Slide 60 text

1 2 600 700 800 900 1000 1100 1200 1300 1400 1500 1600 1700 1800 1900 500 say wait read wait goBackTapped Time (ms) inputs outputs state trace wait corrected state trace lookingAtRobot Error Correction program sketch, K, Θ interaction0
 I0, O0 corrected0
 b Oi AAAB9HicbVBNS8NAEJ34WetX1aOXxSJ4KokIeix68WYF+wFtKJvNtl262cTdSaWE/g4vHhTx6o/x5r9x2+agrQ8GHu/NMDMvSKQw6Lrfzsrq2vrGZmGruL2zu7dfOjhsmDjVjNdZLGPdCqjhUiheR4GStxLNaRRI3gyGN1O/OeLaiFg94DjhfkT7SvQEo2glv/MkQj6gmN1NuqJbKrsVdwayTLyclCFHrVv66oQxSyOukElqTNtzE/QzqlEwySfFTmp4QtmQ9nnbUkUjbvxsdvSEnFolJL1Y21JIZurviYxGxoyjwHZGFAdm0ZuK/3ntFHtXfiZUkiJXbL6ol0qCMZkmQEKhOUM5toQyLeythA2opgxtTkUbgrf48jJpnFc8t+LdX5Sr13kcBTiGEzgDDy6hCrdQgzoweIRneIU3Z+S8OO/Ox7x1xclnjuAPnM8fGq6STQ== AAAB9HicbVBNS8NAEJ34WetX1aOXxSJ4KokIeix68WYF+wFtKJvNtl262cTdSaWE/g4vHhTx6o/x5r9x2+agrQ8GHu/NMDMvSKQw6Lrfzsrq2vrGZmGruL2zu7dfOjhsmDjVjNdZLGPdCqjhUiheR4GStxLNaRRI3gyGN1O/OeLaiFg94DjhfkT7SvQEo2glv/MkQj6gmN1NuqJbKrsVdwayTLyclCFHrVv66oQxSyOukElqTNtzE/QzqlEwySfFTmp4QtmQ9nnbUkUjbvxsdvSEnFolJL1Y21JIZurviYxGxoyjwHZGFAdm0ZuK/3ntFHtXfiZUkiJXbL6ol0qCMZkmQEKhOUM5toQyLeythA2opgxtTkUbgrf48jJpnFc8t+LdX5Sr13kcBTiGEzgDDy6hCrdQgzoweIRneIU3Z+S8OO/Ox7x1xclnjuAPnM8fGq6STQ== AAAB9HicbVBNS8NAEJ34WetX1aOXxSJ4KokIeix68WYF+wFtKJvNtl262cTdSaWE/g4vHhTx6o/x5r9x2+agrQ8GHu/NMDMvSKQw6Lrfzsrq2vrGZmGruL2zu7dfOjhsmDjVjNdZLGPdCqjhUiheR4GStxLNaRRI3gyGN1O/OeLaiFg94DjhfkT7SvQEo2glv/MkQj6gmN1NuqJbKrsVdwayTLyclCFHrVv66oQxSyOukElqTNtzE/QzqlEwySfFTmp4QtmQ9nnbUkUjbvxsdvSEnFolJL1Y21JIZurviYxGxoyjwHZGFAdm0ZuK/3ntFHtXfiZUkiJXbL6ol0qCMZkmQEKhOUM5toQyLeythA2opgxtTkUbgrf48jJpnFc8t+LdX5Sr13kcBTiGEzgDDy6hCrdQgzoweIRneIU3Z+S8OO/Ox7x1xclnjuAPnM8fGq6STQ== AAAB2XicbZDNSgMxFIXv1L86Vq1rN8EiuCozbnQpuHFZwbZCO5RM5k4bmskMyR2hDH0BF25EfC93vo3pz0JbDwQ+zknIvSculLQUBN9ebWd3b/+gfugfNfzjk9Nmo2fz0gjsilzl5jnmFpXU2CVJCp8LgzyLFfbj6f0i77+gsTLXTzQrMMr4WMtUCk7O6oyaraAdLMW2IVxDC9YaNb+GSS7KDDUJxa0dhEFBUcUNSaFw7g9LiwUXUz7GgUPNM7RRtRxzzi6dk7A0N+5oYkv394uKZ9bOstjdzDhN7Ga2MP/LBiWlt1EldVESarH6KC0Vo5wtdmaJNChIzRxwYaSblYkJN1yQa8Z3HYSbG29D77odBu3wMYA6nMMFXEEIN3AHD9CBLghI4BXevYn35n2suqp569LO4I+8zx84xIo4 AAAB6XicbZBLS8NAFIVvfNZatbp1EyyCq5K40aXgxp0V7APaUCaTm3boZBJnbiol9He4caGIf8id/8bpY6GtBwY+zpnh3jlhJoUhz/t2Nja3tnd2S3vl/crB4VH1uNIyaa45NnkqU90JmUEpFDZJkMROppElocR2OLqd5e0xaiNS9UiTDIOEDZSIBWdkraD3LCIcMirup33Rr9a8ujeXuw7+EmqwVKNf/epFKc8TVMQlM6brexkFBdMkuMRpuZcbzBgfsQF2LSqWoAmK+dJT99w6kRun2h5F7tz9/aJgiTGTJLQ3E0ZDs5rNzP+ybk7xdVAIleWEii8Gxbl0KXVnDbiR0MhJTiwwroXd1eVDphkn21PZluCvfnkdWpd136v7Dx6U4BTO4AJ8uIIbuIMGNIHDE7zAG7w7Y+fV+VjUteEsezuBP3I+fwDNppDu AAAB6XicbZBLS8NAFIVvfNZatbp1EyyCq5K40aXgxp0V7APaUCaTm3boZBJnbiol9He4caGIf8id/8bpY6GtBwY+zpnh3jlhJoUhz/t2Nja3tnd2S3vl/crB4VH1uNIyaa45NnkqU90JmUEpFDZJkMROppElocR2OLqd5e0xaiNS9UiTDIOEDZSIBWdkraD3LCIcMirup33Rr9a8ujeXuw7+EmqwVKNf/epFKc8TVMQlM6brexkFBdMkuMRpuZcbzBgfsQF2LSqWoAmK+dJT99w6kRun2h5F7tz9/aJgiTGTJLQ3E0ZDs5rNzP+ybk7xdVAIleWEii8Gxbl0KXVnDbiR0MhJTiwwroXd1eVDphkn21PZluCvfnkdWpd136v7Dx6U4BTO4AJ8uIIbuIMGNIHDE7zAG7w7Y+fV+VjUteEsezuBP3I+fwDNppDu AAAB9HicbVA9SwNBEN2LXzF+RS1tFoNgFe5stAza2BnBfEByhL29SbJkb+/cnYuEI7/DxkIRW3+Mnf/GTXKFJj4YeLw3w8y8IJHCoOt+O4W19Y3NreJ2aWd3b/+gfHjUNHGqOTR4LGPdDpgBKRQ0UKCEdqKBRYGEVjC6mfmtMWgjYvWAkwT8iA2U6AvO0Ep+90mEMGSY3U17oleuuFV3DrpKvJxUSI56r/zVDWOeRqCQS2ZMx3MT9DOmUXAJ01I3NZAwPmID6FiqWATGz+ZHT+mZVULaj7UthXSu/p7IWGTMJApsZ8RwaJa9mfif10mxf+VnQiUpguKLRf1UUozpLAEaCg0c5cQSxrWwt1I+ZJpxtDmVbAje8surpHlR9dyqd+9Watd5HEVyQk7JOfHIJamRW1InDcLJI3kmr+TNGTsvzrvzsWgtOPnMMfkD5/MHGW6SSQ== AAAB9HicbVBNS8NAEJ34WetX1aOXxSJ4KokIeix68WYF+wFtKJvNtl262cTdSaWE/g4vHhTx6o/x5r9x2+agrQ8GHu/NMDMvSKQw6Lrfzsrq2vrGZmGruL2zu7dfOjhsmDjVjNdZLGPdCqjhUiheR4GStxLNaRRI3gyGN1O/OeLaiFg94DjhfkT7SvQEo2glv/MkQj6gmN1NuqJbKrsVdwayTLyclCFHrVv66oQxSyOukElqTNtzE/QzqlEwySfFTmp4QtmQ9nnbUkUjbvxsdvSEnFolJL1Y21JIZurviYxGxoyjwHZGFAdm0ZuK/3ntFHtXfiZUkiJXbL6ol0qCMZkmQEKhOUM5toQyLeythA2opgxtTkUbgrf48jJpnFc8t+LdX5Sr13kcBTiGEzgDDy6hCrdQgzoweIRneIU3Z+S8OO/Ox7x1xclnjuAPnM8fGq6STQ== AAAB9HicbVBNS8NAEJ34WetX1aOXxSJ4KokIeix68WYF+wFtKJvNtl262cTdSaWE/g4vHhTx6o/x5r9x2+agrQ8GHu/NMDMvSKQw6Lrfzsrq2vrGZmGruL2zu7dfOjhsmDjVjNdZLGPdCqjhUiheR4GStxLNaRRI3gyGN1O/OeLaiFg94DjhfkT7SvQEo2glv/MkQj6gmN1NuqJbKrsVdwayTLyclCFHrVv66oQxSyOukElqTNtzE/QzqlEwySfFTmp4QtmQ9nnbUkUjbvxsdvSEnFolJL1Y21JIZurviYxGxoyjwHZGFAdm0ZuK/3ntFHtXfiZUkiJXbL6ol0qCMZkmQEKhOUM5toQyLeythA2opgxtTkUbgrf48jJpnFc8t+LdX5Sr13kcBTiGEzgDDy6hCrdQgzoweIRneIU3Z+S8OO/Ox7x1xclnjuAPnM8fGq6STQ== AAAB9HicbVBNS8NAEJ34WetX1aOXxSJ4KokIeix68WYF+wFtKJvNtl262cTdSaWE/g4vHhTx6o/x5r9x2+agrQ8GHu/NMDMvSKQw6Lrfzsrq2vrGZmGruL2zu7dfOjhsmDjVjNdZLGPdCqjhUiheR4GStxLNaRRI3gyGN1O/OeLaiFg94DjhfkT7SvQEo2glv/MkQj6gmN1NuqJbKrsVdwayTLyclCFHrVv66oQxSyOukElqTNtzE/QzqlEwySfFTmp4QtmQ9nnbUkUjbvxsdvSEnFolJL1Y21JIZurviYxGxoyjwHZGFAdm0ZuK/3ntFHtXfiZUkiJXbL6ol0qCMZkmQEKhOUM5toQyLeythA2opgxtTkUbgrf48jJpnFc8t+LdX5Sr13kcBTiGEzgDDy6hCrdQgzoweIRneIU3Z+S8OO/Ox7x1xclnjuAPnM8fGq6STQ== AAAB9HicbVBNS8NAEJ34WetX1aOXxSJ4KokIeix68WYF+wFtKJvNtl262cTdSaWE/g4vHhTx6o/x5r9x2+agrQ8GHu/NMDMvSKQw6Lrfzsrq2vrGZmGruL2zu7dfOjhsmDjVjNdZLGPdCqjhUiheR4GStxLNaRRI3gyGN1O/OeLaiFg94DjhfkT7SvQEo2glv/MkQj6gmN1NuqJbKrsVdwayTLyclCFHrVv66oQxSyOukElqTNtzE/QzqlEwySfFTmp4QtmQ9nnbUkUjbvxsdvSEnFolJL1Y21JIZurviYxGxoyjwHZGFAdm0ZuK/3ntFHtXfiZUkiJXbL6ol0qCMZkmQEKhOUM5toQyLeythA2opgxtTkUbgrf48jJpnFc8t+LdX5Sr13kcBTiGEzgDDy6hCrdQgzoweIRneIU3Z+S8OO/Ox7x1xclnjuAPnM8fGq6STQ== AAAB9HicbVBNS8NAEJ34WetX1aOXxSJ4KokIeix68WYF+wFtKJvNtl262cTdSaWE/g4vHhTx6o/x5r9x2+agrQ8GHu/NMDMvSKQw6Lrfzsrq2vrGZmGruL2zu7dfOjhsmDjVjNdZLGPdCqjhUiheR4GStxLNaRRI3gyGN1O/OeLaiFg94DjhfkT7SvQEo2glv/MkQj6gmN1NuqJbKrsVdwayTLyclCFHrVv66oQxSyOukElqTNtzE/QzqlEwySfFTmp4QtmQ9nnbUkUjbvxsdvSEnFolJL1Y21JIZurviYxGxoyjwHZGFAdm0ZuK/3ntFHtXfiZUkiJXbL6ol0qCMZkmQEKhOUM5toQyLeythA2opgxtTkUbgrf48jJpnFc8t+LdX5Sr13kcBTiGEzgDDy6hCrdQgzoweIRneIU3Z+S8OO/Ox7x1xclnjuAPnM8fGq6STQ== AAAB9HicbVBNS8NAEJ34WetX1aOXxSJ4KokIeix68WYF+wFtKJvNtl262cTdSaWE/g4vHhTx6o/x5r9x2+agrQ8GHu/NMDMvSKQw6Lrfzsrq2vrGZmGruL2zu7dfOjhsmDjVjNdZLGPdCqjhUiheR4GStxLNaRRI3gyGN1O/OeLaiFg94DjhfkT7SvQEo2glv/MkQj6gmN1NuqJbKrsVdwayTLyclCFHrVv66oQxSyOukElqTNtzE/QzqlEwySfFTmp4QtmQ9nnbUkUjbvxsdvSEnFolJL1Y21JIZurviYxGxoyjwHZGFAdm0ZuK/3ntFHtXfiZUkiJXbL6ol0qCMZkmQEKhOUM5toQyLeythA2opgxtTkUbgrf48jJpnFc8t+LdX5Sr13kcBTiGEzgDDy6hCrdQgzoweIRneIU3Z+S8OO/Ox7x1xclnjuAPnM8fGq6STQ== 3 60

Slide 61

Slide 61 text

600 700 800 900 1000 1100 1200 1300 1400 1500 1600 1700 1800 1900 500 say wait wait read wait goBackTapped Time (ms) inputs outputs state trace corrected state trace lookingAtRobot Error Correction User feedback 1 2 corrected0
 b Oi AAAB9HicbVBNS8NAEJ34WetX1aOXxSJ4KokIeix68WYF+wFtKJvNtl262cTdSaWE/g4vHhTx6o/x5r9x2+agrQ8GHu/NMDMvSKQw6Lrfzsrq2vrGZmGruL2zu7dfOjhsmDjVjNdZLGPdCqjhUiheR4GStxLNaRRI3gyGN1O/OeLaiFg94DjhfkT7SvQEo2glv/MkQj6gmN1NuqJbKrsVdwayTLyclCFHrVv66oQxSyOukElqTNtzE/QzqlEwySfFTmp4QtmQ9nnbUkUjbvxsdvSEnFolJL1Y21JIZurviYxGxoyjwHZGFAdm0ZuK/3ntFHtXfiZUkiJXbL6ol0qCMZkmQEKhOUM5toQyLeythA2opgxtTkUbgrf48jJpnFc8t+LdX5Sr13kcBTiGEzgDDy6hCrdQgzoweIRneIU3Z+S8OO/Ox7x1xclnjuAPnM8fGq6STQ== AAAB9HicbVBNS8NAEJ34WetX1aOXxSJ4KokIeix68WYF+wFtKJvNtl262cTdSaWE/g4vHhTx6o/x5r9x2+agrQ8GHu/NMDMvSKQw6Lrfzsrq2vrGZmGruL2zu7dfOjhsmDjVjNdZLGPdCqjhUiheR4GStxLNaRRI3gyGN1O/OeLaiFg94DjhfkT7SvQEo2glv/MkQj6gmN1NuqJbKrsVdwayTLyclCFHrVv66oQxSyOukElqTNtzE/QzqlEwySfFTmp4QtmQ9nnbUkUjbvxsdvSEnFolJL1Y21JIZurviYxGxoyjwHZGFAdm0ZuK/3ntFHtXfiZUkiJXbL6ol0qCMZkmQEKhOUM5toQyLeythA2opgxtTkUbgrf48jJpnFc8t+LdX5Sr13kcBTiGEzgDDy6hCrdQgzoweIRneIU3Z+S8OO/Ox7x1xclnjuAPnM8fGq6STQ== AAAB9HicbVBNS8NAEJ34WetX1aOXxSJ4KokIeix68WYF+wFtKJvNtl262cTdSaWE/g4vHhTx6o/x5r9x2+agrQ8GHu/NMDMvSKQw6Lrfzsrq2vrGZmGruL2zu7dfOjhsmDjVjNdZLGPdCqjhUiheR4GStxLNaRRI3gyGN1O/OeLaiFg94DjhfkT7SvQEo2glv/MkQj6gmN1NuqJbKrsVdwayTLyclCFHrVv66oQxSyOukElqTNtzE/QzqlEwySfFTmp4QtmQ9nnbUkUjbvxsdvSEnFolJL1Y21JIZurviYxGxoyjwHZGFAdm0ZuK/3ntFHtXfiZUkiJXbL6ol0qCMZkmQEKhOUM5toQyLeythA2opgxtTkUbgrf48jJpnFc8t+LdX5Sr13kcBTiGEzgDDy6hCrdQgzoweIRneIU3Z+S8OO/Ox7x1xclnjuAPnM8fGq6STQ== AAAB2XicbZDNSgMxFIXv1L86Vq1rN8EiuCozbnQpuHFZwbZCO5RM5k4bmskMyR2hDH0BF25EfC93vo3pz0JbDwQ+zknIvSculLQUBN9ebWd3b/+gfugfNfzjk9Nmo2fz0gjsilzl5jnmFpXU2CVJCp8LgzyLFfbj6f0i77+gsTLXTzQrMMr4WMtUCk7O6oyaraAdLMW2IVxDC9YaNb+GSS7KDDUJxa0dhEFBUcUNSaFw7g9LiwUXUz7GgUPNM7RRtRxzzi6dk7A0N+5oYkv394uKZ9bOstjdzDhN7Ga2MP/LBiWlt1EldVESarH6KC0Vo5wtdmaJNChIzRxwYaSblYkJN1yQa8Z3HYSbG29D77odBu3wMYA6nMMFXEEIN3AHD9CBLghI4BXevYn35n2suqp569LO4I+8zx84xIo4 AAAB6XicbZBLS8NAFIVvfNZatbp1EyyCq5K40aXgxp0V7APaUCaTm3boZBJnbiol9He4caGIf8id/8bpY6GtBwY+zpnh3jlhJoUhz/t2Nja3tnd2S3vl/crB4VH1uNIyaa45NnkqU90JmUEpFDZJkMROppElocR2OLqd5e0xaiNS9UiTDIOEDZSIBWdkraD3LCIcMirup33Rr9a8ujeXuw7+EmqwVKNf/epFKc8TVMQlM6brexkFBdMkuMRpuZcbzBgfsQF2LSqWoAmK+dJT99w6kRun2h5F7tz9/aJgiTGTJLQ3E0ZDs5rNzP+ybk7xdVAIleWEii8Gxbl0KXVnDbiR0MhJTiwwroXd1eVDphkn21PZluCvfnkdWpd136v7Dx6U4BTO4AJ8uIIbuIMGNIHDE7zAG7w7Y+fV+VjUteEsezuBP3I+fwDNppDu AAAB6XicbZBLS8NAFIVvfNZatbp1EyyCq5K40aXgxp0V7APaUCaTm3boZBJnbiol9He4caGIf8id/8bpY6GtBwY+zpnh3jlhJoUhz/t2Nja3tnd2S3vl/crB4VH1uNIyaa45NnkqU90JmUEpFDZJkMROppElocR2OLqd5e0xaiNS9UiTDIOEDZSIBWdkraD3LCIcMirup33Rr9a8ujeXuw7+EmqwVKNf/epFKc8TVMQlM6brexkFBdMkuMRpuZcbzBgfsQF2LSqWoAmK+dJT99w6kRun2h5F7tz9/aJgiTGTJLQ3E0ZDs5rNzP+ybk7xdVAIleWEii8Gxbl0KXVnDbiR0MhJTiwwroXd1eVDphkn21PZluCvfnkdWpd136v7Dx6U4BTO4AJ8uIIbuIMGNIHDE7zAG7w7Y+fV+VjUteEsezuBP3I+fwDNppDu AAAB9HicbVA9SwNBEN2LXzF+RS1tFoNgFe5stAza2BnBfEByhL29SbJkb+/cnYuEI7/DxkIRW3+Mnf/GTXKFJj4YeLw3w8y8IJHCoOt+O4W19Y3NreJ2aWd3b/+gfHjUNHGqOTR4LGPdDpgBKRQ0UKCEdqKBRYGEVjC6mfmtMWgjYvWAkwT8iA2U6AvO0Ep+90mEMGSY3U17oleuuFV3DrpKvJxUSI56r/zVDWOeRqCQS2ZMx3MT9DOmUXAJ01I3NZAwPmID6FiqWATGz+ZHT+mZVULaj7UthXSu/p7IWGTMJApsZ8RwaJa9mfif10mxf+VnQiUpguKLRf1UUozpLAEaCg0c5cQSxrWwt1I+ZJpxtDmVbAje8surpHlR9dyqd+9Watd5HEVyQk7JOfHIJamRW1InDcLJI3kmr+TNGTsvzrvzsWgtOPnMMfkD5/MHGW6SSQ== AAAB9HicbVBNS8NAEJ34WetX1aOXxSJ4KokIeix68WYF+wFtKJvNtl262cTdSaWE/g4vHhTx6o/x5r9x2+agrQ8GHu/NMDMvSKQw6Lrfzsrq2vrGZmGruL2zu7dfOjhsmDjVjNdZLGPdCqjhUiheR4GStxLNaRRI3gyGN1O/OeLaiFg94DjhfkT7SvQEo2glv/MkQj6gmN1NuqJbKrsVdwayTLyclCFHrVv66oQxSyOukElqTNtzE/QzqlEwySfFTmp4QtmQ9nnbUkUjbvxsdvSEnFolJL1Y21JIZurviYxGxoyjwHZGFAdm0ZuK/3ntFHtXfiZUkiJXbL6ol0qCMZkmQEKhOUM5toQyLeythA2opgxtTkUbgrf48jJpnFc8t+LdX5Sr13kcBTiGEzgDDy6hCrdQgzoweIRneIU3Z+S8OO/Ox7x1xclnjuAPnM8fGq6STQ== AAAB9HicbVBNS8NAEJ34WetX1aOXxSJ4KokIeix68WYF+wFtKJvNtl262cTdSaWE/g4vHhTx6o/x5r9x2+agrQ8GHu/NMDMvSKQw6Lrfzsrq2vrGZmGruL2zu7dfOjhsmDjVjNdZLGPdCqjhUiheR4GStxLNaRRI3gyGN1O/OeLaiFg94DjhfkT7SvQEo2glv/MkQj6gmN1NuqJbKrsVdwayTLyclCFHrVv66oQxSyOukElqTNtzE/QzqlEwySfFTmp4QtmQ9nnbUkUjbvxsdvSEnFolJL1Y21JIZurviYxGxoyjwHZGFAdm0ZuK/3ntFHtXfiZUkiJXbL6ol0qCMZkmQEKhOUM5toQyLeythA2opgxtTkUbgrf48jJpnFc8t+LdX5Sr13kcBTiGEzgDDy6hCrdQgzoweIRneIU3Z+S8OO/Ox7x1xclnjuAPnM8fGq6STQ== AAAB9HicbVBNS8NAEJ34WetX1aOXxSJ4KokIeix68WYF+wFtKJvNtl262cTdSaWE/g4vHhTx6o/x5r9x2+agrQ8GHu/NMDMvSKQw6Lrfzsrq2vrGZmGruL2zu7dfOjhsmDjVjNdZLGPdCqjhUiheR4GStxLNaRRI3gyGN1O/OeLaiFg94DjhfkT7SvQEo2glv/MkQj6gmN1NuqJbKrsVdwayTLyclCFHrVv66oQxSyOukElqTNtzE/QzqlEwySfFTmp4QtmQ9nnbUkUjbvxsdvSEnFolJL1Y21JIZurviYxGxoyjwHZGFAdm0ZuK/3ntFHtXfiZUkiJXbL6ol0qCMZkmQEKhOUM5toQyLeythA2opgxtTkUbgrf48jJpnFc8t+LdX5Sr13kcBTiGEzgDDy6hCrdQgzoweIRneIU3Z+S8OO/Ox7x1xclnjuAPnM8fGq6STQ== AAAB9HicbVBNS8NAEJ34WetX1aOXxSJ4KokIeix68WYF+wFtKJvNtl262cTdSaWE/g4vHhTx6o/x5r9x2+agrQ8GHu/NMDMvSKQw6Lrfzsrq2vrGZmGruL2zu7dfOjhsmDjVjNdZLGPdCqjhUiheR4GStxLNaRRI3gyGN1O/OeLaiFg94DjhfkT7SvQEo2glv/MkQj6gmN1NuqJbKrsVdwayTLyclCFHrVv66oQxSyOukElqTNtzE/QzqlEwySfFTmp4QtmQ9nnbUkUjbvxsdvSEnFolJL1Y21JIZurviYxGxoyjwHZGFAdm0ZuK/3ntFHtXfiZUkiJXbL6ol0qCMZkmQEKhOUM5toQyLeythA2opgxtTkUbgrf48jJpnFc8t+LdX5Sr13kcBTiGEzgDDy6hCrdQgzoweIRneIU3Z+S8OO/Ox7x1xclnjuAPnM8fGq6STQ== AAAB9HicbVBNS8NAEJ34WetX1aOXxSJ4KokIeix68WYF+wFtKJvNtl262cTdSaWE/g4vHhTx6o/x5r9x2+agrQ8GHu/NMDMvSKQw6Lrfzsrq2vrGZmGruL2zu7dfOjhsmDjVjNdZLGPdCqjhUiheR4GStxLNaRRI3gyGN1O/OeLaiFg94DjhfkT7SvQEo2glv/MkQj6gmN1NuqJbKrsVdwayTLyclCFHrVv66oQxSyOukElqTNtzE/QzqlEwySfFTmp4QtmQ9nnbUkUjbvxsdvSEnFolJL1Y21JIZurviYxGxoyjwHZGFAdm0ZuK/3ntFHtXfiZUkiJXbL6ol0qCMZkmQEKhOUM5toQyLeythA2opgxtTkUbgrf48jJpnFc8t+LdX5Sr13kcBTiGEzgDDy6hCrdQgzoweIRneIU3Z+S8OO/Ox7x1xclnjuAPnM8fGq6STQ== AAAB9HicbVBNS8NAEJ34WetX1aOXxSJ4KokIeix68WYF+wFtKJvNtl262cTdSaWE/g4vHhTx6o/x5r9x2+agrQ8GHu/NMDMvSKQw6Lrfzsrq2vrGZmGruL2zu7dfOjhsmDjVjNdZLGPdCqjhUiheR4GStxLNaRRI3gyGN1O/OeLaiFg94DjhfkT7SvQEo2glv/MkQj6gmN1NuqJbKrsVdwayTLyclCFHrVv66oQxSyOukElqTNtzE/QzqlEwySfFTmp4QtmQ9nnbUkUjbvxsdvSEnFolJL1Y21JIZurviYxGxoyjwHZGFAdm0ZuK/3ntFHtXfiZUkiJXbL6ol0qCMZkmQEKhOUM5toQyLeythA2opgxtTkUbgrf48jJpnFc8t+LdX5Sr13kcBTiGEzgDDy6hCrdQgzoweIRneIU3Z+S8OO/Ox7x1xclnjuAPnM8fGq6STQ== program sketch, K, Θ interaction0
 I0, O0 3 61

Slide 62

Slide 62 text

(repaired) Interaction1
 I1, O1 ... ... ... ... ... ... At each iteration, do 1. Run the program and collect an interaction trace 2. Obtain corrections from the programmer or the user 3. Re-estimate hole distributions using the input and corrected state traces (posterior) 4. Fill-in holes using maximum a posteriori estimation (MAP) for the next iteration corrected0
 b Oi AAAB9HicbVBNS8NAEJ34WetX1aOXxSJ4KokIeix68WYF+wFtKJvNtl262cTdSaWE/g4vHhTx6o/x5r9x2+agrQ8GHu/NMDMvSKQw6Lrfzsrq2vrGZmGruL2zu7dfOjhsmDjVjNdZLGPdCqjhUiheR4GStxLNaRRI3gyGN1O/OeLaiFg94DjhfkT7SvQEo2glv/MkQj6gmN1NuqJbKrsVdwayTLyclCFHrVv66oQxSyOukElqTNtzE/QzqlEwySfFTmp4QtmQ9nnbUkUjbvxsdvSEnFolJL1Y21JIZurviYxGxoyjwHZGFAdm0ZuK/3ntFHtXfiZUkiJXbL6ol0qCMZkmQEKhOUM5toQyLeythA2opgxtTkUbgrf48jJpnFc8t+LdX5Sr13kcBTiGEzgDDy6hCrdQgzoweIRneIU3Z+S8OO/Ox7x1xclnjuAPnM8fGq6STQ== AAAB9HicbVBNS8NAEJ34WetX1aOXxSJ4KokIeix68WYF+wFtKJvNtl262cTdSaWE/g4vHhTx6o/x5r9x2+agrQ8GHu/NMDMvSKQw6Lrfzsrq2vrGZmGruL2zu7dfOjhsmDjVjNdZLGPdCqjhUiheR4GStxLNaRRI3gyGN1O/OeLaiFg94DjhfkT7SvQEo2glv/MkQj6gmN1NuqJbKrsVdwayTLyclCFHrVv66oQxSyOukElqTNtzE/QzqlEwySfFTmp4QtmQ9nnbUkUjbvxsdvSEnFolJL1Y21JIZurviYxGxoyjwHZGFAdm0ZuK/3ntFHtXfiZUkiJXbL6ol0qCMZkmQEKhOUM5toQyLeythA2opgxtTkUbgrf48jJpnFc8t+LdX5Sr13kcBTiGEzgDDy6hCrdQgzoweIRneIU3Z+S8OO/Ox7x1xclnjuAPnM8fGq6STQ== AAAB9HicbVBNS8NAEJ34WetX1aOXxSJ4KokIeix68WYF+wFtKJvNtl262cTdSaWE/g4vHhTx6o/x5r9x2+agrQ8GHu/NMDMvSKQw6Lrfzsrq2vrGZmGruL2zu7dfOjhsmDjVjNdZLGPdCqjhUiheR4GStxLNaRRI3gyGN1O/OeLaiFg94DjhfkT7SvQEo2glv/MkQj6gmN1NuqJbKrsVdwayTLyclCFHrVv66oQxSyOukElqTNtzE/QzqlEwySfFTmp4QtmQ9nnbUkUjbvxsdvSEnFolJL1Y21JIZurviYxGxoyjwHZGFAdm0ZuK/3ntFHtXfiZUkiJXbL6ol0qCMZkmQEKhOUM5toQyLeythA2opgxtTkUbgrf48jJpnFc8t+LdX5Sr13kcBTiGEzgDDy6hCrdQgzoweIRneIU3Z+S8OO/Ox7x1xclnjuAPnM8fGq6STQ== AAAB2XicbZDNSgMxFIXv1L86Vq1rN8EiuCozbnQpuHFZwbZCO5RM5k4bmskMyR2hDH0BF25EfC93vo3pz0JbDwQ+zknIvSculLQUBN9ebWd3b/+gfugfNfzjk9Nmo2fz0gjsilzl5jnmFpXU2CVJCp8LgzyLFfbj6f0i77+gsTLXTzQrMMr4WMtUCk7O6oyaraAdLMW2IVxDC9YaNb+GSS7KDDUJxa0dhEFBUcUNSaFw7g9LiwUXUz7GgUPNM7RRtRxzzi6dk7A0N+5oYkv394uKZ9bOstjdzDhN7Ga2MP/LBiWlt1EldVESarH6KC0Vo5wtdmaJNChIzRxwYaSblYkJN1yQa8Z3HYSbG29D77odBu3wMYA6nMMFXEEIN3AHD9CBLghI4BXevYn35n2suqp569LO4I+8zx84xIo4 AAAB6XicbZBLS8NAFIVvfNZatbp1EyyCq5K40aXgxp0V7APaUCaTm3boZBJnbiol9He4caGIf8id/8bpY6GtBwY+zpnh3jlhJoUhz/t2Nja3tnd2S3vl/crB4VH1uNIyaa45NnkqU90JmUEpFDZJkMROppElocR2OLqd5e0xaiNS9UiTDIOEDZSIBWdkraD3LCIcMirup33Rr9a8ujeXuw7+EmqwVKNf/epFKc8TVMQlM6brexkFBdMkuMRpuZcbzBgfsQF2LSqWoAmK+dJT99w6kRun2h5F7tz9/aJgiTGTJLQ3E0ZDs5rNzP+ybk7xdVAIleWEii8Gxbl0KXVnDbiR0MhJTiwwroXd1eVDphkn21PZluCvfnkdWpd136v7Dx6U4BTO4AJ8uIIbuIMGNIHDE7zAG7w7Y+fV+VjUteEsezuBP3I+fwDNppDu AAAB6XicbZBLS8NAFIVvfNZatbp1EyyCq5K40aXgxp0V7APaUCaTm3boZBJnbiol9He4caGIf8id/8bpY6GtBwY+zpnh3jlhJoUhz/t2Nja3tnd2S3vl/crB4VH1uNIyaa45NnkqU90JmUEpFDZJkMROppElocR2OLqd5e0xaiNS9UiTDIOEDZSIBWdkraD3LCIcMirup33Rr9a8ujeXuw7+EmqwVKNf/epFKc8TVMQlM6brexkFBdMkuMRpuZcbzBgfsQF2LSqWoAmK+dJT99w6kRun2h5F7tz9/aJgiTGTJLQ3E0ZDs5rNzP+ybk7xdVAIleWEii8Gxbl0KXVnDbiR0MhJTiwwroXd1eVDphkn21PZluCvfnkdWpd136v7Dx6U4BTO4AJ8uIIbuIMGNIHDE7zAG7w7Y+fV+VjUteEsezuBP3I+fwDNppDu AAAB9HicbVA9SwNBEN2LXzF+RS1tFoNgFe5stAza2BnBfEByhL29SbJkb+/cnYuEI7/DxkIRW3+Mnf/GTXKFJj4YeLw3w8y8IJHCoOt+O4W19Y3NreJ2aWd3b/+gfHjUNHGqOTR4LGPdDpgBKRQ0UKCEdqKBRYGEVjC6mfmtMWgjYvWAkwT8iA2U6AvO0Ep+90mEMGSY3U17oleuuFV3DrpKvJxUSI56r/zVDWOeRqCQS2ZMx3MT9DOmUXAJ01I3NZAwPmID6FiqWATGz+ZHT+mZVULaj7UthXSu/p7IWGTMJApsZ8RwaJa9mfif10mxf+VnQiUpguKLRf1UUozpLAEaCg0c5cQSxrWwt1I+ZJpxtDmVbAje8surpHlR9dyqd+9Watd5HEVyQk7JOfHIJamRW1InDcLJI3kmr+TNGTsvzrvzsWgtOPnMMfkD5/MHGW6SSQ== AAAB9HicbVBNS8NAEJ34WetX1aOXxSJ4KokIeix68WYF+wFtKJvNtl262cTdSaWE/g4vHhTx6o/x5r9x2+agrQ8GHu/NMDMvSKQw6Lrfzsrq2vrGZmGruL2zu7dfOjhsmDjVjNdZLGPdCqjhUiheR4GStxLNaRRI3gyGN1O/OeLaiFg94DjhfkT7SvQEo2glv/MkQj6gmN1NuqJbKrsVdwayTLyclCFHrVv66oQxSyOukElqTNtzE/QzqlEwySfFTmp4QtmQ9nnbUkUjbvxsdvSEnFolJL1Y21JIZurviYxGxoyjwHZGFAdm0ZuK/3ntFHtXfiZUkiJXbL6ol0qCMZkmQEKhOUM5toQyLeythA2opgxtTkUbgrf48jJpnFc8t+LdX5Sr13kcBTiGEzgDDy6hCrdQgzoweIRneIU3Z+S8OO/Ox7x1xclnjuAPnM8fGq6STQ== AAAB9HicbVBNS8NAEJ34WetX1aOXxSJ4KokIeix68WYF+wFtKJvNtl262cTdSaWE/g4vHhTx6o/x5r9x2+agrQ8GHu/NMDMvSKQw6Lrfzsrq2vrGZmGruL2zu7dfOjhsmDjVjNdZLGPdCqjhUiheR4GStxLNaRRI3gyGN1O/OeLaiFg94DjhfkT7SvQEo2glv/MkQj6gmN1NuqJbKrsVdwayTLyclCFHrVv66oQxSyOukElqTNtzE/QzqlEwySfFTmp4QtmQ9nnbUkUjbvxsdvSEnFolJL1Y21JIZurviYxGxoyjwHZGFAdm0ZuK/3ntFHtXfiZUkiJXbL6ol0qCMZkmQEKhOUM5toQyLeythA2opgxtTkUbgrf48jJpnFc8t+LdX5Sr13kcBTiGEzgDDy6hCrdQgzoweIRneIU3Z+S8OO/Ox7x1xclnjuAPnM8fGq6STQ== AAAB9HicbVBNS8NAEJ34WetX1aOXxSJ4KokIeix68WYF+wFtKJvNtl262cTdSaWE/g4vHhTx6o/x5r9x2+agrQ8GHu/NMDMvSKQw6Lrfzsrq2vrGZmGruL2zu7dfOjhsmDjVjNdZLGPdCqjhUiheR4GStxLNaRRI3gyGN1O/OeLaiFg94DjhfkT7SvQEo2glv/MkQj6gmN1NuqJbKrsVdwayTLyclCFHrVv66oQxSyOukElqTNtzE/QzqlEwySfFTmp4QtmQ9nnbUkUjbvxsdvSEnFolJL1Y21JIZurviYxGxoyjwHZGFAdm0ZuK/3ntFHtXfiZUkiJXbL6ol0qCMZkmQEKhOUM5toQyLeythA2opgxtTkUbgrf48jJpnFc8t+LdX5Sr13kcBTiGEzgDDy6hCrdQgzoweIRneIU3Z+S8OO/Ox7x1xclnjuAPnM8fGq6STQ== AAAB9HicbVBNS8NAEJ34WetX1aOXxSJ4KokIeix68WYF+wFtKJvNtl262cTdSaWE/g4vHhTx6o/x5r9x2+agrQ8GHu/NMDMvSKQw6Lrfzsrq2vrGZmGruL2zu7dfOjhsmDjVjNdZLGPdCqjhUiheR4GStxLNaRRI3gyGN1O/OeLaiFg94DjhfkT7SvQEo2glv/MkQj6gmN1NuqJbKrsVdwayTLyclCFHrVv66oQxSyOukElqTNtzE/QzqlEwySfFTmp4QtmQ9nnbUkUjbvxsdvSEnFolJL1Y21JIZurviYxGxoyjwHZGFAdm0ZuK/3ntFHtXfiZUkiJXbL6ol0qCMZkmQEKhOUM5toQyLeythA2opgxtTkUbgrf48jJpnFc8t+LdX5Sr13kcBTiGEzgDDy6hCrdQgzoweIRneIU3Z+S8OO/Ox7x1xclnjuAPnM8fGq6STQ== AAAB9HicbVBNS8NAEJ34WetX1aOXxSJ4KokIeix68WYF+wFtKJvNtl262cTdSaWE/g4vHhTx6o/x5r9x2+agrQ8GHu/NMDMvSKQw6Lrfzsrq2vrGZmGruL2zu7dfOjhsmDjVjNdZLGPdCqjhUiheR4GStxLNaRRI3gyGN1O/OeLaiFg94DjhfkT7SvQEo2glv/MkQj6gmN1NuqJbKrsVdwayTLyclCFHrVv66oQxSyOukElqTNtzE/QzqlEwySfFTmp4QtmQ9nnbUkUjbvxsdvSEnFolJL1Y21JIZurviYxGxoyjwHZGFAdm0ZuK/3ntFHtXfiZUkiJXbL6ol0qCMZkmQEKhOUM5toQyLeythA2opgxtTkUbgrf48jJpnFc8t+LdX5Sr13kcBTiGEzgDDy6hCrdQgzoweIRneIU3Z+S8OO/Ox7x1xclnjuAPnM8fGq6STQ== AAAB9HicbVBNS8NAEJ34WetX1aOXxSJ4KokIeix68WYF+wFtKJvNtl262cTdSaWE/g4vHhTx6o/x5r9x2+agrQ8GHu/NMDMvSKQw6Lrfzsrq2vrGZmGruL2zu7dfOjhsmDjVjNdZLGPdCqjhUiheR4GStxLNaRRI3gyGN1O/OeLaiFg94DjhfkT7SvQEo2glv/MkQj6gmN1NuqJbKrsVdwayTLyclCFHrVv66oQxSyOukElqTNtzE/QzqlEwySfFTmp4QtmQ9nnbUkUjbvxsdvSEnFolJL1Y21JIZurviYxGxoyjwHZGFAdm0ZuK/3ntFHtXfiZUkiJXbL6ol0qCMZkmQEKhOUM5toQyLeythA2opgxtTkUbgrf48jJpnFc8t+LdX5Sr13kcBTiGEzgDDy6hCrdQgzoweIRneIU3Z+S8OO/Ox7x1xclnjuAPnM8fGq6STQ== 3 1 2 program sketch, K, Θ interaction0
 I0, O0 4 62

Slide 63

Slide 63 text

Iterative BayesRepair 63

Slide 64

Slide 64 text

Iterative BayesRepair 64

Slide 65

Slide 65 text

Iterative BayesRepair frees programmers makes it online friendly allows expressing uncertainty 65

Slide 66

Slide 66 text

• Human simulators for sampling input and ground truth state trace • Baseline: search-based repair • 3 scenarios, 5 repair iterations • Measures • % overlap (interaction quality) • Repair time in ms (speed) A top-down human-simulator. Samples from states, substates (intent), and then inputs Simulation Experiment: Setup 66

Slide 67

Slide 67 text

0 1 2 3 4 5 repair iterations 0.0 0.2 0.4 0.6 0.8 1.0 1.2 overlap % Storytelling 0 1 2 3 4 5 repair iterations 0.0 0.2 0.4 0.6 0.8 1.0 1.2 Neck Exercise 0 1 2 3 4 5 repair iterations 0.0 0.2 0.4 0.6 0.8 1.0 1.2 Open-ended Q&A bayesian search method Simulation Experiment: % Overlap Results 67

Slide 68

Slide 68 text

0 1 2 3 4 5 repair iterations 0.0 0.2 0.4 0.6 0.8 1.0 1.2 overlap % Storytelling 0 1 2 3 4 5 repair iterations 0.0 0.2 0.4 0.6 0.8 1.0 1.2 Neck Exercise 0 1 2 3 4 5 repair iterations 0.0 0.2 0.4 0.6 0.8 1.0 1.2 Open-ended Q&A bayesian search method Simulation Experiment: Speed Results 1 2 3 4 5 repair iterations 0 2,000 4,000 6,000 8,000 10,000 12,000 time (ms) Storytelling 1 2 3 4 5 repair iterations 0 1,000 2,000 3,000 4,000 Neck Exercise 1 2 3 4 5 repair iterations 0 20,000 40,000 60,000 80,000 100,000 120,000 Open-ended Q&A bayesian search method 68

Slide 69

Slide 69 text

• 10 participants • 3 females, avg 23.2 yrs • A custom-built social robot • 2 scenarios, 4 iterations Human experiment setup. Human Experiment: Setup 69

Slide 70

Slide 70 text

• Objective • Before/After repair overlap (%) • # of interaction breakdowns • total duration (sec) • Subjective • 3 HRI fluency questionnaire (Hoffman 2019) Human experiment setup. Human Experiment: Measures 70

Slide 71

Slide 71 text

Human Experiment: Objective Measures 0 1 2 3 repair iter. 0 50 100 150 duration (in sec) Neck Ex. 0 1 2 3 repair iter. 0 50 100 150 Open. Q&A 0 1 2 3 repair iter. -1 0 1 2 3 4 5 6 7 number of corrections Neck Ex. 0 1 2 3 repair iter. -1 0 1 2 3 4 5 6 7 Open. Q&A 71

Slide 72

Slide 72 text

Human Experiment: Subjective Measures One of subjective questions asked:
 
 Please rate your agreement with the following statements: • “I trusted the robot to do the right thing at the right time." 1: strongly disagree
 5: strongly agree 0 1 2 3 repair iter. 0 1 2 3 4 5 6 rating Neck Ex. 0 1 2 3 repair iter. 0 1 2 3 4 5 6 Open. Q&A 72

Slide 73

Slide 73 text

Qualitative Results: Before Repair 73

Slide 74

Slide 74 text

Qualitative Results: After Repair 74

Slide 75

Slide 75 text

Directions for Future Research • Starting with Programming Language for HRI • HRI Authoring as Programming Synthesis • Intuitive UIs for Non-Experts 75

Slide 76

Slide 76 text

Summary • Interactive Robot Applications in Commercial Spaces • User-Requirements and Design Implications • Concurrency in Interactive Robots Programming • Parameter Tuning via Iterative Program Repair 76

Slide 77

Slide 77 text

Source: The Independent, 7/16/2020 Source: Mashable, 10/07/2017 77

Slide 78

Slide 78 text

Thank You! • HCR Lab • Savioke • RSE Lab • Neural Systems Lab • CSE Staff • Family and Friends 78