ConCodeIt! A Comparison of Concurrency Interfaces in Block-Based Visual Robot Programming

Authors: Michael Jae-Yoon Chung, Mino Nakura, Sai Harshita Neti, Anthony Lu, Elana Hummel, and Maya Cakmak

Video: https://youtu.be/wnGNpzsvaK0

Abstract: Concurrency makes robot programming challenging even for professional programmers, yet it is essential for rich, interactive social robot behaviors. Visual programming aims to lower the barrier for robot programming but does not support rich concurrent behavior for meaningful robotics applications. In this paper, we explore extensions to block-based visual languages to enable programming of concurrent behavior with (1) asynchronous procedure calls, which encourage imperative programming, (2) callbacks, which encourage event-driven programming, and (3) promise, which also encourages imperative programming by providing event synchronization utilities. We compare these approaches through a systematic analysis of social robot programs with representative concurrency patterns, as well as a user study (N=23) in which participants authored such programs. Our work identifies characteristic differences between these approaches and demonstrates that the promise-based concurrency interface enables more concise programs with fewer errors.

Transcript

1. ConCodeIt! A Comparison of Concurrency Interfaces in Block- based Visual Robot Programming The 29th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN), 2020 Michael Jae-Yoon
 Chung Maya Cakmak Elana Hummel Sai Harshita
 Neti Mino Natura Anthony Lu 1
2. Source: Mashable, 10/07/2017 Source: The Independent, 7/16/2020 2
3. CustomPrograms,
 Huang et al. HRI 2016 CoBlox, Weintrop et al. CHI 2018 Wizard Easy Programming (Credit: ABB) Cozmo Code Lab (Source: THE VERGE) 3
4. What is Concurrency? gesture gaze Hello… speech screen 4
5. Expressing Concurrency “Start sound AND shimmy in PARALLEL” “Go to ‘Lobby 1’ WHILE waiting for a screen touch event” 5
6. Research Question “How should we design a concurrency interface for block-based visual programming for interactive robots?”
 6
7. Robot API • Robot actions as function calls • Designed to achieve use cases Block-Based Visual Robot Programming + Blockly • An open source visual programming interface like Scratch • Wraps a procedural 
 scripting language like Python Adopted from CustomPrograms, Huang et al. HRI 2016 7
9. Robot APIs: Actions Name say(string text) gesture(string expression) displayText(string text, number duration) displayButtons(string[] choices, number duration) sleep(number duration) Name startSaying(string text) startGesture(string expression) startDisplayText(string text, number duration) startDisplayButtons(string[] choices, number duration) startSleep(number duration) Blocking Non-Blocking 9
10. Robot APIs: Events Event Name Value Type speechDetected string buttonPressed string Event Name Value Type sayDone boolean gestureDone boolean displayTextDone boolean displayButtonDone boolean sleepDone boolean Externally
 Triggered Action
 Triggered // register a callback when(
 “speechDetected”,
 function() {...}); 
 // block until event waitFor(“speechDetected”); 
 // read current state value getLastEventValue(
 “speechDetected"); 10
11. Three Block-based Concurrency Interfaces Async • asynchronous procedure calls • imperative
 paradigm • e.g., socket programming Callback • callback
 • event-driven paradigm • e.g., UI programming Waitfor • promise-based event synchronization • imperative 
 paradigm • e.g., async/await pattern 11
12. Three Block-based Concurrency Interfaces Async • asynchronous procedure calls • imperative
 paradigm • e.g., socket programming Callback • callback
 • event-driven paradigm • e.g., UI programming Waitfor • promise-based event synchronization • imperative 
 paradigm • e.g., async/await pattern 12
13. Three Block-based Concurrency Interfaces Async • asynchronous procedure calls • imperative
 paradigm • e.g., socket programming Callback • callback
 • event-driven paradigm • e.g., UI programming Waitfor • promise-based event synchronization • imperative 
 paradigm • e.g., async/await pattern 13
14. Three Block-based Concurrency Interfaces Async • asynchronous procedure calls • imperative
 paradigm • e.g., socket programming Callback • callback
 • event-driven paradigm • e.g., UI programming Waitfor • promise-based event synchronization • imperative 
 paradigm • e.g., async/await pattern 14
15. Example Robot Behavior Step 1: The robot should 
 say “Hello there, my name is Meebo. Goodbye now!” 
 and wait for the human to say something.
 Step 2: When the robot finished speaking 
 or the human says something,
 the robot should display “On standby.” 15
16. Async Example Robot Behavior
 
 Step 1: The robot should 
 say “Hello there, my name is Meebo. Goodbye now!” 
 and wait for the human to say something.
 
 Step 2: When the robot finished speaking 
 or the human says something,
 the robot should display “On standby.” 16
17. Callback Example Robot Behavior
 
 Step 1: The robot should 
 say “Hello there, my name is Meebo. Goodbye now!” 
 and wait for the human to say something.
 
 Step 2: When the robot finished speaking 
 or the human says something,
 the robot should display “On standby.” 17
18. WaitFor Example Robot Behavior
 
 Step 1: The robot should 
 say “Hello there, my name is Meebo. Goodbye now!” 
 and wait for the human to say something.
 
 Step 2: When the robot finished speaking 
 or the human says something,
 the robot should display “On standby.” 18
19. Systematic Analysis • Goal: to compare conciseness • Procedure: expert
 implemented
 6 unit behaviors rep.
 concur. patterns • Measure: 
 conciseness via
 6 block counts 0 5 10 15 20 25 30 Average Number of Blocks async callback waitfor 0 1 2 3 Average Number of Functions async callback waitfor 0 1 2 3 Average Number of Variables async callback waitfor 0 1 2 3 Average Number of Branches async callback waitfor 0 1 2 3 Average Number of Loops async callback waitfor 0 1 2 3 Average Number of Conditions async callback waitfor 6 Block Count Measures Across 6 Unit Behaviors most concise 19
20. User Study • Between-Group design • Study duration: max. 90 min • Participants (N = 21, 14 F) • with some programming experiences • took intro CS, no robotics • Conducted Online “hello” “Bye” Online User Study Web UI 20
21. Task Description • Step 1: On start, the robot should
 display text “Press or say ‘start’ to begin interaction”
 and display a button with “start” text. • Step 2: If the user presses the button or says “start,” 
 then the robot should introduce itself by 
 saying “Hello, my name is Meebo”
 and make a “happy” gesture. • Step 3: … • Step 4: … 21
22. Concurrency Patterns Perform two actions in parallel say(“Hello I'm robot”) gesture(WAVE) next function Wait for all functions to return time Perform an action while monitoring an event goto(ROOM145) detect(BUTTON_PRESSED) next function Wait for one function to return time Monitor two events in parallel detect(BUTTON_PRESSED) next function Wait for one function to return time detect(SPEECH_COMMAND) 22
23. Results 0 20 40 60 80 100 120 Average Number of Blocks async callback waitfor -2 0 2 4 6 8 10 12 async callback waitfor -2 0 2 4 6 8 10 12 Average Number of Branches async callback waitfor -2 0 2 4 6 8 10 12 async callback waitfor counted # of blocks 0 5 10 15 Average Score async callback waitfor scored using a rubric most concise most correct 23
24. Results 0 20 40 60 80 100 120 Average Number of Blocks async callback waitfor -2 0 2 4 6 8 10 12 async callback waitfor -2 0 2 4 6 8 10 12 Average Number of Branches async callback waitfor -2 0 2 4 6 8 10 12 async callback waitfor counted # of blocks 0 5 10 15 Average Score async callback waitfor scored using a rubric why? 24
25. Example “Callback” User Program A loop for checking a change introduce change 25
26. Summary 1. Robot API designed with concurrency in mind 2. Three representative interfaces: • asynch, callback, waitfor 3. Comparative evaluation • systematic analysis • user study https://github.com/mjyc/concodeit {mjyc,mcakmak}@cs.washington.edu 26