Designing Hybrid Intelligence: From Interaction to Co-Learning

My Job Talk @ TU/e Industrial Design
I shared my research and teaching vision in this talk.

Transcript

1. Designing Hybrid Intelligence Yi-Ching (Janet) Huang 2019.12.18 From Interaction to Co-Learning
2. PhD, National Taiwan University Research Scientist, KAIST PostDoc, TU/e 2018.12-now 2018.07-2018.10 Jane Yung-jen Hsu Juho Kim Lin-Lin Chen 2018.06 Research Topics: Human-AI Interaction, CSCW, Computer-Assisted Learning, Creativity Support Tools, AI Toolkits for Design, Crowdsourcing 2 Hao-Chuan Wang Yi-Ching (Janet) Huang AI, Multi-Agents Systems, Activity Recognition, Human Computation CSCW, HCI, Computer- Supported Creativity HCI, Learning at Scale, Data-Driven Interaction, Crowdsourcing Design Research, Design Innovation Strategy Professor Acting Associate Professor Assistant Professor Full Professor Chair of Design Innovation Strategy Dean of the Industrial Design Dept. Director of the NTU IoX Center
3. Collaboration with Artists 3 choreographer digital artist sound artist director IxD designer architect IxD designer digital artist stage designer maker/artist
4. Dinner of luciérnaga (2011)
5. Outline - Part I: Research - Introduction - Hybrid intelligence - Three contexts of hybrid intelligence systems - Human-AI co-learning - Part II: Teaching - Teaching philosophy - Prior experiences for teaching
6. Outline - Part I: Research - Introduction - Hybrid intelligence - Three contexts of hybrid intelligence systems - Human-AI co-learning - Part II: Teaching - Teaching philosophy - Prior experiences for teaching
7. https://www.facebook.com/womaninthestriped/ 7 My Story about Academic Life PhD research Intro | Hybrid Intelligence | Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning
8. https://www.facebook.com/womaninthestriped/ 7 First, open a new word file Intro | Hybrid Intelligence | Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning
9. https://www.facebook.com/womaninthestriped/ 7 First, open a new word file Only one word-“index” Intro | Hybrid Intelligence | Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning
10. https://www.facebook.com/womaninthestriped/ 7 First, open a new word file My brain is all blank, just like everything I do in life. Only one word-“index” Intro | Hybrid Intelligence | Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning
11. https://www.facebook.com/womaninthestriped/ 7 First, open a new word file My brain is all blank, just like everything I do in life. I feel useless Only one word-“index” … And nobody loves me Intro | Hybrid Intelligence | Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning
12. A Rejection Letter 8
13. A Rejection Letter 8 Why is writing so challenging?
14. Productivity Creativity 9 Ideas Structure Metaphor Grammar Spelling Tone Examples Intro | Hybrid Intelligence | Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning + Writing is one type of creative tasks
15. Complex Creative Process Uncertainty A Concrete Solution 10 Intro | Hybrid Intelligence | Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning
16. Properties of Creative Tasks 1. Open-ended and ill-defined 3. Quality is usually evaluated by multiple criteria 4. Quality can be improved by iterative refinement 2. Answer is not true or false, but how good the answer is 11 (summarized from prior literature) Intro | Hybrid Intelligence | Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning
17. Intro | Hybrid Intelligence | Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning Dellermann et al., Hybrid Intelligence, Business & Information Systems Engineering, 2019. 12 Creativity Productivity
18. Intro | Hybrid Intelligence | Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning Human Intelligence Machine Intelligence Common Sense Empathy & Creativity Pattern Recognition Probabilistic Consistency Speed & Efficiency Analytic Intuitive Flexibility & Transer Annotation of Arbitrary Data Dellermann et al., Hybrid Intelligence, Business & Information Systems Engineering, 2019. 13
19. Human Intelligence Machine Intelligence + 14 Intro | Hybrid Intelligence | Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning = Hybrid Intelligence
20. How can we design effective Hybrid Intelligence systems? Does Hybrid Intelligence systems outperform a single machine or human, even an expert? 1 2 Intro | Hybrid Intelligence | Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning 15
21. Three Contexts of Hybrid Intelligence Systems Crowd-Machine System for Writing Support Human-Machine System for Drawing Support Human-AI System for Smart Environment Intro | Hybrid Intelligence | Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning 16
22. Crowd-Machine System for Writing Support Human-Machine System for Drawing Support Human-AI System for Smart Environment Three Contexts of Hybrid Intelligence Systems Intro | Hybrid Intelligence | Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning 17
23. Crowd-Machine System for Feedback Generation Yi-Ching Huang, Jiunn-Chia Huang, and Jane Yung-jen Hsu. Supporting ESL writing by prompting crowdsourced structural feedback. In Proceedings of the Fifth AAAI Conference on Human Computation and Crowdsourcing (HCOMP 2017) Yi-Ching Huang, Hao-Chuan Wang, and Jane Yung-jen Hsu. Bridging learning gap in writing education with a crowd-powered system. CHI 2017 Workshop on Designing for Curiosity, Denver, Colorado, USA, 2017. 18 StructFeed (Hybrid Intelligence System) Revision Feedback Crowd Machine Revision annotation User Writing workflow Intro | Hybrid Intelligence | Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning
24. 19 Intro | Hybrid Intelligence | Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning
25. topic sentence 19 Intro | Hybrid Intelligence | Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning
26. topic sentence relevant keywords 19 Intro | Hybrid Intelligence | Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning
27. topic sentence relevant keywords 19 Intro | Hybrid Intelligence | Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning
28. topic sentence relevant keywords irrelevant sentence 19 Intro | Hybrid Intelligence | Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning
29. topic sentence relevant keywords irrelevant sentence feedback summary 19 Intro | Hybrid Intelligence | Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning
30. vs vs Study 1: topic sentence prediction Study 2: irrelevant sentence prediction Study 3: writing revision based on feedback Hybrid Intelligence vs Machine Intelligence Hybrid Intelligence vs Human Intelligence Intro | Hybrid Intelligence | Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning 20
31. TF-IDF
 (Word2vec) 0.269 0.240 0.253 ASS
 (Wordnet) 0.403 0.360 0.380 ASS
 (Word2vec) 0.343 0.307 0.324 Rule-based 0.658 0.587 0.620 Crowd-based 0.607 0.720 0.659 0 0.2 0.4 0.6 0.8 TF-IDF
 (Wordnet) TF-IDF
 (Word2vec) ASS
 (Wordnet) ASS
 (Word2vec) Rule Crowd 0.66 0.62 0.32 0.38 0.25 0.27 0.72 0.59 0.31 0.36 0.24 0.25 0.61 0.66 0.34 0.40 0.27 0.28 Precision Recall F1-score Results of Topic Sentence Prediction Crowd vs Machine Intro | Hybrid Intelligence | Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning 21 +
32. Precision Recall F1-score Path Similarity
 (Wordnet) 0.133 0.083 0.103 Cosine Similarity
 (Word2vec) 0.118 0.100 0.108 Crowd-based 0.206 0.326 0.252 0 0.1 0.2 0.3 0.4 Path Similarity
 (Wordnet) Cosine Similarity
 (Word2vec) Crowd-based 0.25 0.11 0.10 0.33 0.10 0.08 0.21 0.12 0.13 Precision Recall F1-score Results of Irrelevant Sentence Prediction Crowd+Machine vs Machine Intro | Hybrid Intelligence | Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning 22 +
33. Expert Random Crowd Worker vs vs 23 Writing Improvement based on three types of feedback StructFeed Intro | Hybrid Intelligence | Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning
34. Expert feedback (C1) Crowd feedback (C2) StructFeed (C3) Cost US$16 US$2 $1-$2 Time 1-2 days 30 mins 1-5 hrs Avg diff rating 0.15 0.21 0.43 Avg standard deviation 0.32 0.32 0.44 # of decreased diff rating 3 2 0 # of equal rating 7 7 5 ! ! ! ! ! ! 24 Intro | Hybrid Intelligence | Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning Expert Random Crowd Worker vs vs Writing improvement based on three types of feedback StructFeed
35. Crowd-Machine System for Writing Support Human-Machine System for Drawing Support Human-AI System for Smart Environment Three Contexts of Hybrid Intelligence Systems Intro | Hybrid Intelligence | Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning 25
36. Crowd-Machine System for Writing Support Human-Machine System for Drawing Support Human-AI System for Smart Environment Three Contexts of Hybrid Intelligence Systems Intro | Hybrid Intelligence | Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning
37. Human-Machine Guidance For Drawing 27 1. human observes the object/picture 2. machine collects, aggregates, and generates gazed-based guidance 3. human uses these guidances to facilitate their drawing Mon-Chu Chen, Yi-Ching Huang, and Kuan-Ying Wu. Gaze-based drawing assistant. In ACM SIGGRAPH 2014 Posters. Intro | Hybrid Intelligence | Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning
38. 28 with guidance with guidance with guidance Intro | Hybrid Intelligence | Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning subject #1 subject #2 subject #3
39. 29 with guidance with guidance with guidance Intro | Hybrid Intelligence | Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning subject #1 subject #2 subject #3 with guidance with guidance with guidance Help proportion and perspective
40. 30 with guidance with guidance with guidance Intro | Hybrid Intelligence | Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning subject #1 subject #2 subject #3 with guidance with guidance with guidance Create new style
41. 31 with guidance with guidance with guidance Intro | Hybrid Intelligence | Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning subject #1 subject #2 subject #3 with guidance with guidance with guidance Create new style Machine is not replacing human in creative drawing, but only supporting their activity.
42. Crowd-Machine System for Writing Support Human-Machine System for Drawing Support Human-AI System for Smart Environment Three Contexts of Hybrid Intelligence Systems Intro | Hybrid Intelligence | Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning
43. Crowd-Machine System for Writing Support Human-Machine System for Drawing Support Human-AI System for Smart Environment Three Contexts of Hybrid Intelligence Systems Intro | Hybrid Intelligence | Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning
44. A micro-volunteering situated platform which allows passersby to report a room status by clicking a button. 34
45. CrowdButton is a tangible platform for collecting human knowle and facilitating system performance Open Simple Lightwe CrowdButton A micro-volunteering situated platform which allows passersby to report a room status by clicking a button. 35 15-20 contributions per device, per day 56% prediction accuracy Yi-Ching Huang. Designing a micro-volunteering platform for situated crowdsourcing. In Proceedings of the 18th ACM Conference Companion on Computer Supported Cooperative Work and Social Computing, CSCW’15 Companion. What kind of activity is happening in the room? Empty 1 2 3 4 Meeting Lecture Study Intro | Hybrid Intelligence | Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning
46. CrowdButton with Machine Feedback Machine senses human presence and generate a prediction, and passersby can confirm or correct the predicted answer, and LED display show the feedback Happy Surprise when human input is as same as machine’s prediction when human input is different from machine’s prediction 36 Intro | Hybrid Intelligence | Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning 3 1 4 2
47. 0.00 0.23 0.45 0.68 0.90 #1 #2 #3 #4 0.87 0.86 0.87 0.56 0.80 0.79 0.67 0.62 0.82 0.62 0.82 0.68 Contribution Aggregated Contribution Prediction Experiment: - People generate more accurate labels by getting machine feedback (#2, #3, #4) than without feedback (#1). - The accuracy of room status prediction increases from 56% (without feedback) to 87% (with feedback) + Intro | Hybrid Intelligence | Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning 37 Machine feedback helps generate accurate labels
48. Structuring Machine Computation with Human Annotations Appropriately Enables Superior Outcomes >> or 38 Intro | Hybrid Intelligence | Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning
49. 39 Hybrid Intelligence for Complex Creative Tasks Intro | Hybrid Intelligence | Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning Feedback Artifact drawing design writing User Revision Human Machine Hybrid Intelligence System knowledge, patterns, strategies workflow
50. Feedback Artifact drawing design writing User Revision Human Machine 40 Hybrid Intelligence System Structure Leveraging “structure” to separate, distribute, and aggregate tasks 1 knowledge, patterns, strategies workflow Intro | Hybrid Intelligence | Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning (1) Separate the complex task into small pieces (2) Distribute these small pieces to humans or machines (3) Aggregate results from difference resources
51. Feedback Artifact drawing design writing User Revision Human Machine 41 Hybrid Intelligence System Techniques for enabling “collaborations” between users and hybrid intelligence systems 2 knowledge, patterns, strategies workflow Intro | Hybrid Intelligence | Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning Collaboration
52. Future Research Agenda
53. To Co-Learning From Interaction 43 v v Intro | Hybrid Intelligence | Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning
54. Human-AI Interaction 44 Intro | Hybrid Intelligence | Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning
55. Human-AI Co-Learning 01. Mutual Understanding 02. Mutual Benefits 03. Mutual Growth 45 Intro | Hybrid Intelligence | Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning
56. AI Toolkits for Data Sensemaking 46 Intro | Hybrid Intelligence | Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning
57. 47 Productivity Creativity Intro | Hybrid Intelligence | Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning
58. Outline - Part I: Research - Introduction - Hybrid intelligence - Three contexts of hybrid intelligence systems - Human-AI co-learning - Part II: Teaching - Teaching philosophy - Prior experiences for teaching
59. Teaching Philosophy 49 Bring a story into materials Learning by doing Constructive feedback Constant conversations
60. Prior Teaching Experiences 50 Workshop Experience Outside Campus Teaching Experience at NTU Artificial Intelligence Multi-Agents Systems AI and IoT Design
61. Designer Data Data Scientist (AI Expert) Users Machine (AI/ML systems) intelligibility transparency Explainable AI interact with new materials understand people Accessible ML uncertainty evolving learning trust control user expectation understand acceptance bias Engineer sketching prototyping understand data Designing Data and AI Vision for Education
62. Thank You Q&A HCI & AI Researcher, Interaction Designer, Digital Artist http://janetyc.github.io Yi-Ching (Janet) Huang