Designing Hybrid Intelligence: From Interaction to Co-Learning

Fc1b497e61d3adca89516740944ad1e5?s=47 janetyc
December 18, 2019

Designing Hybrid Intelligence: From Interaction to Co-Learning

My Job Talk @ TU/e Industrial Design
I shared my research and teaching vision in this talk.

Fc1b497e61d3adca89516740944ad1e5?s=128

janetyc

December 18, 2019
Tweet

Transcript

  1. Designing Hybrid Intelligence Yi-Ching (Janet) Huang 2019.12.18 From Interaction to

    Co-Learning
  2. PhD, National Taiwan University Research Scientist, KAIST PostDoc, TU/e 2018.12-now

    2018.07-2018.10 Jane Yung-jen Hsu Juho Kim Lin-Lin Chen 2018.06 Research Topics: Human-AI Interaction, CSCW, Computer-Assisted Learning, Creativity Support Tools, AI Toolkits for Design, Crowdsourcing 2 Hao-Chuan Wang Yi-Ching (Janet) Huang AI, Multi-Agents Systems, Activity Recognition, Human Computation CSCW, HCI, Computer- Supported Creativity HCI, Learning at Scale, Data-Driven Interaction, Crowdsourcing Design Research, Design Innovation Strategy Professor Acting Associate Professor Assistant Professor Full Professor Chair of Design Innovation Strategy Dean of the Industrial Design Dept. Director of the NTU IoX Center
  3. Collaboration with Artists 3 choreographer digital artist sound artist director

    IxD designer architect IxD designer digital artist stage designer maker/artist
  4. Dinner of luciérnaga (2011)

  5. Outline - Part I: Research - Introduction - Hybrid intelligence

    - Three contexts of hybrid intelligence systems - Human-AI co-learning - Part II: Teaching - Teaching philosophy - Prior experiences for teaching
  6. Outline - Part I: Research - Introduction - Hybrid intelligence

    - Three contexts of hybrid intelligence systems - Human-AI co-learning - Part II: Teaching - Teaching philosophy - Prior experiences for teaching
  7. https://www.facebook.com/womaninthestriped/ 7 My Story about Academic Life PhD research Intro

    | Hybrid Intelligence | Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning
  8. https://www.facebook.com/womaninthestriped/ 7 First, open a new word file Intro |

    Hybrid Intelligence | Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning
  9. https://www.facebook.com/womaninthestriped/ 7 First, open a new word file Only one

    word-“index” Intro | Hybrid Intelligence | Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning
  10. https://www.facebook.com/womaninthestriped/ 7 First, open a new word file My brain

    is all blank, just like everything I do in life. Only one word-“index” Intro | Hybrid Intelligence | Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning
  11. https://www.facebook.com/womaninthestriped/ 7 First, open a new word file My brain

    is all blank, just like everything I do in life. I feel useless Only one word-“index” … And nobody loves me Intro | Hybrid Intelligence | Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning
  12. A Rejection Letter 8

  13. A Rejection Letter 8 Why is writing so challenging?

  14. Productivity Creativity 9 Ideas Structure Metaphor Grammar Spelling Tone Examples

    Intro | Hybrid Intelligence | Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning + Writing is one type of creative tasks
  15. Complex Creative Process Uncertainty A Concrete Solution 10 Intro |

    Hybrid Intelligence | Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning
  16. Properties of Creative Tasks 1. Open-ended and ill-defined 3. Quality

    is usually evaluated by multiple criteria 4. Quality can be improved by iterative refinement 2. Answer is not true or false, but how good the answer is 11 (summarized from prior literature) Intro | Hybrid Intelligence | Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning
  17. Intro | Hybrid Intelligence | Three Contexts of Hybrid Intelligence

    Systems | Human-AI Co-Learning Dellermann et al., Hybrid Intelligence, Business & Information Systems Engineering, 2019. 12 Creativity Productivity
  18. Intro | Hybrid Intelligence | Three Contexts of Hybrid Intelligence

    Systems | Human-AI Co-Learning Human Intelligence Machine Intelligence Common Sense Empathy & Creativity Pattern Recognition Probabilistic Consistency Speed & Efficiency Analytic Intuitive Flexibility & Transer Annotation of Arbitrary Data Dellermann et al., Hybrid Intelligence, Business & Information Systems Engineering, 2019. 13
  19. Human Intelligence Machine Intelligence + 14 Intro | Hybrid Intelligence

    | Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning = Hybrid Intelligence
  20. How can we design effective Hybrid Intelligence systems? Does Hybrid

    Intelligence systems outperform a single machine or human, even an expert? 1 2 Intro | Hybrid Intelligence | Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning 15
  21. Three Contexts of Hybrid Intelligence Systems Crowd-Machine System for Writing

    Support Human-Machine System for Drawing Support Human-AI System for Smart Environment Intro | Hybrid Intelligence | Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning 16
  22. Crowd-Machine System for Writing Support Human-Machine System for Drawing Support

    Human-AI System for Smart Environment Three Contexts of Hybrid Intelligence Systems Intro | Hybrid Intelligence | Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning 17
  23. Crowd-Machine System for Feedback Generation Yi-Ching Huang, Jiunn-Chia Huang, and

    Jane Yung-jen Hsu. Supporting ESL writing by prompting crowdsourced structural feedback. In Proceedings of the Fifth AAAI Conference on Human Computation and Crowdsourcing (HCOMP 2017) Yi-Ching Huang, Hao-Chuan Wang, and Jane Yung-jen Hsu. Bridging learning gap in writing education with a crowd-powered system. CHI 2017 Workshop on Designing for Curiosity, Denver, Colorado, USA, 2017. 18 StructFeed (Hybrid Intelligence System) Revision Feedback Crowd Machine Revision annotation User Writing workflow Intro | Hybrid Intelligence | Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning
  24. 19 Intro | Hybrid Intelligence | Three Contexts of Hybrid

    Intelligence Systems | Human-AI Co-Learning
  25. topic sentence 19 Intro | Hybrid Intelligence | Three Contexts

    of Hybrid Intelligence Systems | Human-AI Co-Learning
  26. topic sentence relevant keywords 19 Intro | Hybrid Intelligence |

    Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning
  27. topic sentence relevant keywords 19 Intro | Hybrid Intelligence |

    Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning
  28. topic sentence relevant keywords irrelevant sentence 19 Intro | Hybrid

    Intelligence | Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning
  29. topic sentence relevant keywords irrelevant sentence feedback summary 19 Intro

    | Hybrid Intelligence | Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning
  30. vs vs Study 1: topic sentence prediction Study 2: irrelevant

    sentence prediction Study 3: writing revision based on feedback Hybrid Intelligence vs Machine Intelligence Hybrid Intelligence vs Human Intelligence Intro | Hybrid Intelligence | Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning 20
  31. TF-IDF
 (Word2vec) 0.269 0.240 0.253 ASS
 (Wordnet) 0.403 0.360 0.380

    ASS
 (Word2vec) 0.343 0.307 0.324 Rule-based 0.658 0.587 0.620 Crowd-based 0.607 0.720 0.659 0 0.2 0.4 0.6 0.8 TF-IDF
 (Wordnet) TF-IDF
 (Word2vec) ASS
 (Wordnet) ASS
 (Word2vec) Rule Crowd 0.66 0.62 0.32 0.38 0.25 0.27 0.72 0.59 0.31 0.36 0.24 0.25 0.61 0.66 0.34 0.40 0.27 0.28 Precision Recall F1-score Results of Topic Sentence Prediction Crowd vs Machine Intro | Hybrid Intelligence | Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning 21 +
  32. Precision Recall F1-score Path Similarity
 (Wordnet) 0.133 0.083 0.103 Cosine

    Similarity
 (Word2vec) 0.118 0.100 0.108 Crowd-based 0.206 0.326 0.252 0 0.1 0.2 0.3 0.4 Path Similarity
 (Wordnet) Cosine Similarity
 (Word2vec) Crowd-based 0.25 0.11 0.10 0.33 0.10 0.08 0.21 0.12 0.13 Precision Recall F1-score Results of Irrelevant Sentence Prediction Crowd+Machine vs Machine Intro | Hybrid Intelligence | Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning 22 +
  33. Expert Random Crowd Worker vs vs 23 Writing Improvement based

    on three types of feedback StructFeed Intro | Hybrid Intelligence | Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning
  34. Expert feedback (C1) Crowd feedback (C2) StructFeed (C3) Cost US$16

    US$2 $1-$2 Time 1-2 days 30 mins 1-5 hrs Avg diff rating 0.15 0.21 0.43 Avg standard deviation 0.32 0.32 0.44 # of decreased diff rating 3 2 0 # of equal rating 7 7 5 ! ! ! ! ! ! 24 Intro | Hybrid Intelligence | Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning Expert Random Crowd Worker vs vs Writing improvement based on three types of feedback StructFeed
  35. Crowd-Machine System for Writing Support Human-Machine System for Drawing Support

    Human-AI System for Smart Environment Three Contexts of Hybrid Intelligence Systems Intro | Hybrid Intelligence | Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning 25
  36. Crowd-Machine System for Writing Support Human-Machine System for Drawing Support

    Human-AI System for Smart Environment Three Contexts of Hybrid Intelligence Systems Intro | Hybrid Intelligence | Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning
  37. Human-Machine Guidance For Drawing 27 1. human observes the object/picture

    2. machine collects, aggregates, and generates gazed-based guidance 3. human uses these guidances to facilitate their drawing Mon-Chu Chen, Yi-Ching Huang, and Kuan-Ying Wu. Gaze-based drawing assistant. In ACM SIGGRAPH 2014 Posters. Intro | Hybrid Intelligence | Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning
  38. 28 with guidance with guidance with guidance Intro | Hybrid

    Intelligence | Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning subject #1 subject #2 subject #3
  39. 29 with guidance with guidance with guidance Intro | Hybrid

    Intelligence | Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning subject #1 subject #2 subject #3 with guidance with guidance with guidance Help proportion and perspective
  40. 30 with guidance with guidance with guidance Intro | Hybrid

    Intelligence | Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning subject #1 subject #2 subject #3 with guidance with guidance with guidance Create new style
  41. 31 with guidance with guidance with guidance Intro | Hybrid

    Intelligence | Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning subject #1 subject #2 subject #3 with guidance with guidance with guidance Create new style Machine is not replacing human in creative drawing, but only supporting their activity.
  42. Crowd-Machine System for Writing Support Human-Machine System for Drawing Support

    Human-AI System for Smart Environment Three Contexts of Hybrid Intelligence Systems Intro | Hybrid Intelligence | Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning
  43. Crowd-Machine System for Writing Support Human-Machine System for Drawing Support

    Human-AI System for Smart Environment Three Contexts of Hybrid Intelligence Systems Intro | Hybrid Intelligence | Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning
  44. A micro-volunteering situated platform which allows passersby to report a

    room status by clicking a button. 34
  45. CrowdButton is a tangible platform for collecting human knowle and

    facilitating system performance Open Simple Lightwe CrowdButton A micro-volunteering situated platform which allows passersby to report a room status by clicking a button. 35 15-20 contributions per device, per day 56% prediction accuracy Yi-Ching Huang. Designing a micro-volunteering platform for situated crowdsourcing. In Proceedings of the 18th ACM Conference Companion on Computer Supported Cooperative Work and Social Computing, CSCW’15 Companion. What kind of activity is happening in the room? Empty 1 2 3 4 Meeting Lecture Study Intro | Hybrid Intelligence | Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning
  46. CrowdButton with Machine Feedback Machine senses human presence and generate

    a prediction, and passersby can confirm or correct the predicted answer, and LED display show the feedback Happy Surprise when human input is as same as machine’s prediction when human input is different from machine’s prediction 36 Intro | Hybrid Intelligence | Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning 3 1 4 2
  47. 0.00 0.23 0.45 0.68 0.90 #1 #2 #3 #4 0.87

    0.86 0.87 0.56 0.80 0.79 0.67 0.62 0.82 0.62 0.82 0.68 Contribution Aggregated Contribution Prediction Experiment: - People generate more accurate labels by getting machine feedback (#2, #3, #4) than without feedback (#1). - The accuracy of room status prediction increases from 56% (without feedback) to 87% (with feedback) + Intro | Hybrid Intelligence | Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning 37 Machine feedback helps generate accurate labels
  48. Structuring Machine Computation with Human Annotations Appropriately Enables Superior Outcomes

    >> or 38 Intro | Hybrid Intelligence | Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning
  49. 39 Hybrid Intelligence for Complex Creative Tasks Intro | Hybrid

    Intelligence | Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning Feedback Artifact drawing design writing User Revision Human Machine Hybrid Intelligence System knowledge, patterns, strategies workflow
  50. Feedback Artifact drawing design writing User Revision Human Machine 40

    Hybrid Intelligence System Structure Leveraging “structure” to separate, distribute, and aggregate tasks 1 knowledge, patterns, strategies workflow Intro | Hybrid Intelligence | Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning (1) Separate the complex task into small pieces (2) Distribute these small pieces to humans or machines (3) Aggregate results from difference resources
  51. Feedback Artifact drawing design writing User Revision Human Machine 41

    Hybrid Intelligence System Techniques for enabling “collaborations” between users and hybrid intelligence systems 2 knowledge, patterns, strategies workflow Intro | Hybrid Intelligence | Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning Collaboration
  52. Future Research Agenda

  53. To Co-Learning From Interaction 43 v v Intro | Hybrid

    Intelligence | Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning
  54. Human-AI Interaction 44 Intro | Hybrid Intelligence | Three Contexts

    of Hybrid Intelligence Systems | Human-AI Co-Learning
  55. Human-AI Co-Learning 01. Mutual Understanding 02. Mutual Benefits 03. Mutual

    Growth 45 Intro | Hybrid Intelligence | Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning
  56. AI Toolkits for Data Sensemaking 46 Intro | Hybrid Intelligence

    | Three Contexts of Hybrid Intelligence Systems | Human-AI Co-Learning
  57. 47 Productivity Creativity Intro | Hybrid Intelligence | Three Contexts

    of Hybrid Intelligence Systems | Human-AI Co-Learning
  58. Outline - Part I: Research - Introduction - Hybrid intelligence

    - Three contexts of hybrid intelligence systems - Human-AI co-learning - Part II: Teaching - Teaching philosophy - Prior experiences for teaching
  59. Teaching Philosophy 49 Bring a story into materials Learning by

    doing Constructive feedback Constant conversations
  60. Prior Teaching Experiences 50 Workshop Experience Outside Campus Teaching Experience

    at NTU Artificial Intelligence Multi-Agents Systems AI and IoT Design
  61. Designer Data Data Scientist (AI Expert) Users Machine (AI/ML systems)

    intelligibility transparency Explainable AI interact with new materials understand people Accessible ML uncertainty evolving learning trust control user expectation understand acceptance bias Engineer sketching prototyping understand data Designing Data and AI Vision for Education
  62. Thank You Q&A HCI & AI Researcher, Interaction Designer, Digital

    Artist http://janetyc.github.io Yi-Ching (Janet) Huang