Human-AI Co-Learning for Data-Driven AI Janet Huang 2019.10.18 

Conversation with Design Materials 

AI/ML as New Design Materials pix2pix style transfer SketchRNN 

pix2pix style transfer SketchRNN Challenges for Designers 1. Inserting ML to Design Practice and Workflow 2. Sensitizing Designers to Existing ML Design Opportunities 3. Developing a Designerly Understanding of ML (Yang, 2018) 

AI toolkits 

(1) How to sketch language interactions abstractly? (2) How to design with data scientists without data at hand? (3) How to understand and stretch NLP’s technical limits? (4) Within these limits, how to envision novel, less obvious applications of NLP? (5) How to prototype an intelligently flawed UX? Five Challenges (1) a new form of wireframes: illustrate abstract language-interaction design ideas (2) a set of NLP technical properties: are relevant to UX design (3) a new wizard-of-oz-based prototyping method: rapidly simulate various kinds of NLP errors Three Contributions Sketching NLP (Yang et al., 2019) 

Data-enabled Design (1) Family Toolkit (2) HCP Dashboard (3) Researcher Dashboard Data-enabled Design Canvas A Two-Step Approach Step 1: Contextual step - goal: gain contextualized understanding of the design space - an open exploration Step 2: Informed step - goal: iteratively design while the prototype(s) stay in the field - an open and dynamic set of tools for shaping the design space based on remotely collected insights - a situated exploration (an on-the-fly exploration) (2016, 2018) 

NETLabTK: Tools for Tangible Design (2003) 

Animistic Design - Not Human Centered Design - Multiple Perspectives, Shared Data - Humble AI - Distributed Cognition - Colleagues not Slaves (2016) 

- “personality control panel” to shape the system’s behaviors and data resources - “visual marionette system” that allows designers to “wizard-of-oz” AI behaviors in real time while observing users Step 2: Simulate, then implement hardware Step 1: Simulate, then implement AI - support AR simulations of AI products, so designers can rapidly test and update the form/ behavior of an embodied experience Delft AI Toolkit (van Allen, 2018) 

Data-enable design Delft AI Toolkit Sketch NLP Task Domain physical intelligent objects physical intelligent objects NLP-involved systems component-based component-based a set of wireframe and NLP properties interact w/ users in the wild WoZ + AR Hybrid WoZ and off-the- shelf prototyping Data or not design along with data - design w/o data Algorithm N/A, might be simple rule decision tree NN-models Visualization HCP dashboard AR Toolkit Family toolkit flow-based programming toolkit Notebook (as a form of wireframe) 

Designer Data Data Scientist (AI Expert) Users Machine (AI/ML systems) intelligibility transparency Explainable AI interact with new materials understand people Accessible ML uncertainty evolving learning trust control user expectation understand acceptance bias Engineer sketching prototyping understand data 1 2 3 

Human-AI Co-Learning 

Human-AI Interaction 

01. Mutual Understanding 02. Mutual Benefits 03. Mutual Growth Human-AI Co-Learning 

1. Mutual Understanding • Human and AI are developing common knowledge (i.e., a shared mental model) through an iterative, interactive process. 2. Mutual Benefits • Human and AI as a team achieves superior results that a single human or AI cannot achieve alone. 3. Mutual Growth • Human and AI both have a growth mindset —i.e. they learn together, learn from each other, learn with each other, and grow and evolve over time. Key Concepts 

Human-Perceived AI Success & Failure (a) AI-Perceived Human Success & Failure (b) 

1. Computer as Nanny 2. Computer as Pen-Pal 3. Computer as Coach 4. Computer as Colleague Lubart’s Framework for Human-AI Interaction (Lubart, 2005) Todd Lubart. How can computers be partners in the creative process: Classification and commentary on the special issue. Int. J. Hum.-Comput. Stud., 63(4-5):365–369, October 2005. 

Action Feedback Learn & Reflect Learn & Reflect Feedback Action Human-AI Collaboration Framework 2 AI’s Active learning Human’s Active Learning Collaboration 1 1 2 3 3 

Human wants to learn 1 Request signals 2 Reason 3 Teach 4 Learn Human-Initiated Learning 

Human wants to advise 1 Act signals 2 Learn data label 3 Feedback Reflect 4 E.g. Interactive Machine Learning Human-Initiated Advising 1. Direct suggestion 2. Prediction 3. Clustering 4. Optimization 1. Teach by Demonstrating 2. Teach by Labeling 3. Teach by Troubleshooting 4. Teach by Verifying 5. Teach by Annotating Meta-data 

AI wants to learn 1 Request signals 3 Teach 2 Reason Learn 4 E.g. Active learning AI-Initiated Learning 

AI wants to advise 1 Act signals 3 Feedback 2 Learn data Reflect 4 AI-Initiated Advising 

long-term learning and negotiation Interactive conversation process 1. interactive explanation AI explains their decisions to human 1 r-1 2 r-2 3 r-3 (Dan, 2019) 2. interactive teaching 1 r-1 2 r-2 3 r-3 human teaches AI what/why/how he do 

An interactive explanatory dialog for gaining insight into a DOG/FISH image classifier Daniel S. Weld and Gagan Bansal. The challenge of crafting intelligible intelligence. Commun. ACM, 62(6):70–79, May 2019. Interactive Explanation Support different follow-up and drill-down action after an initial explanation: 1. Redirecting the answer by changing the foil 2. Asking for more detail 3. Asking for a decision's rationale 4. Query the model’s sensitivity 5. Changing the vocabulary 6. Perturbing the input examples 7. Adjusting the model 

To Learn Something, “Teach it” teach Interactive Teaching 

Algorithms a lot of focus on explaining “algorithmic reasoning” + Data Opportunities to consider “perception differences” encoded in data Model = Machine Intelligence 

Co-Learning Examples 

Human-AI Team Complex Tasks AI-advised human decision making - AI provides a recommendation - Human makes the final decision Examples - high-stakes domains: - (1) recidivism prediction - (2) in-hospital mortality prediction - (3) credit-risk assessment healthcare, criminal justice (Bansal et al., 2019) advise problem decision Human-AI Team 

Human-AI Teams Gagan Bansal, Besmira Nushi, Ece Kamar, Dan Weld, Walter Lasecki, and Eric Horvitz. Updates in human-ai teams: Understanding and addressing the performance/ compatibility tradeoff. In AAAI Conference on Artificial Intelligence. AAAI, January 2019. 

Communication Between AI and Improvising Musicians Beyond Sound In a Silent Way Jon McCormack, Toby Gifford, Patrick Hutchings, Maria Teresa Llano Rodriguez, Matthew Yee-King, and Mark d’Inverno. In a silent way: Communication between ai and improvising musicians beyond sound. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI ’19) Co-Learning for Music Performance 

DuetDraw Changhoon Oh, Jungwoo Song, Jinhan Choi, Seonghyeon Kim, Sungwoo Lee, and Bongwon Suh. I lead, you help but only with enough details: Understanding user experience of co-creation with artificial intelligence. In Proc. of CHI 2018. Human-AI Collaboration (1) Let human takes the Initiative (2) Provide just enough instruction (3) Embed interesting elements in the interaction (4) Ensure balance Co-Learning for Drawing 

AI-Driven Game Level Editor Co-Learning for Game Design Matthew Guzdial, Nicholas Liao, Jonathan Chen, Shao-Yu Chen, Shukan Shah, Vishwa Shah, Joshua Reno, Gillian Smith, and Mark O. Riedl. Friend, collaborator, student, manager: How design of an ai-driven game level editor affects creators. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, CHI ’19 

AI-Assisted Tool For Medical Decision-Making Carrie J. Cai, Emily Reif, Narayan Hegde, Jason Hipp, Been Kim, Daniel Smilkov, Martin Wattenberg, Fernanda Viegas, Greg S. Corrado, Martin C. Stumpe, and Michael Terry. Human- centered tools for coping with imperfect algorithms during medical decision-making. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, CHI ’19 Co-Learning for Medical Decision-Making 

Creative Writing with a Machine in the Loop Elizabeth Clark, Anne Spencer Ross, Chenhao Tan, Yangfeng Ji, and Noah A. Smith. Creative writing with a machine in the loop: Case studies on slogans and stories. In 23rd International Conference on Intelligent User Interfaces, IUI ’18 Co-Learning for Writing Support (I) 

LISA: Lexically Intelligent Story Assistant Rushit Sanghrajka, Daniel Hidalgo, Patrick P. Chen, and Mubbasir Kapadia. LISA: lexically intelligent story assistant. In Proceedings of the Thirteenth AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment (AIIDE-17) Co-Learning for Writing Support (II) 

Co-Learning for Creativity Back-and-white photos colorized by Emil’s algorithm 

Co-Learning for Fashion Design Kato et al., DeepWear: a Case Study of Collaborative Design between Human and Artificial Intelligence. TEI ’18 Art track. 

Co-Learning for Scheduling Justin Cranshaw, Emad Elwany, Todd Newman, Rafal Kocielnik, Bowen Yu, Sandeep Soni, Jaime Teevan, and Andr ́es Monroy-Hern ́andez. Calendar.help: Designing a workflow-based scheduling agent with humans in the loop. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. 

Co-Learning for Making Pizza https://howtogeneratealmostanything.com/food/2018/08/30/episode2.html