NLP for HCI: 人の行動変更を促すためのNLPベースのプロンプトの導入

日常における人の行動変容は学習や健康などwell-beingの向上を目的としたHCI研究の大きなテーマの一つです。大規模モデルをはじめとしたNLP技術の発達を、コンピュータから人への介入 (intervention) の革新と考えて、私たちは人の行動変容のプロンプトを組み込んだアプリケーションを作成し、その評価を行ってきました。本トークではNLP技術をHCI分野に展開した実例、特に、人が信頼 (trust) してシステムを使い続けるためのAIデザインやその評価について紹介しようと思います。

https://nlp-colloquium-jp.github.io/schedule/2023-04-19_riku-arakawa_hiromu-yakura/

Transcript

1. Riku Arakawa *
   
   
   Carnegie Mellon University
   Hiromu Yakura *
   
   
   University of Tsukuba
   NLP for HCI: ਓͷߦಈมߋΛଅͨ͢ΊͷNLP
   ϕʔεͷϓϩϯϓτͷಋೖ
   NLP ίϩΩΞϜ, 2023/4
   

   View Slide

2. What's HCI?
   Human-Computer Interaction: ਓؒத৺తͳ؍఺͔Β
   

   ίϯϐϡʔλͷ৽ͨͳ࢖͍ํΛݟ͚ͭΔֶࡍతͳྖҬ
   ૲ͷࠜతͳ৽ͨͳ࢖͍ํΛ
   

   ݟ͚ͭΔݚڀ
   ৽ͨͳ࢖͍ํΛࣗΒఏҊ͠
   

   ͦͷՄೳੑΛݕূ͢Δݚڀ
   ೔ຊͷΞΠυϧ͕
   

   ͲͷΑ͏ʹΦϯϥΠϯձٞ
   

   πʔϧΛ࢖͍ͬͯΔ͔
   AlphaGoҎ߱ɺAI͕
   

   ϓϩع࢜ͷ࿅श΍
   

   ଧͪखΛͲ͏ม͔͑ͨ
   खʹ360౓ΧϝϥΛ͚ͭΔͰ
   

   ͲΜͳΠϯλϥΫγϣϯΛ
   

   ৽ͨʹ࣮ݱͰ͖Δ͔
   H. Yakura. No More Handshaking: How have COVID-19 pushed the expansion of computer-mediated communication in Japanese idol culture? ACM CHI '21.
   
   
   R. Arakawa, et al. Hand with Sensing Sphere: Body-Centered Spatial Interactions with a Hand-Worn Spherical Camera. ACI SUI '20.
   
   
   J. Kang, et al. How AI-Based Training A
   ff
   ected the Performance of Professional Go Players. ACM CHI '22.
   

   R. Arakawa and Y. Zhang. Low-Cost Millimeter-Wave Interactive Sensing through Origami Re
   fl
   ectors. CHIIoT Workshop '21.
   ϛϦ೾ϨʔμͰંΓࢴͷมܗΛ
   

   ݕग़͢Δ͜ͱͰͲΜͳ௿ίετ
   IoTΛ৽ͨʹ࣮ݱͰ͖Δ͔
   
   

   View Slide

3. NLP for HCI
   ࣗવݴޠॲཧͷ෼໺Ͱଟ͘ͷٕज़͕
   

   ೔ʑڻҟతͳൃలΛݟ͍ͤͯΔ (e.g., จষੜ੒)
   ͦ͏ٕͨ͠ज़ͷൃలΏ͑ʹՄೳͱͳΔ
   

   ৽ͨͳ࢖͍ํ͸͋Δ͔ʁ
   
   

   View Slide

4. Our focus
   ߦಈม༰Ϟσϧ by BJ Fogg
   
   
   Motivation x Ability xPrompt
   NLP౳ͷػցֶशٕज़Λ׆༻ͨ͠γεςϜΛσβΠϯ͠ɺ
   

   ਓͷߦಈͷมԽΛҾ͖ى͜͢͜ͱͰɺWell-Being ʹߩݙ͢Δ
   
   

   View Slide

5. Today we introduce…
   1. େن໛ੜ੒Ϟσϧʹجͮ͘ʮ࡞ۀ΁ͷ෮ؼʯΛଅ͢հೖ
   
   
   2. ػց຋༁Λ׆༻ͨ͠ʮ୯ޠֶशʯΛଅ͢γεςϜ
   
   

   View Slide

6. CatAlyst: Domain-Extensible Intervention for Preventing Task Procrastination
   
   
   Using Large Generative Models
   Riku Arakawa*1, Hiromu Yakura*2,3, Masataka Goto3
   * : equal contribution
   
   
   1 : Carnegie Mellon University
   
   
   2 : University of Tsukuba
   
   
   3 : National Institute of Advanced Industrial Science and Technology (AIST)
   

   View Slide

7. While large generative models show surprising performances,
   
   
   they are not always perfect to alternate our intellectual tasks.
   writing domain-speci
   fi
   c
   
   
   documents
   Is it possible to bene
   fi
   t from them in various tasks?
   providing novel ideas
   
   

   View Slide

8. We hypothesized that such models can
   

   help us avoid procrastination.
   Assumption: even imperfect content generated
   

   can be used to guide users' interests to their tasks.
   Conventional
   

   approaches:
   visual feedback of
   

   task progress [40]
   site blockers [33]
   Our approach:
   [33] G. Kovacs, et al. 2018. Rotating Online Behavior Change Interventions Increases E
   ff
   ectiveness But Also Increases Attrition. ACM CSCW.
   
   
   [40] Y. Liu, et al. 2014. Supporting Task Resumption Using Visual Feedback. ACM CSCW.
   
   

   View Slide

9. Demo of how CatAIyst supports a worker in slide-editing
   
   

   View Slide

10. 
    

    View Slide

11. CatAlyst: Overview
    The pipeline design is independent from task domain.
    
    
    We developed two prototypes: writing and slide-editing.
    
    

    View Slide

12. CatAlyst: Implementation of prototype for slide-editing
    • GPT-3 to generate the continuation of text
    
    
    • It also generates a caption of an image to be used, which is provided to a di
    ff
    usion model
    for image generation.
    
    

    View Slide

13. CatAlyst: Strategy
    💡 Prompt workers to face the task even for a short time.
    encouraging message
    Can we improve the e
    ff
    ectiveness?
    
    

    View Slide

14. CatAlyst: Strategy
    [21] J. Clear. 2015. The Chemistry of Building Better Habits. https://jamesclear.com/chemistry-habits.
    
    

    View Slide

15. User Study: Comparison to conventional site blockers
    Writing Slide-editing
    • H1: CatAlyst is an e
    ff
    ective means to keep attracting the interest of workers who are away
    from the task by presenting the continuation of interrupted work as an intervention.
    
    
    • H2: CatAlyst can induce workers’ behavior to resume the original task e
    ff
    ectively through
    the intervention.
    
    
    • H3: CatAlyst can improve worker productivity by helping them avoid procrastination while
    performing tasks.
    
    
    • H4: CatAlyst can lower the cognitive load imposed on workers while performing a task,
    thereby being favorably accepted by them.
    
    

    View Slide

16. Measures & Results
    Writing Slide-editing
    • Ignorance rate: a rate of noti
    fi
    cations ignored
    
    
    • Interest retrieval time: duration passed before resumption
    
    
    • Progress after resumption: progress made within T s after resumption
    
    
    • Total time: time spent on completing the assigned task
    
    
    • Subjective quality: product quality rated by crowdworkers
    
    
    • Cognitive load: NASA-TLX score responded by participants
    
    
    • System usability: SUS score responded by participants
    ?
    ?
    While total time and subjective quality didn't changed signi
    fi
    cantly
    for writing, CatAlyst holistically exhibited its e
    ffi
    cacy.
    
    

    View Slide

17. User Study: Long-term e
    ff
    ect
    • 5-day use of CatAlyst in their writing tasks
    
    
    • Uncontrolled setting
    
    
    • Semi-structured interviews
    
    

    View Slide

18. User Study: Long-term e
    ff
    ect
    • E
    ff
    ects on behavior
    
    
    • Feelings about AI’s accuracy
    
    
    • Role of CatAlyst
    
    
    • as a reminder
    
    
    • as an ideator
    
    
    • as a peer
    
    
    • Room for further improvements
    Interview Result
    Usage Result
    • No signi
    fi
    cant di
    ff
    erence in the interest
    retrieval time across the
    fi
    ve days
    
    
    • Continued use of CatAlyst over the days
    Suggest long-term e
    ff
    i
    cacy of CatAlyst
    Please refer to the paper!
    
    

    View Slide

19. CatAlyst: Domain-Extensible Intervention for Preventing Task Procrastination
    
    
    Using Large Generative Models
    Writing Slide-Editing
    Instead of pursuing accuracy via tuning for a higher level of task delegation,
    
    
    CatAlyst utilizes large generative models publicly available but imperfect for each individual domain
    
    
    to contribute to workers’ digital well-being by in
    fl
    uencing their behavior.
    Composition
    ɾɾɾ
    
    

    View Slide

20. VocabEncounter: NMT-powered Vocabulary Learning by
    Presenting Computer-Generated Usages of Foreign Words into
    Users' Daily Lives
    bogus (adj.)
    ڏِͷɺ͍Μ͖ͪͷ
    ʜʜ੓෎ͷڅ෇ۚͰ͕͢ɺ
    its applicants were mostly
    occupied by bogus companies.
    ͜ΕΛड͚ͯʜʜ
    ?
    bogus (adj.)
    ڏِͷɺ͍Μ͖ͪͷ
    NLP techniques
    (NMT with constrained decoding, etc.)
    Repeated exposure to
    word usages is crucial in
    vocabulary learning. VocabEncounter achieves it by
    encapsulating foreign words in
    materials the user is reading in native language.
    Various daily activities can be transformed
    into the eld of learning.
    working
    commuting
    strolling
    watching movies
    Riku Arakawa *
    
    
    Carnegie Mellon University
    Hiromu Yakura *
    
    
    University of Tsukuba
    Sosuke Kobayashi
    
    
    Tohoku University
    * Equal contribution
    

    View Slide

21. 
    

    View Slide

22. 
    

    View Slide

23. Θ͟Θֶ͟शͷ࣌ؒΛऔΔඞཁੑ΍ɺଟ༷ͳ༻๏ʹ৮ΕΒΕͳ͍ͱ͍͏໰୊
    
    

    View Slide

24. VocabEncounter: NMT-powered Vocabulary Learning by Presenting
    Computer-Generated Usages of Foreign Words into Users' Daily Lives
    bogus (adj.)
    ڏِͷɺ͍Μ͖ͪͷ
    ʜʜ੓෎ͷڅ෇ۚͰ͕͢ɺ
    its applicants were mostly
    occupied by bogus companies.
    ͜ΕΛड͚ͯʜʜ
    ?
    bogus (adj.)
    ڏِͷɺ͍Μ͖ͪͷ
    NLP techniques
    (NMT with constrained decoding, etc.)
    Repeated exposure to
    word usages is crucial in
    vocabulary learning. VocabEncounter achieves it by
    encapsulating foreign words in
    materials the user is reading in native language.
    Various daily activities can be transformed
    into the eld of learning.
    working
    commuting
    strolling
    watching movies
    
    

    View Slide

25. 
    

    View Slide

26. Implementation Challenges
    • How to choose an appropriate phrase to translate from web pages
    • How to avoid presenting unnatural or mistranslated phrases
    • How to obtain translations that contain the words to remember
    
    

    View Slide

27. Implementation Challenges
    • How to choose an appropriate phrase to translate from web pages
    • How to avoid presenting unnatural or mistranslated phrases
    • How to obtain translations that contain the words to remember
    NMT with Constrained Decoding
    Multilingual word embedding
    Backtranslation + Sentence-BERT
    
    

    View Slide

28. Key Feature of VocabEncounter: Encapsulation
    materials
    
    
    (native language)
    foreign word to remember
    
    

    View Slide

29. materials
    
    
    (native language)
    search for words having a similar meaning
    

    from web pages by multilingual word embedding
    Key Feature of VocabEncounter: Encapsulation
    foreign word to remember
    MUSE
    

    [Conneau and Lample+, ICLR’18]
    
    

    View Slide

30. materials
    
    
    (native language)
    extract a phrase with an appropriate length
    

    around the detected word
    Key Feature of VocabEncounter: Encapsulation
    foreign word to remember
    dependency
    

    structure analysis
    
    

    View Slide

31. materials
    
    
    (native language)
    extract a phrase with an appropriate length
    

    around the detected word
    Key Feature of VocabEncounter: Encapsulation
    foreign word to remember
    dependency
    

    structure analysis
    
    

    View Slide

32. Key Feature of VocabEncounter: Encapsulation
    materials
    
    
    (native language)
    extract a phrase with an appropriate length
    

    around the detected word
    
    

    View Slide

33. generate a translated phrase containing the word
    
    
    by neural mechanical translation with constrained decoding
    Key Feature of VocabEncounter: Encapsulation
    materials
    
    
    (native language)
    NMT with
    
    
    constrained decoding
    

    [Hu+, NAACL’19]
    
    

    View Slide

34. generate a translated phrase containing the word
    
    
    by neural mechanical translation with constrained decoding
    NMT with
    
    
    constrained decoding
    

    [Hu+, NAACL’19]
    Key Feature of VocabEncounter: Encapsulation
    materials
    
    
    (native language)
    foreign
    

    language
    
    

    View Slide

35. translate backwardly to con
    fi
    rm that
    

    the phrase does not lose its original meaning
    NMT with
    
    
    constrained decoding
    

    [Hu+, NAACL’19]
    Key Feature of VocabEncounter: Encapsulation
    materials
    
    
    (native language)
    foreign
    

    language
    original
    

    language
    
    

    View Slide

36. Key Feature of VocabEncounter: Encapsulation
    materials
    
    
    (native language)
    foreign
    

    language
    original
    

    language
    Sentence-BERT
    
    
    [Reimers and Gurevych, EMNLP’19]
    translate backwardly to con
    fi
    rm that
    

    the phrase does not lose its original meaning
    
    

    View Slide

37. replace the original phrase in the material
    

    with the generated phrase if it has a certain "quality"
    Key Feature of VocabEncounter: Encapsulation
    materials
    
    
    (native language)
    foreign
    

    language
    original
    

    language
    Sentence-BERT
    
    
    [Reimers and Gurevych, EMNLP’19]
    
    

    View Slide

38. Key Feature of VocabEncounter: Encapsulation
    materials
    
    
    (native language)
    replace the original phrase in the material
    

    with the generated phrase if it has a certain "quality"
    
    

    View Slide

39. Quality of the Generated Translations
    Similarity between an original phrase
    
    
    and its backtranslated phrase
    Likelihood of the (back)translated phrases
    

    (grammatically broken phrases exhibit low score)
    
    

    View Slide

40. Questions
    • It is unsure whether our approach really helps learners memorizing
    new words e
    ff
    ectively.
    • We also need to examine the experience of learning with
    VocabEncounter in their daily lives.
    • It has a risk of presenting unnatural or mistranslated phrases.
    
    

    View Slide

41. Evaluation 1: Human-Compatible Quality of Translation
    • 60 crowd workers rating (naturalness and meaning preservation)
    
    
    • Human-compatible quality of translation
    
    
    • Meaning-preservation correlates with a designed score using Sentence-BERT.
    
    
    • Filtering is possible.
    
    

    View Slide

42. Evaluation 2: Signi
    fi
    cant Learning E
    ff
    ects
    • 10 participants compared their correct rate between pre- and post- vocabulary test.
    
    
    • VocabEncounter helped them memorize the words to learn.
    
    
    • The e
    ff
    ect of presenting generated usages was
    

    much stronger than presenting only the words.
    
    

    View Slide

43. Evaluation 3: Preferable Experience in 1-week Use
    Semi-structured interviews
    5 participants
    Please refer to the paper!
    • Bene
    fi
    t of Micro Learning
    
    
    • Bene
    fi
    t of Usage-Based Learning
    
    

    View Slide

44. We demonstrate example usages of VocabEncounter.
    
    

    View Slide

45. The news article is distributed under Creative Commons 2.1 by NHN Japan
    

    according to https://www.rondhuit.com/download.html#ldcc
    
    

    View Slide

46. The movie is distributed under Creative Commons 3.0 by WebTB ASO
    

    at https://www.youtube.com/watch?v=Wxh5-NRLxi4
    
    

    View Slide

47. 
    

    View Slide

48. Summary
    • We introduce a new paradigm of vocabulary learning
    

    by leveraging ML-based generation techniques.
    
    
    • We show its feasibility and e
    ff
    ectiveness by implementing
    VocabEncounter, which encapsulates the words to remember
    into materials a user is reading.
    
    
    • We believe that VocabEncounter provides a good example of
    how new ML technology expands the way of interaction.
    
    

    View Slide

49. Take-away
    1. ߦಈม༰ͷ࣮ݱʹ͸௕ظతʹγεςϜΛ࢖༻ͯ͠΋Β͏ඞཁ͕͋Γɺ৴
    པͯ͠ܧଓతʹར༻Ͱ͖ΔAIγεςϜͷσβΠϯ͕ॏཁ
    

    2. ʮෆ׬શʯͳAIΛ͏·͘׆༻͢ΔͨΊͷσβΠϯํ਑ͱͯ͠ɺྫ͑͹ɺ
    a. λʔήοτͱ͢Δߦಈม༰ΛϦσβΠϯ͢Δ
    

    b. ϢʔβʹAI͔Βͷհೖͷ౓߹͍ͷίϯτϩʔϧΛ༩͑Δ
    We are always open for collaboration!
    
    

    View Slide