Upgrade to Pro — share decks privately, control downloads, hide ads and more …

NLP for HCI: 人の行動変更を促すためのNLPベースのプロンプトの導入

NLP for HCI: 人の行動変更を促すためのNLPベースのプロンプトの導入

日常における人の行動変容は学習や健康などwell-beingの向上を目的としたHCI研究の大きなテーマの一つです。大規模モデルをはじめとしたNLP技術の発達を、コンピュータから人への介入 (intervention) の革新と考えて、私たちは人の行動変容のプロンプトを組み込んだアプリケーションを作成し、その評価を行ってきました。本トークではNLP技術をHCI分野に展開した実例、特に、人が信頼 (trust) してシステムを使い続けるためのAIデザインやその評価について紹介しようと思います。

https://nlp-colloquium-jp.github.io/schedule/2023-04-19_riku-arakawa_hiromu-yakura/

Hiromu Yakura

April 19, 2023
Tweet

Other Decks in Research

Transcript

  1. Riku Arakawa *


    Carnegie Mellon University
    Hiromu Yakura *


    University of Tsukuba
    NLP for HCI: ਓͷߦಈมߋΛଅͨ͢ΊͷNLP
    ϕʔεͷϓϩϯϓτͷಋೖ
    NLP ίϩΩΞϜ, 2023/4

    View Slide

  2. What's HCI?
    Human-Computer Interaction: ਓؒத৺తͳ؍఺͔Β

    ίϯϐϡʔλͷ৽ͨͳ࢖͍ํΛݟ͚ͭΔֶࡍతͳྖҬ
    ૲ͷࠜతͳ৽ͨͳ࢖͍ํΛ

    ݟ͚ͭΔݚڀ
    ৽ͨͳ࢖͍ํΛࣗΒఏҊ͠

    ͦͷՄೳੑΛݕূ͢Δݚڀ
    ೔ຊͷΞΠυϧ͕

    ͲͷΑ͏ʹΦϯϥΠϯձٞ

    πʔϧΛ࢖͍ͬͯΔ͔
    AlphaGoҎ߱ɺAI͕

    ϓϩع࢜ͷ࿅श΍

    ଧͪखΛͲ͏ม͔͑ͨ
    खʹ360౓ΧϝϥΛ͚ͭΔͰ

    ͲΜͳΠϯλϥΫγϣϯΛ

    ৽ͨʹ࣮ݱͰ͖Δ͔
    H. Yakura. No More Handshaking: How have COVID-19 pushed the expansion of computer-mediated communication in Japanese idol culture? ACM CHI '21.


    R. Arakawa, et al. Hand with Sensing Sphere: Body-Centered Spatial Interactions with a Hand-Worn Spherical Camera. ACI SUI '20.


    J. Kang, et al. How AI-Based Training A
    ff
    ected the Performance of Professional Go Players. ACM CHI '22.

    R. Arakawa and Y. Zhang. Low-Cost Millimeter-Wave Interactive Sensing through Origami Re
    fl
    ectors. CHIIoT Workshop '21.
    ϛϦ೾ϨʔμͰંΓࢴͷมܗΛ

    ݕग़͢Δ͜ͱͰͲΜͳ௿ίετ
    IoTΛ৽ͨʹ࣮ݱͰ͖Δ͔

    View Slide

  3. NLP for HCI
    ࣗવݴޠॲཧͷ෼໺Ͱଟ͘ͷٕज़͕

    ೔ʑڻҟతͳൃలΛݟ͍ͤͯΔ (e.g., จষੜ੒)
    ͦ͏ٕͨ͠ज़ͷൃలΏ͑ʹՄೳͱͳΔ

    ৽ͨͳ࢖͍ํ͸͋Δ͔ʁ

    View Slide

  4. Our focus
    ߦಈม༰Ϟσϧ by BJ Fogg


    Motivation x Ability xPrompt
    NLP౳ͷػցֶशٕज़Λ׆༻ͨ͠γεςϜΛσβΠϯ͠ɺ

    ਓͷߦಈͷมԽΛҾ͖ى͜͢͜ͱͰɺWell-Being ʹߩݙ͢Δ

    View Slide

  5. Today we introduce…
    1. େن໛ੜ੒Ϟσϧʹجͮ͘ʮ࡞ۀ΁ͷ෮ؼʯΛଅ͢հೖ


    2. ػց຋༁Λ׆༻ͨ͠ʮ୯ޠֶशʯΛଅ͢γεςϜ

    View Slide

  6. CatAlyst: Domain-Extensible Intervention for Preventing Task Procrastination


    Using Large Generative Models
    Riku Arakawa*1, Hiromu Yakura*2,3, Masataka Goto3
    * : equal contribution


    1 : Carnegie Mellon University


    2 : University of Tsukuba


    3 : National Institute of Advanced Industrial Science and Technology (AIST)

    View Slide

  7. While large generative models show surprising performances,


    they are not always perfect to alternate our intellectual tasks.
    writing domain-speci
    fi
    c


    documents
    Is it possible to bene
    fi
    t from them in various tasks?
    providing novel ideas

    View Slide

  8. We hypothesized that such models can

    help us avoid procrastination.
    Assumption: even imperfect content generated

    can be used to guide users' interests to their tasks.
    Conventional

    approaches:
    visual feedback of

    task progress [40]
    site blockers [33]
    Our approach:
    [33] G. Kovacs, et al. 2018. Rotating Online Behavior Change Interventions Increases E
    ff
    ectiveness But Also Increases Attrition. ACM CSCW.


    [40] Y. Liu, et al. 2014. Supporting Task Resumption Using Visual Feedback. ACM CSCW.

    View Slide

  9. Demo of how CatAIyst supports a worker in slide-editing

    View Slide


  10. View Slide

  11. CatAlyst: Overview
    The pipeline design is independent from task domain.


    We developed two prototypes: writing and slide-editing.

    View Slide

  12. CatAlyst: Implementation of prototype for slide-editing
    • GPT-3 to generate the continuation of text


    • It also generates a caption of an image to be used, which is provided to a di
    ff
    usion model
    for image generation.

    View Slide

  13. CatAlyst: Strategy
    💡 Prompt workers to face the task even for a short time.
    encouraging message
    Can we improve the e
    ff
    ectiveness?

    View Slide

  14. CatAlyst: Strategy
    [21] J. Clear. 2015. The Chemistry of Building Better Habits. https://jamesclear.com/chemistry-habits.

    View Slide

  15. User Study: Comparison to conventional site blockers
    Writing Slide-editing
    • H1: CatAlyst is an e
    ff
    ective means to keep attracting the interest of workers who are away
    from the task by presenting the continuation of interrupted work as an intervention.


    • H2: CatAlyst can induce workers’ behavior to resume the original task e
    ff
    ectively through
    the intervention.


    • H3: CatAlyst can improve worker productivity by helping them avoid procrastination while
    performing tasks.


    • H4: CatAlyst can lower the cognitive load imposed on workers while performing a task,
    thereby being favorably accepted by them.

    View Slide

  16. Measures & Results
    Writing Slide-editing
    • Ignorance rate: a rate of noti
    fi
    cations ignored


    • Interest retrieval time: duration passed before resumption


    • Progress after resumption: progress made within T s after resumption


    • Total time: time spent on completing the assigned task


    • Subjective quality: product quality rated by crowdworkers


    • Cognitive load: NASA-TLX score responded by participants


    • System usability: SUS score responded by participants
    ?
    ?
    While total time and subjective quality didn't changed signi
    fi
    cantly
    for writing, CatAlyst holistically exhibited its e
    ffi
    cacy.

    View Slide

  17. User Study: Long-term e
    ff
    ect
    • 5-day use of CatAlyst in their writing tasks


    • Uncontrolled setting


    • Semi-structured interviews

    View Slide

  18. User Study: Long-term e
    ff
    ect
    • E
    ff
    ects on behavior


    • Feelings about AI’s accuracy


    • Role of CatAlyst


    • as a reminder


    • as an ideator


    • as a peer


    • Room for further improvements
    Interview Result
    Usage Result
    • No signi
    fi
    cant di
    ff
    erence in the interest
    retrieval time across the
    fi
    ve days


    • Continued use of CatAlyst over the days
    Suggest long-term e
    ff
    i
    cacy of CatAlyst
    Please refer to the paper!

    View Slide

  19. CatAlyst: Domain-Extensible Intervention for Preventing Task Procrastination


    Using Large Generative Models
    Writing Slide-Editing
    Instead of pursuing accuracy via tuning for a higher level of task delegation,


    CatAlyst utilizes large generative models publicly available but imperfect for each individual domain


    to contribute to workers’ digital well-being by in
    fl
    uencing their behavior.
    Composition
    ɾɾɾ

    View Slide

  20. VocabEncounter: NMT-powered Vocabulary Learning by
    Presenting Computer-Generated Usages of Foreign Words into
    Users' Daily Lives
    bogus (adj.)
    ڏِͷɺ͍Μ͖ͪͷ
    ʜʜ੓෎ͷڅ෇ۚͰ͕͢ɺ
    its applicants were mostly
    occupied by bogus companies.
    ͜ΕΛड͚ͯʜʜ
    ?
    bogus (adj.)
    ڏِͷɺ͍Μ͖ͪͷ
    NLP techniques
    (NMT with constrained decoding, etc.)
    Repeated exposure to
    word usages is crucial in
    vocabulary learning. VocabEncounter achieves it by
    encapsulating foreign words in
    materials the user is reading in native language.
    Various daily activities can be transformed
    into the eld of learning.
    working
    commuting
    strolling
    watching movies
    Riku Arakawa *


    Carnegie Mellon University
    Hiromu Yakura *


    University of Tsukuba
    Sosuke Kobayashi


    Tohoku University
    * Equal contribution

    View Slide


  21. View Slide


  22. View Slide

  23. Θ͟Θֶ͟शͷ࣌ؒΛऔΔඞཁੑ΍ɺଟ༷ͳ༻๏ʹ৮ΕΒΕͳ͍ͱ͍͏໰୊

    View Slide

  24. VocabEncounter: NMT-powered Vocabulary Learning by Presenting
    Computer-Generated Usages of Foreign Words into Users' Daily Lives
    bogus (adj.)
    ڏِͷɺ͍Μ͖ͪͷ
    ʜʜ੓෎ͷڅ෇ۚͰ͕͢ɺ
    its applicants were mostly
    occupied by bogus companies.
    ͜ΕΛड͚ͯʜʜ
    ?
    bogus (adj.)
    ڏِͷɺ͍Μ͖ͪͷ
    NLP techniques
    (NMT with constrained decoding, etc.)
    Repeated exposure to
    word usages is crucial in
    vocabulary learning. VocabEncounter achieves it by
    encapsulating foreign words in
    materials the user is reading in native language.
    Various daily activities can be transformed
    into the eld of learning.
    working
    commuting
    strolling
    watching movies

    View Slide


  25. View Slide

  26. Implementation Challenges
    • How to choose an appropriate phrase to translate from web pages
    • How to avoid presenting unnatural or mistranslated phrases
    • How to obtain translations that contain the words to remember

    View Slide

  27. Implementation Challenges
    • How to choose an appropriate phrase to translate from web pages
    • How to avoid presenting unnatural or mistranslated phrases
    • How to obtain translations that contain the words to remember
    NMT with Constrained Decoding
    Multilingual word embedding
    Backtranslation + Sentence-BERT

    View Slide

  28. Key Feature of VocabEncounter: Encapsulation
    materials


    (native language)
    foreign word to remember

    View Slide

  29. materials


    (native language)
    search for words having a similar meaning

    from web pages by multilingual word embedding
    Key Feature of VocabEncounter: Encapsulation
    foreign word to remember
    MUSE

    [Conneau and Lample+, ICLR’18]

    View Slide

  30. materials


    (native language)
    extract a phrase with an appropriate length

    around the detected word
    Key Feature of VocabEncounter: Encapsulation
    foreign word to remember
    dependency

    structure analysis

    View Slide

  31. materials


    (native language)
    extract a phrase with an appropriate length

    around the detected word
    Key Feature of VocabEncounter: Encapsulation
    foreign word to remember
    dependency

    structure analysis

    View Slide

  32. Key Feature of VocabEncounter: Encapsulation
    materials


    (native language)
    extract a phrase with an appropriate length

    around the detected word

    View Slide

  33. generate a translated phrase containing the word


    by neural mechanical translation with constrained decoding
    Key Feature of VocabEncounter: Encapsulation
    materials


    (native language)
    NMT with


    constrained decoding

    [Hu+, NAACL’19]

    View Slide

  34. generate a translated phrase containing the word


    by neural mechanical translation with constrained decoding
    NMT with


    constrained decoding

    [Hu+, NAACL’19]
    Key Feature of VocabEncounter: Encapsulation
    materials


    (native language)
    foreign

    language

    View Slide

  35. translate backwardly to con
    fi
    rm that

    the phrase does not lose its original meaning
    NMT with


    constrained decoding

    [Hu+, NAACL’19]
    Key Feature of VocabEncounter: Encapsulation
    materials


    (native language)
    foreign

    language
    original

    language

    View Slide

  36. Key Feature of VocabEncounter: Encapsulation
    materials


    (native language)
    foreign

    language
    original

    language
    Sentence-BERT


    [Reimers and Gurevych, EMNLP’19]
    translate backwardly to con
    fi
    rm that

    the phrase does not lose its original meaning

    View Slide

  37. replace the original phrase in the material

    with the generated phrase if it has a certain "quality"
    Key Feature of VocabEncounter: Encapsulation
    materials


    (native language)
    foreign

    language
    original

    language
    Sentence-BERT


    [Reimers and Gurevych, EMNLP’19]

    View Slide

  38. Key Feature of VocabEncounter: Encapsulation
    materials


    (native language)
    replace the original phrase in the material

    with the generated phrase if it has a certain "quality"

    View Slide

  39. Quality of the Generated Translations
    Similarity between an original phrase


    and its backtranslated phrase
    Likelihood of the (back)translated phrases

    (grammatically broken phrases exhibit low score)

    View Slide

  40. Questions
    • It is unsure whether our approach really helps learners memorizing
    new words e
    ff
    ectively.
    • We also need to examine the experience of learning with
    VocabEncounter in their daily lives.
    • It has a risk of presenting unnatural or mistranslated phrases.

    View Slide

  41. Evaluation 1: Human-Compatible Quality of Translation
    • 60 crowd workers rating (naturalness and meaning preservation)


    • Human-compatible quality of translation


    • Meaning-preservation correlates with a designed score using Sentence-BERT.


    • Filtering is possible.

    View Slide

  42. Evaluation 2: Signi
    fi
    cant Learning E
    ff
    ects
    • 10 participants compared their correct rate between pre- and post- vocabulary test.


    • VocabEncounter helped them memorize the words to learn.


    • The e
    ff
    ect of presenting generated usages was

    much stronger than presenting only the words.

    View Slide

  43. Evaluation 3: Preferable Experience in 1-week Use
    Semi-structured interviews
    5 participants
    Please refer to the paper!
    • Bene
    fi
    t of Micro Learning


    • Bene
    fi
    t of Usage-Based Learning

    View Slide

  44. We demonstrate example usages of VocabEncounter.

    View Slide

  45. The news article is distributed under Creative Commons 2.1 by NHN Japan

    according to https://www.rondhuit.com/download.html#ldcc

    View Slide

  46. The movie is distributed under Creative Commons 3.0 by WebTB ASO

    at https://www.youtube.com/watch?v=Wxh5-NRLxi4

    View Slide


  47. View Slide

  48. Summary
    • We introduce a new paradigm of vocabulary learning

    by leveraging ML-based generation techniques.


    • We show its feasibility and e
    ff
    ectiveness by implementing
    VocabEncounter, which encapsulates the words to remember
    into materials a user is reading.


    • We believe that VocabEncounter provides a good example of
    how new ML technology expands the way of interaction.

    View Slide

  49. Take-away
    1. ߦಈม༰ͷ࣮ݱʹ͸௕ظతʹγεςϜΛ࢖༻ͯ͠΋Β͏ඞཁ͕͋Γɺ৴
    པͯ͠ܧଓతʹར༻Ͱ͖ΔAIγεςϜͷσβΠϯ͕ॏཁ

    2. ʮෆ׬શʯͳAIΛ͏·͘׆༻͢ΔͨΊͷσβΠϯํ਑ͱͯ͠ɺྫ͑͹ɺ
    a. λʔήοτͱ͢Δߦಈม༰ΛϦσβΠϯ͢Δ

    b. ϢʔβʹAI͔Βͷհೖͷ౓߹͍ͷίϯτϩʔϧΛ༩͑Δ
    We are always open for collaboration!

    View Slide