Upgrade to Pro — share decks privately, control downloads, hide ads and more …

EmoBalloon - Conveying Emotional Arousal in Text Chats with Speech Balloons

EmoBalloon - Conveying Emotional Arousal in Text Chats with Speech Balloons

Toshiki Aoki*, Rintaro Chujo*, Katsufumi Matsui, Saemi Choi, and Ari Hautasaari. 2022. EmoBalloon - Conveying Emotional Arousal in Text Chats with Speech Balloons. In CHI Conference on Human Factors in Computing Systems (CHI '22). Association for Computing Machinery, New York, NY, USA, Article 527, 1–16. https://doi.org/10.1145/3491102.3501920 (🏆 Best Paper Awards, top 1%) (*: equal contribution)

63587cd10a174f1a6b52f9b2606a1ee0?s=128

Rintaro Chujo

May 09, 2022
Tweet

Other Decks in Research

Transcript

  1. EmoBalloon Conveying Emotional Arousal in Text Chats with Speech Balloons

    * Both authors contributed equally to this research. TOSHIKI AOKI* @hci_cg_student RINTARO CHUJO* @rintaro_chujo KATSUFUMI MATSUI SAEMI CHOI ARI HAUTASAARI
  2. EmoBalloon is a novel method to convey emotional arousal in

    text chats
  3. EmoBalloon Introduction

  4. WhatsApp Messenger Wechat LINE etc.. 10:33 am Lorem ipsum dolor

    sit 10:33 am Lorem ipsum dolor sit amet, 10:42 am 10:33 am Lorem Lorem ipsum 10:35 am 10:33 am 10:33 am Lorem ipsum dolor Lorem ipsum Lorem
  5. Lorem ipsum 10:35 am WhatsApp Lorem ipsum dolor sit amet,

    10:42 am Messenger Lorem ipsum dolor 10:33 am Wechat Lorem ipsum 10:33 am Lorem LINE 10:33 am Lorem etc.. 10:33 am Lorem ipsum dolor sit 10:33 am Problem: Limitations to the transfer of emotional information
  6. 10:33 am Lorem ipsum dolor sit 10:33 am Lorem ipsum

    dolor sit amet, 10:42 am 10:33 am Lorem Lorem ipsum 10:35 am 10:33 am 10:33 am Lorem ipsum dolor Lorem ipsum Lorem in Text Chat Letters Emoticons Stickers ... (is it enough?)
  7. Face-to-Face and richer media Contents of talk Tone of voice

    Gestures Facial expressions etc...
  8. Lorem ipsum 10:35 am Lorem ipsum dolor sit amet, 10:42

    am Lorem ipsum dolor 10:33 am Lorem ipsum 10:33 am Lorem 10:33 am Lorem 10:33 am Lorem ipsum dolor sit 10:33 am Face-to-Face and richer media is BETTER 😊 in emotional communication Text Chat is WORSE ☹ in emotional communication
  9. Emoticons Stickers Colors Typeface etc... (iMessage, Apple) (LINE)

  10. Emoticons Stickers Colors Typeface etc... (Chen et. al., 2021) (Choi

    and Aizawa, 2019)
  11. (c) Ken Yagami (cited from the Mange109) We focused on

    Japanese manga, rich in emotional expression.
  12. Typeface Font size Onomatopoeia Speech Balloon etc... (c) Ken Yagami

    (cited from the Mange109) Japanese Manga
  13. Typeface Font size Onomatopoeia Speech Balloon etc... (c) Ken Yagami

    (cited from the Mange109) Japanese Manga
  14. EmoBalloon Approach

  15. Implement text chat system with generated speech balloons called “EmoBalloon”

    Analyze Validate ACGAN by crowdsourcing the shape of speech balloons in Japanese manga dataset Develop an emotional speech balloon generator with ACGAN
  16. Analyze the shape of speech balloons in Japanese manga dataset

    Detect and classify speech balloons into three types based on Tanaka et. al., (2015) Others (c) Ken Yagami, Masakazu Ooi, Masako Yoshi, Motoei Shinzawa (cited from the Manga109) High Arousal Emotion Explosion Shape Round Shape Neutral Emotion
  17. Analyze the shape of speech balloons in Japanese manga dataset

    Detect and classify speech balloons into three types based on Tanaka et. al., (2015) (c) Ken Yagami, Masakazu Ooi, Masako Yoshi, Motoei Shinzawa (cited from the Manga109) High Arousal Emotion Explosion Shape Round Shape Neutral Emotion
  18. Develop an emotional speech balloon generator with ACGAN Trained a

    generative model (ACGAN) with rendered 100,000 speech balloon images (c) Ken Yagami, Masakazu Ooi, Masako Yoshi, Motoei Shinzawa (cited from the Manga109)
  19. Crowdworkers (n=210) complete a pairwise comparison of the generated speech

    balloons arousal level 0.1 0.3 0.7 0.9 0.5 Validate ACGAN by crowdsourcing
  20. Crowdworkers (n=210) complete a pairwise comparison of the generated speech

    balloons The more explosion-like shape the speech balloons are, the higher the arousal level they are associated with. Validate ACGAN by crowdsourcing
  21. Implement text chat system with generated speech balloons called “EmoBalloon”

    Web Speech API SpeechRecognition ACGAN Generated Speech Balloon Serif Text “HELLO” HELLO Maximum Volume based on Frick (1985) Voice Input
  22. EmoBalloon Expreriments

  23. We conducted a controlled experiment to investigate the effects that

    emotional speech balloons have on text based communication in interactive settings.
  24. Voice input EmoBalloon Speech balloons were automatically generated according to

    the voice input Manual selection EmoBalloon The user manually selects speech balloons Emoticon (control) Text Only (within subjects)
  25. Voice input EmoBalloon Speech balloons were automatically generated according to

    the voice input Manual selection EmoBalloon The user manually selects speech balloons Emoticon (control) Text Only (within subjects)
  26. Voice input EmoBalloon Speech balloons were automatically generated according to

    the voice input Manual selection EmoBalloon The user manually selects speech balloons Emoticon (control) Text Only (within subjects)
  27. Voice input EmoBalloon Speech balloons were automatically generated according to

    the voice input Manual selection EmoBalloon The user manually selects speech balloons Emoticon (control) Text Only (within subjects)
  28. Voice input EmoBalloon Speech balloons were automatically generated according to

    the voice input Manual selection EmoBalloon The user manually selects speech balloons Emoticon (control) Text Only (within subjects)
  29. Dialogue script Japanese participants (n=62) role-played a dialogue script derived

    from rong-chang ESL Inc. (2021) with a partner and rated the emotional Arousal (Calm to Excited) and Valence (Negative to Positive) each time a message was sent or received.
  30. EmoBalloon Results

  31. Difference in arousal evaluations between a message sender and a

    receiver Text only Paralinguistic cues Voice input EmoBalloon 1.2 1.0 0.8 0.6 0.4 0.2 0.0 Manual selection EmoBalloon Emoticon
  32. 0.4 0.2 0.0 Voice input Manual selection Emoticon EmoBalloon EmoBalloon

    1.2 1.0 0.8 0.6 Difference in arousal evaluations between a message sender and a receiver Text only Paralinguistic cues EmoBalloon may be a m or e effective way to convey arousal than emoticons
  33. Voice input EmoBalloon 1 2 3 Manual selection EmoBalloon Emoticon

    Mean arousal evaluations by the sender Text only Paralinguistic cues
  34. 1 Voice input Manual selection Emoticon EmoBalloon EmoBalloon 2 3

    Mean arousal evaluations by the sender Text only Paralinguistic cues Senders using voice input EmoBalloon may have been worried about a type of “negative emotional leak”
  35. Mean valence evaluations We found that EmoBalloon had no significant

    effect on receivers’ perceptions regarding the emotional valence, unlike emoticons.
  36. EmoBalloon Conclusion

  37. We focused on Japanese manga as a means of conveying

    emotion and (c) Ken Yagami (cited from the Mange109) We propose EmoBalloon a novel method to convey emotional arousal with speech balloons
  38. Takeaway 1. Emotional speech balloons are an effective way to

    convey emotional arousal in text chats 2. Speech balloon shapes do not affect emotional valence in text chats
  39. For more details and results, please see our paper. https://emoballoon.emocomm.net

  40. EmoBalloon Conveying Emotional Arousal in Text Chats with Speech Balloons

    TOSHIKI AOKI* RINTARO CHUJO* KATSUFUMI MATSUI SAEMI CHOI ARI HAUTASAARI * Both authors contributed equally to this research. This work was supported by JSPS KAKENHI Grant Number JP18K18085. This research is part of the results of Value Exchange Engineering, a joint research project between Mercari, Inc. and the RIISE.