EmoBalloon - Conveying Emotional Arousal in Text Chats with Speech Balloons

Toshiki Aoki*, Rintaro Chujo*, Katsufumi Matsui, Saemi Choi, and Ari Hautasaari. 2022. EmoBalloon - Conveying Emotional Arousal in Text Chats with Speech Balloons. In CHI Conference on Human Factors in Computing Systems (CHI '22). Association for Computing Machinery, New York, NY, USA, Article 527, 1–16. https://doi.org/10.1145/3491102.3501920 (🏆 Best Paper Awards, top 1%) (*: equal contribution)

Transcript

1. EmoBalloon Conveying Emotional Arousal in Text Chats with Speech Balloons * Both authors contributed equally to this research. TOSHIKI AOKI* @hci_cg_student RINTARO CHUJO* @rintaro_chujo KATSUFUMI MATSUI SAEMI CHOI ARI HAUTASAARI
2. EmoBalloon is a novel method to convey emotional arousal in text chats
3. EmoBalloon Introduction
4. WhatsApp Messenger Wechat LINE etc.. 10:33 am Lorem ipsum dolor sit 10:33 am Lorem ipsum dolor sit amet, 10:42 am 10:33 am Lorem Lorem ipsum 10:35 am 10:33 am 10:33 am Lorem ipsum dolor Lorem ipsum Lorem
5. Lorem ipsum 10:35 am WhatsApp Lorem ipsum dolor sit amet, 10:42 am Messenger Lorem ipsum dolor 10:33 am Wechat Lorem ipsum 10:33 am Lorem LINE 10:33 am Lorem etc.. 10:33 am Lorem ipsum dolor sit 10:33 am Problem: Limitations to the transfer of emotional information
6. 10:33 am Lorem ipsum dolor sit 10:33 am Lorem ipsum dolor sit amet, 10:42 am 10:33 am Lorem Lorem ipsum 10:35 am 10:33 am 10:33 am Lorem ipsum dolor Lorem ipsum Lorem in Text Chat Letters Emoticons Stickers ... (is it enough?)
7. Face-to-Face and richer media Contents of talk Tone of voice Gestures Facial expressions etc...
8. Lorem ipsum 10:35 am Lorem ipsum dolor sit amet, 10:42 am Lorem ipsum dolor 10:33 am Lorem ipsum 10:33 am Lorem 10:33 am Lorem 10:33 am Lorem ipsum dolor sit 10:33 am Face-to-Face and richer media is BETTER 😊 in emotional communication Text Chat is WORSE ☹ in emotional communication
9. Emoticons Stickers Colors Typeface etc... (iMessage, Apple) (LINE)
10. Emoticons Stickers Colors Typeface etc... (Chen et. al., 2021) (Choi and Aizawa, 2019)
11. (c) Ken Yagami (cited from the Mange109) We focused on Japanese manga, rich in emotional expression.
12. Typeface Font size Onomatopoeia Speech Balloon etc... (c) Ken Yagami (cited from the Mange109) Japanese Manga
13. Typeface Font size Onomatopoeia Speech Balloon etc... (c) Ken Yagami (cited from the Mange109) Japanese Manga
14. EmoBalloon Approach
15. Implement text chat system with generated speech balloons called “EmoBalloon” Analyze Validate ACGAN by crowdsourcing the shape of speech balloons in Japanese manga dataset Develop an emotional speech balloon generator with ACGAN
16. Analyze the shape of speech balloons in Japanese manga dataset Detect and classify speech balloons into three types based on Tanaka et. al., (2015) Others (c) Ken Yagami, Masakazu Ooi, Masako Yoshi, Motoei Shinzawa (cited from the Manga109) High Arousal Emotion Explosion Shape Round Shape Neutral Emotion
17. Analyze the shape of speech balloons in Japanese manga dataset Detect and classify speech balloons into three types based on Tanaka et. al., (2015) (c) Ken Yagami, Masakazu Ooi, Masako Yoshi, Motoei Shinzawa (cited from the Manga109) High Arousal Emotion Explosion Shape Round Shape Neutral Emotion
18. Develop an emotional speech balloon generator with ACGAN Trained a generative model (ACGAN) with rendered 100,000 speech balloon images (c) Ken Yagami, Masakazu Ooi, Masako Yoshi, Motoei Shinzawa (cited from the Manga109)
19. Crowdworkers (n=210) complete a pairwise comparison of the generated speech balloons arousal level 0.1 0.3 0.7 0.9 0.5 Validate ACGAN by crowdsourcing
20. Crowdworkers (n=210) complete a pairwise comparison of the generated speech balloons The more explosion-like shape the speech balloons are, the higher the arousal level they are associated with. Validate ACGAN by crowdsourcing
21. Implement text chat system with generated speech balloons called “EmoBalloon” Web Speech API SpeechRecognition ACGAN Generated Speech Balloon Serif Text “HELLO” HELLO Maximum Volume based on Frick (1985) Voice Input
22. EmoBalloon Expreriments
23. We conducted a controlled experiment to investigate the effects that emotional speech balloons have on text based communication in interactive settings.
24. Voice input EmoBalloon Speech balloons were automatically generated according to the voice input Manual selection EmoBalloon The user manually selects speech balloons Emoticon (control) Text Only (within subjects)
25. Voice input EmoBalloon Speech balloons were automatically generated according to the voice input Manual selection EmoBalloon The user manually selects speech balloons Emoticon (control) Text Only (within subjects)
26. Voice input EmoBalloon Speech balloons were automatically generated according to the voice input Manual selection EmoBalloon The user manually selects speech balloons Emoticon (control) Text Only (within subjects)
27. Voice input EmoBalloon Speech balloons were automatically generated according to the voice input Manual selection EmoBalloon The user manually selects speech balloons Emoticon (control) Text Only (within subjects)
28. Voice input EmoBalloon Speech balloons were automatically generated according to the voice input Manual selection EmoBalloon The user manually selects speech balloons Emoticon (control) Text Only (within subjects)
29. Dialogue script Japanese participants (n=62) role-played a dialogue script derived from rong-chang ESL Inc. (2021) with a partner and rated the emotional Arousal (Calm to Excited) and Valence (Negative to Positive) each time a message was sent or received.
30. EmoBalloon Results
31. Difference in arousal evaluations between a message sender and a receiver Text only Paralinguistic cues Voice input EmoBalloon 1.2 1.0 0.8 0.6 0.4 0.2 0.0 Manual selection EmoBalloon Emoticon
32. 0.4 0.2 0.0 Voice input Manual selection Emoticon EmoBalloon EmoBalloon 1.2 1.0 0.8 0.6 Difference in arousal evaluations between a message sender and a receiver Text only Paralinguistic cues EmoBalloon may be a m or e effective way to convey arousal than emoticons
33. Voice input EmoBalloon 1 2 3 Manual selection EmoBalloon Emoticon Mean arousal evaluations by the sender Text only Paralinguistic cues
34. 1 Voice input Manual selection Emoticon EmoBalloon EmoBalloon 2 3 Mean arousal evaluations by the sender Text only Paralinguistic cues Senders using voice input EmoBalloon may have been worried about a type of “negative emotional leak”
35. Mean valence evaluations We found that EmoBalloon had no significant effect on receivers’ perceptions regarding the emotional valence, unlike emoticons.
36. EmoBalloon Conclusion
37. We focused on Japanese manga as a means of conveying emotion and (c) Ken Yagami (cited from the Mange109) We propose EmoBalloon a novel method to convey emotional arousal with speech balloons
38. Takeaway 1. Emotional speech balloons are an effective way to convey emotional arousal in text chats 2. Speech balloon shapes do not affect emotional valence in text chats
39. For more details and results, please see our paper. https://emoballoon.emocomm.net
40. EmoBalloon Conveying Emotional Arousal in Text Chats with Speech Balloons TOSHIKI AOKI* RINTARO CHUJO* KATSUFUMI MATSUI SAEMI CHOI ARI HAUTASAARI * Both authors contributed equally to this research. This work was supported by JSPS KAKENHI Grant Number JP18K18085. This research is part of the results of Value Exchange Engineering, a joint research project between Mercari, Inc. and the RIISE.