See-Through Captions in a Museum Guided Tour: Exploring Museum Guided Tour for Deaf and Hard-of-Hearing People with Real-Time Captioning on Transparent Display ICCHP-AAATE 2022 #ICCHP_AAATE_22 (Oral presentation by Ippei Suzuki)

This slide was presented in Session "Language Accessibility for the Deaf and Hard-Of-Hearing" at the Joint International Conference on Digital Inclusion, Assistive Technology & Accessibility - ICCHP-AAATE 2022.
https://icchp-aaate.org

【Publication】
Suzuki, I., Yamamoto, K., Shitara, A., Hyakuta, R., Iijima, R., Ochiai, Y. (2022). See-Through Captions in a Museum Guided Tour: Exploring Museum Guided Tour for Deaf and Hard-of-Hearing People with Real-Time Captioning on Transparent Display. In: Miesenberger, K., Kouroupetroglou, G., Mavrou, K., Manduchi, R., Covarrubias Rodriguez, M., Penáz, P. (eds) Computers Helping People with Special Needs. ICCHP-AAATE 2022. Lecture Notes in Computer Science, vol 13341. Springer, Cham. https://doi.org/10.1007/978-3-031-08648-9_64

【Project page】
https://digitalnature.slis.tsukuba.ac.jp/2021/02/see-through-captions/

【Presenter】
Ippei Suzuki (鈴木 一平)
Ph.D. Student (Doctoral Program in Informatics)
Graduate School of Comprehensive Human Sciences
University of Tsukuba (JP)
Digital Nature Group (Yoichi Ochiai)
https://1heisuzuki.com

【Abstract】
Access to audible information for deaf and hard-of-hearing (DHH) people is an essential component as we move towards a diverse society. Real-time captioning is a technology with great potential to help the lives of DHH people, and various applications utilizing mobile devices have been developed. These technologies can improve the daily lives of DHH people and can considerably change the value of audio content provided in public facilities such as museums. We developed a real-time captioning system called See-Through Captions that displays subtitles on a transparent display and conducted a demonstration experiment to apply this system to a guided tour in a museum. Eleven DHH people participated in this demonstration experiment, and through questionnaires and interviews, we explored the possibility of utilizing the transparent subtitle system in a guided tour at the museum.

Transcript

1. © R&D Center for Digital Nature / xDiversity See-Through Captions
 in a Museum Guided Tour: Research and Development Center for Digital Nature, University of Tsukuba, Japan
 *These authors contributed equally to this research. Ippei Suzuki*, Kenta Yamamoto*, Akihisa Shitara, 
 Ryosuke Hyakuta, Ryo Iijima, Yoichi Ochiai Exploring Museum Guided Tour for Deaf and Hard-of-Hearing People with Real-Time Captioning on Transparent Display
2. 2 © R&D Center for Digital Nature / xDiversity
3. 3 © R&D Center for Digital Nature / xDiversity Background & Introduction
4. 4 © R&D Center for Digital Nature / xDiversity How can we update 
 the accessibility of museum guided tours 
 for deaf and hard-of-hearing people?
5. 5 © R&D Center for Digital Nature / xDiversity Background Accessibility of Museum Guided Tour for DHH People Image (Left): Namatame et al. 2020. The Science Communication Tour with a Sign Language Interpreter Image (Right): Namatame et al. 2019. Can Exhibit-Explanations in Sign Language Contribute to the Accessibility of Aquariums? Approaches to accessibility of audible information Sign-language guided tours Auditory information via mobile device
6. 6 © R&D Center for Digital Nature / xDiversity Background Accessibility of Museum Guided Tour for DHH People Image (Left): Namatame et al. 2020. The Science Communication Tour with a Sign Language Interpreter Image (Right): Namatame et al. 2019. Can Exhibit-Explanations in Sign Language Contribute to the Accessibility of Aquariums? One-way information Cannot communicate with guide Difficult to recruit an interpreter Sign-language guided tours Auditory information via mobile device
7. 7 © R&D Center for Digital Nature / xDiversity Background Automatic Speech Recognition (ASR) Video: https://www.android.com/accessibility/live-transcribe/ Automatic Speech Recognition (ASR)
8. 8 © R&D Center for Digital Nature / xDiversity Background Automatic Speech Recognition (ASR) ASR on mobile devices ASR on augmented reality devices Approaches to utilize automatic speech recognition
9. 9 © R&D Center for Digital Nature / xDiversity Background Automatic Speech Recognition (ASR) ASR on mobile devices ASR on augmented reality devices Speaker cannot confirm whether the speech has been correctly recognized. The facial expression and body language of the partner are overlooked
10. 10 © R&D Center for Digital Nature / xDiversity Introduction Our Previous Work Kenta Yamamoto, Ippei Suzuki, Akihisa Shitara, Yoichi Ochiai. ASSETS’21. See-Through Captions: Real-Time Captioning on Transparent Display for Deaf and Hard-of-Hearing People.
11. 11 © R&D Center for Digital Nature / xDiversity Introduction Transparent Display
12. 12 © R&D Center for Digital Nature / xDiversity Implementation
13. 13 © R&D Center for Digital Nature / xDiversity Implementation Overview Microphone Backpack Transparent Display
14. 14 © R&D Center for Digital Nature / xDiversity Implementation Transparent Display
15. 15 © R&D Center for Digital Nature / xDiversity Implementation Transparent Display 7 cm 8 cm Resolution 320 × 360 pixels Number of Colors 4,096 Colors Transmittance 87% Weight Approx. 130 g Brightness (Center) 270 cd/m² Contrast Ratio 20:1 Transparent Display Japan Display Inc.
16. 16 © R&D Center for Digital Nature / xDiversity Image: https://www.shure.com/en-US/products/microphones/wh20 Implementation Microphone Headset Microphone Shure; WH20XLR Unidirectional cardioid directivity Less surrounding noise
17. 17 © R&D Center for Digital Nature / xDiversity Implementation Backpack Display Drive Board Battery Computer Tablet PC Mobile Wi-Fi Hotspot Weight : approx. 3.3 kg *Inserted into backpack Audio Interface
18. 18 © R&D Center for Digital Nature / xDiversity Implementation Automatic Speech Recognition API Image (Google Chrome Logo): https://www.google.com/chrome/ Web Speech API on Google Chrome Javascript API
19. 19 © R&D Center for Digital Nature / xDiversity Implementation Overview Audio I/F Computer Drive Board Battery Mobile Wi-Fi Hotspot Display Microphone
20. 20 © R&D Center for Digital Nature / xDiversity Case Study: Guided Tour in Museum
21. 21 © R&D Center for Digital Nature / xDiversity Case Study: Guided Tour in Museum Image (Right): https://www.miraikan.jst.go.jp/aboutus/ Image (Science Communicators): https://www.miraikan.jst.go.jp/en/aboutus/communicators/ɹ Bunsuke Kawasaki Sakiko Tanaka Science Communicators
22. 22 © R&D Center for Digital Nature / xDiversity Case Study: Guided Tour in Museum Tour theme: “The difference between humans and robots”
23. 23 © R&D Center for Digital Nature / xDiversity Case Study: Guided Tour in Museum Communication Method Guide Person See-Through Captions DHH People DHH People Speech or Writing Guide Person Tours were conducted in Japanese language
24. When ASR system stopped… Guide express ‘’wait’’ in gestures of sign language 24 © R&D Center for Digital Nature / xDiversity Case Study: Guided Tour in Museum Communication Protocol
25. 25 © R&D Center for Digital Nature / xDiversity Case Study: Guided Tour in Museum When participants wanted to talk They raise their hand or notepad Communication Protocol When someone talked one’s idea “Applause” in gestures of sing language
26. 26 © R&D Center for Digital Nature / xDiversity Case Study: Guided Tour in Museum Procedure 1. Participants were asked about the preferred position of display 
 and asked about preferred infection-prevention methods 
 (face shield or face mask) 2. The guide described the theme of the tour 
 and conducted some quiz games about Miraikan 3. Guided tour 4. Participants were asked to fill out the questionnaires 
 and be interviewed
27. 27 © R&D Center for Digital Nature / xDiversity Case Study: Guided Tour in Museum Display Position: Basic
28. 28 © R&D Center for Digital Nature / xDiversity Case Study: Guided Tour in Museum Display Position: Overlay
29. 29 © R&D Center for Digital Nature / xDiversity Case Study: Guided Tour in Museum Display Position: Hands-Free
30. 30 © R&D Center for Digital Nature / xDiversity Case Study: Guided Tour in Museum Display Positions
31. 31 © R&D Center for Digital Nature / xDiversity Case Study: Guided Tour in Museum Participants 11 DHH Participants | 18-53 years old 4 Hearing Participants | 36-56 years old Tour Groups Each tour group contained at least one DHH person; some groups contained a few hearing people. DHH person Hearing person 9 Groups + 1 Hearing Participant | without questionnaires
32. 32 © R&D Center for Digital Nature / xDiversity Results & Discussion
33. 33 © R&D Center for Digital Nature / xDiversity Results Quantitative Evaluation Q1. Readability of the ASR results Q2. Noticeability of misrecognition Q3. Whether they wanted to
 continue utilizing this system DHH Hearing M SD 4.45 .50 4.00 .63 DHH Hearing 4.27 .86 2.80 .75 DHH Hearing 4.73 .45 4.20 .40 Strongly Disagree Strongly Agree
34. 34 © R&D Center for Digital Nature / xDiversity Results & Discussion Automatic Speech Recognition ASR sometimes misrecognize the words Possible solution: The speaker acquire utterances and speaking styles that were easy for the system to recognize correctly Dictionary registration for technical terms / nouns It was difficult to read when misrecognition occurred
35. 35 © R&D Center for Digital Nature / xDiversity Results & Discussion Readability of Captions Difficult to see in some settings 
 especially when there is a strong 
 light in the background The readability affected 
 by background and reflection Possible solution: The guide pays attention to that Easily text design changeable system
36. 36 © R&D Center for Digital Nature / xDiversity Results & Discussion How to Display Captions The character flow was too fast The screen was filled with rephrasing
 when misrecognition occurred Subtitle design is 
 for a larger transparent display Possible solution: Function to look back at the history Little larger transparent display
37. 37 © R&D Center for Digital Nature / xDiversity Results & Discussion Benefits of Transparency Participants could see the subtitles 
 while looking at the contents 
 of the exhibition It was easy to communicate 
 in both directions 
 by being able to see the guide’s face 
 and make eye contact Transparency made it possible to see 
 the whole without obstructing the view, 
 and that they did not feel any gap
38. 38 © R&D Center for Digital Nature / xDiversity Results & Discussion Display Position If the display is held near the face, 
 it is easier because 
 there is only one place to watch.” Handheld setup makes us easy
 to change the position We asked participants 
 which position is preferred “
39. 39 © R&D Center for Digital Nature / xDiversity Results & Discussion Display Type and Size Participants mentioned: As a future work, it is necessary 
 to compare See-Through Captions
 with other methods in detail AR glasses was tiring but
 See-Through Captions was easier Display size was small Example of other methods:
 Two-sided tablet
40. 40 © R&D Center for Digital Nature / xDiversity Results & Discussion Challenges Specific to Guided Tours When multiple people participated,
 their voice is NOT displayed See-Through Captions was originally
 developed as a 1:1 communication Possible solution: Participants also wear microphones Participants also hold displays
41. 41 © R&D Center for Digital Nature / xDiversity Future Work How Can DHH People Communicate with Tour Guide? Some DHH people do not 
 tend to speak by their voice The current system assumes that DHH people speak using voice Possible solution: Additional input interface?
42. 42 © R&D Center for Digital Nature / xDiversity Future Work How Can We Communicate with DHH People Who Prefer Sign Language than Text? Some DHH people prefer to read sign language The current system assumes that DHH people read texts Possible solution: Text <-> Sign language Translator?
43. 43 © R&D Center for Digital Nature / xDiversity 1. Implementation of 
 the smaller version of See-Through Captions 2. Case study: a guided tour in a museum 3. Discussion of findings based on the results Summary of Contributions
44. © R&D Center for Digital Nature / xDiversity Akihisa Shitara Ippei Suzuki* Kenta Yamamoto* Ryosuke Hyakuta Ryo Iijima Yoichi Ochiai See-Through Captions in a Museum Guided Tour: Exploring Museum Guided Tour for Deaf and Hard-of-Hearing People
 with Real-Time Captioning on Transparent Display *These authors contributed equally to this research. Research and Development Center for Digital Nature, University of Tsukuba, Japan
45. 45 © R&D Center for Digital Nature / xDiversity Acknowledgements JST CREST Grant Number JPMJCR19F2, including the AIP Challenge Program, Japan. Transparent display provided by User study support and assisted by This work was supported by Bunsuke Kawasaki, Sakiko Tanaka, Chisa Mitsuhashi