Upgrade to Pro — share decks privately, control downloads, hide ads and more …

See-Through Captions in a Museum Guided Tour: Exploring Museum Guided Tour for Deaf and Hard-of-Hearing People with Real-Time Captioning on Transparent Display ICCHP-AAATE 2022 #ICCHP_AAATE_22 (Oral presentation by Ippei Suzuki)

See-Through Captions in a Museum Guided Tour: Exploring Museum Guided Tour for Deaf and Hard-of-Hearing People with Real-Time Captioning on Transparent Display ICCHP-AAATE 2022 #ICCHP_AAATE_22 (Oral presentation by Ippei Suzuki)

This slide was presented in Session "Language Accessibility for the Deaf and Hard-Of-Hearing" at the Joint International Conference on Digital Inclusion, Assistive Technology & Accessibility - ICCHP-AAATE 2022.
https://icchp-aaate.org

【Publication】
Suzuki, I., Yamamoto, K., Shitara, A., Hyakuta, R., Iijima, R., Ochiai, Y. (2022). See-Through Captions in a Museum Guided Tour: Exploring Museum Guided Tour for Deaf and Hard-of-Hearing People with Real-Time Captioning on Transparent Display. In: Miesenberger, K., Kouroupetroglou, G., Mavrou, K., Manduchi, R., Covarrubias Rodriguez, M., Penáz, P. (eds) Computers Helping People with Special Needs. ICCHP-AAATE 2022. Lecture Notes in Computer Science, vol 13341. Springer, Cham. https://doi.org/10.1007/978-3-031-08648-9_64

【Project page】
https://digitalnature.slis.tsukuba.ac.jp/2021/02/see-through-captions/

【Presenter】
Ippei Suzuki (鈴木 一平)
Ph.D. Student (Doctoral Program in Informatics)
Graduate School of Comprehensive Human Sciences
University of Tsukuba (JP)
Digital Nature Group (Yoichi Ochiai)
https://1heisuzuki.com

【Abstract】
Access to audible information for deaf and hard-of-hearing (DHH) people is an essential component as we move towards a diverse society. Real-time captioning is a technology with great potential to help the lives of DHH people, and various applications utilizing mobile devices have been developed. These technologies can improve the daily lives of DHH people and can considerably change the value of audio content provided in public facilities such as museums. We developed a real-time captioning system called See-Through Captions that displays subtitles on a transparent display and conducted a demonstration experiment to apply this system to a guided tour in a museum. Eleven DHH people participated in this demonstration experiment, and through questionnaires and interviews, we explored the possibility of utilizing the transparent subtitle system in a guided tour at the museum.

6d667e747a4a1cda7d201c3358424257?s=128

Digital Nature Group

July 13, 2022
Tweet

More Decks by Digital Nature Group

Other Decks in Research

Transcript

  1. © R&D Center for Digital Nature / xDiversity See-Through Captions


    in a Museum Guided Tour: Research and Development Center for Digital Nature, University of Tsukuba, Japan
 *These authors contributed equally to this research. Ippei Suzuki*, Kenta Yamamoto*, Akihisa Shitara, 
 Ryosuke Hyakuta, Ryo Iijima, Yoichi Ochiai Exploring Museum Guided Tour for Deaf and Hard-of-Hearing People with Real-Time Captioning on Transparent Display
  2. 2 © R&D Center for Digital Nature / xDiversity

  3. 3 © R&D Center for Digital Nature / xDiversity Background

    & Introduction
  4. 4 © R&D Center for Digital Nature / xDiversity How

    can we update 
 the accessibility of museum guided tours 
 for deaf and hard-of-hearing people?
  5. 5 © R&D Center for Digital Nature / xDiversity Background

    Accessibility of Museum Guided Tour for DHH People Image (Left): Namatame et al. 2020. The Science Communication Tour with a Sign Language Interpreter Image (Right): Namatame et al. 2019. Can Exhibit-Explanations in Sign Language Contribute to the Accessibility of Aquariums? Approaches to accessibility of audible information Sign-language guided tours Auditory information via mobile device
  6. 6 © R&D Center for Digital Nature / xDiversity Background

    Accessibility of Museum Guided Tour for DHH People Image (Left): Namatame et al. 2020. The Science Communication Tour with a Sign Language Interpreter Image (Right): Namatame et al. 2019. Can Exhibit-Explanations in Sign Language Contribute to the Accessibility of Aquariums? One-way information Cannot communicate with guide Difficult to recruit an interpreter Sign-language guided tours Auditory information via mobile device
  7. 7 © R&D Center for Digital Nature / xDiversity Background

    Automatic Speech Recognition (ASR) Video: https://www.android.com/accessibility/live-transcribe/ Automatic Speech Recognition (ASR)
  8. 8 © R&D Center for Digital Nature / xDiversity Background

    Automatic Speech Recognition (ASR) ASR on mobile devices ASR on augmented reality devices Approaches to utilize automatic speech recognition
  9. 9 © R&D Center for Digital Nature / xDiversity Background

    Automatic Speech Recognition (ASR) ASR on mobile devices ASR on augmented reality devices Speaker cannot confirm whether the speech has been correctly recognized. The facial expression and body language of the partner are overlooked
  10. 10 © R&D Center for Digital Nature / xDiversity Introduction

    Our Previous Work Kenta Yamamoto, Ippei Suzuki, Akihisa Shitara, Yoichi Ochiai. ASSETS’21. See-Through Captions: Real-Time Captioning on Transparent Display for Deaf and Hard-of-Hearing People.
  11. 11 © R&D Center for Digital Nature / xDiversity Introduction

    Transparent Display
  12. 12 © R&D Center for Digital Nature / xDiversity Implementation

  13. 13 © R&D Center for Digital Nature / xDiversity Implementation

    Overview Microphone Backpack Transparent Display
  14. 14 © R&D Center for Digital Nature / xDiversity Implementation

    Transparent Display
  15. 15 © R&D Center for Digital Nature / xDiversity Implementation

    Transparent Display 7 cm 8 cm Resolution 320 × 360 pixels Number of Colors 4,096 Colors Transmittance 87% Weight Approx. 130 g Brightness (Center) 270 cd/m² Contrast Ratio 20:1 Transparent Display Japan Display Inc.
  16. 16 © R&D Center for Digital Nature / xDiversity Image:

    https://www.shure.com/en-US/products/microphones/wh20 Implementation Microphone Headset Microphone Shure; WH20XLR Unidirectional cardioid directivity Less surrounding noise
  17. 17 © R&D Center for Digital Nature / xDiversity Implementation

    Backpack Display Drive Board Battery Computer Tablet PC Mobile Wi-Fi Hotspot Weight : approx. 3.3 kg *Inserted into backpack Audio Interface
  18. 18 © R&D Center for Digital Nature / xDiversity Implementation

    Automatic Speech Recognition API Image (Google Chrome Logo): https://www.google.com/chrome/ Web Speech API on Google Chrome Javascript API
  19. 19 © R&D Center for Digital Nature / xDiversity Implementation

    Overview Audio I/F Computer Drive Board Battery Mobile Wi-Fi Hotspot Display Microphone
  20. 20 © R&D Center for Digital Nature / xDiversity Case

    Study: Guided Tour in Museum
  21. 21 © R&D Center for Digital Nature / xDiversity Case

    Study: Guided Tour in Museum Image (Right): https://www.miraikan.jst.go.jp/aboutus/ Image (Science Communicators): https://www.miraikan.jst.go.jp/en/aboutus/communicators/ɹ Bunsuke Kawasaki Sakiko Tanaka Science Communicators
  22. 22 © R&D Center for Digital Nature / xDiversity Case

    Study: Guided Tour in Museum Tour theme: “The difference between humans and robots”
  23. 23 © R&D Center for Digital Nature / xDiversity Case

    Study: Guided Tour in Museum Communication Method Guide Person See-Through Captions DHH People DHH People Speech or Writing Guide Person Tours were conducted in Japanese language
  24. When ASR system stopped… Guide express ‘’wait’’ in gestures of

    sign language 24 © R&D Center for Digital Nature / xDiversity Case Study: Guided Tour in Museum Communication Protocol
  25. 25 © R&D Center for Digital Nature / xDiversity Case

    Study: Guided Tour in Museum When participants wanted to talk They raise their hand or notepad Communication Protocol When someone talked one’s idea “Applause” in gestures of sing language
  26. 26 © R&D Center for Digital Nature / xDiversity Case

    Study: Guided Tour in Museum Procedure 1. Participants were asked about the preferred position of display 
 and asked about preferred infection-prevention methods 
 (face shield or face mask) 2. The guide described the theme of the tour 
 and conducted some quiz games about Miraikan 3. Guided tour 4. Participants were asked to fill out the questionnaires 
 and be interviewed
  27. 27 © R&D Center for Digital Nature / xDiversity Case

    Study: Guided Tour in Museum Display Position: Basic
  28. 28 © R&D Center for Digital Nature / xDiversity Case

    Study: Guided Tour in Museum Display Position: Overlay
  29. 29 © R&D Center for Digital Nature / xDiversity Case

    Study: Guided Tour in Museum Display Position: Hands-Free
  30. 30 © R&D Center for Digital Nature / xDiversity Case

    Study: Guided Tour in Museum Display Positions
  31. 31 © R&D Center for Digital Nature / xDiversity Case

    Study: Guided Tour in Museum Participants 11 DHH Participants | 18-53 years old 4 Hearing Participants | 36-56 years old Tour Groups Each tour group contained at least one DHH person; some groups contained a few hearing people. DHH person Hearing person 9 Groups + 1 Hearing Participant | without questionnaires
  32. 32 © R&D Center for Digital Nature / xDiversity Results

    & Discussion
  33. 33 © R&D Center for Digital Nature / xDiversity Results

    Quantitative Evaluation Q1. Readability of the ASR results Q2. Noticeability of misrecognition Q3. Whether they wanted to
 continue utilizing this system DHH Hearing M SD 4.45 .50 4.00 .63 DHH Hearing 4.27 .86 2.80 .75 DHH Hearing 4.73 .45 4.20 .40 Strongly Disagree Strongly Agree
  34. 34 © R&D Center for Digital Nature / xDiversity Results

    & Discussion Automatic Speech Recognition ASR sometimes misrecognize the words Possible solution: The speaker acquire utterances and speaking styles that were easy for the system to recognize correctly Dictionary registration for technical terms / nouns It was difficult to read when misrecognition occurred
  35. 35 © R&D Center for Digital Nature / xDiversity Results

    & Discussion Readability of Captions Difficult to see in some settings 
 especially when there is a strong 
 light in the background The readability affected 
 by background and reflection Possible solution: The guide pays attention to that Easily text design changeable system
  36. 36 © R&D Center for Digital Nature / xDiversity Results

    & Discussion How to Display Captions The character flow was too fast The screen was filled with rephrasing
 when misrecognition occurred Subtitle design is 
 for a larger transparent display Possible solution: Function to look back at the history Little larger transparent display
  37. 37 © R&D Center for Digital Nature / xDiversity Results

    & Discussion Benefits of Transparency Participants could see the subtitles 
 while looking at the contents 
 of the exhibition It was easy to communicate 
 in both directions 
 by being able to see the guide’s face 
 and make eye contact Transparency made it possible to see 
 the whole without obstructing the view, 
 and that they did not feel any gap
  38. 38 © R&D Center for Digital Nature / xDiversity Results

    & Discussion Display Position If the display is held near the face, 
 it is easier because 
 there is only one place to watch.” Handheld setup makes us easy
 to change the position We asked participants 
 which position is preferred “
  39. 39 © R&D Center for Digital Nature / xDiversity Results

    & Discussion Display Type and Size Participants mentioned: As a future work, it is necessary 
 to compare See-Through Captions
 with other methods in detail AR glasses was tiring but
 See-Through Captions was easier Display size was small Example of other methods:
 Two-sided tablet
  40. 40 © R&D Center for Digital Nature / xDiversity Results

    & Discussion Challenges Specific to Guided Tours When multiple people participated,
 their voice is NOT displayed See-Through Captions was originally
 developed as a 1:1 communication Possible solution: Participants also wear microphones Participants also hold displays
  41. 41 © R&D Center for Digital Nature / xDiversity Future

    Work How Can DHH People Communicate with Tour Guide? Some DHH people do not 
 tend to speak by their voice The current system assumes that DHH people speak using voice Possible solution: Additional input interface?
  42. 42 © R&D Center for Digital Nature / xDiversity Future

    Work How Can We Communicate with DHH People Who Prefer Sign Language than Text? Some DHH people prefer to read sign language The current system assumes that DHH people read texts Possible solution: Text <-> Sign language Translator?
  43. 43 © R&D Center for Digital Nature / xDiversity 1.

    Implementation of 
 the smaller version of See-Through Captions 2. Case study: a guided tour in a museum 3. Discussion of findings based on the results Summary of Contributions
  44. © R&D Center for Digital Nature / xDiversity Akihisa Shitara

    Ippei Suzuki* Kenta Yamamoto* Ryosuke Hyakuta Ryo Iijima Yoichi Ochiai See-Through Captions in a Museum Guided Tour: Exploring Museum Guided Tour for Deaf and Hard-of-Hearing People
 with Real-Time Captioning on Transparent Display *These authors contributed equally to this research. Research and Development Center for Digital Nature, University of Tsukuba, Japan
  45. 45 © R&D Center for Digital Nature / xDiversity Acknowledgements

    JST CREST Grant Number JPMJCR19F2, including the AIP Challenge Program, Japan. Transparent display provided by User study support and assisted by This work was supported by Bunsuke Kawasaki, Sakiko Tanaka, Chisa Mitsuhashi