Upgrade to Pro — share decks privately, control downloads, hide ads and more …

See-Through Captions in a Museum Guided Tour: Exploring Museum Guided Tour for Deaf and Hard-of-Hearing People with Real-Time Captioning on Transparent Display ICCHP-AAATE 2022 #ICCHP_AAATE_22 (Oral presentation by Ippei Suzuki)

See-Through Captions in a Museum Guided Tour: Exploring Museum Guided Tour for Deaf and Hard-of-Hearing People with Real-Time Captioning on Transparent Display ICCHP-AAATE 2022 #ICCHP_AAATE_22 (Oral presentation by Ippei Suzuki)

This slide was presented in Session "Language Accessibility for the Deaf and Hard-Of-Hearing" at the Joint International Conference on Digital Inclusion, Assistive Technology & Accessibility - ICCHP-AAATE 2022.
https://icchp-aaate.org

【Publication】
Suzuki, I., Yamamoto, K., Shitara, A., Hyakuta, R., Iijima, R., Ochiai, Y. (2022). See-Through Captions in a Museum Guided Tour: Exploring Museum Guided Tour for Deaf and Hard-of-Hearing People with Real-Time Captioning on Transparent Display. In: Miesenberger, K., Kouroupetroglou, G., Mavrou, K., Manduchi, R., Covarrubias Rodriguez, M., Penáz, P. (eds) Computers Helping People with Special Needs. ICCHP-AAATE 2022. Lecture Notes in Computer Science, vol 13341. Springer, Cham. https://doi.org/10.1007/978-3-031-08648-9_64

【Project page】
https://digitalnature.slis.tsukuba.ac.jp/2021/02/see-through-captions/

【Presenter】
Ippei Suzuki (鈴木 一平)
Ph.D. Student (Doctoral Program in Informatics)
Graduate School of Comprehensive Human Sciences
University of Tsukuba (JP)
Digital Nature Group (Yoichi Ochiai)
https://1heisuzuki.com

【Abstract】
Access to audible information for deaf and hard-of-hearing (DHH) people is an essential component as we move towards a diverse society. Real-time captioning is a technology with great potential to help the lives of DHH people, and various applications utilizing mobile devices have been developed. These technologies can improve the daily lives of DHH people and can considerably change the value of audio content provided in public facilities such as museums. We developed a real-time captioning system called See-Through Captions that displays subtitles on a transparent display and conducted a demonstration experiment to apply this system to a guided tour in a museum. Eleven DHH people participated in this demonstration experiment, and through questionnaires and interviews, we explored the possibility of utilizing the transparent subtitle system in a guided tour at the museum.

Digital Nature Group

July 13, 2022
Tweet

More Decks by Digital Nature Group

Other Decks in Research

Transcript

  1. © R&D Center for Digital Nature / xDiversity
    See-Through Captions

    in a Museum Guided Tour:
    Research and Development Center for Digital Nature, University of Tsukuba, Japan

    *These authors contributed equally to this research.
    Ippei Suzuki*, Kenta Yamamoto*, Akihisa Shitara, 

    Ryosuke Hyakuta, Ryo Iijima, Yoichi Ochiai
    Exploring Museum Guided Tour
    for Deaf and Hard-of-Hearing People
    with Real-Time Captioning on Transparent Display

    View full-size slide

  2. 2
    © R&D Center for Digital Nature / xDiversity

    View full-size slide

  3. 3
    © R&D Center for Digital Nature / xDiversity
    Background & Introduction

    View full-size slide

  4. 4
    © R&D Center for Digital Nature / xDiversity
    How can we update 

    the accessibility of museum guided tours 

    for deaf and hard-of-hearing people?

    View full-size slide

  5. 5
    © R&D Center for Digital Nature / xDiversity
    Background Accessibility of Museum Guided Tour for DHH People
    Image (Left): Namatame et al. 2020. The Science Communication Tour with a Sign Language Interpreter

    Image (Right): Namatame et al. 2019. Can Exhibit-Explanations in Sign Language Contribute to the Accessibility of Aquariums?
    Approaches to accessibility of audible information
    Sign-language guided tours Auditory information via mobile device

    View full-size slide

  6. 6
    © R&D Center for Digital Nature / xDiversity
    Background Accessibility of Museum Guided Tour for DHH People
    Image (Left): Namatame et al. 2020. The Science Communication Tour with a Sign Language Interpreter

    Image (Right): Namatame et al. 2019. Can Exhibit-Explanations in Sign Language Contribute to the Accessibility of Aquariums?
    One-way information

    Cannot communicate with guide
    Difficult to recruit an interpreter
    Sign-language guided tours Auditory information via mobile device

    View full-size slide

  7. 7
    © R&D Center for Digital Nature / xDiversity
    Background Automatic Speech Recognition (ASR)
    Video: https://www.android.com/accessibility/live-transcribe/
    Automatic Speech Recognition (ASR)

    View full-size slide

  8. 8
    © R&D Center for Digital Nature / xDiversity
    Background Automatic Speech Recognition (ASR)
    ASR on mobile devices ASR on augmented reality devices
    Approaches to utilize automatic speech recognition

    View full-size slide

  9. 9
    © R&D Center for Digital Nature / xDiversity
    Background Automatic Speech Recognition (ASR)
    ASR on mobile devices ASR on augmented reality devices
    Speaker cannot confirm

    whether the speech has been

    correctly recognized.
    The facial expression and

    body language of the partner

    are overlooked

    View full-size slide

  10. 10
    © R&D Center for Digital Nature / xDiversity
    Introduction Our Previous Work
    Kenta Yamamoto, Ippei Suzuki, Akihisa Shitara, Yoichi Ochiai. ASSETS’21.

    See-Through Captions: Real-Time Captioning on Transparent Display for Deaf and Hard-of-Hearing People.

    View full-size slide

  11. 11
    © R&D Center for Digital Nature / xDiversity
    Introduction Transparent Display

    View full-size slide

  12. 12
    © R&D Center for Digital Nature / xDiversity
    Implementation

    View full-size slide

  13. 13
    © R&D Center for Digital Nature / xDiversity
    Implementation Overview
    Microphone
    Backpack
    Transparent Display

    View full-size slide

  14. 14
    © R&D Center for Digital Nature / xDiversity
    Implementation Transparent Display

    View full-size slide

  15. 15
    © R&D Center for Digital Nature / xDiversity
    Implementation Transparent Display
    7 cm
    8 cm
    Resolution 320 × 360 pixels
    Number of Colors 4,096 Colors
    Transmittance 87%
    Weight Approx. 130 g
    Brightness (Center) 270 cd/m²
    Contrast Ratio 20:1
    Transparent Display
    Japan Display Inc.

    View full-size slide

  16. 16
    © R&D Center for Digital Nature / xDiversity Image: https://www.shure.com/en-US/products/microphones/wh20
    Implementation Microphone
    Headset Microphone
    Shure; WH20XLR
    Unidirectional cardioid directivity
    Less surrounding noise

    View full-size slide

  17. 17
    © R&D Center for Digital Nature / xDiversity
    Implementation Backpack
    Display Drive Board
    Battery
    Computer
    Tablet PC
    Mobile Wi-Fi Hotspot
    Weight : approx. 3.3 kg
    *Inserted into backpack
    Audio Interface

    View full-size slide

  18. 18
    © R&D Center for Digital Nature / xDiversity
    Implementation Automatic Speech Recognition API
    Image (Google Chrome Logo): https://www.google.com/chrome/
    Web Speech API

    on Google Chrome
    Javascript API

    View full-size slide

  19. 19
    © R&D Center for Digital Nature / xDiversity
    Implementation Overview
    Audio I/F

    Computer

    Drive Board

    Battery

    Mobile Wi-Fi Hotspot
    Display
    Microphone

    View full-size slide

  20. 20
    © R&D Center for Digital Nature / xDiversity
    Case Study:

    Guided Tour in Museum

    View full-size slide

  21. 21
    © R&D Center for Digital Nature / xDiversity
    Case Study: Guided Tour in Museum
    Image (Right): https://www.miraikan.jst.go.jp/aboutus/

    Image (Science Communicators): https://www.miraikan.jst.go.jp/en/aboutus/communicators/ɹ
    Bunsuke Kawasaki Sakiko Tanaka
    Science Communicators

    View full-size slide

  22. 22
    © R&D Center for Digital Nature / xDiversity
    Case Study: Guided Tour in Museum
    Tour theme: “The difference between humans and robots”

    View full-size slide

  23. 23
    © R&D Center for Digital Nature / xDiversity
    Case Study: Guided Tour in Museum
    Communication Method
    Guide Person
    See-Through Captions
    DHH People
    DHH People
    Speech or Writing
    Guide Person
    Tours were conducted in Japanese language

    View full-size slide

  24. When ASR system stopped…
    Guide express ‘’wait’’

    in gestures of sign language
    24
    © R&D Center for Digital Nature / xDiversity
    Case Study: Guided Tour in Museum
    Communication Protocol

    View full-size slide

  25. 25
    © R&D Center for Digital Nature / xDiversity
    Case Study: Guided Tour in Museum
    When participants wanted to talk
    They raise their hand or notepad
    Communication Protocol
    When someone talked one’s idea
    “Applause” in gestures of sing language

    View full-size slide

  26. 26
    © R&D Center for Digital Nature / xDiversity
    Case Study: Guided Tour in Museum
    Procedure
    1. Participants were asked about the preferred position of display 

    and asked about preferred infection-prevention methods 

    (face shield or face mask)

    2. The guide described the theme of the tour 

    and conducted some quiz games about Miraikan

    3. Guided tour

    4. Participants were asked to fill out the questionnaires 

    and be interviewed

    View full-size slide

  27. 27
    © R&D Center for Digital Nature / xDiversity
    Case Study: Guided Tour in Museum
    Display Position: Basic

    View full-size slide

  28. 28
    © R&D Center for Digital Nature / xDiversity
    Case Study: Guided Tour in Museum
    Display Position: Overlay

    View full-size slide

  29. 29
    © R&D Center for Digital Nature / xDiversity
    Case Study: Guided Tour in Museum
    Display Position: Hands-Free

    View full-size slide

  30. 30
    © R&D Center for Digital Nature / xDiversity
    Case Study: Guided Tour in Museum
    Display Positions

    View full-size slide

  31. 31
    © R&D Center for Digital Nature / xDiversity
    Case Study: Guided Tour in Museum
    Participants
    11 DHH Participants | 18-53 years old

    4 Hearing Participants | 36-56 years old
    Tour Groups
    Each tour group contained at least one DHH person; some groups contained a few hearing people.
    DHH person
    Hearing person
    9 Groups
    + 1 Hearing Participant
    |
    without questionnaires

    View full-size slide

  32. 32
    © R&D Center for Digital Nature / xDiversity
    Results & Discussion

    View full-size slide

  33. 33
    © R&D Center for Digital Nature / xDiversity
    Results Quantitative Evaluation
    Q1.

    Readability of the ASR results
    Q2.

    Noticeability of misrecognition
    Q3.

    Whether they wanted to

    continue utilizing this system
    DHH
    Hearing
    M SD
    4.45 .50
    4.00 .63
    DHH
    Hearing
    4.27 .86
    2.80 .75
    DHH
    Hearing
    4.73 .45
    4.20 .40
    Strongly Disagree Strongly Agree

    View full-size slide

  34. 34
    © R&D Center for Digital Nature / xDiversity
    Results & Discussion Automatic Speech Recognition
    ASR sometimes misrecognize the words
    Possible solution:
    The speaker acquire utterances and speaking styles

    that were easy for the system to recognize correctly
    Dictionary registration for technical terms / nouns
    It was difficult to read when misrecognition occurred

    View full-size slide

  35. 35
    © R&D Center for Digital Nature / xDiversity
    Results & Discussion Readability of Captions
    Difficult to see in some settings 

    especially when there is a strong 

    light in the background
    The readability affected 

    by background and reflection
    Possible solution:
    The guide pays attention to that

    Easily text design changeable system

    View full-size slide

  36. 36
    © R&D Center for Digital Nature / xDiversity
    Results & Discussion How to Display Captions
    The character flow was too fast
    The screen was filled with rephrasing

    when misrecognition occurred
    Subtitle design is 

    for a larger transparent display
    Possible solution:
    Function to look back at the history

    Little larger transparent display

    View full-size slide

  37. 37
    © R&D Center for Digital Nature / xDiversity
    Results & Discussion Benefits of Transparency
    Participants could see the subtitles 

    while looking at the contents 

    of the exhibition
    It was easy to communicate 

    in both directions 

    by being able to see the guide’s face 

    and make eye contact
    Transparency made it possible to see 

    the whole without obstructing the view, 

    and that they did not feel any gap

    View full-size slide

  38. 38
    © R&D Center for Digital Nature / xDiversity
    Results & Discussion Display Position
    If the display is held near the face, 

    it is easier because 

    there is only one place to watch.”
    Handheld setup makes us easy

    to change the position
    We asked participants 

    which position is preferred

    View full-size slide

  39. 39
    © R&D Center for Digital Nature / xDiversity
    Results & Discussion Display Type and Size
    Participants mentioned:
    As a future work, it is necessary 

    to compare See-Through Captions

    with other methods in detail
    AR glasses was tiring but

    See-Through Captions was easier
    Display size was small
    Example of other methods:

    Two-sided tablet

    View full-size slide

  40. 40
    © R&D Center for Digital Nature / xDiversity
    Results & Discussion Challenges Specific to Guided Tours
    When multiple people participated,

    their voice is NOT displayed
    See-Through Captions was originally

    developed as a 1:1 communication
    Possible solution:
    Participants also wear microphones

    Participants also hold displays

    View full-size slide

  41. 41
    © R&D Center for Digital Nature / xDiversity
    Future Work How Can DHH People Communicate with Tour Guide?
    Some DHH people do not 

    tend to speak by their voice
    The current system assumes that

    DHH people speak using voice
    Possible solution:
    Additional input interface?

    View full-size slide

  42. 42
    © R&D Center for Digital Nature / xDiversity
    Future Work How Can We Communicate with DHH People Who Prefer Sign Language than Text?
    Some DHH people

    prefer to read sign language
    The current system assumes that

    DHH people read texts
    Possible solution:
    Text <-> Sign language Translator?

    View full-size slide

  43. 43
    © R&D Center for Digital Nature / xDiversity
    1. Implementation of 

    the smaller version of See-Through Captions

    2. Case study: a guided tour in a museum

    3. Discussion of findings based on the results
    Summary of Contributions

    View full-size slide

  44. © R&D Center for Digital Nature / xDiversity
    Akihisa Shitara
    Ippei Suzuki* Kenta Yamamoto* Ryosuke Hyakuta Ryo Iijima Yoichi Ochiai
    See-Through Captions in a Museum Guided Tour:
    Exploring Museum Guided Tour for Deaf and Hard-of-Hearing People

    with Real-Time Captioning on Transparent Display
    *These authors contributed equally to this research.
    Research and Development Center for Digital Nature, University of Tsukuba, Japan

    View full-size slide

  45. 45
    © R&D Center for Digital Nature / xDiversity
    Acknowledgements
    JST CREST Grant Number JPMJCR19F2, including the AIP Challenge Program, Japan.
    Transparent display provided by User study support and assisted by
    This work was supported by
    Bunsuke Kawasaki, Sakiko Tanaka, Chisa Mitsuhashi

    View full-size slide