Upgrade to Pro — share decks privately, control downloads, hide ads and more …

AI filter assistance in video call systems

AI filter assistance in video call systems


LINE DevDay 2020

November 25, 2020


  1. None
  2. Increasing demand on video communication › Increasing use of video

    chats for New Normal lifestyles › Diversified purposes of use beyond online meetings › Online-Nomikai (drinking parties) › Online-Kisei (homecoming) › Online-Konkatsu (partner matching)
  3. Players in online video communication media Private Public Casual Formal

    Instagram TikTok LINE LIVE Webinars Zoom Microsoft Teams Skype Youtube Live
  4. The hole: casual chats for private groups Private Public Casual

    Formal Instagram TikTok LINE LIVE Webinars Microsoft Teams Skype Youtube Live Project masquerade Zoom
  5. AR (augmented reality) filters in video chats › Beautifying selfies

    › Just for fun › Instagram, TikTok, etc… › As ice breakers or conversation cues › As effects in collaborative games A strong potential to be a critical visual element for future casual video chats.
  6. Broken flows in interaction with AR filters › Tap an

    icon to launch the AR filters menus › Scroll the menu until you find the filter you like › Apply the filter and re-do the process if the filter is not really what you want. This interaction can break ongoing conversations because user’s attention needs to be directed to filter selection.
  7. Project Masquerade: allowing users to engage in AR filters through

    gestures and speech › Users perform physical or on-screen gestures along with speech command. › The system immediately chooses an AR filter that the users want, and automatically applies to the screen. › This is a collaboration project with Prof. Koji Yatani at University of Tokyo, and IIS Lab, supported by Center for Robust Intelligence and Social Technology (CRIS).
  8. Project Masquerade early prototype demo

  9. Our interaction design: gestures set where to apply, speech sets

    what to apply. › Physical gestures define the scope where an AR filter is going to be applied. › Hand pose estimation › On-screen gestures for applying an AR filter to the chat partner. (currently under development) › Speech identifies what AR filter the user wants to apply. › Speech Recognition Cat Cat
  10. AR filter categories Person-centric Background-centric Others

  11. What’s next? › Further interface design explorations for more complex

    AR filters › Improvements on filter selection algorithms › User experience validation through a deployment study
  12. Thank you! Project Masquerade Re-shaping casual video chats for private

    groups by offering a quick and easy access to AR filters Contact: Michihiko Ueno Koji Yatani (koji@iis-lab.org)