Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Opening Mobile Phone to Multimodal Interaction

Opening Mobile Phone to Multimodal Interaction

OpenInterface European Research Project presentation about the Python Multimodal Hub.

40fa6431238df7db2ba245186beb9d64?s=128

Nilo Ney Coutinho Menezes

November 19, 2008
Tweet

Transcript

  1. Opening Mobile Phones to Multimodal Interaction - The Multimodal Hub

    Approach Nilo Menezes (Multitel ASBL) Sophia Antipolis, France, November 19, 2008
  2. Agenda • Our case • The OI Kernel • Multimodal

    Applications • The Mobile Barrier • The Multimodal Hub • Multimodal Middleware Protocol • Game example • Open Source • New standard • Conclusions
  3. Our case • In the OpenInterface project, we needed to

    validate Multimodal components in mobile phones • Our existing solution, the OpenInterface Kernel was not able to run or even to be ported to mobile phones • A first intermediary solution was built, but it still need a link to a computer
  4. The OI Kernel • Supports components written in Java, C++,

    Python, Mathlab and .net. • Runs on Linux and Windows • Components integrated as libraries • Pipeline is built to glue all components in a new application OpenInterface Kernel Component A C++ Component B Java Pipeline Component C Python
  5. Multimodal Applications OIDE Visual Editor OI Kernel Runtime Pipeline

  6. The Mobile Barrier • Low resources (CPU and Memory) •

    No support for external libraries (applications are installed in separate slots, only system libraries are shared) • No capability to start effectively external applications • Very difficult integration of components written in different programming languages
  7. The Multimodal Hub Multimodal Hub Component Component Component Multimodal Browser

  8. Multimodal Hub • Central connection point for all components •

    Able to route messages between components based on publish/subscribe mechanism • Uses the Multimodal Middleware Protocol • Can use multiple network protocols • Multidevice Multimodal Hub Compone nt Compone nt Compone nt Multimodal Browser
  9. Multimodal Browser • Visual Interface to the Multimodal Hub •

    Can be seen as a special component • Or even attached to the Hub in the same application • Enable user configuration • Supports run time changes without restarting the component network Multimodal Hub Compone nt Compone nt Compone nt Multimodal Browser
  10. Components • Connect to the Multimodal Hub • Produce events

    • Consume events • Can be written in any language • Use the Multimodal Middleware Protocol • Translate • Compose • Filter • Log and Inspect Multimodal Hub Compone nt Compone nt Compone nt Multimodal Browser
  11. Multimodal Middleware Protocol • Defines a cross platform data serialization

    mechanism • Extensible to support new messages • Events described using XML, but exchanged in binary format for performance reasons • Uses the OI Repository to store new event formats • Enables message routing between components using three fields: UCID, DeviceID, EventID. • Support standard rules for event routing
  12. Multimodal Middleware Protocol • All components connect to the Multimodal

    Hub • Once connected, they announce their list of produced and consumed events. • The same component can consume or produce any number of events • The Hub store these lists and try to automatically connect components based on a simple match algorithm • Sophisticated operations can be made using a configuration file, enabling even direct connections between components • Uses the best protocol for the service, does not re-specify existing protocols.
  13. Multimodal Middleware Protocol • Lightweight • Suitable for Mobile and

    Embedded devices • Cross platform • As simple as possible • Logic is implemented using components • Multidevice • Provides a standard interface between components, allowing component reuse, exchange and experimentation
  14. Game Example Game Produces: Start of Game End of Game

    Player Wins Player Dies Consumes: Navigational Commands
  15. Game Example Game Acceleromet er GPS Gesture Recognizer Multimodal Hub

  16. Game Example Game 3D Accelerometer GPS Gesture Recognizer Game-GPS Translator

  17. Open Source • Python Multimodal Hub: • LGPL License •

    Hosted at SourceForge.net • Implemented in the Python programming language • Ready to run on Windows, Linux and Mac OS X
  18. New Standards • To extend and improve the actual proposition

    • To support new modalities • To provide end users with configurable multimodality • To enable multimodal components and applications
  19. Conclusions • A real standard is made with adoption •

    Open to extend and modify the protocol • A standard can create opportunity to the development of new components • End user configurations will enable new configurations and uses for existing modalities • Adaptation to user needs • User scriptable components can enhance multimodal experience
  20. Thank you!