Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Opening Mobile Phone to Multimodal Interaction

Opening Mobile Phone to Multimodal Interaction

OpenInterface European Research Project presentation about the Python Multimodal Hub.


Nilo Ney Coutinho Menezes

November 19, 2008


  1. Opening Mobile Phones to Multimodal Interaction - The Multimodal Hub

    Approach Nilo Menezes (Multitel ASBL) Sophia Antipolis, France, November 19, 2008
  2. Agenda • Our case • The OI Kernel • Multimodal

    Applications • The Mobile Barrier • The Multimodal Hub • Multimodal Middleware Protocol • Game example • Open Source • New standard • Conclusions
  3. Our case • In the OpenInterface project, we needed to

    validate Multimodal components in mobile phones • Our existing solution, the OpenInterface Kernel was not able to run or even to be ported to mobile phones • A first intermediary solution was built, but it still need a link to a computer
  4. The OI Kernel • Supports components written in Java, C++,

    Python, Mathlab and .net. • Runs on Linux and Windows • Components integrated as libraries • Pipeline is built to glue all components in a new application OpenInterface Kernel Component A C++ Component B Java Pipeline Component C Python
  5. Multimodal Applications OIDE Visual Editor OI Kernel Runtime Pipeline

  6. The Mobile Barrier • Low resources (CPU and Memory) •

    No support for external libraries (applications are installed in separate slots, only system libraries are shared) • No capability to start effectively external applications • Very difficult integration of components written in different programming languages
  7. The Multimodal Hub Multimodal Hub Component Component Component Multimodal Browser

  8. Multimodal Hub • Central connection point for all components •

    Able to route messages between components based on publish/subscribe mechanism • Uses the Multimodal Middleware Protocol • Can use multiple network protocols • Multidevice Multimodal Hub Compone nt Compone nt Compone nt Multimodal Browser
  9. Multimodal Browser • Visual Interface to the Multimodal Hub •

    Can be seen as a special component • Or even attached to the Hub in the same application • Enable user configuration • Supports run time changes without restarting the component network Multimodal Hub Compone nt Compone nt Compone nt Multimodal Browser
  10. Components • Connect to the Multimodal Hub • Produce events

    • Consume events • Can be written in any language • Use the Multimodal Middleware Protocol • Translate • Compose • Filter • Log and Inspect Multimodal Hub Compone nt Compone nt Compone nt Multimodal Browser
  11. Multimodal Middleware Protocol • Defines a cross platform data serialization

    mechanism • Extensible to support new messages • Events described using XML, but exchanged in binary format for performance reasons • Uses the OI Repository to store new event formats • Enables message routing between components using three fields: UCID, DeviceID, EventID. • Support standard rules for event routing
  12. Multimodal Middleware Protocol • All components connect to the Multimodal

    Hub • Once connected, they announce their list of produced and consumed events. • The same component can consume or produce any number of events • The Hub store these lists and try to automatically connect components based on a simple match algorithm • Sophisticated operations can be made using a configuration file, enabling even direct connections between components • Uses the best protocol for the service, does not re-specify existing protocols.
  13. Multimodal Middleware Protocol • Lightweight • Suitable for Mobile and

    Embedded devices • Cross platform • As simple as possible • Logic is implemented using components • Multidevice • Provides a standard interface between components, allowing component reuse, exchange and experimentation
  14. Game Example Game Produces: Start of Game End of Game

    Player Wins Player Dies Consumes: Navigational Commands
  15. Game Example Game Acceleromet er GPS Gesture Recognizer Multimodal Hub

  16. Game Example Game 3D Accelerometer GPS Gesture Recognizer Game-GPS Translator

  17. Open Source • Python Multimodal Hub: • LGPL License •

    Hosted at SourceForge.net • Implemented in the Python programming language • Ready to run on Windows, Linux and Mac OS X
  18. New Standards • To extend and improve the actual proposition

    • To support new modalities • To provide end users with configurable multimodality • To enable multimodal components and applications
  19. Conclusions • A real standard is made with adoption •

    Open to extend and modify the protocol • A standard can create opportunity to the development of new components • End user configurations will enable new configurations and uses for existing modalities • Adaptation to user needs • User scriptable components can enhance multimodal experience
  20. Thank you!