Midas: A Declarative Multi-Touch Interaction Framework

Presentation given at TEI 2011, 5th International Conference on Tangible, Embedded and Embodied Interaction, Madeira, Portugal.

ABSTRACT: Over the past few years, multi-touch user interfaces emerged from research prototypes into mass market products. This evolution has been mainly driven by innovative devices such as Apple’s iPhone or Microsoft’s Surface tabletop computer. Unfortunately, there seems to be a lack of software engineering abstractions in existing multi-touch development frameworks. Many multi-touch applications are based on hardcoded procedural low level event processing. This leads to proprietary solutions with a lack of gesture extensibility and cross-application reusability. We present Midas, a declarative model for the definition and detection of multi-touch gestures where gestures are expressed via logical rules over a set of input facts. We highlight how our rule-based language approach leads to improvements in gesture extensibility and reusability. Last but not least, we introduce JMidas, an instantiation of Midas for the Java programming language and describe how JMidas has been applied to implement a number of innovative multi-touch gestures.

Research paper: https://beatsigner.com/publications/midas-a-declarative-multi-touch-interaction-framework.pdf

Transcript

1. A
   Declarative
   Multi-Touch
   Interaction
   Framework
   CHRISTOPHE SCHOLLIERS
   LODE HOSTE
   BEAT SIGNER
   WOLFGANG DE MEUTER
   VRIJE UNIVERSITEIT BRUSSEL
   #MidasFW
   MIDAS
   

   View Slide

2. MULTI-TOUCH IS EVERYWHERE!
   

   View Slide

3. HARDWARE CAPABILITIES VS
   GESTURE RECOGNITION SOFTWARE
   Flick
   Drag
   Rotate
   Pinch Stretch
   

   View Slide

4. WHAT’S THE FUNDAMENTAL PROBLEM?
   update 0 0,565000 0,335417
   update 1 0,387500 0,450000
   update 0 0,560000 0,345833
   update 1 0,390000 0,462500
   update 0 0,558750 0,352083
   update 1 0,391250 0,479167
   update 0 0,556250 0,358333
   update 1 0,391250 0,489583
   update 0 0,553750 0,366667
   update 1 0,392500 0,502083
   update 0 0,550000 0,377083
   update 1 0,392500 0,516667
   update 0 0,548750 0,385417
   update 1 0,392500 0,531250
   update 0 0,546250 0,395833
   update 1 0,390000 0,543750
   update 0 0,543750 0,408333
   update 1 0,388750 0,556250
   update 0 0,542500 0,422917
   update 1 0,386250 0,566667
   update 0 0,543750 0,429167
   update 1 0,385000 0,577083
   update 0 0,543750 0,441667
   update 1 0,378750 0,595833
   update 0 0,543750 0,450000
   update 1 0,367500 0,633333
   update 0 0,547500 0,475000
   update 1 0,348750 0,660417
   update 0 0,551250 0,504167
   update 1 0,341250 0,666667
   update 0 0,553750 0,514583
   update 1 0,336250 0,675000
   update 0 0,557500 0,527083
   update 1 0,331250 0,679167
   update 0 0,561250 0,541667
   update 1 0,326250 0,683333
   update 0 0,565000 0,552083
   update 1 0,321250 0,685417
   update 0 0,570000 0,572917
   update 1 0,316250 0,685417
   update 0 0,575000 0,583333
   update 1 0,311250 0,685417
   update 0 0,581250 0,595833
   update 1 0,307500 0,683333
   update 0 0,590000 0,612500
   update 1 0,302500 0,681250
   update 0 0,595000 0,618750
   update 1 0,296250 0,679167
   update 0 0,606250 0,633333
   update 1 0,290000 0,675000
   update 0 0,612500 0,639583
   update 1 0,281250 0,664583
   update 0 0,622500 0,654167
   update 1 0,272500 0,656250
   update 0 0,631250 0,660417
   update 1 0,268750 0,647917
   update 0 0,636250 0,662500
   update 1 0,262500 0,639583
   update 0 0,643750 0,664583
   update 1 0,257500 0,629167
   update 0 0,647500 0,664583
   update 1 0,255000 0,627083
   update 0 0,653750 0,666667
   update 1 0,166250 0,343750
   update 0 0,728750 0,306250
   update 1 0,172500 0,331250
   update 0 0,725000 0,300000
   update 1 0,177500 0,318750
   update 0 0,723750 0,295833
   update 1 0,181250 0,314583
   update 0 0,720000 0,291667
   update 1 0,185000 0,310417
   update 0 0,718750 0,289583
   update 1 0,186250 0,304167
   update 0 0,716250 0,287500
   update 1 0,190000 0,302083
   update 0 0,713750 0,281250
   update 1 0,192500 0,300000
   update 0 0,708750 0,277083
   update 1 0,198750 0,293750
   update 0 0,702500 0,270833
   update 1 0,205000 0,289583
   update 0 0,692500 0,266667
   update 1 0,208750 0,289583
   update 0 0,688750 0,264583
   update 1 0,212500 0,287500
   update 0 0,685000 0,264583
   update 1 0,216250 0,285417
   update 0 0,678750 0,256250
   update 1 0,221250 0,283333
   update 0 0,675000 0,252083
   update 1 0,227500 0,283333
   update 0 0,671250 0,250000
   update 1 0,231250 0,281250
   update 0 0,667500 0,250000
   update 1 0,233750 0,281250
   update 0 0,662500 0,247917
   update 1 0,238750 0,279167
   update 0 0,658750 0,247917
   update 1 0,245000 0,279167
   update 0 0,655000 0,245833
   update 1 0,248750 0,277083
   update 0 0,651250 0,245833
   update 1 0,253750 0,277083
   update 0 0,646250 0,250000
   update 1 0,258750 0,275000
   update 0 0,642500 0,250000
   update 1 0,263750 0,272917
   update 0 0,638750 0,254167
   update 1 0,267500 0,270833
   update 0 0,635000 0,256250
   update 1 0,271250 0,270833
   update 0 0,631250 0,258333
   update 1 0,278750 0,270833
   update 0 0,625000 0,262500
   update 1 0,282500 0,270833
   update 0 0,622500 0,262500
   update 1 0,286250 0,270833
   update 0 0,621250 0,264583
   update 1 0,288750 0,268750
   update 0 0,620000 0,266667
   update 1 0,291250 0,268750
   update 0 0,618750 0,268750
   update 1 0,292500 0,268750
   update 0 0,617500 0,268750
   update 1 0,293750 0,268750
   update 1 0,251250 0,620833
   update 0 0,656250 0,666667
   update 1 0,242500 0,604167
   update 0 0,663750 0,662500
   update 1 0,237500 0,591667
   update 0 0,667500 0,658333
   update 1 0,232500 0,583333
   update 0 0,672500 0,656250
   update 1 0,215000 0,558333
   update 0 0,691250 0,637500
   update 1 0,206250 0,543750
   update 0 0,702500 0,612500
   update 1 0,187500 0,510417
   update 0 0,711250 0,589583
   update 1 0,167500 0,485417
   update 0 0,716250 0,568750
   update 1 0,161250 0,477083
   update 0 0,720000 0,556250
   update 1 0,152500 0,466667
   update 0 0,725000 0,539583
   update 1 0,146250 0,460417
   update 0 0,727500 0,527083
   update 1 0,143750 0,456250
   update 0 0,728750 0,510417
   update 1 0,138750 0,447917
   update 0 0,731250 0,502083
   update 1 0,136250 0,445833
   update 0 0,733750 0,485417
   update 1 0,133750 0,441667
   update 0 0,736250 0,472917
   update 1 0,132500 0,439583
   update 0 0,740000 0,458333
   update 1 0,131250 0,435417
   update 0 0,741250 0,447917
   update 0 0,741250 0,439583
   update 1 0,130000 0,431250
   update 0 0,742500 0,422917
   update 1 0,130000 0,429167
   update 0 0,742500 0,416667
   update 1 0,130000 0,427083
   update 0 0,743750 0,402083
   update 1 0,130000 0,425000
   update 0 0,745000 0,395833
   update 1 0,131250 0,422917
   update 0 0,745000 0,387500
   update 1 0,132500 0,418750
   update 0 0,745000 0,383333
   update 1 0,133750 0,416667
   update 0 0,743750 0,377083
   update 1 0,135000 0,412500
   update 0 0,743750 0,360417
   update 1 0,136250 0,408333
   update 0 0,741250 0,354167
   update 1 0,141250 0,400000
   update 0 0,738750 0,337500
   update 1 0,145000 0,389583
   update 0 0,738750 0,333333
   update 1 0,147500 0,383333
   update 0 0,735000 0,325000
   update 1 0,151250 0,372917
   update 0 0,731250 0,316667
   update 1 0,157500 0,360417
   update 0 0,728750 0,308333
   add 0 0,633750 0,293750
   update 0 0,635000 0,293750
   update 0 0,633750 0,293750
   update 0 0,633750 0,295833
   update 0 0,633750 0,297917
   update 0 0,632500 0,300000
   update 0 0,631250 0,300000
   update 0 0,630000 0,300000
   update 0 0,628750 0,300000
   update 0 0,627500 0,300000
   add 1 0,246250 0,289583
   update 1 0,247500 0,289583
   update 1 0,248750 0,291667
   update 1 0,250000 0,291667
   update 1 0,252500 0,291667
   update 1 0,256250 0,291667
   update 1 0,258750 0,291667
   update 1 0,263750 0,291667
   update 0 0,626250 0,300000
   update 1 0,276250 0,291667
   update 0 0,625000 0,300000
   update 1 0,285000 0,295833
   update 0 0,622500 0,300000
   update 1 0,293750 0,300000
   update 0 0,621250 0,302083
   update 1 0,300000 0,304167
   update 1 0,310000 0,310417
   update 0 0,618750 0,302083
   update 1 0,318750 0,316667
   update 0 0,616250 0,302083
   update 1 0,325000 0,320833
   update 0 0,613750 0,302083
   update 1 0,330000 0,322917
   update 0 0,611250 0,302083
   update 1 0,335000 0,325000
   update 0 0,610000 0,302083
   update 1 0,338750 0,329167
   update 0 0,607500 0,304167
   update 1 0,341250 0,333333
   update 0 0,605000 0,304167
   update 1 0,345000 0,335417
   update 0 0,602500 0,304167
   update 1 0,347500 0,337500
   update 0 0,600000 0,304167
   update 1 0,351250 0,341667
   update 0 0,596250 0,304167
   update 1 0,355000 0,347917
   update 0 0,593750 0,306250
   update 1 0,358750 0,352083
   update 0 0,590000 0,308333
   update 1 0,361250 0,358333
   update 0 0,586250 0,312500
   update 1 0,365000 0,368750
   update 0 0,583750 0,314583
   update 1 0,368750 0,379167
   update 0 0,580000 0,316667
   update 1 0,372500 0,389583
   update 0 0,577500 0,318750
   update 1 0,378750 0,410417
   update 0 0,571250 0,329167
   update 1 0,382500 0,420833
   update 0 0,568750 0,333333
   update 1 0,383750 0,435417
   

   View Slide

5.  Sequential code for the
   processing of concurrent events
    Find spatial and temporal
   relations in an immense pool
   of x/y coordinates
   ▪ ~25 events / second / finger
    Approximative detection
    GUI-Event correlation
    Modularisation and composition
   ▪ implementing gestures is hard
   ▪ combining gestures is even harder
   WHAT’S THE FUNDAMENTAL PROBLEM?
   

   View Slide

6. THE MIDAS APPROACH
   SELECT * FROM Audience
   WHERE interest > 0
   Facts
   Fact Base Rules
   

   View Slide

7. Midas
   Application Layer
   Core Layer
   Infrastructure Layer
   GUI
   Hardware
   Bridge
   * Translator
   Fact
   Base
   Inference
   Engine
   Rule
   Base
   Shadows Model
   MIDAS ARCHITECTURE
    Infrastructure layer
   ▪ Transform low level
   data into facts
    Core layer
   ▪ temporal and spatial
   operators
   ▪ continuous queries via
   Rete algorithm
    Application layer
   ▪ regular program
   augmented with rules
   

   View Slide

8. MIDAS USES
   (defrule PrintCursor
   ?c <- (Cursor)
   =>
   (printout t ‘‘A cursor is moving at: ’’ ?c.x ‘‘,’’ ?c.y))
   

   View Slide

9. MIDAS FILTERS
   +
   (defrule Tap
   ?appear <- (Cursor (finger ?f) (state ?*APPEAR*))
   ?disappear <- (Cursor (finger ?f) (state ?*DISAPPEAR*))
   (test (tShortlyAfter ?disappear ?appear))
   =>
   (printout t “A single tap detected at: “ ?appear.x “,” ?appear.y))
   only a few lines of declarative code
   

   View Slide

10. MIDAS FINDS
    +
    (defrule Tap
    ?appear <- (Cursor (finger ?f) (state ?*APPEAR*))
    ?disappear <- (Cursor (finger ?f) (state ?*DISAPPEAR*))
    (test (tShortlyAfter ?disappear ?appear))
    (test (sVeryNear ?disappear ?appear))
    =>
    (printout t “A single tap detected at: “ ?appear.x “,” ?appear.y)
    (assert (Tap (x ?appear.x) (y ?appear.y) (on ?appear.on))
    (retract ?appear ?disappear))
    

    View Slide

11. MIDAS ENCOURAGES
    (defrule DoubleTap
    ?tap1 <- (Tap)
    ?tap2 <- (Tap)
    (test (tShortlyAfter ?tap2 ?tap1))
    (test (sVeryNear ?tap2 ?tap1))
    =>
    (printout t “A double tap detected at: “ ?tap1.x“,” ?tap2.y)
    (retract ?tap1 ?tap2))
    small and reusable building blocks
    

    View Slide

12. MIDAS ENCOURAGES
    (defrule FixedUpUp
    ?fixed <- (Fixed)
    ?up1 <- (Up)
    ?up2 <- (Up)
    (test (tEqual3 ?fixed ?up1 ?up2))
    (test (sLeftOf ?fixed ?up1))
    (test (sLeftOf ?up1 ?up2))
    =>
    (printout t “Complex scroll detected!”)
    (retract ?fixed ?up1 ?up2))
    

    View Slide

13. update 0 0,565000 0,335417
    update 1 0,387500 0,450000
    update 0 0,560000 0,345833
    update 1 0,390000 0,462500
    update 0 0,558750 0,352083
    update 1 0,391250 0,479167
    update 0 0,556250 0,358333
    update 1 0,391250 0,489583
    update 0 0,553750 0,366667
    update 1 0,392500 0,502083
    update 0 0,550000 0,377083
    update 1 0,392500 0,516667
    update 0 0,548750 0,385417
    update 1 0,392500 0,531250
    update 0 0,546250 0,395833
    update 1 0,390000 0,543750
    MIDAS CORRELATES
    (defrule Gui
    ?tap<- (Tap)
    ?planet <- (Planet)
    (test (sInside ?tap ?planet))
    =>
    (printout t “Tap inside planet!”))
    add 0 0,633750 0,293750
    update 0 0,635000 0,293750
    update 0 0,633750 0,293750
    update 0 0,633750 0,295833
    update 0 0,633750 0,297917
    update 0 0,632500 0,300000
    update 0 0,631250 0,300000
    update 0 0,630000 0,300000
    update 0 0,628750 0,300000
    update 0 0,627500 0,300000
    add 1 0,246250 0,289583
    update 1 0,247500 0,289583
    update 1 0,248750 0,291667
    update 1 0,250000 0,291667
    update 1 0,252500 0,291667
    update 1 0,256250 0,291667
    Fact Base
    Shadows
    (GUI elements)
    

    View Slide

14. MIDAS SUPPORTS
    (defrule nextSlide
    ?goRight1 <- (FlickRight (tuioId ?*RH*))
    ?goLeft <- (FlickLeft (tuioId ?*LH*))
    ?goRight2 <- (FlickRight (tuioId ?*RH*))
    (test (tMeets ?goRight1 ?goLeft ?goRight2))
    =>
    (call PowerPoint showNextSlide)
    (retract ?goRight1 ?goLeft ?goRight2))
    Implemented online gestures
    Templates
    Kinect example
    

    View Slide

15. CONCLUSION
     Midas multi-touch interaction framework
    ▪ Declarative description of gestures (fact base and rule engine)
    ▪ extensibility and reusability of gestures
    ▪ correlation of GUI elements and events via shadow facts
    ▪ JMidas Java integration
     Ongoing work
    ▪ integration of new devices: Kinect, Digital Pen & Paper, ...
    ▪ extend Midas to support multimodal gestures
     Rapid prototyping of multi-touch gestures
    Research: http://goo.gl/OFpej - Youtube: http://goo.gl/MNDkT - Twitter: #MidasFW
    Midas Demo, Tuesday @ 15.00-18.30 -- ID: 251
    

    View Slide