Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Midas: A Declarative Multi-Touch Interaction Framework

Beat Signer
January 24, 2011

Midas: A Declarative Multi-Touch Interaction Framework

Presentation given at TEI 2011, 5th International Conference on Tangible, Embedded and Embodied Interaction, Madeira, Portugal.

ABSTRACT: Over the past few years, multi-touch user interfaces emerged from research prototypes into mass market products. This evolution has been mainly driven by innovative devices such as Apple’s iPhone or Microsoft’s Surface tabletop computer. Unfortunately, there seems to be a lack of software engineering abstractions in existing multi-touch development frameworks. Many multi-touch applications are based on hardcoded procedural low level event processing. This leads to proprietary solutions with a lack of gesture extensibility and cross-application reusability. We present Midas, a declarative model for the definition and detection of multi-touch gestures where gestures are expressed via logical rules over a set of input facts. We highlight how our rule-based language approach leads to improvements in gesture extensibility and reusability. Last but not least, we introduce JMidas, an instantiation of Midas for the Java programming language and describe how JMidas has been applied to implement a number of innovative multi-touch gestures.

Research paper: https://beatsigner.com/publications/midas-a-declarative-multi-touch-interaction-framework.pdf

Beat Signer

January 24, 2011
Tweet

More Decks by Beat Signer

Other Decks in Science

Transcript

  1. A
    Declarative
    Multi-Touch
    Interaction
    Framework
    CHRISTOPHE SCHOLLIERS
    LODE HOSTE
    BEAT SIGNER
    WOLFGANG DE MEUTER
    VRIJE UNIVERSITEIT BRUSSEL
    #MidasFW
    MIDAS

    View full-size slide

  2. MULTI-TOUCH IS EVERYWHERE!

    View full-size slide

  3. HARDWARE CAPABILITIES VS
    GESTURE RECOGNITION SOFTWARE
    Flick
    Drag
    Rotate
    Pinch Stretch

    View full-size slide

  4. WHAT’S THE FUNDAMENTAL PROBLEM?
    update 0 0,565000 0,335417
    update 1 0,387500 0,450000
    update 0 0,560000 0,345833
    update 1 0,390000 0,462500
    update 0 0,558750 0,352083
    update 1 0,391250 0,479167
    update 0 0,556250 0,358333
    update 1 0,391250 0,489583
    update 0 0,553750 0,366667
    update 1 0,392500 0,502083
    update 0 0,550000 0,377083
    update 1 0,392500 0,516667
    update 0 0,548750 0,385417
    update 1 0,392500 0,531250
    update 0 0,546250 0,395833
    update 1 0,390000 0,543750
    update 0 0,543750 0,408333
    update 1 0,388750 0,556250
    update 0 0,542500 0,422917
    update 1 0,386250 0,566667
    update 0 0,543750 0,429167
    update 1 0,385000 0,577083
    update 0 0,543750 0,441667
    update 1 0,378750 0,595833
    update 0 0,543750 0,450000
    update 1 0,367500 0,633333
    update 0 0,547500 0,475000
    update 1 0,348750 0,660417
    update 0 0,551250 0,504167
    update 1 0,341250 0,666667
    update 0 0,553750 0,514583
    update 1 0,336250 0,675000
    update 0 0,557500 0,527083
    update 1 0,331250 0,679167
    update 0 0,561250 0,541667
    update 1 0,326250 0,683333
    update 0 0,565000 0,552083
    update 1 0,321250 0,685417
    update 0 0,570000 0,572917
    update 1 0,316250 0,685417
    update 0 0,575000 0,583333
    update 1 0,311250 0,685417
    update 0 0,581250 0,595833
    update 1 0,307500 0,683333
    update 0 0,590000 0,612500
    update 1 0,302500 0,681250
    update 0 0,595000 0,618750
    update 1 0,296250 0,679167
    update 0 0,606250 0,633333
    update 1 0,290000 0,675000
    update 0 0,612500 0,639583
    update 1 0,281250 0,664583
    update 0 0,622500 0,654167
    update 1 0,272500 0,656250
    update 0 0,631250 0,660417
    update 1 0,268750 0,647917
    update 0 0,636250 0,662500
    update 1 0,262500 0,639583
    update 0 0,643750 0,664583
    update 1 0,257500 0,629167
    update 0 0,647500 0,664583
    update 1 0,255000 0,627083
    update 0 0,653750 0,666667
    update 1 0,166250 0,343750
    update 0 0,728750 0,306250
    update 1 0,172500 0,331250
    update 0 0,725000 0,300000
    update 1 0,177500 0,318750
    update 0 0,723750 0,295833
    update 1 0,181250 0,314583
    update 0 0,720000 0,291667
    update 1 0,185000 0,310417
    update 0 0,718750 0,289583
    update 1 0,186250 0,304167
    update 0 0,716250 0,287500
    update 1 0,190000 0,302083
    update 0 0,713750 0,281250
    update 1 0,192500 0,300000
    update 0 0,708750 0,277083
    update 1 0,198750 0,293750
    update 0 0,702500 0,270833
    update 1 0,205000 0,289583
    update 0 0,692500 0,266667
    update 1 0,208750 0,289583
    update 0 0,688750 0,264583
    update 1 0,212500 0,287500
    update 0 0,685000 0,264583
    update 1 0,216250 0,285417
    update 0 0,678750 0,256250
    update 1 0,221250 0,283333
    update 0 0,675000 0,252083
    update 1 0,227500 0,283333
    update 0 0,671250 0,250000
    update 1 0,231250 0,281250
    update 0 0,667500 0,250000
    update 1 0,233750 0,281250
    update 0 0,662500 0,247917
    update 1 0,238750 0,279167
    update 0 0,658750 0,247917
    update 1 0,245000 0,279167
    update 0 0,655000 0,245833
    update 1 0,248750 0,277083
    update 0 0,651250 0,245833
    update 1 0,253750 0,277083
    update 0 0,646250 0,250000
    update 1 0,258750 0,275000
    update 0 0,642500 0,250000
    update 1 0,263750 0,272917
    update 0 0,638750 0,254167
    update 1 0,267500 0,270833
    update 0 0,635000 0,256250
    update 1 0,271250 0,270833
    update 0 0,631250 0,258333
    update 1 0,278750 0,270833
    update 0 0,625000 0,262500
    update 1 0,282500 0,270833
    update 0 0,622500 0,262500
    update 1 0,286250 0,270833
    update 0 0,621250 0,264583
    update 1 0,288750 0,268750
    update 0 0,620000 0,266667
    update 1 0,291250 0,268750
    update 0 0,618750 0,268750
    update 1 0,292500 0,268750
    update 0 0,617500 0,268750
    update 1 0,293750 0,268750
    update 1 0,251250 0,620833
    update 0 0,656250 0,666667
    update 1 0,242500 0,604167
    update 0 0,663750 0,662500
    update 1 0,237500 0,591667
    update 0 0,667500 0,658333
    update 1 0,232500 0,583333
    update 0 0,672500 0,656250
    update 1 0,215000 0,558333
    update 0 0,691250 0,637500
    update 1 0,206250 0,543750
    update 0 0,702500 0,612500
    update 1 0,187500 0,510417
    update 0 0,711250 0,589583
    update 1 0,167500 0,485417
    update 0 0,716250 0,568750
    update 1 0,161250 0,477083
    update 0 0,720000 0,556250
    update 1 0,152500 0,466667
    update 0 0,725000 0,539583
    update 1 0,146250 0,460417
    update 0 0,727500 0,527083
    update 1 0,143750 0,456250
    update 0 0,728750 0,510417
    update 1 0,138750 0,447917
    update 0 0,731250 0,502083
    update 1 0,136250 0,445833
    update 0 0,733750 0,485417
    update 1 0,133750 0,441667
    update 0 0,736250 0,472917
    update 1 0,132500 0,439583
    update 0 0,740000 0,458333
    update 1 0,131250 0,435417
    update 0 0,741250 0,447917
    update 0 0,741250 0,439583
    update 1 0,130000 0,431250
    update 0 0,742500 0,422917
    update 1 0,130000 0,429167
    update 0 0,742500 0,416667
    update 1 0,130000 0,427083
    update 0 0,743750 0,402083
    update 1 0,130000 0,425000
    update 0 0,745000 0,395833
    update 1 0,131250 0,422917
    update 0 0,745000 0,387500
    update 1 0,132500 0,418750
    update 0 0,745000 0,383333
    update 1 0,133750 0,416667
    update 0 0,743750 0,377083
    update 1 0,135000 0,412500
    update 0 0,743750 0,360417
    update 1 0,136250 0,408333
    update 0 0,741250 0,354167
    update 1 0,141250 0,400000
    update 0 0,738750 0,337500
    update 1 0,145000 0,389583
    update 0 0,738750 0,333333
    update 1 0,147500 0,383333
    update 0 0,735000 0,325000
    update 1 0,151250 0,372917
    update 0 0,731250 0,316667
    update 1 0,157500 0,360417
    update 0 0,728750 0,308333
    add 0 0,633750 0,293750
    update 0 0,635000 0,293750
    update 0 0,633750 0,293750
    update 0 0,633750 0,295833
    update 0 0,633750 0,297917
    update 0 0,632500 0,300000
    update 0 0,631250 0,300000
    update 0 0,630000 0,300000
    update 0 0,628750 0,300000
    update 0 0,627500 0,300000
    add 1 0,246250 0,289583
    update 1 0,247500 0,289583
    update 1 0,248750 0,291667
    update 1 0,250000 0,291667
    update 1 0,252500 0,291667
    update 1 0,256250 0,291667
    update 1 0,258750 0,291667
    update 1 0,263750 0,291667
    update 0 0,626250 0,300000
    update 1 0,276250 0,291667
    update 0 0,625000 0,300000
    update 1 0,285000 0,295833
    update 0 0,622500 0,300000
    update 1 0,293750 0,300000
    update 0 0,621250 0,302083
    update 1 0,300000 0,304167
    update 1 0,310000 0,310417
    update 0 0,618750 0,302083
    update 1 0,318750 0,316667
    update 0 0,616250 0,302083
    update 1 0,325000 0,320833
    update 0 0,613750 0,302083
    update 1 0,330000 0,322917
    update 0 0,611250 0,302083
    update 1 0,335000 0,325000
    update 0 0,610000 0,302083
    update 1 0,338750 0,329167
    update 0 0,607500 0,304167
    update 1 0,341250 0,333333
    update 0 0,605000 0,304167
    update 1 0,345000 0,335417
    update 0 0,602500 0,304167
    update 1 0,347500 0,337500
    update 0 0,600000 0,304167
    update 1 0,351250 0,341667
    update 0 0,596250 0,304167
    update 1 0,355000 0,347917
    update 0 0,593750 0,306250
    update 1 0,358750 0,352083
    update 0 0,590000 0,308333
    update 1 0,361250 0,358333
    update 0 0,586250 0,312500
    update 1 0,365000 0,368750
    update 0 0,583750 0,314583
    update 1 0,368750 0,379167
    update 0 0,580000 0,316667
    update 1 0,372500 0,389583
    update 0 0,577500 0,318750
    update 1 0,378750 0,410417
    update 0 0,571250 0,329167
    update 1 0,382500 0,420833
    update 0 0,568750 0,333333
    update 1 0,383750 0,435417

    View full-size slide

  5.  Sequential code for the
    processing of concurrent events
     Find spatial and temporal
    relations in an immense pool
    of x/y coordinates
    ▪ ~25 events / second / finger
     Approximative detection
     GUI-Event correlation
     Modularisation and composition
    ▪ implementing gestures is hard
    ▪ combining gestures is even harder
    WHAT’S THE FUNDAMENTAL PROBLEM?

    View full-size slide

  6. THE MIDAS APPROACH
    SELECT * FROM Audience
    WHERE interest > 0
    Facts
    Fact Base Rules

    View full-size slide

  7. Midas
    Application Layer
    Core Layer
    Infrastructure Layer
    GUI
    Hardware
    Bridge
    * Translator
    Fact
    Base
    Inference
    Engine
    Rule
    Base
    Shadows Model
    MIDAS ARCHITECTURE
     Infrastructure layer
    ▪ Transform low level
    data into facts
     Core layer
    ▪ temporal and spatial
    operators
    ▪ continuous queries via
    Rete algorithm
     Application layer
    ▪ regular program
    augmented with rules

    View full-size slide

  8. MIDAS USES
    (defrule PrintCursor
    ?c <- (Cursor)
    =>
    (printout t ‘‘A cursor is moving at: ’’ ?c.x ‘‘,’’ ?c.y))

    View full-size slide

  9. MIDAS FILTERS
    +
    (defrule Tap
    ?appear <- (Cursor (finger ?f) (state ?*APPEAR*))
    ?disappear <- (Cursor (finger ?f) (state ?*DISAPPEAR*))
    (test (tShortlyAfter ?disappear ?appear))
    =>
    (printout t “A single tap detected at: “ ?appear.x “,” ?appear.y))
    only a few lines of declarative code

    View full-size slide

  10. MIDAS FINDS
    +
    (defrule Tap
    ?appear <- (Cursor (finger ?f) (state ?*APPEAR*))
    ?disappear <- (Cursor (finger ?f) (state ?*DISAPPEAR*))
    (test (tShortlyAfter ?disappear ?appear))
    (test (sVeryNear ?disappear ?appear))
    =>
    (printout t “A single tap detected at: “ ?appear.x “,” ?appear.y)
    (assert (Tap (x ?appear.x) (y ?appear.y) (on ?appear.on))
    (retract ?appear ?disappear))

    View full-size slide

  11. MIDAS ENCOURAGES
    (defrule DoubleTap
    ?tap1 <- (Tap)
    ?tap2 <- (Tap)
    (test (tShortlyAfter ?tap2 ?tap1))
    (test (sVeryNear ?tap2 ?tap1))
    =>
    (printout t “A double tap detected at: “ ?tap1.x“,” ?tap2.y)
    (retract ?tap1 ?tap2))
    small and reusable building blocks

    View full-size slide

  12. MIDAS ENCOURAGES
    (defrule FixedUpUp
    ?fixed <- (Fixed)
    ?up1 <- (Up)
    ?up2 <- (Up)
    (test (tEqual3 ?fixed ?up1 ?up2))
    (test (sLeftOf ?fixed ?up1))
    (test (sLeftOf ?up1 ?up2))
    =>
    (printout t “Complex scroll detected!”)
    (retract ?fixed ?up1 ?up2))

    View full-size slide

  13. update 0 0,565000 0,335417
    update 1 0,387500 0,450000
    update 0 0,560000 0,345833
    update 1 0,390000 0,462500
    update 0 0,558750 0,352083
    update 1 0,391250 0,479167
    update 0 0,556250 0,358333
    update 1 0,391250 0,489583
    update 0 0,553750 0,366667
    update 1 0,392500 0,502083
    update 0 0,550000 0,377083
    update 1 0,392500 0,516667
    update 0 0,548750 0,385417
    update 1 0,392500 0,531250
    update 0 0,546250 0,395833
    update 1 0,390000 0,543750
    MIDAS CORRELATES
    (defrule Gui
    ?tap<- (Tap)
    ?planet <- (Planet)
    (test (sInside ?tap ?planet))
    =>
    (printout t “Tap inside planet!”))
    add 0 0,633750 0,293750
    update 0 0,635000 0,293750
    update 0 0,633750 0,293750
    update 0 0,633750 0,295833
    update 0 0,633750 0,297917
    update 0 0,632500 0,300000
    update 0 0,631250 0,300000
    update 0 0,630000 0,300000
    update 0 0,628750 0,300000
    update 0 0,627500 0,300000
    add 1 0,246250 0,289583
    update 1 0,247500 0,289583
    update 1 0,248750 0,291667
    update 1 0,250000 0,291667
    update 1 0,252500 0,291667
    update 1 0,256250 0,291667
    Fact Base
    Shadows
    (GUI elements)

    View full-size slide

  14. MIDAS SUPPORTS
    (defrule nextSlide
    ?goRight1 <- (FlickRight (tuioId ?*RH*))
    ?goLeft <- (FlickLeft (tuioId ?*LH*))
    ?goRight2 <- (FlickRight (tuioId ?*RH*))
    (test (tMeets ?goRight1 ?goLeft ?goRight2))
    =>
    (call PowerPoint showNextSlide)
    (retract ?goRight1 ?goLeft ?goRight2))
    Implemented online gestures
    Templates
    Kinect example

    View full-size slide

  15. CONCLUSION
     Midas multi-touch interaction framework
    ▪ Declarative description of gestures (fact base and rule engine)
    ▪ extensibility and reusability of gestures
    ▪ correlation of GUI elements and events via shadow facts
    ▪ JMidas Java integration
     Ongoing work
    ▪ integration of new devices: Kinect, Digital Pen & Paper, ...
    ▪ extend Midas to support multimodal gestures
     Rapid prototyping of multi-touch gestures
    Research: http://goo.gl/OFpej - Youtube: http://goo.gl/MNDkT - Twitter: #MidasFW
    Midas Demo, Tuesday @ 15.00-18.30 -- ID: 251

    View full-size slide