Midas: A Declarative Multi-Touch Interaction Framework

Presentation given at TEI 2011, 5th International Conference on Tangible, Embedded and Embodied Interaction, Madeira, Portugal.

ABSTRACT: Over the past few years, multi-touch user interfaces emerged from research prototypes into mass market products. This evolution has been mainly driven by innovative devices such as Apple’s iPhone or Microsoft’s Surface tabletop computer. Unfortunately, there seems to be a lack of software engineering abstractions in existing multi-touch development frameworks. Many multi-touch applications are based on hardcoded procedural low level event processing. This leads to proprietary solutions with a lack of gesture extensibility and cross-application reusability. We present Midas, a declarative model for the definition and detection of multi-touch gestures where gestures are expressed via logical rules over a set of input facts. We highlight how our rule-based language approach leads to improvements in gesture extensibility and reusability. Last but not least, we introduce JMidas, an instantiation of Midas for the Java programming language and describe how JMidas has been applied to implement a number of innovative multi-touch gestures.

Research paper: https://beatsigner.com/publications/midas-a-declarative-multi-touch-interaction-framework.pdf

Transcript

1. A Declarative Multi-Touch Interaction Framework CHRISTOPHE SCHOLLIERS LODE HOSTE BEAT SIGNER WOLFGANG DE MEUTER VRIJE UNIVERSITEIT BRUSSEL #MidasFW MIDAS
2. MULTI-TOUCH IS EVERYWHERE!
3. HARDWARE CAPABILITIES VS GESTURE RECOGNITION SOFTWARE Flick Drag Rotate Pinch Stretch
4. WHAT’S THE FUNDAMENTAL PROBLEM? update 0 0,565000 0,335417 update 1 0,387500 0,450000 update 0 0,560000 0,345833 update 1 0,390000 0,462500 update 0 0,558750 0,352083 update 1 0,391250 0,479167 update 0 0,556250 0,358333 update 1 0,391250 0,489583 update 0 0,553750 0,366667 update 1 0,392500 0,502083 update 0 0,550000 0,377083 update 1 0,392500 0,516667 update 0 0,548750 0,385417 update 1 0,392500 0,531250 update 0 0,546250 0,395833 update 1 0,390000 0,543750 update 0 0,543750 0,408333 update 1 0,388750 0,556250 update 0 0,542500 0,422917 update 1 0,386250 0,566667 update 0 0,543750 0,429167 update 1 0,385000 0,577083 update 0 0,543750 0,441667 update 1 0,378750 0,595833 update 0 0,543750 0,450000 update 1 0,367500 0,633333 update 0 0,547500 0,475000 update 1 0,348750 0,660417 update 0 0,551250 0,504167 update 1 0,341250 0,666667 update 0 0,553750 0,514583 update 1 0,336250 0,675000 update 0 0,557500 0,527083 update 1 0,331250 0,679167 update 0 0,561250 0,541667 update 1 0,326250 0,683333 update 0 0,565000 0,552083 update 1 0,321250 0,685417 update 0 0,570000 0,572917 update 1 0,316250 0,685417 update 0 0,575000 0,583333 update 1 0,311250 0,685417 update 0 0,581250 0,595833 update 1 0,307500 0,683333 update 0 0,590000 0,612500 update 1 0,302500 0,681250 update 0 0,595000 0,618750 update 1 0,296250 0,679167 update 0 0,606250 0,633333 update 1 0,290000 0,675000 update 0 0,612500 0,639583 update 1 0,281250 0,664583 update 0 0,622500 0,654167 update 1 0,272500 0,656250 update 0 0,631250 0,660417 update 1 0,268750 0,647917 update 0 0,636250 0,662500 update 1 0,262500 0,639583 update 0 0,643750 0,664583 update 1 0,257500 0,629167 update 0 0,647500 0,664583 update 1 0,255000 0,627083 update 0 0,653750 0,666667 update 1 0,166250 0,343750 update 0 0,728750 0,306250 update 1 0,172500 0,331250 update 0 0,725000 0,300000 update 1 0,177500 0,318750 update 0 0,723750 0,295833 update 1 0,181250 0,314583 update 0 0,720000 0,291667 update 1 0,185000 0,310417 update 0 0,718750 0,289583 update 1 0,186250 0,304167 update 0 0,716250 0,287500 update 1 0,190000 0,302083 update 0 0,713750 0,281250 update 1 0,192500 0,300000 update 0 0,708750 0,277083 update 1 0,198750 0,293750 update 0 0,702500 0,270833 update 1 0,205000 0,289583 update 0 0,692500 0,266667 update 1 0,208750 0,289583 update 0 0,688750 0,264583 update 1 0,212500 0,287500 update 0 0,685000 0,264583 update 1 0,216250 0,285417 update 0 0,678750 0,256250 update 1 0,221250 0,283333 update 0 0,675000 0,252083 update 1 0,227500 0,283333 update 0 0,671250 0,250000 update 1 0,231250 0,281250 update 0 0,667500 0,250000 update 1 0,233750 0,281250 update 0 0,662500 0,247917 update 1 0,238750 0,279167 update 0 0,658750 0,247917 update 1 0,245000 0,279167 update 0 0,655000 0,245833 update 1 0,248750 0,277083 update 0 0,651250 0,245833 update 1 0,253750 0,277083 update 0 0,646250 0,250000 update 1 0,258750 0,275000 update 0 0,642500 0,250000 update 1 0,263750 0,272917 update 0 0,638750 0,254167 update 1 0,267500 0,270833 update 0 0,635000 0,256250 update 1 0,271250 0,270833 update 0 0,631250 0,258333 update 1 0,278750 0,270833 update 0 0,625000 0,262500 update 1 0,282500 0,270833 update 0 0,622500 0,262500 update 1 0,286250 0,270833 update 0 0,621250 0,264583 update 1 0,288750 0,268750 update 0 0,620000 0,266667 update 1 0,291250 0,268750 update 0 0,618750 0,268750 update 1 0,292500 0,268750 update 0 0,617500 0,268750 update 1 0,293750 0,268750 update 1 0,251250 0,620833 update 0 0,656250 0,666667 update 1 0,242500 0,604167 update 0 0,663750 0,662500 update 1 0,237500 0,591667 update 0 0,667500 0,658333 update 1 0,232500 0,583333 update 0 0,672500 0,656250 update 1 0,215000 0,558333 update 0 0,691250 0,637500 update 1 0,206250 0,543750 update 0 0,702500 0,612500 update 1 0,187500 0,510417 update 0 0,711250 0,589583 update 1 0,167500 0,485417 update 0 0,716250 0,568750 update 1 0,161250 0,477083 update 0 0,720000 0,556250 update 1 0,152500 0,466667 update 0 0,725000 0,539583 update 1 0,146250 0,460417 update 0 0,727500 0,527083 update 1 0,143750 0,456250 update 0 0,728750 0,510417 update 1 0,138750 0,447917 update 0 0,731250 0,502083 update 1 0,136250 0,445833 update 0 0,733750 0,485417 update 1 0,133750 0,441667 update 0 0,736250 0,472917 update 1 0,132500 0,439583 update 0 0,740000 0,458333 update 1 0,131250 0,435417 update 0 0,741250 0,447917 update 0 0,741250 0,439583 update 1 0,130000 0,431250 update 0 0,742500 0,422917 update 1 0,130000 0,429167 update 0 0,742500 0,416667 update 1 0,130000 0,427083 update 0 0,743750 0,402083 update 1 0,130000 0,425000 update 0 0,745000 0,395833 update 1 0,131250 0,422917 update 0 0,745000 0,387500 update 1 0,132500 0,418750 update 0 0,745000 0,383333 update 1 0,133750 0,416667 update 0 0,743750 0,377083 update 1 0,135000 0,412500 update 0 0,743750 0,360417 update 1 0,136250 0,408333 update 0 0,741250 0,354167 update 1 0,141250 0,400000 update 0 0,738750 0,337500 update 1 0,145000 0,389583 update 0 0,738750 0,333333 update 1 0,147500 0,383333 update 0 0,735000 0,325000 update 1 0,151250 0,372917 update 0 0,731250 0,316667 update 1 0,157500 0,360417 update 0 0,728750 0,308333 add 0 0,633750 0,293750 update 0 0,635000 0,293750 update 0 0,633750 0,293750 update 0 0,633750 0,295833 update 0 0,633750 0,297917 update 0 0,632500 0,300000 update 0 0,631250 0,300000 update 0 0,630000 0,300000 update 0 0,628750 0,300000 update 0 0,627500 0,300000 add 1 0,246250 0,289583 update 1 0,247500 0,289583 update 1 0,248750 0,291667 update 1 0,250000 0,291667 update 1 0,252500 0,291667 update 1 0,256250 0,291667 update 1 0,258750 0,291667 update 1 0,263750 0,291667 update 0 0,626250 0,300000 update 1 0,276250 0,291667 update 0 0,625000 0,300000 update 1 0,285000 0,295833 update 0 0,622500 0,300000 update 1 0,293750 0,300000 update 0 0,621250 0,302083 update 1 0,300000 0,304167 update 1 0,310000 0,310417 update 0 0,618750 0,302083 update 1 0,318750 0,316667 update 0 0,616250 0,302083 update 1 0,325000 0,320833 update 0 0,613750 0,302083 update 1 0,330000 0,322917 update 0 0,611250 0,302083 update 1 0,335000 0,325000 update 0 0,610000 0,302083 update 1 0,338750 0,329167 update 0 0,607500 0,304167 update 1 0,341250 0,333333 update 0 0,605000 0,304167 update 1 0,345000 0,335417 update 0 0,602500 0,304167 update 1 0,347500 0,337500 update 0 0,600000 0,304167 update 1 0,351250 0,341667 update 0 0,596250 0,304167 update 1 0,355000 0,347917 update 0 0,593750 0,306250 update 1 0,358750 0,352083 update 0 0,590000 0,308333 update 1 0,361250 0,358333 update 0 0,586250 0,312500 update 1 0,365000 0,368750 update 0 0,583750 0,314583 update 1 0,368750 0,379167 update 0 0,580000 0,316667 update 1 0,372500 0,389583 update 0 0,577500 0,318750 update 1 0,378750 0,410417 update 0 0,571250 0,329167 update 1 0,382500 0,420833 update 0 0,568750 0,333333 update 1 0,383750 0,435417
5.  Sequential code for the processing of concurrent events  Find spatial and temporal relations in an immense pool of x/y coordinates ▪ ~25 events / second / finger  Approximative detection  GUI-Event correlation  Modularisation and composition ▪ implementing gestures is hard ▪ combining gestures is even harder WHAT’S THE FUNDAMENTAL PROBLEM?
6. THE MIDAS APPROACH SELECT * FROM Audience WHERE interest > 0 Facts Fact Base Rules
7. Midas Application Layer Core Layer Infrastructure Layer GUI Hardware Bridge * Translator Fact Base Inference Engine Rule Base Shadows Model MIDAS ARCHITECTURE  Infrastructure layer ▪ Transform low level data into facts  Core layer ▪ temporal and spatial operators ▪ continuous queries via Rete algorithm  Application layer ▪ regular program augmented with rules
8. MIDAS USES (defrule PrintCursor ?c <- (Cursor) => (printout t ‘‘A cursor is moving at: ’’ ?c.x ‘‘,’’ ?c.y))
9. MIDAS FILTERS + (defrule Tap ?appear <- (Cursor (finger ?f) (state ?*APPEAR*)) ?disappear <- (Cursor (finger ?f) (state ?*DISAPPEAR*)) (test (tShortlyAfter ?disappear ?appear)) => (printout t “A single tap detected at: “ ?appear.x “,” ?appear.y)) only a few lines of declarative code
10. MIDAS FINDS + (defrule Tap ?appear <- (Cursor (finger ?f) (state ?*APPEAR*)) ?disappear <- (Cursor (finger ?f) (state ?*DISAPPEAR*)) (test (tShortlyAfter ?disappear ?appear)) (test (sVeryNear ?disappear ?appear)) => (printout t “A single tap detected at: “ ?appear.x “,” ?appear.y) (assert (Tap (x ?appear.x) (y ?appear.y) (on ?appear.on)) (retract ?appear ?disappear))
11. MIDAS ENCOURAGES (defrule DoubleTap ?tap1 <- (Tap) ?tap2 <- (Tap) (test (tShortlyAfter ?tap2 ?tap1)) (test (sVeryNear ?tap2 ?tap1)) => (printout t “A double tap detected at: “ ?tap1.x“,” ?tap2.y) (retract ?tap1 ?tap2)) small and reusable building blocks
12. MIDAS ENCOURAGES (defrule FixedUpUp ?fixed <- (Fixed) ?up1 <- (Up) ?up2 <- (Up) (test (tEqual3 ?fixed ?up1 ?up2)) (test (sLeftOf ?fixed ?up1)) (test (sLeftOf ?up1 ?up2)) => (printout t “Complex scroll detected!”) (retract ?fixed ?up1 ?up2))
13. update 0 0,565000 0,335417 update 1 0,387500 0,450000 update 0 0,560000 0,345833 update 1 0,390000 0,462500 update 0 0,558750 0,352083 update 1 0,391250 0,479167 update 0 0,556250 0,358333 update 1 0,391250 0,489583 update 0 0,553750 0,366667 update 1 0,392500 0,502083 update 0 0,550000 0,377083 update 1 0,392500 0,516667 update 0 0,548750 0,385417 update 1 0,392500 0,531250 update 0 0,546250 0,395833 update 1 0,390000 0,543750 MIDAS CORRELATES (defrule Gui ?tap<- (Tap) ?planet <- (Planet) (test (sInside ?tap ?planet)) => (printout t “Tap inside planet!”)) add 0 0,633750 0,293750 update 0 0,635000 0,293750 update 0 0,633750 0,293750 update 0 0,633750 0,295833 update 0 0,633750 0,297917 update 0 0,632500 0,300000 update 0 0,631250 0,300000 update 0 0,630000 0,300000 update 0 0,628750 0,300000 update 0 0,627500 0,300000 add 1 0,246250 0,289583 update 1 0,247500 0,289583 update 1 0,248750 0,291667 update 1 0,250000 0,291667 update 1 0,252500 0,291667 update 1 0,256250 0,291667 Fact Base Shadows (GUI elements)
14. MIDAS SUPPORTS (defrule nextSlide ?goRight1 <- (FlickRight (tuioId ?*RH*)) ?goLeft <- (FlickLeft (tuioId ?*LH*)) ?goRight2 <- (FlickRight (tuioId ?*RH*)) (test (tMeets ?goRight1 ?goLeft ?goRight2)) => (call PowerPoint showNextSlide) (retract ?goRight1 ?goLeft ?goRight2)) Implemented online gestures Templates Kinect example
15. CONCLUSION  Midas multi-touch interaction framework ▪ Declarative description of gestures (fact base and rule engine) ▪ extensibility and reusability of gestures ▪ correlation of GUI elements and events via shadow facts ▪ JMidas Java integration  Ongoing work ▪ integration of new devices: Kinect, Digital Pen & Paper, ... ▪ extend Midas to support multimodal gestures  Rapid prototyping of multi-touch gestures Research: http://goo.gl/OFpej - Youtube: http://goo.gl/MNDkT - Twitter: #MidasFW Midas Demo, Tuesday @ 15.00-18.30 -- ID: 251