Gestural Query Specification 
 Arnab Nandi, Lilong Jiang, Michael Mandel
 Computer Science & Engineering The Ohio State University interactive data systems research group at the ohio state university papers, videos & more at http://interact.osu.edu/gesturedb 

What does a computer " look like? 

No content

No content

No content

Computing and Keyboards 550   351   0   100   200   300   400   500   600   Desktops, Laptops & Netbooks Smartphones & Tablets (1.5x) 2011 Worldwide Shipments, in Millions Canalys. Worldwide Smartphone and Client PC Shipment Estimates. 2012. 

Computing and Keyboards 185   93.7   0   50   100   150   200   Desktops, Laptops & Netbooks Smartphones & Tablets (1.9x) Q4’11 Worldwide Shipments, in Millions Canalys. Worldwide Smartphone and Client PC Shipment Estimates. 2012. 

No content

There will soon be more computers without keyboards than with. 

SELECT * FROM SQL?! 

Need: " Databases that can meet these challenges &" serve such interactions 

Challenges •  Direct manipulation •  Interaction is continuous •  Infinite number of gestures •  Feedback is required to keep user oriented •  100-1000ms latencies 

Gesture •  A movement of the hand or body •  Represented as a set of time-series of points •  pi = •  t: timestamp •  l: locational information •  m: metadata information 

gestureDB •  Our attempt at building an 
 interactive database •  Principled approach
 •  Serve multiple modes 
 of interaction •  Touch •  Motion tracking •  Mouse •  Combinations 

Outline •  Challenges •  An Interactive Database: gestureDB •  Query Paradigm •  Query Language •  System Architecture
 •  Experiments and Evaluation 

Outline •  Challenges •  An Interactive Database: gestureDB •  Query Paradigm •  Query Language •  System Architecture
 •  Experiments and Evaluation 

Query Paradigm •  Query Intent •  Space of articulated queries •  User goes from vague to specific intent •  Intent Transition •  Query Context •  Direct Manipulation •  Session of multiple Query Intent Transitions 

Query Paradigm •  Query Intent •  Space of articulate queries •  User goes from vague to specific intent •  Intent Transition •  Query Context •  Direct Manipulation •  Session of multiple Query Intent Transitions Q1 Q2 Q3 Q4 Q5 1.0   0                                   

Query Paradigm •  Query Intent •  Space of articulate queries •  User goes from vague to specific intent •  Intent Transition •  Query Context •  Direct Manipulation •  Session of multiple Query Intent Transitions Q1 Q2 Q3 Q4 Q5 1.0   0                                   

Query Paradigm •  Query Intent •  Space of articulate queries •  User goes from vague to specific intent •  Intent Transition •  Query Context •  Direct Manipulation •  Session of multiple Query Intent Transitions Q1 Q2 Q3 Q4 Q5 1.0   0                                   

Query Paradigm •  Query Intent •  Space of articulate queries •  User goes from vague to specific intent •  Gesture Articulation •  Query Context •  Direct Manipulation •  Session of multiple Gesture Articulations 

Query Paradigm (contd.) •  Query Gesture •  Library of user gestures •  Mapped to parameterized query •  Intent Feedback •  During Gesture Articulation •  Constantly provide user with feedback • Query • Result 

Gesture Query Language Direct Manipulation · Closure and Composition · Feedback · Expressivity 1 " 2 " NYC" 22 " 2 " 2 " SF" 31 " 3 " 4 " ATL" 3 " Employee" id " projectId " loca:on" deptId " "" "" "" "" "" "" "" "" "" "" 1 " 2 " NYC" 22 " 2 " 2 " SF" 31 " 3 " 4 " ATL" 3 " Employee" id " projectId " loca:on" deptId " "" "" 1 " 2 " NYC" 22 " 2 " 2 " SF" 31 " 3 " 4 " ATL" 3 " Employee" id " projectId " loca:on" deptId " "" "" !"#$ %$ &%%$ WHERE emp.age > 60 distribution
 of values 
 in column! Employee ! id ! projectId ! loca.on! deptId ! !! !! !! !! !! !! ! ! ! ! Person ! id ! projId ! loca.on! department ! !! !! !! !!!!!! ! ! ! ! Employee   id   projectId   loca9on   deptId   9tle   deptId           Employee   id   projectId   loca9on   deptId   9tle   loca9on           projectId           1   2   NYC   22   2   2   SF   31   3   4   ATL   3   Employee   id   projectId   loca9on   deptId   Employee   id   projectId   loca9on   deptId               1   2   NYC   22   2   2   SF   31   3   4   ATL   3   Employee   id   projectId   loca9on   deptId                               Preview   Sort   Filter   Range  Filter   Union   Rearrange   Aggregate   Update  schema  &  update  value   Projec9on   Insert   MIN   MAX   AVG   SUM   

Using Design: Exploratory Join •  Challenge •  M x N join …Why not use interaction & design? Employee id projectId location deptId Person id projId location department Employee id projectId location deptId Person id projId location department Employee id projectId location deptId Person id projId location department     O(M * N) O(Min(M, N)) O(M * N) 

Intent Interpretation: Design Radial, Tabular and Arc Layouts 
 The responsive arc layout allows all queries to be expressed unambiguously while still achieving readability.   Employee id projectId location deptId title Project id projId managerId location name projectId location Employee id deptId managerId projId Project id location name title projId managerId Project id projectId location deptId title Employee Radial Vertical Arc 

Demo" gestureDB w/ iPad frontend Note: There can be 
 many different frontends Using Database Explorer as an example (Could have built a “social media explorer” too!) 
   

gestureDB Architecture Feedback     Genera9on   Data   Store   User   Context   Cache   User  Interface   Context   Cache   Network   projectId* loca-on* !Employee! ** ** ** ** ** ** ** ** id* deptId* parentProjectId- supervisorID- Project( id- managerId- -- -- -------- -- -- Gesture   Query   Language   Intent     Interpreta9on   Interac9ve   Query   Execu9on   John   SF   Audit   Tax   Fin   Joe   John   LA   Review   Acc   Fin   Jen   ⋈   1:2   n=1200   

Classifier d combine them linearly in the argument of an onential. Mathematically, the goodness g ( q ) of a ential query q with feature values fi( q ) is g ( q ) = exp( X i ifi( q )) . eatures can be binary or real-valued, with 0 being th ue of an uninformative feature. Parameters of the ssifier, i , can be learned across a collection of record ning gestures to tune the quality of the classifier. Fo liminary experiments in Section , these parameters a manually, and tuning using training data is scope fo ther improvement in quality. New queries can be •  g(q) = “Goodness” of a query, probability distribution of SQL queries •  fi = features from •  Relative Position information (gesture + objects) •  Schema compatibility •  Data compatibility (distribution) •  λ learned from training 

Classifier Strategies •  Precompute Summaries (send to frontend) •  Data Distribution Comparisons •  Schema Compatibilities •  Skip Computations •  When -∞ is encountered •  Memoize Everything (distance calculations) •  Jitter 

Schema + Data helps" Intent Interpretation “How quickly does the system correctly interpret intent?” 1.0 = correct at the start of the gesture articulation 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Join1A Join1R Join2A Join2R Union1 Union2 Anticipation Proximity Proximity + Schema Proximity + Schema + Data 

Outline •  Challenges •  An Interactive Database: gestureDB •  Experiments and Evaluation •  Completion Time •  Discoverability •  Survey •  Classifier Performance 

is gestural querying of data " a good idea? 

User Studies •  Running on iPad 3rd Gen(16GB) •  Comparisons •  Visual Query Builder vs Console vs GestureDB •  30 users, within-subjects •  15 naïve, 15 expert 

User Study: Avoiding Biases •  Fatigue •  Learning Effects •  Carryover Effects •  Consistency 

Datasets •  Chinook •  11 tables, 64 columns, 15K tuples, 1MB •  http://chinookdatabase.codeplex.com •  TPC-H •  9 tables, 76 columns, 86K tuples, 15MB •  http://www.tpc.org/tpch 

Completion Time Task •  Preview Preview the table Invoice •  Filter Filter table Invoice according the BillingCountry value of the tuple whose InvoiceId is 5 •  Sort Sort table Album in descending order on AlbumId •  Group by & Aggregate For table PART, group by the attribute P_TYPE and aggregate on the attribute P_SIZE with max function •  Join Invoice Join InvoiceLine on Invoice.InvoiceId = InvoiceLine.InvoiceId •  Union Album1 Union Album2 

Gestural Querying is Faster 0" 10" 20" 30" 40" 50" 60" 70" Preview" Filter" Sort" Group"by"&" Aggregate" Join" Union" Predic'on*(s)* ConsoleCbased"SQL" Visual"Query"Builder" GestureQuery" 

80 108 111 128 169 178 196 218 256 294 305 329 340 384 446 447 476 514 550 563 616 679 745 900 900 900 900 900 900 900 900 893 596 900 900 313 731 268 455 240 900 198 525 900 900 900 402 900 185 781 900 654 240 205 58 429 900 900 900 900 0 100 200 300 400 500 600 700 800 900 1000 Discoverability(s) Users GestureQuery Visual Query Builder Gestural Queries are Discoverable •  14 / 30 fail with Visual Query Builder •  21 / 30 perform better with GestureQuery 

Users prefer Gestural Querying learn (learnability) for each query type. Systems are CONSOLE- BASED SQL, VISUAL QUERY BUILDER (VQB), and GES- TUREQUERY (GQ – our system). For users claiming equal preference for multiple systems, their votes were split equally amongst the systems. Survey Action Console VQB GQ Usability PREVIEW 1 5 24 Usability FILTER 5 1.5 23.5 Usability SORT 0 2 28 Usability AGGREGATE 5 4 21 Usability JOIN 3 15 12 Usability UNION 2 0 28 Learnability PREVIEW 1.66 2.16 26.16 Learnability FILTER 5 4.5 20.5 Learnability SORT 1.33 1.33 27.33 Learnability AGGREGATE 2 10.5 17.5 Learnability JOIN 1 15 14 Learnability UNION 3 0 27 Table 3: Interactivity: percentage of touch classifications that take less than 33 ms. Figu by 3 sific sent ture low for 5.3 A clas part JOI Join the 

Classifier Performance •  180 gestures (from studies) •  Performance: Almost always < 33ms •  > 95% •  30Hz •  Interactivity 

Summary •  Challenges
 •  An Interactive Database: gestureDB •  Experiments and Evaluation 

Summary •  Challenges •  An Interactive Database: gestureDB •  Query Paradigm •  Gesture Query Language •  System Architecture
 •  Experiments and Evaluation 

Summary •  Challenges •  An Interactive Database: gestureDB •  Experiments and Evaluation •  Completion Time •  Discoverability •  Survey •  Classifier Performance 

Thank you! interactive data systems research group at the ohio state university papers, videos & more at http://interact.osu.edu/gesturedb