Gestural Query Specification: Querying without Keyboards

Computing devices that use non-traditional methods to interact with data are becoming more popular than those that use keyboard-based interaction. Applications and query interfaces for such devices pose a fundamentally different set of workloads on the underlying databases. We characterize these differences and show how today's query representation and evaluation techniques are not well-suited to these new non-keyboard interaction modalities. We propose a new database architecture to address these issues, and demonstrate that it can be built using already existing components.

For more details, visit: http://interact.osu.edu/gesturedb/

Transcript

1. Gestural Query Specification 
 Arnab Nandi, Lilong Jiang, Michael Mandel
 Computer Science & Engineering The Ohio State University interactive data systems research group at the ohio state university papers, videos & more at http://interact.osu.edu/gesturedb
2. What does a computer " look like?
6. Computing and Keyboards 550   351   0   100   200   300   400   500   600   Desktops, Laptops & Netbooks Smartphones & Tablets (1.5x) 2011 Worldwide Shipments, in Millions Canalys. Worldwide Smartphone and Client PC Shipment Estimates. 2012.
7. Computing and Keyboards 185   93.7   0   50   100   150   200   Desktops, Laptops & Netbooks Smartphones & Tablets (1.9x) Q4’11 Worldwide Shipments, in Millions Canalys. Worldwide Smartphone and Client PC Shipment Estimates. 2012.
9. There will soon be more computers without keyboards than with.
10. SELECT * FROM SQL?!
11. Need: " Databases that can meet these challenges &" serve such interactions
12. Challenges •  Direct manipulation •  Interaction is continuous •  Infinite number of gestures •  Feedback is required to keep user oriented •  100-1000ms latencies
13. Gesture •  A movement of the hand or body •  Represented as a set of time-series of points •  pi = <t, l, m> •  t: timestamp •  l: locational information •  m: metadata information
14. gestureDB •  Our attempt at building an 
 interactive database •  Principled approach
 •  Serve multiple modes 
 of interaction •  Touch •  Motion tracking •  Mouse •  Combinations
15. Outline •  Challenges •  An Interactive Database: gestureDB •  Query Paradigm •  Query Language •  System Architecture
 •  Experiments and Evaluation
16. Outline •  Challenges •  An Interactive Database: gestureDB •  Query Paradigm •  Query Language •  System Architecture
 •  Experiments and Evaluation
17. Query Paradigm •  Query Intent •  Space of articulated queries •  User goes from vague to specific intent •  Intent Transition •  Query Context •  Direct Manipulation •  Session of multiple Query Intent Transitions
18. Query Paradigm •  Query Intent •  Space of articulate queries •  User goes from vague to specific intent •  Intent Transition •  Query Context •  Direct Manipulation •  Session of multiple Query Intent Transitions Q1 Q2 Q3 Q4 Q5 1.0   0                                  
19. Query Paradigm •  Query Intent •  Space of articulate queries •  User goes from vague to specific intent •  Intent Transition •  Query Context •  Direct Manipulation •  Session of multiple Query Intent Transitions Q1 Q2 Q3 Q4 Q5 1.0   0                                  
20. Query Paradigm •  Query Intent •  Space of articulate queries •  User goes from vague to specific intent •  Intent Transition •  Query Context •  Direct Manipulation •  Session of multiple Query Intent Transitions Q1 Q2 Q3 Q4 Q5 1.0   0                                  
21. Query Paradigm •  Query Intent •  Space of articulate queries •  User goes from vague to specific intent •  Gesture Articulation •  Query Context •  Direct Manipulation •  Session of multiple Gesture Articulations
22. Query Paradigm (contd.) •  Query Gesture •  Library of user gestures •  Mapped to parameterized query •  Intent Feedback •  During Gesture Articulation •  Constantly provide user with feedback • Query • Result
23. Gesture Query Language Direct Manipulation · Closure and Composition · Feedback · Expressivity 1 " 2 " NYC" 22 " 2 " 2 " SF" 31 " 3 " 4 " ATL" 3 " Employee" id " projectId " loca:on" deptId " "" "" "" "" "" "" "" "" "" "" 1 " 2 " NYC" 22 " 2 " 2 " SF" 31 " 3 " 4 " ATL" 3 " Employee" id " projectId " loca:on" deptId " "" "" 1 " 2 " NYC" 22 " 2 " 2 " SF" 31 " 3 " 4 " ATL" 3 " Employee" id " projectId " loca:on" deptId " "" "" !"#$ %$ &%%$ WHERE emp.age > 60 distribution
 of values 
 in column! Employee ! id ! projectId ! loca.on! deptId ! !! !! !! !! !! !! ! ! ! ! Person ! id ! projId ! loca.on! department ! !! !! !! !!!!!! ! ! ! ! Employee   id   projectId   loca9on   deptId   9tle   deptId           Employee   id   projectId   loca9on   deptId   9tle   loca9on           projectId           1   2   NYC   22   2   2   SF   31   3   4   ATL   3   Employee   id   projectId   loca9on   deptId   Employee   id   projectId   loca9on   deptId               1   2   NYC   22   2   2   SF   31   3   4   ATL   3   Employee   id   projectId   loca9on   deptId                               Preview   Sort   Filter   Range  Filter   Union   Rearrange   Aggregate   Update  schema  &  update  value   Projec9on   Insert   MIN   MAX   AVG   SUM  
24. Using Design: Exploratory Join •  Challenge •  M x N join …Why not use interaction & design? Employee id projectId location deptId Person id projId location department Employee id projectId location deptId Person id projId location department Employee id projectId location deptId Person id projId location department     O(M * N) O(Min(M, N)) O(M * N)
25. Intent Interpretation: Design Radial, Tabular and Arc Layouts 
 The responsive arc layout allows all queries to be expressed unambiguously while still achieving readability.   Employee id projectId location deptId title Project id projId managerId location name projectId location Employee id deptId managerId projId Project id location name title projId managerId Project id projectId location deptId title Employee Radial Vertical Arc
26. Demo" gestureDB w/ iPad frontend Note: There can be 
 many different frontends Using Database Explorer as an example (Could have built a “social media explorer” too!) 
  
27. gestureDB Architecture Feedback     Genera9on   Data   Store   User   Context   Cache   User  Interface   Context   Cache   Network   projectId* loca-on* !Employee! ** ** ** ** ** ** ** ** id* deptId* parentProjectId- supervisorID- Project( id- managerId- -- -- -------- -- -- Gesture   Query   Language   Intent     Interpreta9on   Interac9ve   Query   Execu9on   John   SF   Audit   Tax   Fin   Joe   John   LA   Review   Acc   Fin   Jen   ⋈   1:2   n=1200  
28. Classifier d combine them linearly in the argument of an onential. Mathematically, the goodness g ( q ) of a ential query q with feature values fi( q ) is g ( q ) = exp( X i ifi( q )) . eatures can be binary or real-valued, with 0 being th ue of an uninformative feature. Parameters of the ssifier, i , can be learned across a collection of record ning gestures to tune the quality of the classifier. Fo liminary experiments in Section , these parameters a manually, and tuning using training data is scope fo ther improvement in quality. New queries can be •  g(q) = “Goodness” of a query, probability distribution of SQL queries •  fi = features from •  Relative Position information (gesture + objects) •  Schema compatibility •  Data compatibility (distribution) •  λ learned from training
29. Classifier Strategies •  Precompute Summaries (send to frontend) •  Data Distribution Comparisons •  Schema Compatibilities •  Skip Computations •  When -∞ is encountered •  Memoize Everything (distance calculations) •  Jitter
30. Schema + Data helps" Intent Interpretation “How quickly does the system correctly interpret intent?” 1.0 = correct at the start of the gesture articulation 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Join1A Join1R Join2A Join2R Union1 Union2 Anticipation Proximity Proximity + Schema Proximity + Schema + Data
31. Outline •  Challenges •  An Interactive Database: gestureDB •  Experiments and Evaluation •  Completion Time •  Discoverability •  Survey •  Classifier Performance
32. is gestural querying of data " a good idea?
33. User Studies •  Running on iPad 3rd Gen(16GB) •  Comparisons •  Visual Query Builder vs Console vs GestureDB •  30 users, within-subjects •  15 naïve, 15 expert
34. User Study: Avoiding Biases •  Fatigue •  Learning Effects •  Carryover Effects •  Consistency
35. Datasets •  Chinook •  11 tables, 64 columns, 15K tuples, 1MB •  http://chinookdatabase.codeplex.com •  TPC-H •  9 tables, 76 columns, 86K tuples, 15MB •  http://www.tpc.org/tpch
36. Completion Time Task •  Preview Preview the table Invoice •  Filter Filter table Invoice according the BillingCountry value of the tuple whose InvoiceId is 5 •  Sort Sort table Album in descending order on AlbumId •  Group by & Aggregate For table PART, group by the attribute P_TYPE and aggregate on the attribute P_SIZE with max function •  Join Invoice Join InvoiceLine on Invoice.InvoiceId = InvoiceLine.InvoiceId •  Union Album1 Union Album2
37. Gestural Querying is Faster 0" 10" 20" 30" 40" 50" 60" 70" Preview" Filter" Sort" Group"by"&" Aggregate" Join" Union" Predic'on*(s)* ConsoleCbased"SQL" Visual"Query"Builder" GestureQuery"
38. 80 108 111 128 169 178 196 218 256 294 305 329 340 384 446 447 476 514 550 563 616 679 745 900 900 900 900 900 900 900 900 893 596 900 900 313 731 268 455 240 900 198 525 900 900 900 402 900 185 781 900 654 240 205 58 429 900 900 900 900 0 100 200 300 400 500 600 700 800 900 1000 Discoverability(s) Users GestureQuery Visual Query Builder Gestural Queries are Discoverable •  14 / 30 fail with Visual Query Builder •  21 / 30 perform better with GestureQuery
39. Users prefer Gestural Querying learn (learnability) for each query type. Systems are CONSOLE- BASED SQL, VISUAL QUERY BUILDER (VQB), and GES- TUREQUERY (GQ – our system). For users claiming equal preference for multiple systems, their votes were split equally amongst the systems. Survey Action Console VQB GQ Usability PREVIEW 1 5 24 Usability FILTER 5 1.5 23.5 Usability SORT 0 2 28 Usability AGGREGATE 5 4 21 Usability JOIN 3 15 12 Usability UNION 2 0 28 Learnability PREVIEW 1.66 2.16 26.16 Learnability FILTER 5 4.5 20.5 Learnability SORT 1.33 1.33 27.33 Learnability AGGREGATE 2 10.5 17.5 Learnability JOIN 1 15 14 Learnability UNION 3 0 27 Table 3: Interactivity: percentage of touch classifications that take less than 33 ms. Figu by 3 sific sent ture low for 5.3 A clas part JOI Join the
40. Classifier Performance •  180 gestures (from studies) •  Performance: Almost always < 33ms •  > 95% •  30Hz •  Interactivity
41. Summary •  Challenges
 •  An Interactive Database: gestureDB •  Experiments and Evaluation
42. Summary •  Challenges •  An Interactive Database: gestureDB •  Query Paradigm •  Gesture Query Language •  System Architecture
 •  Experiments and Evaluation
43. Summary •  Challenges •  An Interactive Database: gestureDB •  Experiments and Evaluation •  Completion Time •  Discoverability •  Survey •  Classifier Performance
44. Thank you! interactive data systems research group at the ohio state university papers, videos & more at http://interact.osu.edu/gesturedb