Gestural Query Specification: Querying without Keyboards

D49c0d235689fe04b1dd4630cabc2aa8?s=47 Arnab Nandi
September 04, 2014

Gestural Query Specification: Querying without Keyboards

Computing devices that use non-traditional methods to interact with data are becoming more popular than those that use keyboard-based interaction. Applications and query interfaces for such devices pose a fundamentally different set of workloads on the underlying databases. We characterize these differences and show how today's query representation and evaluation techniques are not well-suited to these new non-keyboard interaction modalities. We propose a new database architecture to address these issues, and demonstrate that it can be built using already existing components.

For more details, visit: http://interact.osu.edu/gesturedb/

D49c0d235689fe04b1dd4630cabc2aa8?s=128

Arnab Nandi

September 04, 2014
Tweet

Transcript

  1. 1.

    Gestural Query Specification 
 Arnab Nandi, Lilong Jiang, Michael Mandel


    Computer Science & Engineering The Ohio State University interactive data systems research group at the ohio state university papers, videos & more at http://interact.osu.edu/gesturedb
  2. 3.
  3. 4.
  4. 5.
  5. 6.

    Computing and Keyboards 550   351   0   100

      200   300   400   500   600   Desktops, Laptops & Netbooks Smartphones & Tablets (1.5x) 2011 Worldwide Shipments, in Millions Canalys. Worldwide Smartphone and Client PC Shipment Estimates. 2012.
  6. 7.

    Computing and Keyboards 185   93.7   0   50

      100   150   200   Desktops, Laptops & Netbooks Smartphones & Tablets (1.9x) Q4’11 Worldwide Shipments, in Millions Canalys. Worldwide Smartphone and Client PC Shipment Estimates. 2012.
  7. 8.
  8. 12.

    Challenges •  Direct manipulation •  Interaction is continuous •  Infinite

    number of gestures •  Feedback is required to keep user oriented •  100-1000ms latencies
  9. 13.

    Gesture •  A movement of the hand or body • 

    Represented as a set of time-series of points •  pi = <t, l, m> •  t: timestamp •  l: locational information •  m: metadata information
  10. 14.

    gestureDB •  Our attempt at building an 
 interactive database

    •  Principled approach
 •  Serve multiple modes 
 of interaction •  Touch •  Motion tracking •  Mouse •  Combinations
  11. 15.

    Outline •  Challenges •  An Interactive Database: gestureDB •  Query

    Paradigm •  Query Language •  System Architecture
 •  Experiments and Evaluation
  12. 16.

    Outline •  Challenges •  An Interactive Database: gestureDB •  Query

    Paradigm •  Query Language •  System Architecture
 •  Experiments and Evaluation
  13. 17.

    Query Paradigm •  Query Intent •  Space of articulated queries

    •  User goes from vague to specific intent •  Intent Transition •  Query Context •  Direct Manipulation •  Session of multiple Query Intent Transitions
  14. 18.

    Query Paradigm •  Query Intent •  Space of articulate queries

    •  User goes from vague to specific intent •  Intent Transition •  Query Context •  Direct Manipulation •  Session of multiple Query Intent Transitions Q1 Q2 Q3 Q4 Q5 1.0   0                                  
  15. 19.

    Query Paradigm •  Query Intent •  Space of articulate queries

    •  User goes from vague to specific intent •  Intent Transition •  Query Context •  Direct Manipulation •  Session of multiple Query Intent Transitions Q1 Q2 Q3 Q4 Q5 1.0   0                                  
  16. 20.

    Query Paradigm •  Query Intent •  Space of articulate queries

    •  User goes from vague to specific intent •  Intent Transition •  Query Context •  Direct Manipulation •  Session of multiple Query Intent Transitions Q1 Q2 Q3 Q4 Q5 1.0   0                                  
  17. 21.

    Query Paradigm •  Query Intent •  Space of articulate queries

    •  User goes from vague to specific intent •  Gesture Articulation •  Query Context •  Direct Manipulation •  Session of multiple Gesture Articulations
  18. 22.

    Query Paradigm (contd.) •  Query Gesture •  Library of user

    gestures •  Mapped to parameterized query •  Intent Feedback •  During Gesture Articulation •  Constantly provide user with feedback • Query • Result
  19. 23.

    Gesture Query Language Direct Manipulation · Closure and Composition ·

    Feedback · Expressivity 1 " 2 " NYC" 22 " 2 " 2 " SF" 31 " 3 " 4 " ATL" 3 " Employee" id " projectId " loca:on" deptId " "" "" "" "" "" "" "" "" "" "" 1 " 2 " NYC" 22 " 2 " 2 " SF" 31 " 3 " 4 " ATL" 3 " Employee" id " projectId " loca:on" deptId " "" "" 1 " 2 " NYC" 22 " 2 " 2 " SF" 31 " 3 " 4 " ATL" 3 " Employee" id " projectId " loca:on" deptId " "" "" !"#$ %$ &%%$ WHERE emp.age > 60 distribution
 of values 
 in column! Employee ! id ! projectId ! loca.on! deptId ! !! !! !! !! !! !! ! ! ! ! Person ! id ! projId ! loca.on! department ! !! !! !! !!!!!! ! ! ! ! Employee   id   projectId   loca9on   deptId   9tle   deptId           Employee   id   projectId   loca9on   deptId   9tle   loca9on           projectId           1   2   NYC   22   2   2   SF   31   3   4   ATL   3   Employee   id   projectId   loca9on   deptId   Employee   id   projectId   loca9on   deptId               1   2   NYC   22   2   2   SF   31   3   4   ATL   3   Employee   id   projectId   loca9on   deptId                               Preview   Sort   Filter   Range  Filter   Union   Rearrange   Aggregate   Update  schema  &  update  value   Projec9on   Insert   MIN   MAX   AVG   SUM  
  20. 24.

    Using Design: Exploratory Join •  Challenge •  M x N

    join …Why not use interaction & design? Employee id projectId location deptId Person id projId location department Employee id projectId location deptId Person id projId location department Employee id projectId location deptId Person id projId location department     O(M * N) O(Min(M, N)) O(M * N)
  21. 25.

    Intent Interpretation: Design Radial, Tabular and Arc Layouts 
 The

    responsive arc layout allows all queries to be expressed unambiguously while still achieving readability.   Employee id projectId location deptId title Project id projId managerId location name projectId location Employee id deptId managerId projId Project id location name title projId managerId Project id projectId location deptId title Employee Radial Vertical Arc
  22. 26.

    Demo" gestureDB w/ iPad frontend Note: There can be 


    many different frontends Using Database Explorer as an example (Could have built a “social media explorer” too!) 
  
  23. 27.

    gestureDB Architecture Feedback     Genera9on   Data   Store

      User   Context   Cache   User  Interface   Context   Cache   Network   projectId* loca-on* !Employee! ** ** ** ** ** ** ** ** id* deptId* parentProjectId- supervisorID- Project( id- managerId- -- -- -------- -- -- Gesture   Query   Language   Intent     Interpreta9on   Interac9ve   Query   Execu9on   John   SF   Audit   Tax   Fin   Joe   John   LA   Review   Acc   Fin   Jen   ⋈   1:2   n=1200  
  24. 28.

    Classifier d combine them linearly in the argument of an

    onential. Mathematically, the goodness g ( q ) of a ential query q with feature values fi( q ) is g ( q ) = exp( X i ifi( q )) . eatures can be binary or real-valued, with 0 being th ue of an uninformative feature. Parameters of the ssifier, i , can be learned across a collection of record ning gestures to tune the quality of the classifier. Fo liminary experiments in Section , these parameters a manually, and tuning using training data is scope fo ther improvement in quality. New queries can be •  g(q) = “Goodness” of a query, probability distribution of SQL queries •  fi = features from •  Relative Position information (gesture + objects) •  Schema compatibility •  Data compatibility (distribution) •  λ learned from training
  25. 29.

    Classifier Strategies •  Precompute Summaries (send to frontend) •  Data

    Distribution Comparisons •  Schema Compatibilities •  Skip Computations •  When -∞ is encountered •  Memoize Everything (distance calculations) •  Jitter
  26. 30.

    Schema + Data helps" Intent Interpretation “How quickly does the

    system correctly interpret intent?” 1.0 = correct at the start of the gesture articulation 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Join1A Join1R Join2A Join2R Union1 Union2 Anticipation Proximity Proximity + Schema Proximity + Schema + Data
  27. 31.

    Outline •  Challenges •  An Interactive Database: gestureDB •  Experiments

    and Evaluation •  Completion Time •  Discoverability •  Survey •  Classifier Performance
  28. 33.

    User Studies •  Running on iPad 3rd Gen(16GB) •  Comparisons

    •  Visual Query Builder vs Console vs GestureDB •  30 users, within-subjects •  15 naïve, 15 expert
  29. 35.

    Datasets •  Chinook •  11 tables, 64 columns, 15K tuples,

    1MB •  http://chinookdatabase.codeplex.com •  TPC-H •  9 tables, 76 columns, 86K tuples, 15MB •  http://www.tpc.org/tpch
  30. 36.

    Completion Time Task •  Preview Preview the table Invoice • 

    Filter Filter table Invoice according the BillingCountry value of the tuple whose InvoiceId is 5 •  Sort Sort table Album in descending order on AlbumId •  Group by & Aggregate For table PART, group by the attribute P_TYPE and aggregate on the attribute P_SIZE with max function •  Join Invoice Join InvoiceLine on Invoice.InvoiceId = InvoiceLine.InvoiceId •  Union Album1 Union Album2
  31. 37.

    Gestural Querying is Faster 0" 10" 20" 30" 40" 50"

    60" 70" Preview" Filter" Sort" Group"by"&" Aggregate" Join" Union" Predic'on*(s)* ConsoleCbased"SQL" Visual"Query"Builder" GestureQuery"
  32. 38.

    80 108 111 128 169 178 196 218 256 294

    305 329 340 384 446 447 476 514 550 563 616 679 745 900 900 900 900 900 900 900 900 893 596 900 900 313 731 268 455 240 900 198 525 900 900 900 402 900 185 781 900 654 240 205 58 429 900 900 900 900 0 100 200 300 400 500 600 700 800 900 1000 Discoverability(s) Users GestureQuery Visual Query Builder Gestural Queries are Discoverable •  14 / 30 fail with Visual Query Builder •  21 / 30 perform better with GestureQuery
  33. 39.

    Users prefer Gestural Querying learn (learnability) for each query type.

    Systems are CONSOLE- BASED SQL, VISUAL QUERY BUILDER (VQB), and GES- TUREQUERY (GQ – our system). For users claiming equal preference for multiple systems, their votes were split equally amongst the systems. Survey Action Console VQB GQ Usability PREVIEW 1 5 24 Usability FILTER 5 1.5 23.5 Usability SORT 0 2 28 Usability AGGREGATE 5 4 21 Usability JOIN 3 15 12 Usability UNION 2 0 28 Learnability PREVIEW 1.66 2.16 26.16 Learnability FILTER 5 4.5 20.5 Learnability SORT 1.33 1.33 27.33 Learnability AGGREGATE 2 10.5 17.5 Learnability JOIN 1 15 14 Learnability UNION 3 0 27 Table 3: Interactivity: percentage of touch classifications that take less than 33 ms. Figu by 3 sific sent ture low for 5.3 A clas part JOI Join the
  34. 40.

    Classifier Performance •  180 gestures (from studies) •  Performance: Almost

    always < 33ms •  > 95% •  30Hz •  Interactivity
  35. 42.

    Summary •  Challenges •  An Interactive Database: gestureDB •  Query

    Paradigm •  Gesture Query Language •  System Architecture
 •  Experiments and Evaluation
  36. 43.

    Summary •  Challenges •  An Interactive Database: gestureDB •  Experiments

    and Evaluation •  Completion Time •  Discoverability •  Survey •  Classifier Performance
  37. 44.

    Thank you! interactive data systems research group at the ohio

    state university papers, videos & more at http://interact.osu.edu/gesturedb