Gestural Query Specification: Querying without Keyboards

Computing devices that use non-traditional methods to interact with data are becoming more popular than those that use keyboard-based interaction. Applications and query interfaces for such devices pose a fundamentally different set of workloads on the underlying databases. We characterize these differences and show how today's query representation and evaluation techniques are not well-suited to these new non-keyboard interaction modalities. We propose a new database architecture to address these issues, and demonstrate that it can be built using already existing components.

For more details, visit: http://interact.osu.edu/gesturedb/

Transcript

1. Gestural Query Specification
   

   Arnab Nandi, Lilong Jiang, Michael Mandel

   Computer Science & Engineering
   The Ohio State University
   interactive data systems
   research group at the ohio state university
   papers, videos & more at http://interact.osu.edu/gesturedb
   
   

   View Slide

2. What does a computer "
   look like?
   

   View Slide

3. View Slide

4. View Slide

5. View Slide

6. Computing and Keyboards
   550  
   351  
   0   100   200   300   400   500   600  
   Desktops, Laptops & Netbooks
   Smartphones & Tablets (1.5x)
   2011
   Worldwide Shipments, in Millions
   Canalys. Worldwide Smartphone and Client PC Shipment Estimates. 2012.
   

   View Slide

7. Computing and Keyboards
   185  
   93.7  
   0   50   100   150   200  
   Desktops, Laptops & Netbooks
   Smartphones & Tablets (1.9x)
   Q4’11
   Worldwide Shipments, in Millions
   Canalys. Worldwide Smartphone and Client PC Shipment Estimates. 2012.
   

   View Slide

8. View Slide

9. There will soon be more
   computers without keyboards
   than with.
   

   View Slide

10. SELECT *
    FROM SQL?!
    

    View Slide

11. Need: "
    Databases that can meet
    these challenges &"
    serve such interactions
    

    View Slide

12. Challenges
    •  Direct manipulation
    •  Interaction is continuous
    
    •  Infinite number of gestures
    •  Feedback is required to keep user oriented
    •  100-1000ms latencies
    
    

    View Slide

13. Gesture
    •  A movement of the hand or body
    
    •  Represented as a set of time-series of
    points
    •  pi
    =
    •  t: timestamp
    •  l: locational information
    •  m: metadata information
    

    View Slide

14. gestureDB
    •  Our attempt at building an 

    interactive database
    •  Principled approach

    
    •  Serve multiple modes 

    of interaction
    •  Touch
    •  Motion tracking
    •  Mouse
    •  Combinations
    
    

    View Slide

15. Outline
    •  Challenges
    
    •  An Interactive Database: gestureDB
    •  Query Paradigm
    •  Query Language
    •  System Architecture

    
    •  Experiments and Evaluation
    

    View Slide

16. Outline
    •  Challenges
    
    •  An Interactive Database: gestureDB
    •  Query Paradigm
    •  Query Language
    •  System Architecture

    
    •  Experiments and Evaluation
    
    

    View Slide

17. Query Paradigm
    •  Query Intent
    •  Space of articulated queries
    •  User goes from vague to specific intent
    •  Intent Transition
    •  Query Context
    •  Direct Manipulation
    •  Session of multiple Query Intent Transitions
    

    View Slide

18. Query Paradigm
    •  Query Intent
    •  Space of articulate queries
    •  User goes from vague to specific intent
    •  Intent Transition
    •  Query Context
    •  Direct Manipulation
    •  Session of multiple Query Intent Transitions
    Q1 Q2 Q3 Q4 Q5
    1.0  
    0  
       
       
           
       
       
       
       
    

    View Slide

19. Query Paradigm
    •  Query Intent
    •  Space of articulate queries
    •  User goes from vague to specific intent
    •  Intent Transition
    •  Query Context
    •  Direct Manipulation
    •  Session of multiple Query Intent Transitions
    Q1 Q2 Q3 Q4 Q5
    1.0  
    0  
       
       
           
       
       
       
       
    

    View Slide

20. Query Paradigm
    •  Query Intent
    •  Space of articulate queries
    •  User goes from vague to specific intent
    •  Intent Transition
    •  Query Context
    •  Direct Manipulation
    •  Session of multiple Query Intent Transitions
    Q1 Q2 Q3 Q4 Q5
    1.0  
    0  
       
       
           
       
       
       
       
    

    View Slide

21. Query Paradigm
    •  Query Intent
    •  Space of articulate queries
    •  User goes from vague to specific intent
    •  Gesture Articulation
    •  Query Context
    •  Direct Manipulation
    •  Session of multiple Gesture Articulations
    

    View Slide

22. Query Paradigm (contd.)
    •  Query Gesture
    •  Library of user gestures
    •  Mapped to parameterized query
    •  Intent Feedback
    •  During Gesture Articulation
    •  Constantly provide user with feedback
    
    
    • Query
    • Result
    
    

    View Slide

23. Gesture Query Language
    Direct Manipulation · Closure and Composition · Feedback · Expressivity
    1
    "
    2
    "
    NYC"
    22
    "
    2
    "
    2
    "
    SF"
    31
    "
    3
    "
    4
    "
    ATL"
    3
    "
    Employee"
    id
    "
    projectId
    "
    loca:on"
    deptId
    "
    ""
    ""
    ""
    "" "" ""
    "" ""
    ""
    ""
    1
    "
    2
    "
    NYC"
    22
    "
    2
    "
    2
    "
    SF"
    31
    "
    3
    "
    4
    "
    ATL"
    3
    "
    Employee"
    id
    "
    projectId
    "
    loca:on"
    deptId
    "
    ""
    ""
    1
    "
    2
    "
    NYC"
    22
    "
    2
    "
    2
    "
    SF"
    31
    "
    3
    "
    4
    "
    ATL"
    3
    "
    Employee"
    id
    "
    projectId
    "
    loca:on"
    deptId
    "
    ""
    ""
    !"#$
    %$
    &%%$
    WHERE emp.age > 60
    distribution

    of values 

    in column!
    Employee
    !
    id
    !
    projectId
    !
    loca.on!
    deptId
    !
    !!
    !!
    !! !!
    !!
    !!
    !
    !
    !
    !
    Person
    !
    id
    !
    projId
    !
    loca.on!
    department
    !
    !!
    !!
    !! !!!!!!
    !
    ! !
    !
    Employee  
    id  
    projectId  
    loca9on  
    deptId  
    9tle  
    deptId  
       
       
    Employee  
    id  
    projectId  
    loca9on  
    deptId  
    9tle  
    loca9on  
       
       
    projectId  
       
       
    1  
    2  
    NYC  
    22  
    2  
    2  
    SF  
    31  
    3  
    4  
    ATL  
    3  
    Employee  
    id  
    projectId  
    loca9on  
    deptId  
    Employee  
    id  
    projectId  
    loca9on  
    deptId  
       
       
       
    1  
    2  
    NYC  
    22  
    2  
    2  
    SF  
    31  
    3  
    4  
    ATL  
    3  
    Employee  
    id  
    projectId  
    loca9on  
    deptId      
       
       
       
       
       
       
    Preview   Sort   Filter   Range  Filter  
    Union   Rearrange   Aggregate   Update  schema  &  update  value  
    Projec9on  
    Insert  
    MIN  
    MAX  
    AVG  
    SUM  
    

    View Slide

24. Using Design: Exploratory Join
    •  Challenge
    •  M x N join
    …Why not use interaction & design?
    Employee
    id
    projectId
    location
    deptId
    Person
    id
    projId
    location
    department
    Employee
    id
    projectId
    location
    deptId
    Person
    id
    projId
    location
    department
    Employee
    id
    projectId
    location
    deptId
    Person
    id
    projId
    location
    department
       
    O(M * N) O(Min(M, N)) O(M * N)
    

    View Slide

25. Intent Interpretation: Design
    Radial, Tabular and Arc Layouts
    

    The responsive arc layout allows all queries to be
    expressed unambiguously while still achieving readability.
     
    Employee
    id
    projectId
    location
    deptId
    title
    Project
    id
    projId
    managerId
    location
    name
    projectId
    location
    Employee
    id
    deptId
    managerId
    projId
    Project
    id
    location
    name
    title
    projId
    managerId
    Project
    id
    projectId
    location
    deptId
    title
    Employee
    Radial
    Vertical Arc
    

    View Slide

26. Demo"
    gestureDB w/ iPad frontend
    Note: There can be 

    many different frontends
    
    Using Database Explorer as an example
    (Could have built a “social media explorer” too!)
    

     
    

    View Slide

27. gestureDB Architecture
    Feedback    
    Genera9on  
    Data  
    Store  
    User  
    Context  
    Cache  
    User  Interface  
    Context  
    Cache  
    Network  
    projectId*
    loca-on*
    !Employee!
    **
    **
    ** **
    **
    **
    **
    **
    id*
    deptId*
    parentProjectId-
    supervisorID-
    Project(
    id-
    managerId-
    --
    --
    --------
    -- -- Gesture  
    Query  
    Language  
    Intent    
    Interpreta9on  
    Interac9ve  
    Query  
    Execu9on  
    John   SF   Audit   Tax   Fin   Joe  
    John   LA   Review   Acc   Fin   Jen  
    ⋈  
    1:2  
    n=1200  
    

    View Slide

28. Classifier
    d combine them linearly in the argument of an
    onential. Mathematically, the goodness g
    (
    q
    )
    of a
    ential query q with feature values fi(
    q
    )
    is
    g
    (
    q
    ) = exp(
    X
    i
    ifi(
    q
    ))
    .
    eatures can be binary or real-valued, with 0 being th
    ue of an uninformative feature. Parameters of the
    ssifier, i
    , can be learned across a collection of record
    ning gestures to tune the quality of the classifier. Fo
    liminary experiments in Section , these parameters a
    manually, and tuning using training data is scope fo
    ther improvement in quality. New queries can be
    •  g(q) = “Goodness” of a query, probability distribution of SQL
    queries
    
    •  fi
    = features from
    •  Relative Position information (gesture + objects)
    •  Schema compatibility
    •  Data compatibility (distribution)
    •  λ learned from training
    

    View Slide

29. Classifier Strategies
    •  Precompute Summaries (send to frontend)
    •  Data Distribution Comparisons
    •  Schema Compatibilities
    •  Skip Computations
    •  When -∞ is encountered
    •  Memoize Everything (distance calculations)
    •  Jitter
    

    View Slide

30. Schema + Data helps"
    Intent Interpretation
    “How quickly does the system correctly interpret intent?”
    
    1.0 = correct at the start of the gesture articulation
    
    0.0
    0.1
    0.2
    0.3
    0.4
    0.5
    0.6
    0.7
    0.8
    0.9
    1.0
    Join1A Join1R Join2A Join2R Union1 Union2
    Anticipation
    Proximity
    Proximity + Schema
    Proximity + Schema + Data
    

    View Slide

31. Outline
    •  Challenges
    
    •  An Interactive Database: gestureDB
    •  Experiments and Evaluation
    •  Completion Time
    •  Discoverability
    •  Survey
    •  Classifier Performance
    
    

    View Slide

32. is gestural querying of data "
    a good idea?
    

    View Slide

33. User Studies
    •  Running on iPad 3rd Gen(16GB)
    •  Comparisons
    •  Visual Query Builder vs Console vs GestureDB
    •  30 users, within-subjects
    •  15 naïve, 15 expert
    
    
    

    View Slide

34. User Study: Avoiding Biases
    •  Fatigue
    •  Learning Effects
    •  Carryover Effects
    •  Consistency
    

    View Slide

35. Datasets
    •  Chinook
    •  11 tables, 64 columns, 15K tuples, 1MB
    •  http://chinookdatabase.codeplex.com
    
    •  TPC-H
    •  9 tables, 76 columns, 86K tuples, 15MB
    •  http://www.tpc.org/tpch
    

    View Slide

36. Completion Time Task
    •  Preview
    Preview the table Invoice
    •  Filter
    Filter table Invoice according the BillingCountry value of the tuple whose InvoiceId is 5
    •  Sort
    Sort table Album in descending order on AlbumId
    •  Group by & Aggregate
    For table PART, group by the attribute P_TYPE and aggregate on the attribute P_SIZE with max function
    •  Join
    Invoice Join InvoiceLine on Invoice.InvoiceId = InvoiceLine.InvoiceId
    •  Union
    Album1 Union Album2
    
    

    View Slide

37. Gestural Querying is Faster
    0"
    10"
    20"
    30"
    40"
    50"
    60"
    70"
    Preview" Filter" Sort" Group"by"&"
    Aggregate"
    Join" Union"
    Predic'on*(s)*
    ConsoleCbased"SQL"
    Visual"Query"Builder"
    GestureQuery"
    

    View Slide

38. 80
    108
    111
    128
    169
    178
    196
    218
    256
    294
    305
    329
    340
    384
    446
    447
    476
    514
    550
    563
    616
    679
    745
    900
    900
    900
    900
    900
    900
    900
    900
    893
    596
    900
    900
    313
    731
    268
    455
    240
    900
    198
    525
    900
    900
    900
    402
    900
    185
    781
    900
    654
    240
    205
    58
    429
    900
    900
    900
    900
    0
    100
    200
    300
    400
    500
    600
    700
    800
    900
    1000
    Discoverability(s)
    Users GestureQuery Visual Query Builder
    Gestural Queries are
    Discoverable
    •  14 / 30 fail with Visual Query Builder
    •  21 / 30 perform better with GestureQuery
    

    View Slide

39. Users prefer Gestural Querying
    learn (learnability) for each query type. Systems are CONSOLE-
    BASED SQL, VISUAL QUERY BUILDER (VQB), and GES-
    TUREQUERY (GQ – our system). For users claiming equal
    preference for multiple systems, their votes were split equally
    amongst the systems.
    Survey Action Console VQB GQ
    Usability PREVIEW 1 5 24
    Usability FILTER 5 1.5 23.5
    Usability SORT 0 2 28
    Usability AGGREGATE 5 4 21
    Usability JOIN 3 15 12
    Usability UNION 2 0 28
    Learnability PREVIEW 1.66 2.16 26.16
    Learnability FILTER 5 4.5 20.5
    Learnability SORT 1.33 1.33 27.33
    Learnability AGGREGATE 2 10.5 17.5
    Learnability JOIN 1 15 14
    Learnability UNION 3 0 27
    Table 3: Interactivity: percentage of touch classifications that
    take less than 33 ms.
    Figu
    by
    3
    sific
    sent
    ture
    low
    for
    5.3
    A
    clas
    part
    JOI
    Join
    the
    

    View Slide

40. Classifier Performance
    •  180 gestures (from studies)
    •  Performance: Almost always < 33ms
    •  > 95%
    •  30Hz
    
    •  Interactivity
    

    View Slide

41. Summary
    •  Challenges

    
    •  An Interactive Database: gestureDB
    •  Experiments and Evaluation
    

    View Slide

42. Summary
    •  Challenges
    
    •  An Interactive Database: gestureDB
    •  Query Paradigm
    •  Gesture Query Language
    •  System Architecture

    
    •  Experiments and Evaluation
    
    

    View Slide

43. Summary
    •  Challenges
    
    •  An Interactive Database: gestureDB
    •  Experiments and Evaluation
    •  Completion Time
    •  Discoverability
    •  Survey
    •  Classifier Performance
    

    View Slide

44. Thank you!
    interactive data systems
    research group at the ohio state university
    papers, videos & more at http://interact.osu.edu/gesturedb
    

    View Slide