Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Gestural Query Specification: Querying without Keyboards

Arnab Nandi
September 04, 2014

Gestural Query Specification: Querying without Keyboards

Computing devices that use non-traditional methods to interact with data are becoming more popular than those that use keyboard-based interaction. Applications and query interfaces for such devices pose a fundamentally different set of workloads on the underlying databases. We characterize these differences and show how today's query representation and evaluation techniques are not well-suited to these new non-keyboard interaction modalities. We propose a new database architecture to address these issues, and demonstrate that it can be built using already existing components.

For more details, visit: http://interact.osu.edu/gesturedb/

Arnab Nandi

September 04, 2014
Tweet

More Decks by Arnab Nandi

Other Decks in Technology

Transcript

  1. Gestural Query Specification

    Arnab Nandi, Lilong Jiang, Michael Mandel

    Computer Science & Engineering
    The Ohio State University
    interactive data systems
    research group at the ohio state university
    papers, videos & more at http://interact.osu.edu/gesturedb

    View Slide

  2. What does a computer "
    look like?

    View Slide

  3. View Slide

  4. View Slide

  5. View Slide

  6. Computing and Keyboards
    550  
    351  
    0   100   200   300   400   500   600  
    Desktops, Laptops & Netbooks
    Smartphones & Tablets (1.5x)
    2011
    Worldwide Shipments, in Millions
    Canalys. Worldwide Smartphone and Client PC Shipment Estimates. 2012.

    View Slide

  7. Computing and Keyboards
    185  
    93.7  
    0   50   100   150   200  
    Desktops, Laptops & Netbooks
    Smartphones & Tablets (1.9x)
    Q4’11
    Worldwide Shipments, in Millions
    Canalys. Worldwide Smartphone and Client PC Shipment Estimates. 2012.

    View Slide

  8. View Slide

  9. There will soon be more
    computers without keyboards
    than with.

    View Slide

  10. SELECT *
    FROM SQL?!

    View Slide

  11. Need: "
    Databases that can meet
    these challenges &"
    serve such interactions

    View Slide

  12. Challenges
    •  Direct manipulation
    •  Interaction is continuous

    •  Infinite number of gestures
    •  Feedback is required to keep user oriented
    •  100-1000ms latencies

    View Slide

  13. Gesture
    •  A movement of the hand or body

    •  Represented as a set of time-series of
    points
    •  pi
    =
    •  t: timestamp
    •  l: locational information
    •  m: metadata information

    View Slide

  14. gestureDB
    •  Our attempt at building an 

    interactive database
    •  Principled approach


    •  Serve multiple modes 

    of interaction
    •  Touch
    •  Motion tracking
    •  Mouse
    •  Combinations

    View Slide

  15. Outline
    •  Challenges

    •  An Interactive Database: gestureDB
    •  Query Paradigm
    •  Query Language
    •  System Architecture


    •  Experiments and Evaluation

    View Slide

  16. Outline
    •  Challenges

    •  An Interactive Database: gestureDB
    •  Query Paradigm
    •  Query Language
    •  System Architecture


    •  Experiments and Evaluation

    View Slide

  17. Query Paradigm
    •  Query Intent
    •  Space of articulated queries
    •  User goes from vague to specific intent
    •  Intent Transition
    •  Query Context
    •  Direct Manipulation
    •  Session of multiple Query Intent Transitions

    View Slide

  18. Query Paradigm
    •  Query Intent
    •  Space of articulate queries
    •  User goes from vague to specific intent
    •  Intent Transition
    •  Query Context
    •  Direct Manipulation
    •  Session of multiple Query Intent Transitions
    Q1 Q2 Q3 Q4 Q5
    1.0  
    0  
       
       
           
       
       
       
       

    View Slide

  19. Query Paradigm
    •  Query Intent
    •  Space of articulate queries
    •  User goes from vague to specific intent
    •  Intent Transition
    •  Query Context
    •  Direct Manipulation
    •  Session of multiple Query Intent Transitions
    Q1 Q2 Q3 Q4 Q5
    1.0  
    0  
       
       
           
       
       
       
       

    View Slide

  20. Query Paradigm
    •  Query Intent
    •  Space of articulate queries
    •  User goes from vague to specific intent
    •  Intent Transition
    •  Query Context
    •  Direct Manipulation
    •  Session of multiple Query Intent Transitions
    Q1 Q2 Q3 Q4 Q5
    1.0  
    0  
       
       
           
       
       
       
       

    View Slide

  21. Query Paradigm
    •  Query Intent
    •  Space of articulate queries
    •  User goes from vague to specific intent
    •  Gesture Articulation
    •  Query Context
    •  Direct Manipulation
    •  Session of multiple Gesture Articulations

    View Slide

  22. Query Paradigm (contd.)
    •  Query Gesture
    •  Library of user gestures
    •  Mapped to parameterized query
    •  Intent Feedback
    •  During Gesture Articulation
    •  Constantly provide user with feedback


    • Query
    • Result

    View Slide

  23. Gesture Query Language
    Direct Manipulation · Closure and Composition · Feedback · Expressivity
    1
    "
    2
    "
    NYC"
    22
    "
    2
    "
    2
    "
    SF"
    31
    "
    3
    "
    4
    "
    ATL"
    3
    "
    Employee"
    id
    "
    projectId
    "
    loca:on"
    deptId
    "
    ""
    ""
    ""
    "" "" ""
    "" ""
    ""
    ""
    1
    "
    2
    "
    NYC"
    22
    "
    2
    "
    2
    "
    SF"
    31
    "
    3
    "
    4
    "
    ATL"
    3
    "
    Employee"
    id
    "
    projectId
    "
    loca:on"
    deptId
    "
    ""
    ""
    1
    "
    2
    "
    NYC"
    22
    "
    2
    "
    2
    "
    SF"
    31
    "
    3
    "
    4
    "
    ATL"
    3
    "
    Employee"
    id
    "
    projectId
    "
    loca:on"
    deptId
    "
    ""
    ""
    !"#$
    %$
    &%%$
    WHERE emp.age > 60
    distribution

    of values 

    in column!
    Employee
    !
    id
    !
    projectId
    !
    loca.on!
    deptId
    !
    !!
    !!
    !! !!
    !!
    !!
    !
    !
    !
    !
    Person
    !
    id
    !
    projId
    !
    loca.on!
    department
    !
    !!
    !!
    !! !!!!!!
    !
    ! !
    !
    Employee  
    id  
    projectId  
    loca9on  
    deptId  
    9tle  
    deptId  
       
       
    Employee  
    id  
    projectId  
    loca9on  
    deptId  
    9tle  
    loca9on  
       
       
    projectId  
       
       
    1  
    2  
    NYC  
    22  
    2  
    2  
    SF  
    31  
    3  
    4  
    ATL  
    3  
    Employee  
    id  
    projectId  
    loca9on  
    deptId  
    Employee  
    id  
    projectId  
    loca9on  
    deptId  
       
       
       
    1  
    2  
    NYC  
    22  
    2  
    2  
    SF  
    31  
    3  
    4  
    ATL  
    3  
    Employee  
    id  
    projectId  
    loca9on  
    deptId      
       
       
       
       
       
       
    Preview   Sort   Filter   Range  Filter  
    Union   Rearrange   Aggregate   Update  schema  &  update  value  
    Projec9on  
    Insert  
    MIN  
    MAX  
    AVG  
    SUM  

    View Slide

  24. Using Design: Exploratory Join
    •  Challenge
    •  M x N join
    …Why not use interaction & design?
    Employee
    id
    projectId
    location
    deptId
    Person
    id
    projId
    location
    department
    Employee
    id
    projectId
    location
    deptId
    Person
    id
    projId
    location
    department
    Employee
    id
    projectId
    location
    deptId
    Person
    id
    projId
    location
    department
       
    O(M * N) O(Min(M, N)) O(M * N)

    View Slide

  25. Intent Interpretation: Design
    Radial, Tabular and Arc Layouts

    The responsive arc layout allows all queries to be
    expressed unambiguously while still achieving readability.
     
    Employee
    id
    projectId
    location
    deptId
    title
    Project
    id
    projId
    managerId
    location
    name
    projectId
    location
    Employee
    id
    deptId
    managerId
    projId
    Project
    id
    location
    name
    title
    projId
    managerId
    Project
    id
    projectId
    location
    deptId
    title
    Employee
    Radial
    Vertical Arc

    View Slide

  26. Demo"
    gestureDB w/ iPad frontend
    Note: There can be 

    many different frontends

    Using Database Explorer as an example
    (Could have built a “social media explorer” too!)

     

    View Slide

  27. gestureDB Architecture
    Feedback    
    Genera9on  
    Data  
    Store  
    User  
    Context  
    Cache  
    User  Interface  
    Context  
    Cache  
    Network  
    projectId*
    loca-on*
    !Employee!
    **
    **
    ** **
    **
    **
    **
    **
    id*
    deptId*
    parentProjectId-
    supervisorID-
    Project(
    id-
    managerId-
    --
    --
    --------
    -- -- Gesture  
    Query  
    Language  
    Intent    
    Interpreta9on  
    Interac9ve  
    Query  
    Execu9on  
    John   SF   Audit   Tax   Fin   Joe  
    John   LA   Review   Acc   Fin   Jen  
    ⋈  
    1:2  
    n=1200  

    View Slide

  28. Classifier
    d combine them linearly in the argument of an
    onential. Mathematically, the goodness g
    (
    q
    )
    of a
    ential query q with feature values fi(
    q
    )
    is
    g
    (
    q
    ) = exp(
    X
    i
    ifi(
    q
    ))
    .
    eatures can be binary or real-valued, with 0 being th
    ue of an uninformative feature. Parameters of the
    ssifier, i
    , can be learned across a collection of record
    ning gestures to tune the quality of the classifier. Fo
    liminary experiments in Section , these parameters a
    manually, and tuning using training data is scope fo
    ther improvement in quality. New queries can be
    •  g(q) = “Goodness” of a query, probability distribution of SQL
    queries

    •  fi
    = features from
    •  Relative Position information (gesture + objects)
    •  Schema compatibility
    •  Data compatibility (distribution)
    •  λ learned from training

    View Slide

  29. Classifier Strategies
    •  Precompute Summaries (send to frontend)
    •  Data Distribution Comparisons
    •  Schema Compatibilities
    •  Skip Computations
    •  When -∞ is encountered
    •  Memoize Everything (distance calculations)
    •  Jitter

    View Slide

  30. Schema + Data helps"
    Intent Interpretation
    “How quickly does the system correctly interpret intent?”

    1.0 = correct at the start of the gesture articulation

    0.0
    0.1
    0.2
    0.3
    0.4
    0.5
    0.6
    0.7
    0.8
    0.9
    1.0
    Join1A Join1R Join2A Join2R Union1 Union2
    Anticipation
    Proximity
    Proximity + Schema
    Proximity + Schema + Data

    View Slide

  31. Outline
    •  Challenges

    •  An Interactive Database: gestureDB
    •  Experiments and Evaluation
    •  Completion Time
    •  Discoverability
    •  Survey
    •  Classifier Performance

    View Slide

  32. is gestural querying of data "
    a good idea?

    View Slide

  33. User Studies
    •  Running on iPad 3rd Gen(16GB)
    •  Comparisons
    •  Visual Query Builder vs Console vs GestureDB
    •  30 users, within-subjects
    •  15 naïve, 15 expert


    View Slide

  34. User Study: Avoiding Biases
    •  Fatigue
    •  Learning Effects
    •  Carryover Effects
    •  Consistency

    View Slide

  35. Datasets
    •  Chinook
    •  11 tables, 64 columns, 15K tuples, 1MB
    •  http://chinookdatabase.codeplex.com

    •  TPC-H
    •  9 tables, 76 columns, 86K tuples, 15MB
    •  http://www.tpc.org/tpch

    View Slide

  36. Completion Time Task
    •  Preview
    Preview the table Invoice
    •  Filter
    Filter table Invoice according the BillingCountry value of the tuple whose InvoiceId is 5
    •  Sort
    Sort table Album in descending order on AlbumId
    •  Group by & Aggregate
    For table PART, group by the attribute P_TYPE and aggregate on the attribute P_SIZE with max function
    •  Join
    Invoice Join InvoiceLine on Invoice.InvoiceId = InvoiceLine.InvoiceId
    •  Union
    Album1 Union Album2

    View Slide

  37. Gestural Querying is Faster
    0"
    10"
    20"
    30"
    40"
    50"
    60"
    70"
    Preview" Filter" Sort" Group"by"&"
    Aggregate"
    Join" Union"
    Predic'on*(s)*
    ConsoleCbased"SQL"
    Visual"Query"Builder"
    GestureQuery"

    View Slide

  38. 80
    108
    111
    128
    169
    178
    196
    218
    256
    294
    305
    329
    340
    384
    446
    447
    476
    514
    550
    563
    616
    679
    745
    900
    900
    900
    900
    900
    900
    900
    900
    893
    596
    900
    900
    313
    731
    268
    455
    240
    900
    198
    525
    900
    900
    900
    402
    900
    185
    781
    900
    654
    240
    205
    58
    429
    900
    900
    900
    900
    0
    100
    200
    300
    400
    500
    600
    700
    800
    900
    1000
    Discoverability(s)
    Users GestureQuery Visual Query Builder
    Gestural Queries are
    Discoverable
    •  14 / 30 fail with Visual Query Builder
    •  21 / 30 perform better with GestureQuery

    View Slide

  39. Users prefer Gestural Querying
    learn (learnability) for each query type. Systems are CONSOLE-
    BASED SQL, VISUAL QUERY BUILDER (VQB), and GES-
    TUREQUERY (GQ – our system). For users claiming equal
    preference for multiple systems, their votes were split equally
    amongst the systems.
    Survey Action Console VQB GQ
    Usability PREVIEW 1 5 24
    Usability FILTER 5 1.5 23.5
    Usability SORT 0 2 28
    Usability AGGREGATE 5 4 21
    Usability JOIN 3 15 12
    Usability UNION 2 0 28
    Learnability PREVIEW 1.66 2.16 26.16
    Learnability FILTER 5 4.5 20.5
    Learnability SORT 1.33 1.33 27.33
    Learnability AGGREGATE 2 10.5 17.5
    Learnability JOIN 1 15 14
    Learnability UNION 3 0 27
    Table 3: Interactivity: percentage of touch classifications that
    take less than 33 ms.
    Figu
    by
    3
    sific
    sent
    ture
    low
    for
    5.3
    A
    clas
    part
    JOI
    Join
    the

    View Slide

  40. Classifier Performance
    •  180 gestures (from studies)
    •  Performance: Almost always < 33ms
    •  > 95%
    •  30Hz

    •  Interactivity

    View Slide

  41. Summary
    •  Challenges


    •  An Interactive Database: gestureDB
    •  Experiments and Evaluation

    View Slide

  42. Summary
    •  Challenges

    •  An Interactive Database: gestureDB
    •  Query Paradigm
    •  Gesture Query Language
    •  System Architecture


    •  Experiments and Evaluation

    View Slide

  43. Summary
    •  Challenges

    •  An Interactive Database: gestureDB
    •  Experiments and Evaluation
    •  Completion Time
    •  Discoverability
    •  Survey
    •  Classifier Performance

    View Slide

  44. Thank you!
    interactive data systems
    research group at the ohio state university
    papers, videos & more at http://interact.osu.edu/gesturedb

    View Slide