Share
Embed
Gestural Query Specification: Querying without Keyboards
Computing devices that use non-traditional methods to interact with data are becoming more popular than those that use keyboard-based interaction. Applications and query interfaces for such devices pose a fundamentally different set of workloads on the underlying databases. We characterize these differences and show how today's query representation and evaluation techniques are not well-suited to these new non-keyboard interaction modalities. We propose a new database architecture to address these issues, and demonstrate that it can be built using already existing components.
For more details, visit: http://interact.osu.edu/gesturedb/
Arnab Nandi
September 04, 2014
Transcript
44 slides
-
Gestural Query Specification Arnab Nandi, Lilong Jiang, Michael Mandel
Computer Science & Engineering The Ohio State University interactive data systems research group at the ohio state university papers, videos & more at http://interact.osu.edu/gesturedb -
Computing and Keyboards 550 351 0 100
200 300 400 500 600 Desktops, Laptops & Netbooks Smartphones & Tablets (1.5x) 2011 Worldwide Shipments, in Millions Canalys. Worldwide Smartphone and Client PC Shipment Estimates. 2012. -
Computing and Keyboards 185 93.7 0 50
100 150 200 Desktops, Laptops & Netbooks Smartphones & Tablets (1.9x) Q4’11 Worldwide Shipments, in Millions Canalys. Worldwide Smartphone and Client PC Shipment Estimates. 2012. -
Need: " Databases that can meet these challenges &" serve
such interactions -
Challenges • Direct manipulation • Interaction is continuous • Infinite
number of gestures • Feedback is required to keep user oriented • 100-1000ms latencies -
Gesture • A movement of the hand or body •
Represented as a set of time-series of points • pi = <t, l, m> • t: timestamp • l: locational information • m: metadata information -
gestureDB • Our attempt at building an interactive database
• Principled approach • Serve multiple modes of interaction • Touch • Motion tracking • Mouse • Combinations -
Outline • Challenges • An Interactive Database: gestureDB • Query
Paradigm • Query Language • System Architecture • Experiments and Evaluation -
Outline • Challenges • An Interactive Database: gestureDB • Query
Paradigm • Query Language • System Architecture • Experiments and Evaluation -
Query Paradigm • Query Intent • Space of articulated queries
• User goes from vague to specific intent • Intent Transition • Query Context • Direct Manipulation • Session of multiple Query Intent Transitions -
Query Paradigm • Query Intent • Space of articulate queries
• User goes from vague to specific intent • Intent Transition • Query Context • Direct Manipulation • Session of multiple Query Intent Transitions Q1 Q2 Q3 Q4 Q5 1.0 0 -
Query Paradigm • Query Intent • Space of articulate queries
• User goes from vague to specific intent • Intent Transition • Query Context • Direct Manipulation • Session of multiple Query Intent Transitions Q1 Q2 Q3 Q4 Q5 1.0 0 -
Query Paradigm • Query Intent • Space of articulate queries
• User goes from vague to specific intent • Intent Transition • Query Context • Direct Manipulation • Session of multiple Query Intent Transitions Q1 Q2 Q3 Q4 Q5 1.0 0 -
Query Paradigm • Query Intent • Space of articulate queries
• User goes from vague to specific intent • Gesture Articulation • Query Context • Direct Manipulation • Session of multiple Gesture Articulations -
Query Paradigm (contd.) • Query Gesture • Library of user
gestures • Mapped to parameterized query • Intent Feedback • During Gesture Articulation • Constantly provide user with feedback • Query • Result -
Gesture Query Language Direct Manipulation · Closure and Composition ·
Feedback · Expressivity 1 " 2 " NYC" 22 " 2 " 2 " SF" 31 " 3 " 4 " ATL" 3 " Employee" id " projectId " loca:on" deptId " "" "" "" "" "" "" "" "" "" "" 1 " 2 " NYC" 22 " 2 " 2 " SF" 31 " 3 " 4 " ATL" 3 " Employee" id " projectId " loca:on" deptId " "" "" 1 " 2 " NYC" 22 " 2 " 2 " SF" 31 " 3 " 4 " ATL" 3 " Employee" id " projectId " loca:on" deptId " "" "" !"#$ %$ &%%$ WHERE emp.age > 60 distribution of values in column! Employee ! id ! projectId ! loca.on! deptId ! !! !! !! !! !! !! ! ! ! ! Person ! id ! projId ! loca.on! department ! !! !! !! !!!!!! ! ! ! ! Employee id projectId loca9on deptId 9tle deptId Employee id projectId loca9on deptId 9tle loca9on projectId 1 2 NYC 22 2 2 SF 31 3 4 ATL 3 Employee id projectId loca9on deptId Employee id projectId loca9on deptId 1 2 NYC 22 2 2 SF 31 3 4 ATL 3 Employee id projectId loca9on deptId Preview Sort Filter Range Filter Union Rearrange Aggregate Update schema & update value Projec9on Insert MIN MAX AVG SUM -
Using Design: Exploratory Join • Challenge • M x N
join …Why not use interaction & design? Employee id projectId location deptId Person id projId location department Employee id projectId location deptId Person id projId location department Employee id projectId location deptId Person id projId location department O(M * N) O(Min(M, N)) O(M * N) -
Intent Interpretation: Design Radial, Tabular and Arc Layouts The
responsive arc layout allows all queries to be expressed unambiguously while still achieving readability. Employee id projectId location deptId title Project id projId managerId location name projectId location Employee id deptId managerId projId Project id location name title projId managerId Project id projectId location deptId title Employee Radial Vertical Arc -
Demo" gestureDB w/ iPad frontend Note: There can be
many different frontends Using Database Explorer as an example (Could have built a “social media explorer” too!) -
gestureDB Architecture Feedback Genera9on Data Store
User Context Cache User Interface Context Cache Network projectId* loca-on* !Employee! ** ** ** ** ** ** ** ** id* deptId* parentProjectId- supervisorID- Project( id- managerId- -- -- -------- -- -- Gesture Query Language Intent Interpreta9on Interac9ve Query Execu9on John SF Audit Tax Fin Joe John LA Review Acc Fin Jen ⋈ 1:2 n=1200 -
Classifier d combine them linearly in the argument of an
onential. Mathematically, the goodness g ( q ) of a ential query q with feature values fi( q ) is g ( q ) = exp( X i ifi( q )) . eatures can be binary or real-valued, with 0 being th ue of an uninformative feature. Parameters of the ssifier, i , can be learned across a collection of record ning gestures to tune the quality of the classifier. Fo liminary experiments in Section , these parameters a manually, and tuning using training data is scope fo ther improvement in quality. New queries can be • g(q) = “Goodness” of a query, probability distribution of SQL queries • fi = features from • Relative Position information (gesture + objects) • Schema compatibility • Data compatibility (distribution) • λ learned from training -
Classifier Strategies • Precompute Summaries (send to frontend) • Data
Distribution Comparisons • Schema Compatibilities • Skip Computations • When -∞ is encountered • Memoize Everything (distance calculations) • Jitter -
Schema + Data helps" Intent Interpretation “How quickly does the
system correctly interpret intent?” 1.0 = correct at the start of the gesture articulation 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Join1A Join1R Join2A Join2R Union1 Union2 Anticipation Proximity Proximity + Schema Proximity + Schema + Data -
Outline • Challenges • An Interactive Database: gestureDB • Experiments
and Evaluation • Completion Time • Discoverability • Survey • Classifier Performance -
User Studies • Running on iPad 3rd Gen(16GB) • Comparisons
• Visual Query Builder vs Console vs GestureDB • 30 users, within-subjects • 15 naïve, 15 expert -
User Study: Avoiding Biases • Fatigue • Learning Effects •
Carryover Effects • Consistency -
Datasets • Chinook • 11 tables, 64 columns, 15K tuples,
1MB • http://chinookdatabase.codeplex.com • TPC-H • 9 tables, 76 columns, 86K tuples, 15MB • http://www.tpc.org/tpch -
Completion Time Task • Preview Preview the table Invoice •
Filter Filter table Invoice according the BillingCountry value of the tuple whose InvoiceId is 5 • Sort Sort table Album in descending order on AlbumId • Group by & Aggregate For table PART, group by the attribute P_TYPE and aggregate on the attribute P_SIZE with max function • Join Invoice Join InvoiceLine on Invoice.InvoiceId = InvoiceLine.InvoiceId • Union Album1 Union Album2 -
Gestural Querying is Faster 0" 10" 20" 30" 40" 50"
60" 70" Preview" Filter" Sort" Group"by"&" Aggregate" Join" Union" Predic'on*(s)* ConsoleCbased"SQL" Visual"Query"Builder" GestureQuery" -
80 108 111 128 169 178 196 218 256 294
305 329 340 384 446 447 476 514 550 563 616 679 745 900 900 900 900 900 900 900 900 893 596 900 900 313 731 268 455 240 900 198 525 900 900 900 402 900 185 781 900 654 240 205 58 429 900 900 900 900 0 100 200 300 400 500 600 700 800 900 1000 Discoverability(s) Users GestureQuery Visual Query Builder Gestural Queries are Discoverable • 14 / 30 fail with Visual Query Builder • 21 / 30 perform better with GestureQuery -
Users prefer Gestural Querying learn (learnability) for each query type.
Systems are CONSOLE- BASED SQL, VISUAL QUERY BUILDER (VQB), and GES- TUREQUERY (GQ – our system). For users claiming equal preference for multiple systems, their votes were split equally amongst the systems. Survey Action Console VQB GQ Usability PREVIEW 1 5 24 Usability FILTER 5 1.5 23.5 Usability SORT 0 2 28 Usability AGGREGATE 5 4 21 Usability JOIN 3 15 12 Usability UNION 2 0 28 Learnability PREVIEW 1.66 2.16 26.16 Learnability FILTER 5 4.5 20.5 Learnability SORT 1.33 1.33 27.33 Learnability AGGREGATE 2 10.5 17.5 Learnability JOIN 1 15 14 Learnability UNION 3 0 27 Table 3: Interactivity: percentage of touch classifications that take less than 33 ms. Figu by 3 sific sent ture low for 5.3 A clas part JOI Join the -
Classifier Performance • 180 gestures (from studies) • Performance: Almost
always < 33ms • > 95% • 30Hz • Interactivity -
Summary • Challenges • An Interactive Database: gestureDB • Query
Paradigm • Gesture Query Language • System Architecture • Experiments and Evaluation -
Summary • Challenges • An Interactive Database: gestureDB • Experiments
and Evaluation • Completion Time • Discoverability • Survey • Classifier Performance -
Thank you! interactive data systems research group at the ohio
state university papers, videos & more at http://interact.osu.edu/gesturedb