ARQuery: Hallucinating Analytics over Real-World Data using Augmented Reality

Work by Codi Burley, Arnab Nandi [ https://interact.osu.edu ]

ARQuery is a query platform that utilizes augmented reality to enable
querying over real-world data. We provide an interaction and visualization grammar that is designed to augment real-world data, and a performant framework that enables query exploration in real-time.

More at: http://cidrdb.org/cidr2019/papers/p93-burley-cidr19.pdf

Transcript

1. ARQuery: Hallucinating Analytics using Augmented Reality Codi Burley Arnab Nandi
2. Motivation: Data in the Real World • Paper • Ledgers • Menus • Labels on Physical Objects • Grocery Store Aisles • Electronic Displays • Third Party Screens • Device Meters
3. Motivation: Querying in the Real World • Airport Gate: “Rebooking to the earliest flight to Monterey” • Other examples: • Subsetting restaurant menu for dietary restrictions • Noting allergies from a prescription medicine bottle • Finding the cheapest crunchy peanut butter in a large grocery aisle
4. Challenge: Hard to Query the Real-World • Involves importing data into a database or spreadsheet • Lose context • Hard to do in ad-hoc, unprepared settings
5. Proposal: Augmented Reality • Augmented Reality • Overlay camera scene with UI & information • Made popular by games • Now becoming mainstream & commodity • Poses multiple challenges • How does one represent query results in augmented reality? • How do you interact with augmented information?
6. Our Contributions • An interaction and visualization grammar • and encoding of query results in augmented reality • A performant framework for querying structured data in AR • End-to-end extraction, mapping, querying, and overlaying • Experimental Evaluations
7. Outline • Motivation & Challenges • Contributions • Query Grammar & Visual Encoding • Demo • Evaluation • Conclusions & Future Work
8. Query Grammar & Visual Encoding Query Operations Visual Encoding Selection Occlusion Projection Occlusion Sort Color Gradient Group Divergent Colors Aggregate Virtual Columns/Legends • Real-world ≠ screens • Static, cannot move around • Important to maintain context and placement • Need to maintain query state
9. Virtual World: It’s okay to manipulate UI Sort Operation: Changes are ok Virtual World: Spreadsheet UI
10. Blending virtual and real-world data: Maintaining Context Changes are too Too disorienting Real-world context
11. Query Grammar & Visual Encoding Query Operations Visual Encoding Selection Occlusion Projection Occlusion Sort Color Gradient Group Divergent Colors Aggregate Virtual Columns/Legends • Real-world ≠ screens • Static, cannot move around • Important to maintain context and placement • Need to maintain query state
12. Augmented Highlights, Virtual Results • Augmented Layer • Augmented Highlights • Virtual Results INTERACTIVE QUERYING AUGMENTED LAYER REAL-WORLD DATA LAYER VIRTUAL RESULTS AUGMENTED HIGHLIGHTS
13. Query Walkthrough SELECT * FROM Products SELECT "Product Name" F Product Name Supplier ID Unit Price Product Name Supplie Chais 1 10 boxes x 20 bags 18 Chais 1 Ikura 4 12 - 200 ml jars 31 Ikura 4 Chang 1 24 - 12 oz bottles 19 Chang 1 Syrup 1 12 - 550 ml bottles 10 Syrup 1 Kobe Niku 4 18 - 500 g pkgs. 97 Kobe Niku 4 Konbu 6 2 kg box 6 Konbu 6 Tofu 6 40 - 100 g pkgs. 23.25 Tofu 6 Geitost 15 500 g 2.5 Geitost 15 SELECT * FROM Products ORDER BY Price SELECT * FROM P Product Name Supplier ID Unit Price Product Name Supplie Chais 1 10 boxes x 20 bags 18 Chais 1
14. Selection & Projection using Occlusion SELECT "Product Name" FROM Products WHERE Price<15 Price Product Name Supplier ID Unit Price 18 Chais 1 10 boxes x 20 bags 18 31 Ikura 4 12 - 200 ml jars 31 19 Chang 1 24 - 12 oz bottles 19 10 Syrup 1 12 - 550 ml bottles 10 97 Kobe Niku 4 18 - 500 g pkgs. 97 6 Konbu 6 2 kg box 6 23.25 Tofu 6 40 - 100 g pkgs. 23.25 2.5 Geitost 15 500 g 2.5 rice SELECT * FROM Products GROUP BY "Supplier ID" Price Product Name Supplier ID Unit Price
15. ORDER BY using Color Gradients Chais 1 10 boxes x 20 bags 18 Chais Ikura 4 12 - 200 ml jars 31 Ikura Chang 1 24 - 12 oz bottles 19 Chang Syrup 1 12 - 550 ml bottles 10 Syrup Kobe Niku 4 18 - 500 g pkgs. 97 Kobe Niku Konbu 6 2 kg box 6 Konbu Tofu 6 40 - 100 g pkgs. 23.25 Tofu Geitost 15 500 g 2.5 Geitost SELECT * FROM Products ORDER BY Price SELECT * F Product Name Supplier ID Unit Price Product Name Chais 1 10 boxes x 20 bags 18 Chais Ikura 4 12 - 200 ml jars 31 Ikura Chang 1 24 - 12 oz bottles 19 Chang Syrup 1 12 - 550 ml bottles 10 Syrup Kobe Niku 4 18 - 500 g pkgs. 97 Kobe Niku Konbu 6 2 kg box 6 Konbu Tofu 6 40 - 100 g pkgs. 23.25 Tofu Geitost 15 500 g 2.5 Geitost AVG(Price) 14 2.5
16. GROUP BY / Agg using diverging colors s x 20 bags 18 Chais 1 10 boxes x 20 bags 18 00 ml jars 31 Ikura 4 12 - 200 ml jars 31 oz bottles 19 Chang 1 24 - 12 oz bottles 19 ml bottles 10 Syrup 1 12 - 550 ml bottles 10 00 g pkgs. 97 Kobe Niku 4 18 - 500 g pkgs. 97 g box 6 Konbu 6 2 kg box 6 00 g pkgs. 23.25 Tofu 6 40 - 100 g pkgs. 23.25 00 g 2.5 Geitost 15 500 g 2.5 RDER BY Price SELECT * FROM Products GROUP BY "Supplier ID" Unit Price Product Name Supplier ID Unit Price s x 20 bags 18 Chais 1 10 boxes x 20 bags 18 00 ml jars 31 Ikura 4 12 - 200 ml jars 31 oz bottles 19 Chang 1 24 - 12 oz bottles 19 ml bottles 10 Syrup 1 12 - 550 ml bottles 10 00 g pkgs. 97 Kobe Niku 4 18 - 500 g pkgs. 97 g box 6 Konbu 6 2 kg box 6 00 g pkgs. 23.25 Tofu 6 40 - 100 g pkgs. 23.25 00 g 2.5 Geitost 15 500 g 2.5 AVG(Price) 14.75 15.66 2.5 64
17. Querying Paper Tables with ARQuery: A Demo
18. Walkthrough demo
19. • Camera First Application • Relation Extraction • Query Mapping • Visual Encoding System Architecture DB Visual Space Encoder Gesture Classifier Interactive Query Session Rendering Table Extraction Vision & Table Tracking User Interface CAMERA HEADSET / SCREEN
20. Navigating Airports with ARQuery: A Demo
21. Airports with ARQuery: Demo
22. Evaluation • Task: Filter, Sort, and Group By / Aggregation • Table printed on paper • Comparisons • ARQuery (iPad implementation) • Manual (pen and paper) • Excel (cost to transfer data not included) • 15 users, measured task completion time
23. Evaluation: Task Completion Time • ARQuery was consistently faster 6.2 5.7 7.1 15.8 6.9 31.8 7.1 6.9 90.0 1 2 4 8 16 32 64 128 Filter Sort GroupBy Completion Time, log2(s) ARQuery Excel Manual
24. Evaluation: Insights • User Confidence • Users trust ARQuery more than even doing things by hand • Worried about making mistakes • Scale • For paper, cognitive challenges increase as data increases • ARQuery does not have this problem
25. Future Work • A full fledged querying framework in augmented reality • Augmented Reality JOINs • More complex queries and viz overlays
26. Future Work • Combining Interaction Modalities • Gestures • Augmented Reality • Speech • Challenges • Trading off speed of interaction vs. accuracy
27. Conclusion • Next frontier: Real-world Data • Augmented Reality • widely available on phones & tablets • Can be used for querying • Challenges: Visual Encoding, Querying Grammar • End-to-end stack for querying in augmented reality
28. Thank you! https://interact.osu.edu