ARQuery: Hallucinating Analytics using Augmented Reality Codi Burley Arnab Nandi 

Motivation: Data in the Real World • Paper • Ledgers • Menus • Labels on Physical Objects • Grocery Store Aisles • Electronic Displays • Third Party Screens • Device Meters 

Motivation: Querying in the Real World • Airport Gate: “Rebooking to the earliest flight to Monterey” • Other examples: • Subsetting restaurant menu for dietary restrictions • Noting allergies from a prescription medicine bottle • Finding the cheapest crunchy peanut butter in a large grocery aisle 

Challenge: Hard to Query the Real-World • Involves importing data into a database or spreadsheet • Lose context • Hard to do in ad-hoc, unprepared settings 

Proposal: Augmented Reality • Augmented Reality • Overlay camera scene with UI & information • Made popular by games • Now becoming mainstream & commodity • Poses multiple challenges • How does one represent query results in augmented reality? • How do you interact with augmented information? 

Our Contributions • An interaction and visualization grammar • and encoding of query results in augmented reality • A performant framework for querying structured data in AR • End-to-end extraction, mapping, querying, and overlaying • Experimental Evaluations 

Outline • Motivation & Challenges • Contributions • Query Grammar & Visual Encoding • Demo • Evaluation • Conclusions & Future Work 

Query Grammar & Visual Encoding Query Operations Visual Encoding Selection Occlusion Projection Occlusion Sort Color Gradient Group Divergent Colors Aggregate Virtual Columns/Legends • Real-world ≠ screens • Static, cannot move around • Important to maintain context and placement • Need to maintain query state 

Virtual World: It’s okay to manipulate UI Sort Operation: Changes are ok Virtual World: Spreadsheet UI 

Blending virtual and real-world data: Maintaining Context Changes are too Too disorienting Real-world context 

Query Grammar & Visual Encoding Query Operations Visual Encoding Selection Occlusion Projection Occlusion Sort Color Gradient Group Divergent Colors Aggregate Virtual Columns/Legends • Real-world ≠ screens • Static, cannot move around • Important to maintain context and placement • Need to maintain query state 

Augmented Highlights, Virtual Results • Augmented Layer • Augmented Highlights • Virtual Results INTERACTIVE QUERYING AUGMENTED LAYER REAL-WORLD DATA LAYER VIRTUAL RESULTS AUGMENTED HIGHLIGHTS 

Query Walkthrough SELECT * FROM Products SELECT "Product Name" F Product Name Supplier ID Unit Price Product Name Supplie Chais 1 10 boxes x 20 bags 18 Chais 1 Ikura 4 12 - 200 ml jars 31 Ikura 4 Chang 1 24 - 12 oz bottles 19 Chang 1 Syrup 1 12 - 550 ml bottles 10 Syrup 1 Kobe Niku 4 18 - 500 g pkgs. 97 Kobe Niku 4 Konbu 6 2 kg box 6 Konbu 6 Tofu 6 40 - 100 g pkgs. 23.25 Tofu 6 Geitost 15 500 g 2.5 Geitost 15 SELECT * FROM Products ORDER BY Price SELECT * FROM P Product Name Supplier ID Unit Price Product Name Supplie Chais 1 10 boxes x 20 bags 18 Chais 1 

Selection & Projection using Occlusion SELECT "Product Name" FROM Products WHERE Price<15 Price Product Name Supplier ID Unit Price 18 Chais 1 10 boxes x 20 bags 18 31 Ikura 4 12 - 200 ml jars 31 19 Chang 1 24 - 12 oz bottles 19 10 Syrup 1 12 - 550 ml bottles 10 97 Kobe Niku 4 18 - 500 g pkgs. 97 6 Konbu 6 2 kg box 6 23.25 Tofu 6 40 - 100 g pkgs. 23.25 2.5 Geitost 15 500 g 2.5 rice SELECT * FROM Products GROUP BY "Supplier ID" Price Product Name Supplier ID Unit Price 

ORDER BY using Color Gradients Chais 1 10 boxes x 20 bags 18 Chais Ikura 4 12 - 200 ml jars 31 Ikura Chang 1 24 - 12 oz bottles 19 Chang Syrup 1 12 - 550 ml bottles 10 Syrup Kobe Niku 4 18 - 500 g pkgs. 97 Kobe Niku Konbu 6 2 kg box 6 Konbu Tofu 6 40 - 100 g pkgs. 23.25 Tofu Geitost 15 500 g 2.5 Geitost SELECT * FROM Products ORDER BY Price SELECT * F Product Name Supplier ID Unit Price Product Name Chais 1 10 boxes x 20 bags 18 Chais Ikura 4 12 - 200 ml jars 31 Ikura Chang 1 24 - 12 oz bottles 19 Chang Syrup 1 12 - 550 ml bottles 10 Syrup Kobe Niku 4 18 - 500 g pkgs. 97 Kobe Niku Konbu 6 2 kg box 6 Konbu Tofu 6 40 - 100 g pkgs. 23.25 Tofu Geitost 15 500 g 2.5 Geitost AVG(Price) 14 2.5 

GROUP BY / Agg using diverging colors s x 20 bags 18 Chais 1 10 boxes x 20 bags 18 00 ml jars 31 Ikura 4 12 - 200 ml jars 31 oz bottles 19 Chang 1 24 - 12 oz bottles 19 ml bottles 10 Syrup 1 12 - 550 ml bottles 10 00 g pkgs. 97 Kobe Niku 4 18 - 500 g pkgs. 97 g box 6 Konbu 6 2 kg box 6 00 g pkgs. 23.25 Tofu 6 40 - 100 g pkgs. 23.25 00 g 2.5 Geitost 15 500 g 2.5 RDER BY Price SELECT * FROM Products GROUP BY "Supplier ID" Unit Price Product Name Supplier ID Unit Price s x 20 bags 18 Chais 1 10 boxes x 20 bags 18 00 ml jars 31 Ikura 4 12 - 200 ml jars 31 oz bottles 19 Chang 1 24 - 12 oz bottles 19 ml bottles 10 Syrup 1 12 - 550 ml bottles 10 00 g pkgs. 97 Kobe Niku 4 18 - 500 g pkgs. 97 g box 6 Konbu 6 2 kg box 6 00 g pkgs. 23.25 Tofu 6 40 - 100 g pkgs. 23.25 00 g 2.5 Geitost 15 500 g 2.5 AVG(Price) 14.75 15.66 2.5 64 

Querying Paper Tables with ARQuery: A Demo 

Walkthrough demo 

• Camera First Application • Relation Extraction • Query Mapping • Visual Encoding System Architecture DB Visual Space Encoder Gesture Classifier Interactive Query Session Rendering Table Extraction Vision & Table Tracking User Interface CAMERA HEADSET / SCREEN 

Navigating Airports with ARQuery: A Demo 

Airports with ARQuery: Demo 

Evaluation • Task: Filter, Sort, and Group By / Aggregation • Table printed on paper • Comparisons • ARQuery (iPad implementation) • Manual (pen and paper) • Excel (cost to transfer data not included) • 15 users, measured task completion time 

Evaluation: Task Completion Time • ARQuery was consistently faster 6.2 5.7 7.1 15.8 6.9 31.8 7.1 6.9 90.0 1 2 4 8 16 32 64 128 Filter Sort GroupBy Completion Time, log2(s) ARQuery Excel Manual 

Evaluation: Insights • User Confidence • Users trust ARQuery more than even doing things by hand • Worried about making mistakes • Scale • For paper, cognitive challenges increase as data increases • ARQuery does not have this problem 

Future Work • A full fledged querying framework in augmented reality • Augmented Reality JOINs • More complex queries and viz overlays 

Future Work • Combining Interaction Modalities • Gestures • Augmented Reality • Speech • Challenges • Trading off speed of interaction vs. accuracy 

Conclusion • Next frontier: Real-world Data • Augmented Reality • widely available on phones & tablets • Can be used for querying • Challenges: Visual Encoding, Querying Grammar • End-to-end stack for querying in augmented reality 

Thank you! https://interact.osu.edu