Pro Yearly is on sale from $80 to $50! »

ARQuery: Hallucinating Analytics over Real-World Data using Augmented Reality

ARQuery: Hallucinating Analytics over Real-World Data using Augmented Reality

Work by Codi Burley, Arnab Nandi [ https://interact.osu.edu ]

ARQuery is a query platform that utilizes augmented reality to enable
querying over real-world data. We provide an interaction and visualization grammar that is designed to augment real-world data, and a performant framework that enables query exploration in real-time.

More at: http://cidrdb.org/cidr2019/papers/p93-burley-cidr19.pdf

D49c0d235689fe04b1dd4630cabc2aa8?s=128

Arnab Nandi

January 15, 2019
Tweet

Transcript

  1. ARQuery: Hallucinating Analytics using Augmented Reality Codi Burley Arnab Nandi

  2. Motivation: Data in the Real World • Paper • Ledgers

    • Menus • Labels on Physical Objects • Grocery Store Aisles • Electronic Displays • Third Party Screens • Device Meters
  3. Motivation: Querying in the Real World • Airport Gate: “Rebooking

    to the earliest flight to Monterey” • Other examples: • Subsetting restaurant menu for dietary restrictions • Noting allergies from a prescription medicine bottle • Finding the cheapest crunchy peanut butter in a large grocery aisle
  4. Challenge: Hard to Query the Real-World • Involves importing data

    into a database or spreadsheet • Lose context • Hard to do in ad-hoc, unprepared settings
  5. Proposal: Augmented Reality • Augmented Reality • Overlay camera scene

    with UI & information • Made popular by games • Now becoming mainstream & commodity • Poses multiple challenges • How does one represent query results in augmented reality? • How do you interact with augmented information?
  6. Our Contributions • An interaction and visualization grammar • and

    encoding of query results in augmented reality • A performant framework for querying structured data in AR • End-to-end extraction, mapping, querying, and overlaying • Experimental Evaluations
  7. Outline • Motivation & Challenges • Contributions • Query Grammar

    & Visual Encoding • Demo • Evaluation • Conclusions & Future Work
  8. Query Grammar & Visual Encoding Query Operations Visual Encoding Selection

    Occlusion Projection Occlusion Sort Color Gradient Group Divergent Colors Aggregate Virtual Columns/Legends • Real-world ≠ screens • Static, cannot move around • Important to maintain context and placement • Need to maintain query state
  9. Virtual World: It’s okay to manipulate UI Sort Operation: Changes

    are ok Virtual World: Spreadsheet UI
  10. Blending virtual and real-world data: Maintaining Context Changes are too

    Too disorienting Real-world context
  11. Query Grammar & Visual Encoding Query Operations Visual Encoding Selection

    Occlusion Projection Occlusion Sort Color Gradient Group Divergent Colors Aggregate Virtual Columns/Legends • Real-world ≠ screens • Static, cannot move around • Important to maintain context and placement • Need to maintain query state
  12. Augmented Highlights, Virtual Results • Augmented Layer • Augmented Highlights

    • Virtual Results INTERACTIVE QUERYING AUGMENTED LAYER REAL-WORLD DATA LAYER VIRTUAL RESULTS AUGMENTED HIGHLIGHTS
  13. Query Walkthrough SELECT * FROM Products SELECT "Product Name" F

    Product Name Supplier ID Unit Price Product Name Supplie Chais 1 10 boxes x 20 bags 18 Chais 1 Ikura 4 12 - 200 ml jars 31 Ikura 4 Chang 1 24 - 12 oz bottles 19 Chang 1 Syrup 1 12 - 550 ml bottles 10 Syrup 1 Kobe Niku 4 18 - 500 g pkgs. 97 Kobe Niku 4 Konbu 6 2 kg box 6 Konbu 6 Tofu 6 40 - 100 g pkgs. 23.25 Tofu 6 Geitost 15 500 g 2.5 Geitost 15 SELECT * FROM Products ORDER BY Price SELECT * FROM P Product Name Supplier ID Unit Price Product Name Supplie Chais 1 10 boxes x 20 bags 18 Chais 1
  14. Selection & Projection using Occlusion SELECT "Product Name" FROM Products

    WHERE Price<15 Price Product Name Supplier ID Unit Price 18 Chais 1 10 boxes x 20 bags 18 31 Ikura 4 12 - 200 ml jars 31 19 Chang 1 24 - 12 oz bottles 19 10 Syrup 1 12 - 550 ml bottles 10 97 Kobe Niku 4 18 - 500 g pkgs. 97 6 Konbu 6 2 kg box 6 23.25 Tofu 6 40 - 100 g pkgs. 23.25 2.5 Geitost 15 500 g 2.5 rice SELECT * FROM Products GROUP BY "Supplier ID" Price Product Name Supplier ID Unit Price
  15. ORDER BY using Color Gradients Chais 1 10 boxes x

    20 bags 18 Chais Ikura 4 12 - 200 ml jars 31 Ikura Chang 1 24 - 12 oz bottles 19 Chang Syrup 1 12 - 550 ml bottles 10 Syrup Kobe Niku 4 18 - 500 g pkgs. 97 Kobe Niku Konbu 6 2 kg box 6 Konbu Tofu 6 40 - 100 g pkgs. 23.25 Tofu Geitost 15 500 g 2.5 Geitost SELECT * FROM Products ORDER BY Price SELECT * F Product Name Supplier ID Unit Price Product Name Chais 1 10 boxes x 20 bags 18 Chais Ikura 4 12 - 200 ml jars 31 Ikura Chang 1 24 - 12 oz bottles 19 Chang Syrup 1 12 - 550 ml bottles 10 Syrup Kobe Niku 4 18 - 500 g pkgs. 97 Kobe Niku Konbu 6 2 kg box 6 Konbu Tofu 6 40 - 100 g pkgs. 23.25 Tofu Geitost 15 500 g 2.5 Geitost AVG(Price) 14 2.5
  16. GROUP BY / Agg using diverging colors s x 20

    bags 18 Chais 1 10 boxes x 20 bags 18 00 ml jars 31 Ikura 4 12 - 200 ml jars 31 oz bottles 19 Chang 1 24 - 12 oz bottles 19 ml bottles 10 Syrup 1 12 - 550 ml bottles 10 00 g pkgs. 97 Kobe Niku 4 18 - 500 g pkgs. 97 g box 6 Konbu 6 2 kg box 6 00 g pkgs. 23.25 Tofu 6 40 - 100 g pkgs. 23.25 00 g 2.5 Geitost 15 500 g 2.5 RDER BY Price SELECT * FROM Products GROUP BY "Supplier ID" Unit Price Product Name Supplier ID Unit Price s x 20 bags 18 Chais 1 10 boxes x 20 bags 18 00 ml jars 31 Ikura 4 12 - 200 ml jars 31 oz bottles 19 Chang 1 24 - 12 oz bottles 19 ml bottles 10 Syrup 1 12 - 550 ml bottles 10 00 g pkgs. 97 Kobe Niku 4 18 - 500 g pkgs. 97 g box 6 Konbu 6 2 kg box 6 00 g pkgs. 23.25 Tofu 6 40 - 100 g pkgs. 23.25 00 g 2.5 Geitost 15 500 g 2.5 AVG(Price) 14.75 15.66 2.5 64
  17. Querying Paper Tables with ARQuery: A Demo

  18. Walkthrough demo

  19. • Camera First Application • Relation Extraction • Query Mapping

    • Visual Encoding System Architecture DB Visual Space Encoder Gesture Classifier Interactive Query Session Rendering Table Extraction Vision & Table Tracking User Interface CAMERA HEADSET / SCREEN
  20. Navigating Airports with ARQuery: A Demo

  21. Airports with ARQuery: Demo

  22. Evaluation • Task: Filter, Sort, and Group By / Aggregation

    • Table printed on paper • Comparisons • ARQuery (iPad implementation) • Manual (pen and paper) • Excel (cost to transfer data not included) • 15 users, measured task completion time
  23. Evaluation: Task Completion Time • ARQuery was consistently faster 6.2

    5.7 7.1 15.8 6.9 31.8 7.1 6.9 90.0 1 2 4 8 16 32 64 128 Filter Sort GroupBy Completion Time, log2(s) ARQuery Excel Manual
  24. Evaluation: Insights • User Confidence • Users trust ARQuery more

    than even doing things by hand • Worried about making mistakes • Scale • For paper, cognitive challenges increase as data increases • ARQuery does not have this problem
  25. Future Work • A full fledged querying framework in augmented

    reality • Augmented Reality JOINs • More complex queries and viz overlays
  26. Future Work • Combining Interaction Modalities • Gestures • Augmented

    Reality • Speech • Challenges • Trading off speed of interaction vs. accuracy
  27. Conclusion • Next frontier: Real-world Data • Augmented Reality •

    widely available on phones & tablets • Can be used for querying • Challenges: Visual Encoding, Querying Grammar • End-to-end stack for querying in augmented reality
  28. Thank you! https://interact.osu.edu