ARQuery: Hallucinating Analytics over Real-World Data using Augmented Reality

Work by Codi Burley, Arnab Nandi [ https://interact.osu.edu ]

ARQuery is a query platform that utilizes augmented reality to enable
querying over real-world data. We provide an interaction and visualization grammar that is designed to augment real-world data, and a performant framework that enables query exploration in real-time.

More at: http://cidrdb.org/cidr2019/papers/p93-burley-cidr19.pdf

Transcript

1. ARQuery:
   Hallucinating Analytics
   using Augmented Reality
   Codi Burley
   Arnab Nandi
   

   View Slide

2. Motivation: Data in the Real World
   • Paper
   • Ledgers
   • Menus
   • Labels on Physical Objects
   • Grocery Store Aisles
   • Electronic Displays
   • Third Party Screens
   • Device Meters
   

   View Slide

3. Motivation: Querying in the Real World
   • Airport Gate: “Rebooking to the
   earliest flight to Monterey”
   • Other examples:
   • Subsetting restaurant menu for
   dietary restrictions
   • Noting allergies from a prescription
   medicine bottle
   • Finding the cheapest crunchy
   peanut butter
   in a large grocery aisle
   

   View Slide

4. Challenge: Hard to Query the Real-World
   • Involves importing data into a
   database or spreadsheet
   • Lose context
   • Hard to do in ad-hoc, unprepared
   settings
   

   View Slide

5. Proposal: Augmented Reality
   • Augmented Reality
   • Overlay camera scene with UI & information
   • Made popular by games
   • Now becoming mainstream & commodity
   • Poses multiple challenges
   • How does one represent query results in
   augmented reality?
   • How do you interact with augmented information?
   

   View Slide

6. Our Contributions
   • An interaction and visualization grammar
   • and encoding of query results in
   augmented reality
   • A performant framework for querying
   structured data in AR
   • End-to-end extraction, mapping,
   querying, and overlaying
   • Experimental Evaluations
   

   View Slide

7. Outline
   • Motivation & Challenges
   • Contributions
   • Query Grammar & Visual Encoding
   • Demo
   • Evaluation
   • Conclusions & Future Work
   

   View Slide

8. Query Grammar & Visual Encoding
   Query Operations Visual Encoding
   Selection Occlusion
   Projection Occlusion
   Sort Color Gradient
   Group Divergent Colors
   Aggregate Virtual Columns/Legends
   • Real-world ≠ screens
   • Static, cannot move around
   • Important to maintain context
   and placement
   • Need to maintain query state
   

   View Slide

9. Virtual World: It’s okay to manipulate UI
   Sort Operation:
   Changes are ok
   Virtual World:
   Spreadsheet UI
   

   View Slide

10. Blending virtual and real-world data:
    Maintaining Context
    Changes are too
    Too disorienting
    Real-world context
    

    View Slide

11. Query Grammar & Visual Encoding
    Query Operations Visual Encoding
    Selection Occlusion
    Projection Occlusion
    Sort Color Gradient
    Group Divergent Colors
    Aggregate Virtual Columns/Legends
    • Real-world ≠ screens
    • Static, cannot move around
    • Important to maintain context
    and placement
    • Need to maintain query state
    

    View Slide

12. Augmented Highlights, Virtual Results
    • Augmented Layer
    • Augmented Highlights
    • Virtual Results
    INTERACTIVE QUERYING
    AUGMENTED LAYER
    REAL-WORLD DATA LAYER
    VIRTUAL
    RESULTS
    AUGMENTED
    HIGHLIGHTS
    

    View Slide

13. Query Walkthrough
    SELECT * FROM Products SELECT "Product Name" F
    Product Name Supplier ID Unit Price Product Name Supplie
    Chais 1 10 boxes x 20 bags 18 Chais 1
    Ikura 4 12 - 200 ml jars 31 Ikura 4
    Chang 1 24 - 12 oz bottles 19 Chang 1
    Syrup 1 12 - 550 ml bottles 10 Syrup 1
    Kobe Niku 4 18 - 500 g pkgs. 97 Kobe Niku 4
    Konbu 6 2 kg box 6 Konbu 6
    Tofu 6 40 - 100 g pkgs. 23.25 Tofu 6
    Geitost 15 500 g 2.5 Geitost 15
    SELECT * FROM Products ORDER BY Price SELECT * FROM P
    Product Name Supplier ID Unit Price Product Name Supplie
    Chais 1 10 boxes x 20 bags 18 Chais 1
    

    View Slide

14. Selection & Projection using Occlusion
    SELECT "Product Name" FROM Products WHERE Price<15
    Price Product Name Supplier ID Unit Price
    18 Chais 1 10 boxes x 20 bags 18
    31 Ikura 4 12 - 200 ml jars 31
    19 Chang 1 24 - 12 oz bottles 19
    10 Syrup 1 12 - 550 ml bottles 10
    97 Kobe Niku 4 18 - 500 g pkgs. 97
    6 Konbu 6 2 kg box 6
    23.25 Tofu 6 40 - 100 g pkgs. 23.25
    2.5 Geitost 15 500 g 2.5
    rice SELECT * FROM Products GROUP BY "Supplier ID"
    Price Product Name Supplier ID Unit Price
    

    View Slide

15. ORDER BY using Color Gradients
    Chais 1 10 boxes x 20 bags 18 Chais
    Ikura 4 12 - 200 ml jars 31 Ikura
    Chang 1 24 - 12 oz bottles 19 Chang
    Syrup 1 12 - 550 ml bottles 10 Syrup
    Kobe Niku 4 18 - 500 g pkgs. 97 Kobe Niku
    Konbu 6 2 kg box 6 Konbu
    Tofu 6 40 - 100 g pkgs. 23.25 Tofu
    Geitost 15 500 g 2.5 Geitost
    SELECT * FROM Products ORDER BY Price SELECT * F
    Product Name Supplier ID Unit Price Product Name
    Chais 1 10 boxes x 20 bags 18 Chais
    Ikura 4 12 - 200 ml jars 31 Ikura
    Chang 1 24 - 12 oz bottles 19 Chang
    Syrup 1 12 - 550 ml bottles 10 Syrup
    Kobe Niku 4 18 - 500 g pkgs. 97 Kobe Niku
    Konbu 6 2 kg box 6 Konbu
    Tofu 6 40 - 100 g pkgs. 23.25 Tofu
    Geitost 15 500 g 2.5 Geitost
    AVG(Price)
    14
    2.5
    

    View Slide

16. GROUP BY / Agg using diverging colors
    s x 20 bags 18 Chais 1 10 boxes x 20 bags 18
    00 ml jars 31 Ikura 4 12 - 200 ml jars 31
    oz bottles 19 Chang 1 24 - 12 oz bottles 19
    ml bottles 10 Syrup 1 12 - 550 ml bottles 10
    00 g pkgs. 97 Kobe Niku 4 18 - 500 g pkgs. 97
    g box 6 Konbu 6 2 kg box 6
    00 g pkgs. 23.25 Tofu 6 40 - 100 g pkgs. 23.25
    00 g 2.5 Geitost 15 500 g 2.5
    RDER BY Price SELECT * FROM Products GROUP BY "Supplier ID"
    Unit Price Product Name Supplier ID Unit Price
    s x 20 bags 18 Chais 1 10 boxes x 20 bags 18
    00 ml jars 31 Ikura 4 12 - 200 ml jars 31
    oz bottles 19 Chang 1 24 - 12 oz bottles 19
    ml bottles 10 Syrup 1 12 - 550 ml bottles 10
    00 g pkgs. 97 Kobe Niku 4 18 - 500 g pkgs. 97
    g box 6 Konbu 6 2 kg box 6
    00 g pkgs. 23.25 Tofu 6 40 - 100 g pkgs. 23.25
    00 g 2.5 Geitost 15 500 g 2.5
    AVG(Price)
    14.75 15.66
    2.5 64
    

    View Slide

17. Querying Paper Tables with ARQuery:
    A Demo
    

    View Slide

18. Walkthrough demo
    

    View Slide

19. • Camera First Application
    • Relation Extraction
    • Query Mapping
    • Visual Encoding
    System Architecture
    DB
    Visual Space
    Encoder
    Gesture
    Classifier
    Interactive
    Query Session
    Rendering
    Table Extraction Vision &
    Table Tracking
    User Interface
    CAMERA
    HEADSET / SCREEN
    

    View Slide

20. Navigating Airports with ARQuery:
    A Demo
    

    View Slide

21. Airports with ARQuery: Demo
    

    View Slide

22. Evaluation
    • Task: Filter, Sort, and Group By / Aggregation
    • Table printed on paper
    • Comparisons
    • ARQuery (iPad implementation)
    • Manual (pen and paper)
    • Excel (cost to transfer data not included)
    • 15 users, measured task completion time
    

    View Slide

23. Evaluation: Task Completion Time
    • ARQuery was consistently faster
    6.2
    5.7
    7.1
    15.8
    6.9
    31.8
    7.1
    6.9
    90.0
    1 2 4 8 16 32 64 128
    Filter Sort GroupBy
    Completion Time, log2(s)
    ARQuery
    Excel
    Manual
    

    View Slide

24. Evaluation: Insights
    • User Confidence
    • Users trust ARQuery more than even doing things by hand
    • Worried about making mistakes
    • Scale
    • For paper, cognitive challenges increase as data increases
    • ARQuery does not have this problem
    

    View Slide

25. Future Work
    • A full fledged querying
    framework in
    augmented reality
    • Augmented Reality JOINs
    • More complex queries
    and viz overlays
    

    View Slide

26. Future Work
    • Combining Interaction Modalities
    • Gestures
    • Augmented Reality
    • Speech
    • Challenges
    • Trading off speed of interaction vs. accuracy
    

    View Slide

27. Conclusion
    • Next frontier: Real-world Data
    • Augmented Reality
    • widely available on phones & tablets
    • Can be used for querying
    • Challenges: Visual Encoding, Querying Grammar
    • End-to-end stack for querying in augmented reality
    

    View Slide

28. Thank you!
    https://interact.osu.edu
    

    View Slide