Upgrade to Pro — share decks privately, control downloads, hide ads and more …

ARQuery: Hallucinating Analytics over Real-World Data using Augmented Reality

Arnab Nandi
January 15, 2019

ARQuery: Hallucinating Analytics over Real-World Data using Augmented Reality

Work by Codi Burley, Arnab Nandi [ https://interact.osu.edu ]

ARQuery is a query platform that utilizes augmented reality to enable
querying over real-world data. We provide an interaction and visualization grammar that is designed to augment real-world data, and a performant framework that enables query exploration in real-time.

More at: http://cidrdb.org/cidr2019/papers/p93-burley-cidr19.pdf

Arnab Nandi

January 15, 2019
Tweet

More Decks by Arnab Nandi

Other Decks in Technology

Transcript

  1. ARQuery:
    Hallucinating Analytics
    using Augmented Reality
    Codi Burley
    Arnab Nandi

    View Slide

  2. Motivation: Data in the Real World
    • Paper
    • Ledgers
    • Menus
    • Labels on Physical Objects
    • Grocery Store Aisles
    • Electronic Displays
    • Third Party Screens
    • Device Meters

    View Slide

  3. Motivation: Querying in the Real World
    • Airport Gate: “Rebooking to the
    earliest flight to Monterey”
    • Other examples:
    • Subsetting restaurant menu for
    dietary restrictions
    • Noting allergies from a prescription
    medicine bottle
    • Finding the cheapest crunchy
    peanut butter
    in a large grocery aisle

    View Slide

  4. Challenge: Hard to Query the Real-World
    • Involves importing data into a
    database or spreadsheet
    • Lose context
    • Hard to do in ad-hoc, unprepared
    settings

    View Slide

  5. Proposal: Augmented Reality
    • Augmented Reality
    • Overlay camera scene with UI & information
    • Made popular by games
    • Now becoming mainstream & commodity
    • Poses multiple challenges
    • How does one represent query results in
    augmented reality?
    • How do you interact with augmented information?

    View Slide

  6. Our Contributions
    • An interaction and visualization grammar
    • and encoding of query results in
    augmented reality
    • A performant framework for querying
    structured data in AR
    • End-to-end extraction, mapping,
    querying, and overlaying
    • Experimental Evaluations

    View Slide

  7. Outline
    • Motivation & Challenges
    • Contributions
    • Query Grammar & Visual Encoding
    • Demo
    • Evaluation
    • Conclusions & Future Work

    View Slide

  8. Query Grammar & Visual Encoding
    Query Operations Visual Encoding
    Selection Occlusion
    Projection Occlusion
    Sort Color Gradient
    Group Divergent Colors
    Aggregate Virtual Columns/Legends
    • Real-world ≠ screens
    • Static, cannot move around
    • Important to maintain context
    and placement
    • Need to maintain query state

    View Slide

  9. Virtual World: It’s okay to manipulate UI
    Sort Operation:
    Changes are ok
    Virtual World:
    Spreadsheet UI

    View Slide

  10. Blending virtual and real-world data:
    Maintaining Context
    Changes are too
    Too disorienting
    Real-world context

    View Slide

  11. Query Grammar & Visual Encoding
    Query Operations Visual Encoding
    Selection Occlusion
    Projection Occlusion
    Sort Color Gradient
    Group Divergent Colors
    Aggregate Virtual Columns/Legends
    • Real-world ≠ screens
    • Static, cannot move around
    • Important to maintain context
    and placement
    • Need to maintain query state

    View Slide

  12. Augmented Highlights, Virtual Results
    • Augmented Layer
    • Augmented Highlights
    • Virtual Results
    INTERACTIVE QUERYING
    AUGMENTED LAYER
    REAL-WORLD DATA LAYER
    VIRTUAL
    RESULTS
    AUGMENTED
    HIGHLIGHTS

    View Slide

  13. Query Walkthrough
    SELECT * FROM Products SELECT "Product Name" F
    Product Name Supplier ID Unit Price Product Name Supplie
    Chais 1 10 boxes x 20 bags 18 Chais 1
    Ikura 4 12 - 200 ml jars 31 Ikura 4
    Chang 1 24 - 12 oz bottles 19 Chang 1
    Syrup 1 12 - 550 ml bottles 10 Syrup 1
    Kobe Niku 4 18 - 500 g pkgs. 97 Kobe Niku 4
    Konbu 6 2 kg box 6 Konbu 6
    Tofu 6 40 - 100 g pkgs. 23.25 Tofu 6
    Geitost 15 500 g 2.5 Geitost 15
    SELECT * FROM Products ORDER BY Price SELECT * FROM P
    Product Name Supplier ID Unit Price Product Name Supplie
    Chais 1 10 boxes x 20 bags 18 Chais 1

    View Slide

  14. Selection & Projection using Occlusion
    SELECT "Product Name" FROM Products WHERE Price<15
    Price Product Name Supplier ID Unit Price
    18 Chais 1 10 boxes x 20 bags 18
    31 Ikura 4 12 - 200 ml jars 31
    19 Chang 1 24 - 12 oz bottles 19
    10 Syrup 1 12 - 550 ml bottles 10
    97 Kobe Niku 4 18 - 500 g pkgs. 97
    6 Konbu 6 2 kg box 6
    23.25 Tofu 6 40 - 100 g pkgs. 23.25
    2.5 Geitost 15 500 g 2.5
    rice SELECT * FROM Products GROUP BY "Supplier ID"
    Price Product Name Supplier ID Unit Price

    View Slide

  15. ORDER BY using Color Gradients
    Chais 1 10 boxes x 20 bags 18 Chais
    Ikura 4 12 - 200 ml jars 31 Ikura
    Chang 1 24 - 12 oz bottles 19 Chang
    Syrup 1 12 - 550 ml bottles 10 Syrup
    Kobe Niku 4 18 - 500 g pkgs. 97 Kobe Niku
    Konbu 6 2 kg box 6 Konbu
    Tofu 6 40 - 100 g pkgs. 23.25 Tofu
    Geitost 15 500 g 2.5 Geitost
    SELECT * FROM Products ORDER BY Price SELECT * F
    Product Name Supplier ID Unit Price Product Name
    Chais 1 10 boxes x 20 bags 18 Chais
    Ikura 4 12 - 200 ml jars 31 Ikura
    Chang 1 24 - 12 oz bottles 19 Chang
    Syrup 1 12 - 550 ml bottles 10 Syrup
    Kobe Niku 4 18 - 500 g pkgs. 97 Kobe Niku
    Konbu 6 2 kg box 6 Konbu
    Tofu 6 40 - 100 g pkgs. 23.25 Tofu
    Geitost 15 500 g 2.5 Geitost
    AVG(Price)
    14
    2.5

    View Slide

  16. GROUP BY / Agg using diverging colors
    s x 20 bags 18 Chais 1 10 boxes x 20 bags 18
    00 ml jars 31 Ikura 4 12 - 200 ml jars 31
    oz bottles 19 Chang 1 24 - 12 oz bottles 19
    ml bottles 10 Syrup 1 12 - 550 ml bottles 10
    00 g pkgs. 97 Kobe Niku 4 18 - 500 g pkgs. 97
    g box 6 Konbu 6 2 kg box 6
    00 g pkgs. 23.25 Tofu 6 40 - 100 g pkgs. 23.25
    00 g 2.5 Geitost 15 500 g 2.5
    RDER BY Price SELECT * FROM Products GROUP BY "Supplier ID"
    Unit Price Product Name Supplier ID Unit Price
    s x 20 bags 18 Chais 1 10 boxes x 20 bags 18
    00 ml jars 31 Ikura 4 12 - 200 ml jars 31
    oz bottles 19 Chang 1 24 - 12 oz bottles 19
    ml bottles 10 Syrup 1 12 - 550 ml bottles 10
    00 g pkgs. 97 Kobe Niku 4 18 - 500 g pkgs. 97
    g box 6 Konbu 6 2 kg box 6
    00 g pkgs. 23.25 Tofu 6 40 - 100 g pkgs. 23.25
    00 g 2.5 Geitost 15 500 g 2.5
    AVG(Price)
    14.75 15.66
    2.5 64

    View Slide

  17. Querying Paper Tables with ARQuery:
    A Demo

    View Slide

  18. Walkthrough demo

    View Slide

  19. • Camera First Application
    • Relation Extraction
    • Query Mapping
    • Visual Encoding
    System Architecture
    DB
    Visual Space
    Encoder
    Gesture
    Classifier
    Interactive
    Query Session
    Rendering
    Table Extraction Vision &
    Table Tracking
    User Interface
    CAMERA
    HEADSET / SCREEN

    View Slide

  20. Navigating Airports with ARQuery:
    A Demo

    View Slide

  21. Airports with ARQuery: Demo

    View Slide

  22. Evaluation
    • Task: Filter, Sort, and Group By / Aggregation
    • Table printed on paper
    • Comparisons
    • ARQuery (iPad implementation)
    • Manual (pen and paper)
    • Excel (cost to transfer data not included)
    • 15 users, measured task completion time

    View Slide

  23. Evaluation: Task Completion Time
    • ARQuery was consistently faster
    6.2
    5.7
    7.1
    15.8
    6.9
    31.8
    7.1
    6.9
    90.0
    1 2 4 8 16 32 64 128
    Filter Sort GroupBy
    Completion Time, log2(s)
    ARQuery
    Excel
    Manual

    View Slide

  24. Evaluation: Insights
    • User Confidence
    • Users trust ARQuery more than even doing things by hand
    • Worried about making mistakes
    • Scale
    • For paper, cognitive challenges increase as data increases
    • ARQuery does not have this problem

    View Slide

  25. Future Work
    • A full fledged querying
    framework in
    augmented reality
    • Augmented Reality JOINs
    • More complex queries
    and viz overlays

    View Slide

  26. Future Work
    • Combining Interaction Modalities
    • Gestures
    • Augmented Reality
    • Speech
    • Challenges
    • Trading off speed of interaction vs. accuracy

    View Slide

  27. Conclusion
    • Next frontier: Real-world Data
    • Augmented Reality
    • widely available on phones & tablets
    • Can be used for querying
    • Challenges: Visual Encoding, Querying Grammar
    • End-to-end stack for querying in augmented reality

    View Slide

  28. Thank you!
    https://interact.osu.edu

    View Slide