Standards in Augmented Reality - Towards Prototyping Haptic Medical AR

Augmented Reality technology has been used in medical visualization applications in various different ways. Haptics, on the other hand, are a popular method of interacting in Augmented and Virtual Reality environments. We present how reliance on standards benefits the fusion of these technologies, through a series of research themes, carried out in Bangor University, UK (and international partners), as well as within the activities domain of the Research Institute of Visual Computing (RIVIC), UK.

Transcript

1. International AR Standards Meeting-March 1-2, 2013
   Standards in Augmented Reality -
   Towards Prototyping Haptic Medical AR
   Panagiotis D. Ritsos  Nigel W. John  Jonathan C. Roberts
   School of Computer Science, Bangor University, UK
   {p.ritsos, n.w.john, j.c.roberts}@bangor.ac.uk
   

   View Slide

2. School of Computer Science
   Ysgol Gwyddorau Cyfrifiadurol
   • Revisit AR/MR definitions – to emphasise the importance of interaction
   • Discuss briefly some popular standards for medical visualization, for
   which researchers in Bangor University have contributed
   • Present HITPROTO, a toolkit for prototyping haptic interactions
   • Suggest an architecture, based on the use of standards and using
   HITPROTO for prototyping haptic interactions in medical AR.
   Presentation Outline
   

   View Slide

3. School of Computer Science
   Ysgol Gwyddorau Cyfrifiadurol
   VMG Group, School of Computer Science, Bangor University, UK
   http://www.vmg.cs.bangor.ac.uk/
   Our research interests include medical visualization, virtual environments, information
   visualization and visual analytics, use of haptic interfaces, segmentation, artificial life,
   high dynamic range imaging, and augmented reality.
   Research Institute of Visual Computing
   http://www.rivic.org.uk
   A collaborative amalgamation of research programmes between the computer science
   departments in Aberystwyth, Bangor, Cardiff and Swansea Universities
   Bangor CS & RIVIC
   

   View Slide

4. School of Computer Science
   Ysgol Gwyddorau Cyfrifiadurol
   • Azuma’s definition: combines real and virtual, interactive in real time,
   registered in 3-D
   • AR is a subset of Milgram’s and Kishino’s MR
   • Broaden MR to include paradigms of varying conformance Azuma’s
   definition, as discussed by Mackay (1998) – maybe include notions of
   Weiser’s Ubicomp (1991)?
   • Emphasis on AR being interactive
   • Using senses beyond vision offers a more organic sense of immersion
   Revisiting AR Definitions
   

   View Slide

5. School of Computer Science
   Ysgol Gwyddorau Cyfrifiadurol
   • Digital Imaging and Communications in Medicine (DICOM) 
   MedX3D, an extension of X3D:
   - the specification of the X3D Volume Rendering Extensions (VRE)
   - the MedX3D profile,
   - segmentation and ontology support,
   - Import/export library and a web browser plugin that reads DICOM data and provided 3D
   visualization through the VRE
   • Building on the above standards, medical simulators benefit from
   increased fidelity, as hardware develops, becomes cheaper and more
   accessible - e.g., using H3DAPI
   - an open-source haptics software development platform that uses OpenGL and X3D.
   - It extends X3D with support for haptic interaction and volume rendering
   - allows users to build applications for haptic devices, combining X3D, C++ and Python.
   - also has support for physics engines such as PhysX and SOFA. The latter is specifically
   designed for surgery simulation.
   X3D in Medical AR
   

   View Slide

6. School of Computer Science
   Ysgol Gwyddorau Cyfrifiadurol
   Haptics & Medical Applications
   E.g., Image Guided Interventional Needle Simulation (ImaGINe-S)
   Bello, Fernando, Andrew Bulpitt, Derek A. Gould, Richard Holbrey, Carrie Hunt, Thien How,
   Nigel W. John et al. "ImaGINe-S: Imaging Guided Interventional Needle Simulation." In
   Eurographics 2009-Medical Prize, pp. 5-8. The Eurographics Association, 2009.
   

   View Slide

7. School of Computer Science
   Ysgol Gwyddorau Cyfrifiadurol
   • HITPROTO enables users to quickly create haptic interactions through a
   graphical programming interface.
   • The toolkit uses a modular approach where the developer drags and
   drops components into a design area (canvas), assembling diagrams
   that translate into haptic interactions.
   • Each component, called ‘Block’, maps to elements and functions of the
   H3D API.
   HITPROTO - I
   Left Panel
   (blocks) Menu Bar
   Bottom Panel
   (block properties)
   Canvas
   

   View Slide

8. School of Computer Science
   Ysgol Gwyddorau Cyfrifiadurol
   • Creating a haptic interaction with HITPROTO is a three step process:
   - (a) The developer builds an interaction diagram by connecting modular blocks together,
   referencing objects in an X3D scene,
   - (b) the diagram is saved in an intermediate .hit XML file
   - (c) HITPROTO outputs a H3D Python file. The H3DAPI viewer is then used to execute the
   scene.
   HITPROTO - II
   HITPROTO Python File
   H3DAPI
   Code
   Generation
   Usage &
   Abstraction
   Execution
   

   View Slide

9. School of Computer Science
   Ysgol Gwyddorau Cyfrifiadurol
   • Built on standards, HITPTOTO can
   be part of an architecture that allows
   developers and researchers to
   prototype haptic interactions for
   medical AR.
   • Reliance on standards allows fusion
   with other APIs or their
   extensions/libraries like HART, an
   bridge between H3D and ARToolkit.
   • At the moment requires support from
   more devices.
   HITPROTO & Prototyping Haptic Medical AR
   HITPROTO
   X3D
   (MedX3D)
   Scene
   Medical AR
   Application
   H3D API
   Interaction
   HART
   3D scan
   data
   

   View Slide

10. School of Computer Science
    Ysgol Gwyddorau Cyfrifiadurol
    Thank you!
    Dydd Gŵyl Dewi Hapus
    

    View Slide