Standards in Augmented Reality - Towards Prototyping Haptic Medical AR

Augmented Reality technology has been used in medical visualization applications in various different ways. Haptics, on the other hand, are a popular method of interacting in Augmented and Virtual Reality environments. We present how reliance on standards benefits the fusion of these technologies, through a series of research themes, carried out in Bangor University, UK (and international partners), as well as within the activities domain of the Research Institute of Visual Computing (RIVIC), UK.

Transcript

1. International AR Standards Meeting-March 1-2, 2013 Standards in Augmented Reality - Towards Prototyping Haptic Medical AR Panagiotis D. Ritsos  Nigel W. John  Jonathan C. Roberts School of Computer Science, Bangor University, UK {p.ritsos, n.w.john, j.c.roberts}@bangor.ac.uk
2. School of Computer Science Ysgol Gwyddorau Cyfrifiadurol • Revisit AR/MR definitions – to emphasise the importance of interaction • Discuss briefly some popular standards for medical visualization, for which researchers in Bangor University have contributed • Present HITPROTO, a toolkit for prototyping haptic interactions • Suggest an architecture, based on the use of standards and using HITPROTO for prototyping haptic interactions in medical AR. Presentation Outline
3. School of Computer Science Ysgol Gwyddorau Cyfrifiadurol VMG Group, School of Computer Science, Bangor University, UK http://www.vmg.cs.bangor.ac.uk/ Our research interests include medical visualization, virtual environments, information visualization and visual analytics, use of haptic interfaces, segmentation, artificial life, high dynamic range imaging, and augmented reality. Research Institute of Visual Computing http://www.rivic.org.uk A collaborative amalgamation of research programmes between the computer science departments in Aberystwyth, Bangor, Cardiff and Swansea Universities Bangor CS & RIVIC
4. School of Computer Science Ysgol Gwyddorau Cyfrifiadurol • Azuma’s definition: combines real and virtual, interactive in real time, registered in 3-D • AR is a subset of Milgram’s and Kishino’s MR • Broaden MR to include paradigms of varying conformance Azuma’s definition, as discussed by Mackay (1998) – maybe include notions of Weiser’s Ubicomp (1991)? • Emphasis on AR being interactive • Using senses beyond vision offers a more organic sense of immersion Revisiting AR Definitions
5. School of Computer Science Ysgol Gwyddorau Cyfrifiadurol • Digital Imaging and Communications in Medicine (DICOM)  MedX3D, an extension of X3D: - the specification of the X3D Volume Rendering Extensions (VRE) - the MedX3D profile, - segmentation and ontology support, - Import/export library and a web browser plugin that reads DICOM data and provided 3D visualization through the VRE • Building on the above standards, medical simulators benefit from increased fidelity, as hardware develops, becomes cheaper and more accessible - e.g., using H3DAPI - an open-source haptics software development platform that uses OpenGL and X3D. - It extends X3D with support for haptic interaction and volume rendering - allows users to build applications for haptic devices, combining X3D, C++ and Python. - also has support for physics engines such as PhysX and SOFA. The latter is specifically designed for surgery simulation. X3D in Medical AR
6. School of Computer Science Ysgol Gwyddorau Cyfrifiadurol Haptics & Medical Applications E.g., Image Guided Interventional Needle Simulation (ImaGINe-S) Bello, Fernando, Andrew Bulpitt, Derek A. Gould, Richard Holbrey, Carrie Hunt, Thien How, Nigel W. John et al. "ImaGINe-S: Imaging Guided Interventional Needle Simulation." In Eurographics 2009-Medical Prize, pp. 5-8. The Eurographics Association, 2009.
7. School of Computer Science Ysgol Gwyddorau Cyfrifiadurol • HITPROTO enables users to quickly create haptic interactions through a graphical programming interface. • The toolkit uses a modular approach where the developer drags and drops components into a design area (canvas), assembling diagrams that translate into haptic interactions. • Each component, called ‘Block’, maps to elements and functions of the H3D API. HITPROTO - I Left Panel (blocks) Menu Bar Bottom Panel (block properties) Canvas
8. School of Computer Science Ysgol Gwyddorau Cyfrifiadurol • Creating a haptic interaction with HITPROTO is a three step process: - (a) The developer builds an interaction diagram by connecting modular blocks together, referencing objects in an X3D scene, - (b) the diagram is saved in an intermediate .hit XML file - (c) HITPROTO outputs a H3D Python file. The H3DAPI viewer is then used to execute the scene. HITPROTO - II HITPROTO Python File H3DAPI Code Generation Usage & Abstraction Execution
9. School of Computer Science Ysgol Gwyddorau Cyfrifiadurol • Built on standards, HITPTOTO can be part of an architecture that allows developers and researchers to prototype haptic interactions for medical AR. • Reliance on standards allows fusion with other APIs or their extensions/libraries like HART, an bridge between H3D and ARToolkit. • At the moment requires support from more devices. HITPROTO & Prototyping Haptic Medical AR HITPROTO X3D (MedX3D) Scene Medical AR Application H3D API Interaction HART 3D scan data
10. School of Computer Science Ysgol Gwyddorau Cyfrifiadurol Thank you! Dydd Gŵyl Dewi Hapus