School of Computer Science Ysgol Gwyddorau Cyfrifiadurol Panagiotis D. Ritsos1  Sabrina A. Panëels2  Peter J. Rodgers3  Jonathan C. Roberts1 [1] School of Computer Science, Bangor University, UK - {p.ritsos, j.c.roberts}@bangor.ac.uk) [2] CEA, LIST, France – [email protected] [3] School of Computer Science, University of Kent, UK – [email protected] VMG OCTOBER 2013 Prototyping Haptic Data Visualizations 

School of Computer Science Ysgol Gwyddorau Cyfrifiadurol  Introduction to Haptic Data Visualization (HDV).  Related work using Tactile Representations  Overview of the concept of prototyping interactions  In-depth description of the HITPROTO toolkit  Examples of HDVs produced using the HITPROTO toolkit  User evaluation of the HITPROTO toolkit  Preliminary investigation with users with vision impairments  Lessons learned Presentation Outline 

School of Computer Science Ysgol Gwyddorau Cyfrifiadurol  S.A. Paneels, P.D. Ritsos, P.J.Rodgers, and J.C.Roberts, Prototyping 3D haptic data visualizations, Computers & Graphics, Volume 37, Issue 3, May 2013, pp. 179-192  P.D. Ritsos, S.A. Paneels, P.J.Rodgers, and J.C. Roberts, Towards a Formalized Process for Creating Haptic Data Visualization, Visweek 2013, Atlanta, Georgia, USA Related Publications 

School of Computer Science Ysgol Gwyddorau Cyfrifiadurol  The use of haptic devices is becoming widespread, particularly with their growth in the home-games market.  Likewise, the number of haptic devices and APIs is increasing.  However, it is still difficult to program and develop haptic applications.  One way to tackle this is by providing prototyping tools or frameworks.  This paper presents a visual prototyping tool for interactions for…  … haptic data visualization (HDV), also called haptification - the use of haptic devices to represent and realize information and data. Introduction 

School of Computer Science Ysgol Gwyddorau Cyfrifiadurol  In software engineering (SE) the term prototyping refers to the rapid development of a software solution and has, typically, two forms:  evolutionary prototyping, where the user refines the prototype in an ongoing process before it turns into the final system  throw-away prototyping where subsequent systems are created and then discarded in turn before the final development and delivery of a software produce.  Our approach fuses both forms through the rapid creation of haptic interactions…  …and the generation of reusable code that can be built-upon by developers. Prototyping 

School of Computer Science Ysgol Gwyddorau Cyfrifiadurol  The haptic visualization display can be considered to consist of three parts:  First a model is created that contains the haptic information that is going to be perceived.  The generation of the model closely follows the visualization process, where the data is enhanced and mapped into an abstract/haptic model.  Second, haptic rendering is used to compute the forces required to realize the model. The Haptic Data Visualization  Third, an electro-mechanical device is used to exert the forces that the user perceives. 

School of Computer Science Ysgol Gwyddorau Cyfrifiadurol  Braille tactile writing system (paper embossing)  microcapsule paper, thermo-form and vacuum-form  Wikkistix, which are strings doped with wax to make the string keep its form Tactile Graphics and Static Representations 

School of Computer Science Ysgol Gwyddorau Cyfrifiadurol  HITPROTO attempts to ameliorate the technical complexities and provide an interface that is closer to a natural language (e.g., “Wait for a button press, then add and start guidance”).  We use a modular approach in HITPROTO, where users drag-and- drop components onto a canvas and connect them together to provide the logic for the haptic visualization.  Parameters of the blocks can be set to describe specific behaviours.  The arrangement of the blocks describes the semantic structure of the haptic interaction The HITPROTO Toolkit - Design 

School of Computer Science Ysgol Gwyddorau Cyfrifiadurol  There are four main parts to the process: 1. The user makes the HDV scenario by connecting modular blocks on the canvas, 2. this information is saved in the .hit XML file, 3. HITPROTO generates a H3D Python file, 4. H3DAPI is used to execute the haptic scene The HITPROTO Toolkit – The process 

School of Computer Science Ysgol Gwyddorau Cyfrifiadurol  HITPROTO uses H3DAPI (h3dapi.org), is implemented in C++ with WxWidgets and combines X3D, C++ and Python  Current version tested with the Geomagic Touch and Touch X (formerly Sensable Phantom Omni and Desktop, respectively), but in theory can be used with devices supported by H3DAPI The HITPROTO Toolkit – The application  GUI is like Lego NXT Mindstorms visual programming language 

School of Computer Science Ysgol Gwyddorau Cyfrifiadurol HITPROTO Blocks • Action blocks - describe the addition, removal, creation and modification of haptic and guidance effects Stop – compulsory block that delimits the end of the ‘interaction scenario’ Guidance Add – creates a guidance instance. Includes a spring to attach to the device and an anchor to visualize the spring and parameters such as path and speed/duration. Haptic Effect – creates a chosen haptic effect. The available haptic effects are: SpringEffect, Magnetic Line(s) and PositionFunctionEffect (model-based). • Flow blocks that control the flow of the data by listening to events. Wait For – enables the interruption of a sequence of actions until a chosen event happens such as a haptic device/mouse button or a keyboard key being pressed/released, an elapsed time or the activation of a spring Switch and Switch End – Checks if a condition is satisfied or not before executing a set of actions contained between the two blocks 

School of Computer Science Ysgol Gwyddorau Cyfrifiadurol 1 2 3 4 5 6 9 10 11 12 13 14 Block Diagram Intermediate Form 

School of Computer Science Ysgol Gwyddorau Cyfrifiadurol • We parse the .hit file sequentially, and the parameters stored as XML attributes are passed to the respective Python code components. • However, code generation is not a linear process. This is because the labels and values may be defined at a lower position in the .hit file • The Python code generated provides a runnable implementation and can be executed directly from HITPROTO’s interface, or the Python file run separately. • For the mentor/blind-user situation this may be enough, and it affords quick development and deployment. • However the code, being standard H3DAPI Python, can be reused and extended for the needs of another application Block Diagram Parsing and Python Output 

School of Computer Science Ysgol Gwyddorau Cyfrifiadurol HITPROTO Examples Irises data set – force model UK Public Spending since 1963 – magnetic lines + X3D visual elements 

School of Computer Science Ysgol Gwyddorau Cyfrifiadurol  HITPROTO was evaluated by postgraduates that fitted the profile of potential support workers for blind users  We used a ‘convenience sample’ of nine participants: ages 18-65, 6 female and male, field study anything but Computer Science  Participants completed four tasks of gradually increasing difficulty and success rates, time for task completion was recorded.  A post-experiment questionnaire was completed, consisting of the System Usability Scale (SUS) and open ended questions. Usability Evaluation - I 

School of Computer Science Ysgol Gwyddorau Cyfrifiadurol  The average SUS score, rating overall usability, was 67%. Individual scores from participants ranged between 50% and 92.5%, except for one at 17.5%.  The results indicate that earlier tasks were completed successfully with no or little help, whereas help was required for latter ones.  This behaviour was expected because the latter tasks were designed to be more challenging than earlier ones.  Overall 88.9% of the attempts at the tasks resulted in a working interaction, with or without help, while only 8.3% of them resulted in failures, despite the help given. Usability Evaluation - II 

School of Computer Science Ysgol Gwyddorau Cyfrifiadurol  We performed a talk-aloud session of the Guided Tour and the magnetic line-graphs  They were very positive over haptic visualization techniques and saw much potential in the tool  They discussed how they explored information in general, and explained that they had used static examples such as thermoformed images. Using HITPROTO with Blind Students  Emphasised the fact that many tasks take them longer to perform in comparison to a sighted user.  Saw the benefit of the Guided Tour example that led them to points of interest.  We discussed how audio annotation could be incorporated into the representations 

School of Computer Science Ysgol Gwyddorau Cyfrifiadurol  Visual dataflow programming assists novice users  Medium granularity required  Block Appearance improves User Experience  Alternative notation (block diagram and Python code) assists expert users  Reuse of models/patterns towards Learning by Example  Reliance on standards (e.g., X3D) and popular APIs (e.g., H3DAPI) allows reuse and extensibility  Usage patterns  Error Checking  Default parameter values Lessons Learnt – Prototyping tools 

School of Computer Science Ysgol Gwyddorau Cyfrifiadurol  Objective mapping - clear mapping from the data to the haptic representation.  Value - enable the user to understand the underlying data quantity. Values may be represented through other modalities (such as sound).  Reproducible - identical if the same data is loaded in another session  Different data can be loaded to be represented by HDV.  Systematic set of interactions that allow exploring in 3D  Provide feature guidance or touring mechanisms  Exploration is encouraged Lessons Learnt – HDVs 

School of Computer Science Ysgol Gwyddorau Cyfrifiadurol  There are many areas that can be improved in HITPROTO.  new block diagram editor with an error-checking parser,  new custom block creator,  ‘hinting’ mechanisms to the dataflow block diagram editor,  use of ‘Patterns’ for users to build typical configurations. Future Work & Conclusions 

School of Computer Science Ysgol Gwyddorau Cyfrifiadurol Thank You!