Evaluating Interpreting in Virtual Reality

New Computing Technologies Animation and Games Workshop- Presentation of the EU LLP project EVIVA, evaluating the performance of VLEs in interpreter-mediated communication and training

1. Panagiotis “Panos” D. Ritsos, Robert Gittins and Jonathan C. Roberts {p.ritsos, r.gittins, j.c.roberts} @bangor.ac.uk NEW COMPUTER TECHNOLOGIES WALES – ANIMATION & GAMES WORKSHOP – 15TH MAY 2014
2. Summary • Introduction to our use-case • Overview of project EVIVA • Brief overview of IVY – Virtual Environment (IVY-VE) • Challenges, solutions and opportunities • Future steps
3. Our use case… Training and simulation for interpreter-mediated communications • The rise of migration and multilingualism in Europe requires professional interpreters in business, legal, medical and many other settings. • Future interpreters therefore need to master an ever broadening range of interpreting scenarios and skills. • This is difficult to achieve with traditional teaching methods and in times of reduced teaching contact hours. • Also, a client-side understanding of what working with an interpreter involves is crucial, but efforts to educate potential clients of interpreters are scarce and normally separate from interpreter education.
4. Our projects… Interpreting in Virtual Reality (2011 – 2012) • …address the needs of future interpreters and users of interpreters in higher education, vocational training and adult learning contexts • …use 3D virtual environment technology to create an adaptive, learning environment that supports the acquisition and application of skills required in interpreter-mediated communication Evaluating the Education of Interpreters and their Clients through Virtual Learning Activities (2013 – 2014) • …evaluate the educational opportunities that three types of virtual learning environments — 3D virtual worlds, videoconference tools and video repositories of training material— offer for future interpreters and their clients
5. Projects relationship Timeline to projects relative to EVIVA Bangor, Surrey, Tubingen, StC Steinbeis, U. Adam Mickiewicz, U. Cyprus, Bar-Ilan, UVEG 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 EVIVA StC, Surrey Surrey, StC, UVEG Bangor, Surrey, StC, UAM, UCY, Bar-llan Bangor, Surrey, StC, UAM, UCY ELISA BACKBONE IVY Evaluation & analysis of user behaviour in relation to UX, pedagogy in VLEs used for interpreting, (including IVY-VE) Implementation of a 3D VLE based on the synergy of Second Life and web tech (Appfuse) through the SL URI namespace Repositories of audiovisual corpus material of interpreting- specific scenarios
6. EVIVA - aims • The aims of EVIVA project are: – to investigate the efficiency of VLEs and how they support different learning activities – to develop innovative evaluation methods for this investigation – to formulate design recommendations and guidelines for VLEs in interpreting • By evaluating educational solutions that reflect real-life situations, EVIVA contributes to bringing together the worlds of work and education. • Outcomes will inform VLE design and will be transferable to other fields of education • One of the evaluated VLEs is IVY-VE
7. Virtual Environment requirements • Re-create scenarios, using a large audio corpus, using a multi-user virtual world – ~75 scenarios shared around 20 locations • Enrich the corpus with more languages (for each scenario) – EN, GE, FR, GR, HE, RU, PL to begin with, followed by CH, IT, SP • Allow interpreting tutors (who can not program, sigh!) to add content on their own • Built three modes into the system: a training/simulation mode, a live mode and an exhibition mode. • Get the environment working in six months for students to use…
8. IVY-VE – our plan (circa 2011) • Due to time limitations, previous experience and existing assets we chose to built our IVY-VE around Second Life virtual world, – Advantages • easy to fabricate locations for our scenarios • easy to `built’ a multiplayer world • no special needs for infrastructure • wide use in academia and education – Disadvantages • No instancing, zoning and replication (like in MMOs) • No mature implementations for NPCs • A bit erratic at the time with maintenance service breaks
9. IVY-VE – Scenarios Interpreter Actor Observer Interpreter Actor A Actor B Observer Monologue Dialogue • Actors can be users or Non-player Characters (NPCs) • User is the interpreter – his audio is not recorded at this point
10. IVY-VE – architecture • IVY-VE comprises of three components (plus a database) – A scenario management system (SMS) that allows tutors to upload, edit and manage content. The scenarios are associated with locations in SL (user class: administrators) – The IVY island where most settings are built (user class: all + observers) – A Heads-up display, (using the HTML-on-a-prim feature) working as an extension to the SL GUI, giving access to the scenarios and initiating in-world events through the SL URI namespace (user class: interpreter) Virtual World Scenario Selection Menu (HUD) Scenes Scenario Management IVY-VE Interface Second Life Web User Administrator Observer
11. IVY-VE – our actors • Two types of actors used – `dummies’ and ‘bots’ – Bots have better appearance, can be controlled by whispers and through a telnet server... … but seem to have a mind on their own on zone restarts – Dummies are more stable but not realistically animated and of mediocre quality – In-house built dummies, (imported from Autodesk Pinocchio Project, tweaked in Maya etc.)
12. IVY-VE – HUD
13. IVY-VE – without instancing… • Each location can hold more than one scenarios, e.g., more than one taking place in an office setting • Due to that, we replicated popular locations into our skyboxes • Once a scenario is selected by a user all scenarios taking place on that location become unavailable to the remaining users • Upon scenario selection the HUD becomes a player • Teleportation events (to the scenarios location) are initiated • The user is required to press a teleport button
14. Preliminary evaluation • IVY-VE has been used in Interpreter training and modules, since 01/2012, in various sites: – University of Surrey, Center of Translation Studies – Adam Mickiewicz, Department of Translation Studies – University of Cyprus, Department of English Studies • Preliminary UX evaluations where conducted towards the end of 2012 • Interpreters appeared quite excited to have IVY-VE… • …but are they biased from having a dedicated tool (which they did not in the past)? • Longer, more thorough evaluation is required, as well as comparison with other tools Interpreters may use • This led to the EVIVA project…
15. EVIVA – adapting IVY-VE • Two new scenario hubs added on a seperate SL island. – A visitors centre, used for project promotion, access to the public etc. – An evaluation centre, used for the evaluation sessions taking place in 2014 – `Vanilla’ IVY-VE is still used by postgraduate students • New locations, using higher quality 3D assets added throughout the two additional hubs • Separate SMS and HUDs are used for each island
16. EVIVA– locations Replication of locations on skyboxes Visitors Centre Evaluation Centre Additional scenario locations
17. EVIVA - evaluation
18. EVIVA – opportunities - I • In EVIVA we can explore UX using new interfaces, such as Oculus Rift and the CtrlAltStudio Viewer (and soon the SL viewer) and the Razer Hydra
19. EVIVA – opportunities - II • We have also been experimenting with the implementation of a new bespoke system using various technologies, like WebGL/Three.js and Unity3D
20. Future challenges • We still have lots to do regarding: – Avatars, movement, realism – Customizable NPCs – Multiplayer functionality (instancing, replication, zoning etc.) – Reuse of our Scenario Management System – Synchronized facial animations and gestures • If this is your area and expertise or share the same questions… lets have a chat about it!
21. Thank you! http://www.virtual-interpreting.net @vr_interpreting @ritsos_p