A Framework for Remote Usability Evaluation on Mobile Devices

Presentation slides for my B.Sc. thesis at TU München.

Transcript

1. Technische Universität München A Framework for Remote Usability Evaluation on Mobile Devices Bachelorarbeit in Informatik Daniel Bader
2. Technische Universität München Daniel Bader 15.04.2011 Introduction Usability - An indicator for the ease of use and acceptability of a system Usability Evaluation - Methods for determining the usability of a system - Usability data are collected and analyzed by human evaluators Common activities (Ivory and Hearst, 2001) 2 Common usability evaluation activities Capture Analysis Critique Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1)
3. Technische Universität München Daniel Bader 15.04.2011 Introduction Automated Usability Evaluation - Some or all phases of an Usability Evaluation are automated - Various degrees of automation exist Remote Usability Evaluation - Evaluators and test users are separated in space and/or time Automated Remote Usability Evaluation - One of the “ultimate goals” in the field of usability evaluation 3
4. Technische Universität München Daniel Bader 15.04.2011 Introduction Usability Evaluation is especially important on mobile devices - User interfaces on mobile devices became more complex - Mobile context creates new challenges: - different environments, distractions, ... Current mobile devices have a rich ability to track their surroundings - Microphone, GPS, accelerometer, video camera, ... Usability Evaluation can be (partly) automated 4
5. Technische Universität München Daniel Bader 15.04.2011 Problem statement A lack of software support for Usability Evaluation on current mobile platforms - No automation support in existing tools - No support for built-in tracking capabilities (microphone, accelerometer, GPS, ...) - No support for collecting user interactions graphically (screenshots, video) 5
6. Technische Universität München Daniel Bader 15.04.2011 Proposed solution The muEvaluationFramework (mobile usability Evaluation Framework) An application-independent software framework for remote usability evaluation on mobile platforms which supports automation in all phases of a usability evaluation. 6 Common usability evaluation activities Capture Analysis Critique Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1)
7. Technische Universität München Daniel Bader 15.04.2011 Outline ✓ Introduction ✓ Problem statement • Proposed solution – Requirements specification – System design – Object design • Prototype demo • Future work 7
8. Common usability evaluation activities Capture Analysis Critique Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1) Technische Universität München Daniel Bader 15.04.2011 Functional requirements Capture phase - Session-based usability data collection - Support for multiple usability-data sources - User input events - Application events - Device sensor events - Live preview for collected events 8
9. Common usability evaluation activities Capture Analysis Critique Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1) Technische Universität München Daniel Bader 15.04.2011 Functional requirements Analysis phase - Interpretation of the captured data - Detection of usability problems - e.g. Human Interface Guidelines (HIG) violations - Usability summary generation - Most used views and UI widgets - Heat maps for user input events - ... 9
10. Common usability evaluation activities Capture Analysis Critique Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1) Technische Universität München Daniel Bader 15.04.2011 Functional requirements Critique phase - Report generation - Reports summarize the interpretation results - Reports can be viewed using a web browser - Report configuration - Selection of the report sections 10
11. Technische Universität München Daniel Bader 15.04.2011 Nonfunctional requirements Minimal setup work required by the developer - Low entry barrier - Quick setup for existing projects - Goal: “link against one library and make one method call” Wireless communication with the mobile device - Test users can move freely and do not feel restricted - Evaluation can be performed outside of a laboratory Extensibility - Easily add new sensors, interpreters, and sections - Framework intended as a research platform 11
12. Technische Universität München Daniel Bader 15.04.2011 Use case model 12 <<actor>> HostApplication ControlSession ConfigureReport ViewReport CaptureUsabilityData muEvaluationFramework TestUser Developer <<initiate>> <<initiate>> <<initiate>> <<participate>> <<participate>> <<initiate>> Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1)
13. Technische Universität München Daniel Bader 15.04.2011 Object model and Dynamic model 13 Capture Analysis Critique Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1) Conceptual sub-models for each of the three phases
14. Technische Universität München Daniel Bader 15.04.2011 Object model and Dynamic model 13 Capture Analysis Critique Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1) Conceptual sub-models for each of the three phases
15. Technische Universität München Daniel Bader 15.04.2011 Capture phase 14 HostApplication TestUser EvaluationSession Developer 4: stop() 3: interactWith() 2: participate() 1: start() Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1) The life cycle of an EvaluationSession
16. Technische Universität München Daniel Bader 15.04.2011 Capture phase 15 +start() +stop() +selectSensors(sensorSet : Sensor []) EvaluationSession TestUser Developer +launch() +quit() HostApplication UsabilityData MobileDevice +attachTo(t : SensorTarget) +enable() +disable() Sensor monitors collects runs on uses is collected in is analyzed in participates in initiates interacts with Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1)
17. -timestamp : Time <<abstract>> Event +attachTo(t : SensorTarget) +enable() +disable() <<abstract>> Sensor +addEvent(e : Event) SessionJournal UsabilityData * * {ordered} collects <<represents>> creates Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1) Technische Universität München Daniel Bader 15.04.2011 Capture phase 16 Sensors and Events
18. Technische Universität München Daniel Bader 15.04.2011 Capture phase 17 <<abstract>> Sensor TouchInputSensor AudioSensor TextEntrySensor ScreenSensor Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1) Sensor specializations
19. <<abstract>> Event UserInputEvent CustomEvent ApplicationEvent UserInterfaceEvent DeviceSensorEvent Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1) Technische Universität München Daniel Bader 15.04.2011 Capture phase 18 Event specializations
20. Technische Universität München Daniel Bader 15.04.2011 Object model and Dynamic model 19 Capture Analysis Critique Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1)
21. Technische Universität München Daniel Bader 15.04.2011 Analysis phase 20 <<abstract>> InterpretationResult <<abstract>> Interpreter InterpreterController +addResult(r : InterpretationResult) ResultStore -timestamp : Time <<abstract>> Event * * * 1..* stores results in controls * interprets interprets generates Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1) Interpreters work on data collected in the Capture phase and generate InterpretationResults.
22. <<abstract>> Interpreter OverviewInterpreter ViewChangesInterpreter ScreenshotInterpreter UIGuidelineChecker ViewDurationInterpreter TouchHeatmapInterpreter AnalysisStatisticsInterpreter Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1) Technische Universität München Daniel Bader 15.04.2011 Analysis phase 21 Interpreter specializations
23. Technische Universität München Daniel Bader 15.04.2011 Object model and Dynamic model 22 Capture Analysis Critique Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1)
24. Technische Universität München Daniel Bader 15.04.2011 Critique phase 23 Report InterpretationResult Developer HostApplication recommends changes to views summarizes Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1)
25. +addSection(s : Section) Report <<abstract>> Section OverviewSection EventTimeline TouchHeatmaps UserNavigationPath 1..* {ordered} Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1) Technische Universität München Daniel Bader 15.04.2011 Critique phase 24 Report document model
26. Technische Universität München Daniel Bader 15.04.2011 High-level dynamic model 25 Capture Analysis Critique TestUser Report ResultStore SessionJournal UsabilityData InterpretationResults Events Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1) Inputs and outputs of the three phases
27. Technische Universität München Daniel Bader 15.04.2011 System design 26 Capture Critique Analysis ReportController ResultStorage AnalysisController Interpretation SectionGenerator CaptureLibrary CaptureServer Communication Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1) Subsystem decomposition of the framework
28. Technische Universität München Daniel Bader 15.04.2011 System design 27 <<device>> Mobile device <<device>> Desktop computer <<component>> CaptureLibrary <<component>> HostApplication <<component>> Critique <<component>> Analysis <<component>> CaptureServer <<artifact>> CaptureServer.py <<artifact>> libCapture.a <<artifact>> ReportGenerator.py <<manifest>> <<wireless network>> <<manifest>> <<manifest>> <<manifest>> Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1) Deployment of the framework
29. Technische Universität München Daniel Bader 15.04.2011 Object design 28 Capture Critique Analysis ReportController ResultStorage AnalysisController Interpretation SectionGenerator CaptureLibrary CaptureServer Communication Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1)
30. Technische Universität München Daniel Bader 15.04.2011 CaptureLibrary - Method interception 29 UIWindow +sendEvent() +... +... Original implementation of sendEvent() Event UIWindow sendEvent() behavior before method interception
31. Technische Universität München Daniel Bader 15.04.2011 CaptureLibrary - Method interception 29 UIWindow +sendEvent() +... +... Original implementation of sendEvent() Event UIWindow sendEvent() behavior before method interception
32. Technische Universität München Daniel Bader 15.04.2011 CaptureLibrary - Method interception 29 UIWindow +sendEvent() +... +... Original implementation of sendEvent() Event UIWindow sendEvent() behavior before method interception
33. Technische Universität München Daniel Bader 15.04.2011 CaptureLibrary - Method interception 30 UIWindow +sendEvent() +... +... Original implementation of sendEvent() UIWindow sendEvent() behavior after method interception Our implementation of sendEvent() Event CaptureLibrary
34. Technische Universität München Daniel Bader 15.04.2011 CaptureLibrary - Method interception 30 UIWindow +sendEvent() +... +... Original implementation of sendEvent() UIWindow sendEvent() behavior after method interception Our implementation of sendEvent() Event CaptureLibrary
35. Technische Universität München Daniel Bader 15.04.2011 CaptureLibrary - Method interception 30 UIWindow +sendEvent() +... +... Original implementation of sendEvent() UIWindow sendEvent() behavior after method interception Our implementation of sendEvent() Event CaptureLibrary
36. Technische Universität München Daniel Bader 15.04.2011 CaptureLibrary - Method interception 30 UIWindow +sendEvent() +... +... Original implementation of sendEvent() UIWindow sendEvent() behavior after method interception Our implementation of sendEvent() Event CaptureLibrary Event
37. Technische Universität München Daniel Bader 15.04.2011 Object design 31 Capture Critique Analysis ReportController ResultStorage AnalysisController Interpretation SectionGenerator CaptureLibrary CaptureServer Communication Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1)
38. Technische Universität München Daniel Bader 15.04.2011 Analysis subsystem 32 Usage of the Blackboard design pattern in the Analysis subsystem ResultStore InterpreterController Interpreter +getResultsOfType() +addResult() <<Interface>> Blackboard +canContribute() +updateBlackboard() <<Interface>> BlackboardExpert +loop() +getReadyExperts() <<Interface>> BlackboardSupervisor supervises 1..* controls updates and accesses Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1)
39. Technische Universität München Daniel Bader 15.04.2011 Analysis subsystem 33 Dependencies between Interpreters are resolved automatically InterpreterA InterpreterB InterpreterC InterpreterD Generated order of execution:
40. Technische Universität München Daniel Bader 15.04.2011 Analysis subsystem 33 Dependencies between Interpreters are resolved automatically InterpreterA InterpreterB InterpreterC InterpreterD InterpreterC InterpreterB InterpreterA InterpreterD Generated order of execution: {C} → {B} → {A, D}
41. Technische Universität München Daniel Bader 15.04.2011 Object design 34 Capture Critique Analysis ReportController ResultStorage AnalysisController Interpretation SectionGenerator CaptureLibrary CaptureServer Communication Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1)
42. Technische Universität München Daniel Bader 15.04.2011 Critique subsystem 35 -body : String Element -title : string -header : string -elements : Element[] +__init__(title) +add_element(e) +generate_html(result_store) : string <<abstract>> Section OverviewSection NavigationPathSection TouchHeatmapSection ViewDurationSection UIGuidelineIssuesSection -title : string -header : string -footer : string -sections : Section[] +add_section(s) +generate_html(result_store) : string +generate(result_store, filename) Report -reportFilename : string -eventsFilename : string -interpreters : string[] -sections : string[] +addInterpreter(i : string) +addSection(s : string) +setEventsFilename(s : string) +setReportFilename(s : string) +generateReport() ReportGeneratorFacade 1..* 1..* Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1)
43. Technische Universität München Daniel Bader 15.04.2011 Prototypical implementation Full vertical prototype for Apple iOS - Implemented in Objective-C and Python - Tested with the applications Wordpress for iOS and PlainNote Capture support - Automatic network setup using Zeroconf/Bonjour - Custom, stateless XML-based network protocol - Data transmission is performed on background thread Analysis support - Six interpreters were implemented Critique support - Generates HTML output - Five section types were implemented 36
44. Technische Universität München Daniel Bader 15.04.2011 Demo 37
45. Technische Universität München Daniel Bader 15.04.2011 Future work Add more sensors and interpreters - e.g. audio and video recording Use the framework in a real world scenario Add support for Google Android - Java has rich support for reflection Add ability to playback sessions - Play back captured user interactions on the device Integrate framework in FastFix - Perform usability error detection - Provide access to user interaction data and context information 38
46. Technische Universität München Daniel Bader 15.04.2011 Thank you! Questions? 39
47. Technische Universität München Daniel Bader 15.04.2011 Backup slides 40
48. <<actor>> HostApplication PreviewEvents SelectSensors StopSession StartSession ControlSession ConfigureReport ViewReport CaptureUsabilityData muEvaluationFramework TestUser Developer <<Extend>> <<initiate>> <<initiate>> <<initiate>> <<participate>> <<participate>> <<initiate>> Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1) Technische Universität München Daniel Bader 15.04.2011 Use case model (with refinements) 41
49. Technische Universität München Daniel Bader 15.04.2011 Capture phase 42 Sensors and SensorTargets -isActive : boolean +attachTo(t : SensorTarget) +enable() +disable() <<abstract>> Sensor HostApplication UsabilityData SensorTarget * 1..* * observes collects monitors Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1)
50. SessionJournal e1 : UserInputEvent TestUser a1 : Action Sensor 2: detect() 4: addEvent(e1) 3: create(time) 1: perform() Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1) Technische Universität München Daniel Bader 15.04.2011 Capture phase 43 Event detection example
51. Technische Universität München Daniel Bader 15.04.2011 Capture phase 44 Capture phase Initialize SessionJournal Initialize Sensors Create Event Observe SensorTargets Continue with next Sensor ActiveSensors <<structured>> for each Sensor SessionJournal [targets did not change] Event [targets changed] Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1)
52. Run analysis phase Initialize ResultStore Initialize Interpreters Update ResultStore Run Interpreter ResultStore <<structured>> Loop SessionJournal InterpretationResult Event Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1) Technische Universität München Daniel Bader 15.04.2011 Analysis phase 45
53. Generate Report Add Section to Report Generate Section Select Section to generate ReportConfiguration <<structured>> Loop Report ResultStore InterpretationResult Section Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1) Technische Universität München Daniel Bader 15.04.2011 Critique phase 46
54. Technische Universität München Daniel Bader 15.04.2011 System design 47 The Capture package Capture CaptureLibrary CaptureServer Communication <<abstract>> Sensor SensorTarget <<abstract>> Event SessionJournal NetworkConnection NetworkMessage CaptureManager ServerController 1..* 1..* * * * receives Events sends Events observes creates Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1)
55. Technische Universität München Daniel Bader 15.04.2011 System design 48 The Analysis package Analysis Capture AnalysisController Interpretation Storage InterpretationResult Interpreter InterpreterController ResultStore stores results in 1..* * * generates SessionJournal Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1)
56. Technische Universität München Daniel Bader 15.04.2011 System design 49 The Critique package Critique Analysis SectionGenerator ReportController ReportConfiguration ReportGenerator Report Section SectionGenerator 1..* 1..* uses generates ResultStore uses generates Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1)