A Framework for Remote Usability Evaluation on Mobile Devices

Slide 1

Slide 1 text

Technische Universität München A Framework for Remote Usability Evaluation on Mobile Devices Bachelorarbeit in Informatik Daniel Bader

Slide 2

Slide 2 text

Technische Universität München Daniel Bader 15.04.2011 Introduction Usability - An indicator for the ease of use and acceptability of a system Usability Evaluation - Methods for determining the usability of a system - Usability data are collected and analyzed by human evaluators Common activities (Ivory and Hearst, 2001) 2 Common usability evaluation activities Capture Analysis Critique Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1)

Slide 3

Slide 3 text

Technische Universität München Daniel Bader 15.04.2011 Introduction Automated Usability Evaluation - Some or all phases of an Usability Evaluation are automated - Various degrees of automation exist Remote Usability Evaluation - Evaluators and test users are separated in space and/or time Automated Remote Usability Evaluation - One of the “ultimate goals” in the ﬁeld of usability evaluation 3

Slide 4

Slide 4 text

Technische Universität München Daniel Bader 15.04.2011 Introduction Usability Evaluation is especially important on mobile devices - User interfaces on mobile devices became more complex - Mobile context creates new challenges: - different environments, distractions, ... Current mobile devices have a rich ability to track their surroundings - Microphone, GPS, accelerometer, video camera, ... Usability Evaluation can be (partly) automated 4

Slide 5

Slide 5 text

Technische Universität München Daniel Bader 15.04.2011 Problem statement A lack of software support for Usability Evaluation on current mobile platforms - No automation support in existing tools - No support for built-in tracking capabilities (microphone, accelerometer, GPS, ...) - No support for collecting user interactions graphically (screenshots, video) 5

Slide 6

Slide 6 text

Technische Universität München Daniel Bader 15.04.2011 Proposed solution The muEvaluationFramework (mobile usability Evaluation Framework) An application-independent software framework for remote usability evaluation on mobile platforms which supports automation in all phases of a usability evaluation. 6 Common usability evaluation activities Capture Analysis Critique Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1)

Slide 7

Slide 7 text

Technische Universität München Daniel Bader 15.04.2011 Outline ✓ Introduction ✓ Problem statement • Proposed solution – Requirements speciﬁcation – System design – Object design • Prototype demo • Future work 7

Slide 8

Slide 8 text

Common usability evaluation activities Capture Analysis Critique Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1) Technische Universität München Daniel Bader 15.04.2011 Functional requirements Capture phase - Session-based usability data collection - Support for multiple usability-data sources - User input events - Application events - Device sensor events - Live preview for collected events 8

Slide 9

Slide 9 text

Common usability evaluation activities Capture Analysis Critique Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1) Technische Universität München Daniel Bader 15.04.2011 Functional requirements Analysis phase - Interpretation of the captured data - Detection of usability problems - e.g. Human Interface Guidelines (HIG) violations - Usability summary generation - Most used views and UI widgets - Heat maps for user input events - ... 9

Slide 10

Slide 10 text

Common usability evaluation activities Capture Analysis Critique Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1) Technische Universität München Daniel Bader 15.04.2011 Functional requirements Critique phase - Report generation - Reports summarize the interpretation results - Reports can be viewed using a web browser - Report conﬁguration - Selection of the report sections 10

Slide 11

Slide 11 text

Technische Universität München Daniel Bader 15.04.2011 Nonfunctional requirements Minimal setup work required by the developer - Low entry barrier - Quick setup for existing projects - Goal: “link against one library and make one method call” Wireless communication with the mobile device - Test users can move freely and do not feel restricted - Evaluation can be performed outside of a laboratory Extensibility - Easily add new sensors, interpreters, and sections - Framework intended as a research platform 11

Slide 12

Slide 12 text

Technische Universität München Daniel Bader 15.04.2011 Use case model 12 <> HostApplication ControlSession ConfigureReport ViewReport CaptureUsabilityData muEvaluationFramework TestUser Developer <> <> <> <> <> <> Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1)

Slide 13

Slide 13 text

Technische Universität München Daniel Bader 15.04.2011 Object model and Dynamic model 13 Capture Analysis Critique Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1) Conceptual sub-models for each of the three phases

Slide 14

Slide 14 text

Slide 15

Slide 15 text

Technische Universität München Daniel Bader 15.04.2011 Capture phase 14 HostApplication TestUser EvaluationSession Developer 4: stop() 3: interactWith() 2: participate() 1: start() Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1) The life cycle of an EvaluationSession

Slide 16

Slide 16 text

Technische Universität München Daniel Bader 15.04.2011 Capture phase 15 +start() +stop() +selectSensors(sensorSet : Sensor []) EvaluationSession TestUser Developer +launch() +quit() HostApplication UsabilityData MobileDevice +attachTo(t : SensorTarget) +enable() +disable() Sensor monitors collects runs on uses is collected in is analyzed in participates in initiates interacts with Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1)

Slide 17

Slide 17 text

-timestamp : Time <> Event +attachTo(t : SensorTarget) +enable() +disable() <> Sensor +addEvent(e : Event) SessionJournal UsabilityData * * {ordered} collects <> creates Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1) Technische Universität München Daniel Bader 15.04.2011 Capture phase 16 Sensors and Events

Slide 18

Slide 18 text

Technische Universität München Daniel Bader 15.04.2011 Capture phase 17 <> Sensor TouchInputSensor AudioSensor TextEntrySensor ScreenSensor Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1) Sensor specializations

Slide 19

Slide 19 text

<> Event UserInputEvent CustomEvent ApplicationEvent UserInterfaceEvent DeviceSensorEvent Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1) Technische Universität München Daniel Bader 15.04.2011 Capture phase 18 Event specializations

Slide 20

Slide 20 text

Technische Universität München Daniel Bader 15.04.2011 Object model and Dynamic model 19 Capture Analysis Critique Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1)

Slide 21

Slide 21 text

Technische Universität München Daniel Bader 15.04.2011 Analysis phase 20 <> InterpretationResult <> Interpreter InterpreterController +addResult(r : InterpretationResult) ResultStore -timestamp : Time <> Event * * * 1..* stores results in controls * interprets interprets generates Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1) Interpreters work on data collected in the Capture phase and generate InterpretationResults.

Slide 22

Slide 22 text

<> Interpreter OverviewInterpreter ViewChangesInterpreter ScreenshotInterpreter UIGuidelineChecker ViewDurationInterpreter TouchHeatmapInterpreter AnalysisStatisticsInterpreter Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1) Technische Universität München Daniel Bader 15.04.2011 Analysis phase 21 Interpreter specializations

Slide 23

Slide 23 text

Technische Universität München Daniel Bader 15.04.2011 Object model and Dynamic model 22 Capture Analysis Critique Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1)

Slide 24

Slide 24 text

Technische Universität München Daniel Bader 15.04.2011 Critique phase 23 Report InterpretationResult Developer HostApplication recommends changes to views summarizes Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1)

Slide 25

Slide 25 text

+addSection(s : Section) Report <> Section OverviewSection EventTimeline TouchHeatmaps UserNavigationPath 1..* {ordered} Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1) Technische Universität München Daniel Bader 15.04.2011 Critique phase 24 Report document model

Slide 26

Slide 26 text

Technische Universität München Daniel Bader 15.04.2011 High-level dynamic model 25 Capture Analysis Critique TestUser Report ResultStore SessionJournal UsabilityData InterpretationResults Events Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1) Inputs and outputs of the three phases

Slide 27

Slide 27 text

Technische Universität München Daniel Bader 15.04.2011 System design 26 Capture Critique Analysis ReportController ResultStorage AnalysisController Interpretation SectionGenerator CaptureLibrary CaptureServer Communication Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1) Subsystem decomposition of the framework

Slide 28

Slide 28 text

Technische Universität München Daniel Bader 15.04.2011 System design 27 <> Mobile device <> Desktop computer <> CaptureLibrary <> HostApplication <> Critique <> Analysis <> CaptureServer <> CaptureServer.py <> libCapture.a <> ReportGenerator.py <> <> <> <> <> Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1) Deployment of the framework

Slide 29

Slide 29 text

Technische Universität München Daniel Bader 15.04.2011 Object design 28 Capture Critique Analysis ReportController ResultStorage AnalysisController Interpretation SectionGenerator CaptureLibrary CaptureServer Communication Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1)

Slide 30

Slide 30 text

Technische Universität München Daniel Bader 15.04.2011 CaptureLibrary - Method interception 29 UIWindow +sendEvent() +... +... Original implementation of sendEvent() Event UIWindow sendEvent() behavior before method interception

Slide 31

Slide 31 text

Slide 32

Slide 32 text

Slide 33

Slide 33 text

Technische Universität München Daniel Bader 15.04.2011 CaptureLibrary - Method interception 30 UIWindow +sendEvent() +... +... Original implementation of sendEvent() UIWindow sendEvent() behavior after method interception Our implementation of sendEvent() Event CaptureLibrary

Slide 34

Slide 34 text

Slide 35

Slide 35 text

Slide 36

Slide 36 text

Slide 37

Slide 37 text

Technische Universität München Daniel Bader 15.04.2011 Object design 31 Capture Critique Analysis ReportController ResultStorage AnalysisController Interpretation SectionGenerator CaptureLibrary CaptureServer Communication Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1)

Slide 38

Slide 38 text

Technische Universität München Daniel Bader 15.04.2011 Analysis subsystem 32 Usage of the Blackboard design pattern in the Analysis subsystem ResultStore InterpreterController Interpreter +getResultsOfType() +addResult() <> Blackboard +canContribute() +updateBlackboard() <> BlackboardExpert +loop() +getReadyExperts() <> BlackboardSupervisor supervises 1..* controls updates and accesses Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1)

Slide 39

Slide 39 text

Technische Universität München Daniel Bader 15.04.2011 Analysis subsystem 33 Dependencies between Interpreters are resolved automatically InterpreterA InterpreterB InterpreterC InterpreterD Generated order of execution:

Slide 40

Slide 40 text

Slide 41

Slide 41 text

Technische Universität München Daniel Bader 15.04.2011 Object design 34 Capture Critique Analysis ReportController ResultStorage AnalysisController Interpretation SectionGenerator CaptureLibrary CaptureServer Communication Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1)

Slide 42

Slide 42 text

Technische Universität München Daniel Bader 15.04.2011 Critique subsystem 35 -body : String Element -title : string -header : string -elements : Element[] +__init__(title) +add_element(e) +generate_html(result_store) : string <> Section OverviewSection NavigationPathSection TouchHeatmapSection ViewDurationSection UIGuidelineIssuesSection -title : string -header : string -footer : string -sections : Section[] +add_section(s) +generate_html(result_store) : string +generate(result_store, filename) Report -reportFilename : string -eventsFilename : string -interpreters : string[] -sections : string[] +addInterpreter(i : string) +addSection(s : string) +setEventsFilename(s : string) +setReportFilename(s : string) +generateReport() ReportGeneratorFacade 1..* 1..* Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1)

Slide 43

Slide 43 text

Technische Universität München Daniel Bader 15.04.2011 Prototypical implementation Full vertical prototype for Apple iOS - Implemented in Objective-C and Python - Tested with the applications Wordpress for iOS and PlainNote Capture support - Automatic network setup using Zeroconf/Bonjour - Custom, stateless XML-based network protocol - Data transmission is performed on background thread Analysis support - Six interpreters were implemented Critique support - Generates HTML output - Five section types were implemented 36

Slide 44

Slide 44 text

Technische Universität München Daniel Bader 15.04.2011 Demo 37

Slide 45

Slide 45 text

Technische Universität München Daniel Bader 15.04.2011 Future work Add more sensors and interpreters - e.g. audio and video recording Use the framework in a real world scenario Add support for Google Android - Java has rich support for reﬂection Add ability to playback sessions - Play back captured user interactions on the device Integrate framework in FastFix - Perform usability error detection - Provide access to user interaction data and context information 38

Slide 46

Slide 46 text

Technische Universität München Daniel Bader 15.04.2011 Thank you! Questions? 39

Slide 47

Slide 47 text

Technische Universität München Daniel Bader 15.04.2011 Backup slides 40

Slide 48

Slide 48 text

<> HostApplication PreviewEvents SelectSensors StopSession StartSession ControlSession ConfigureReport ViewReport CaptureUsabilityData muEvaluationFramework TestUser Developer <> <> <> <> <> <> <> Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1) Technische Universität München Daniel Bader 15.04.2011 Use case model (with reﬁnements) 41

Slide 49

Slide 49 text

Technische Universität München Daniel Bader 15.04.2011 Capture phase 42 Sensors and SensorTargets -isActive : boolean +attachTo(t : SensorTarget) +enable() +disable() <> Sensor HostApplication UsabilityData SensorTarget * 1..* * observes collects monitors Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1)

Slide 50

Slide 50 text

SessionJournal e1 : UserInputEvent TestUser a1 : Action Sensor 2: detect() 4: addEvent(e1) 3: create(time) 1: perform() Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1) Technische Universität München Daniel Bader 15.04.2011 Capture phase 43 Event detection example

Slide 51

Slide 51 text

Technische Universität München Daniel Bader 15.04.2011 Capture phase 44 Capture phase Initialize SessionJournal Initialize Sensors Create Event Observe SensorTargets Continue with next Sensor ActiveSensors <> for each Sensor SessionJournal [targets did not change] Event [targets changed] Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1)

Slide 52

Slide 52 text

Run analysis phase Initialize ResultStore Initialize Interpreters Update ResultStore Run Interpreter ResultStore <> Loop SessionJournal InterpretationResult Event Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1) Technische Universität München Daniel Bader 15.04.2011 Analysis phase 45

Slide 53

Slide 53 text

Generate Report Add Section to Report Generate Section Select Section to generate ReportConfiguration <> Loop Report ResultStore InterpretationResult Section Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1) Technische Universität München Daniel Bader 15.04.2011 Critique phase 46

Slide 54

Slide 54 text

Technische Universität München Daniel Bader 15.04.2011 System design 47 The Capture package Capture CaptureLibrary CaptureServer Communication <> Sensor SensorTarget <> Event SessionJournal NetworkConnection NetworkMessage CaptureManager ServerController 1..* 1..* * * * receives Events sends Events observes creates Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1)

Slide 55

Slide 55 text

Technische Universität München Daniel Bader 15.04.2011 System design 48 The Analysis package Analysis Capture AnalysisController Interpretation Storage InterpretationResult Interpreter InterpreterController ResultStore stores results in 1..* * * generates SessionJournal Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1)

Slide 56

Slide 56 text

Technische Universität München Daniel Bader 15.04.2011 System design 49 The Critique package Critique Analysis SectionGenerator ReportController ReportConfiguration ReportGenerator Report Section SectionGenerator 1..* 1..* uses generates ResultStore uses generates Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1)