Usability testing presentation

Presenting usability results for www.onetravel.com

Transcript

1. Reecha  Bharali   HsiaoWen  Wu   Aeshvarya  Verma   Chin  Powit  Rungsangthiwakorn   User  Testing   Presentation-­‐  I543   Team   OneTravel.com  
2. Presentation  Outline   •  Tes$ng  Procedure   •  Quan$ta$ve  Results  Comparison   •  Qualita$ve  Results  Comparison   •  Recommenda$on   •  Conclusion  
3. Test  Procedure   Compara=ve  Test   OneTravel  v.s.  Hipmunk     •  Within-­‐Subject  Design   •  Order  Effect:  Alterna$ng  User  Star$ng  Website   •  Post-­‐Task  Ques$onnaire   •  SUS  Score  
4. User  Pro;ile   10  Users  tested     •  Average  User  Tes$ng  Session:  80  minutes   •  Equipment:  Audio-­‐Video  Recording  with  Screen  Capture   Demographics   •  Age  18-­‐40  years   •  Internet  Usage  Over  6hours   •  English  Proficiency  –  Good-­‐Fair-­‐Fluent   •  5  Competent  Users  and  5  Beginners  
5. Tasks   The  users  had  to  perform  5  tasks  with  the  facilitator     Task1:  How  …  cheap  book  flight  and  hotel?   Task2:  How  …  payment?   Task3:  How  …  compare  trips?   Task4:  How  …  find  flight  with  specific  =me?   Task5:  How  …  find  hotel  with  specific  loca=on?     •  All  the  task  are  aimed  at  the  main  tasks  that  the  website   is  supposed  to  perform.   •  Conforming  the  heuris;cs    
6. Quantitative  Result   •  Success  rate   •  Time-­‐on-­‐task   •  Post-­‐Task   •  SUS  
7. Quantitative  Result   •  Success  Rate   •  4/5  tasks  in  Hipmunk  have  higher  success  rate  than  in  Ontravel   •  Only  “Find  the  how  to  pay”  in  Onetravel  is  more  than  Hipmunk   0   2   4   6   8   10   12   Find  Flight   and  Hotel   Find  the   how  to  pay   Compare   Price   Find  Flight   w/  specific   $me   Find  Hotel   w/  specific   loca$on   Number  of  success Task Success  Rate  for  Each  Task OneTravel   Hipmunk  
8. Quantitative  Result   •  Users  spend  more  $me  in  Onetravel  for  4  tasks   •  Learning  from  task            (Find  Flight  and  Hotel  vs.  Compare  Price)   •  Anomaly  in  “Find  the  how  to  pay”   Find   Flight  and   Hotel   Find  the   how  to   pay   Compare   Prices   Find   Flight  w/   specific   $me   Find   Hotel  w/   specific   loca$on   Onetravel   8.5   0.9   6.0   5.7   4.8   Hipmunk   6.1   1.5   5.6   3.1   2.9   0.0   2.0   4.0   6.0   8.0   10.0   12.0   14.0   Time  (min) Average  Time  on  Task
9. Quantitative  Result   •  Post-­‐Task  Ques$onnaire   •  Q1:  How  was  the  overall  task?              (1  very  easy  -­‐  7  Very  Difficult)   0   1   2   3   4   5   6   7   Find  Flight  and   Hotel   Find  the  how  to   pay   Compare  Prices   Find  Flight  w/   specific  $me   Find  Hotel  w/   specific  loca$on   Average  Score   Task Perceived  Difficulty   OneTravel   Hipmunk  
10. Quantitative  Result •  Q1:    How  was  the  overall  task?  (1  very  easy  -­‐  7  Very  Difficult)   •  Q2:  the  ease  of  comple$ng  this  task.            (1  Strongly  Disagree  -­‐  7  Strongly  Agree)   •  Q3:  the  amount  of  =me  it  took  to  complete  the  task            (1  Strongly  Disagree  -­‐  7  Strongly  Agree) 0   2   4   6   8   Find  Flight   and  Hotel   Find  the   how  to  pay   Compare   Prices   Find  Flight   w/  specific   $me   Find  Hotel   w/  specific   loca$on   Average Task Average  Score  for  Onetravel Q1   Q2   Q3  
11. Quantitative  Result   •  Average  SUS  score   •  Onetravel=  50.3  ±  16   •  Hipmunk=  54.2  ±  21   •  6/9  users  don’t  like  the  site  they  start  with   P1   P2   P3   P4   P5   P6   P7   P8   P10   OneTravel   32.5   52.5   62.5   42.5   35   45   37.5   65   80   Hipmunk   62.5   25   82.5   55   32.5   42.5   70   80   37.5   0   20   40   60   80   100   SUS    Score SUS  Score  of  Par=cipants  
12. Qualitative  Results/  Findings   •  Home  Page   •  Search  Result  Page   •  Loading  Page   •  Comparing  Page  
13. Usability  Testing  Results   Homepage   Loading  Page   Search  Result  Page  
14. Homepage   STATUS   PROBLEM   SEVERITY   RECOMMENDATION   Confirmed   The  majority  of  the   par$cipants  do  not  no$ce   the  "Vaca$on”  tab.   4   Make  the  label  more   obvious  e.g.  Flight  +  Hotel  or   give  a  pop  up  descrip$on   when  user  put  the  mouse   over  the  tab)  
15. Homepage   STATUS   PROBLEM   SEVERITY   RECOMMENDATION   New   Search tab reset to “Flight” every time the user refreshes or comeback to the homepage. 2   Remember the user’s last used preference.
16. Homepage   STATUS   PROBLEM   SEVERITY   RECOMMENDATION   New   OneTravel allow only Single Session.(e.g. user cannot open new tab to do multiple search) 2   Provide multiple session to a single user.
17. Loading  Page   STATUS   PROBLEM   SEVERITY   RECOMMENDATION   Confirmed   The loading page do not show the progress and appears static. 3   Add animation to show progress.
18. Search  Result  Page   STATUS   PROBLEM   SEVERITY   RECOMMENDATION   Lower   severity   Booking  steps  are   confusing  (the  user  do   not  know  that  their   flight  is  pre-­‐selected,   and  have  trouble  finding   a  way  to  change  the   flight)   4  =>  2   Make  it  more  clear  for  the   user  to  change  flight.  
19. Search  Result  Page   STATUS   PROBLEM   SEVERITY   REASON   Higher   Severity   Lot  of  informa$on  in  the   page  that  the  user  missed   many  things,  (can't  find   the  “tripAdvisor”  score,   can't    find  show  map   buron,  can't  find  the   Change  Flight  link)   2  =>  3   The  User  struggle  in  finding   informa$on  e.g.  One  user   did  not  know  they  did  wrong   search  
20. Search  Result  Page   STATUS   PROBLEM   SEVERITY   REASON   Lower   Severity   Checkout  buron  is  not   clearly  visible,  there  is   not  explana$on  about   the     4  =>  1   All  user  manage  to  find  the   way  to  make  payment  quite   easier  using  buron  at  the   buron  of  the  page.  
21. Search  Result  Page   STATUS   PROBLEM   SEVERITY   REASON   Lower   Severity   Unclickable breadcrumb. “How should I go to the next step”   3  =>  2   It  seems  that  users  are   naturally  press  browser   back  buron  to  go  back   instead.  
22. Search  Result  Page   STATUS   PROBLEM   SEVERITY   RECOMMENDATION   New   Browser  Back   Naviga$on  does  not   work  properly:  it  go   back  to  the  loading  page   and  send  the  user  back   to  the  result  page   2   Allow  user  to  go  back  each   step  using  this  browser  back   buron.  
23. Search  Result:  Comparing  Hotels  
24. Search  Result:  Comparing  Hotels   STATUS   PROBLEM   SEVERITY   RECOMMENDATION   Confirmed   No  single  map  showing   the  loca$on  of  the   hotels.     4 Adding  a  single  map   illustrate  loca$on   New   Confusion  between   Star  Ra$ng  and   TripAdvisor   3 Confusion  between  Star   Ra$ng  and  TripAdvisor   New   Difficult  to  compare   when  choosing  hotels   with  2  or  more   quali$es   4 Adding  filter  or  secondary   sort  func$on.  
25. Comparing  Hotels:  Quotes   "It's too difficult to find the hotel from this website"   “  (sigh)  It  is  going  to  be   a  pain  in  the  ass   because  I  have  to    check   every  maps…”   “May  be  I  will  go  to   another  website.”  
28. Search  Result:  Comparing   Flight   PROBLEM   SEVERITY   RECOMMENDATION   New   Unclear  label  for  Selected   Search  Date,  Des$na$on,     2   Make  the  Search  Date  more   prominent   Con   firmed   Sort  by  requested  $me   does  not  sort  by   requested  $me.       4   Sort  by  requested  $me   Raise   Severity   Near-­‐by  airport  result     2  =>  4   Only  show  result  from   selected  airport,  or  give   op$on  for  user  to  hide   near-­‐by  airport.  
29. Comparing  Flight:  Quotes   "The  one  that  doesn't   go  to  JFK  and  go  to   Newark  is  bull  s@*t"   "I  hate  this   website"  
30. User  Perception   •  Despite  the  pain,  many  users  s$ll  perceived  OneTravel  as   more  useful,  or  more  reliable.   “I  think  I  am  willing  to   spend  $me  learning  how  to   use  OneTravel,  because  I   Think  it  is  more  useful   “I  will  rather  trust  Hipmunk   than  OneTravel.  Hipmunk   look  like  student  project”  
31. Conclusion   •  For  most  task,  user  performed  berer  on  Hipmunk.   •  Order  Effect:  Most  user  prefer  the  second  website   •  Perceived  Usefulness   •  2  Users  who  started  with     •  Learning:    
32. Conclusion:  5  Problems  to  ;ix?   Problem   Severity   RECOMMENDATION   1   “Vaca$on”  Label       4   Choose  a  word  which   describe  the  func$on  more   clearly   2   Sta$c-­‐Looking  Loading  Page     3   Add  clear  progress   indica$on     3   Sort  by  requested  $me  does   not  sort  by  requested  $me.     4   Sort  by  requested  $me   4   Near-­‐by  airport  result     2  =>  4   Only  show  result  from   selected  airport   5   No  single  map  showing  the   loca$on  of  the  hotels.     4   Add  a  Map  View  
33. Thank  you  J  
34. Appendix •  Average  $me  on  task Site Find  Flight   and  Hotel   Find  Flight   and  Hotel   Find  Flight   and  Hotel   Find  Flight   and  Hotel   Find  Flight   and  Hotel   Onetravel   8.5±4.4     0.9±0.6     6.0±3.2   5.7±3.4     4.8±3.4   Hipmunk   6.1±3.3     1.5±1.2     5.6±3.1   3.1±2.7     2.9±1.8   (Unit:  min)
35. •  In  Ontravel   •  There  are  two  ways  to  checkout   •  The  buron  as  right  side  is  a  shortcut  to  pay.   •  Another  one  is  at  the  end  of  page,  user  has  to  scroll  down  to  click  it   and  read  through  page.   0.0     0.5     1.0     1.5     2.0     2.5     3.0     3.5     1   2   3   4   5   6   7   8   9   10   Time  on  Task  (min) Par=cipant Task  =me  of  Par=cipants       (Task  2)   Onetravel   Hipmunk