PuReWidgets toolkit

A presentation about the initial version of the PuReWidgets toolkit for interactive public display applications.

1. The  PuReWidgets  toolkit  for   interac6ve  public  display   applica6ons   Jorge  C.  S.  Cardoso   jorgecardoso@ieee.org     Ubicomp@UMinho     May  30,  2012   1  
2. Content   •  Mo6va6on   •  PuReWidgets  features   •  Architecture   •  Applica6on  models   •  Programming  library  overview   •  HelloWorld   2  
3. Mo6va6on   Complexity  of  incorpora6ng  interac6ve  features  in  public  display  applica6ons.   Too  many  things  to  answer  when  designing  the  interac6on  for  a  public  display   applica6on:   •  What  controls  can  we  use  to  offer  interac6ve  features  to  users?   –  No  standard  interac6ve  controls  for  public  displays   •  What  interac6on  mechanisms  can  be  used  to  provide  those  features?   –  Various  possible  interac6on  mechanisms  (SMS,  BT  naming,  OBEX,  Email,  IM,  mobile  app,   desktop,  QR  codes,  touch,  etc.)   –  Different  loca6ons  may  have  different  interac6on  resources   •  How  will  those  features  work?   –  Desktop  interac6on  paradigm  does  not  generally  apply   –  Applica6ons  may  not  always  be  visible   •  How  can  the  applica6on  react  to  different  users?   –  The  interac6on  environment  is  mul6-­‐user   –  We  need  to  iden6fy/differen6ate  users   •  How  can  users  tell  what  they  can  do  with  the  applica6on?   –  Where  are  the  features  presented?   3  
4. Mo6va6on   We  can  solve  all  those  ques6ons  for  ad  hoc  applica6ons,  but   •  This  means  wasted  effort  in  developing  applica6ons   –  Programmers  must  deal  with  issues  extraneous  to  the  main   applica6on  logic   •  Will  certainly  lead  to  inconsistencies  in  the  interac6on   model   –  Each  solu6on  will  behave  slightly  differently   –  Will  confuse  users   We  need  a  programming  toolkit  for  interac6ve  public  display   applica6ons!   4  
5. Toolkit   •  The  toolkit  should  provide  a  consistent  model   for  user  interac6on  and  applica6on   development   – But  s6ll  allow  some  diversity  in  solu6ons   5  
6. PuReWidgets   •  PuReWidgets  tries  to  solve  these  problems.   •  To  make  sure  it  covers  a  wide  range  of   applica6ons,  we  looked  at  various  interac6ve   public  display  systems  to  learn   – General  requirements   – Types  of  controls   – Types  of  input  mechanisms   6  
7. PuReWidgets   Widget-­‐based  toolkit.     •  A  widget  represents  an  interac6ve  feature.   •  Is  represented  by  class  in  an  object-­‐oriented  programming  model.   •  Applica6ons  instan6ate  widgets,  define  a  callback,  and  receive   interac6on  event  through  the  callback     •  Main  features   –  Mul6ple,  extensible,  controls   –  Independence  of  input  mechanisms  and  modali6es   –  Automa6cally  generated  graphical  interfaces   –  Asynchronous  interac6on   –  Concurrent  interac6on         –  Graphical  affordances   –  Server  &  client  libraries   7  
8. Features   Mul6ple,  extensible,  controls.       •  Allows  developers  to  create  rich  interac6ve  applica6ons.  Controls  can  be  extended  with  custom   func6onality.   •  We  studied  various  interac6ve  display  systems  to  define  a  set  of  basic  interac6ve  controls,  common  to  the   majority  of  applica6ons:   •  Ac6on  bugon   –  Trigger  an  ac6on  in  the  applica6on,  e.g.,  play  a  video   •  Op6on  selec6on   –  Selects  among  a  set  of  op6ons,  e.g.,  to  vote   •  Text  entry   –  Sends  text  to  the  applica6on,  e.g.,  a  comment,  tag,  search  keyword   •  Download   –  Receives  a  media  file  from  the  applica6on,  e.g.,  download  poster   •  Upload   –  Uploads  a  media  file  to  the  applica6on,  e.g.,  a  photo  to  be  displayed   •  Check-­‐in   –  Says  “I’m  here”   8  
9. Features   Independence  of  input  mechanisms  and  modali6es.     •  Allows  developers  to  focus  on  the  core   applica6on  func6onality  and  not  on  the  low-­‐level   interac6on  details.   •  Allows  applica6ons  to  work  in  heterogeneous   loca6ons,  with  different  mechanism     •  Interac6on  can  be  accomplished  with  various   mechanisms  (SMS,  Bluetooth  naming,  QR  codes,   and  automa6cally  generated  graphical   interfaces).   9  
10. Features   Automa6cally  generated  graphical  interfaces.     •  Provide  a  richer  interac6on  for  desktop  and   mobile  devices.   •  PuReWidgets  generates  web-­‐based  interfaces   for  each  applica6on.     •  It  also  generates  QR  codes  for  each   applica6on's  feature     10  
11. Features   Asynchronous  interac6on.     •  Allows  applica6ons  to  receive  input  events   that  were  generated  when  the  applica6on   was  not  listening.   •  PuReWidgets  provides  a  persistent  input   queue  for  each  applica6on   •  Applica6ons  can  request  past  input  at  any   6me   11  
12. Features   Concurrent  interac6on.     •  Allows  various  users  to  interact  at  the  same   6me,  while  providing  applica6ons  with   iden6ty  informa6on  that  allows  them  to   differen6ate  users.   •  Every  input  event  carries  a  user  id  (if  available)   – Where  possible,  anonymous  interac6ons  are  s6ll   “iden6fied”  (more  on  this  later)   12  
13. Features   Graphical  affordances.     •  Allows  users  to  immediately  recognize  interac6ve   features  and  input  feedback.   •  Controls  have  an  op6onal  graphical   representa6on  that  can  be  used  on  the  public   display  interface.   –  Applica6ons  may  not  show  the  features  on  the  public   display,  but  they  will  s6ll  be  available     •  Input  feedback  is  also  (op6onally)  displayed  on   the  public  display.   13  
14. Features   Server  &  client  libraries.     •  Programmers  can  choose  to  use  just  the   server,  just  the  client,  or  both  libraries   together.   •  Allows  developers  to  choose  the  best   applica6on  model.   14  
15. PuReWidgets  Architecture   15  
16. Interac6on  services   •  PuReWidgets  service   –  Keeps  informa6on  about  every  widget   –  Generates  graphical  interfaces  for  desktop,  mobile,  touch   plaporms     •  These  allow  anonymous  interac6ons,  but  they  generate  persistent   cookie-­‐based  random  ids.   –  Generates  QR  codes  for  every  widget   –  Generates  unique  textural  reference  codes  for  data-­‐based   interac6ons  (SMS,  BT  naming,  OBEX,  etc.)   •  IO  infrastructure   –  (from  Instant  places)   –  Provides  low-­‐level  data-­‐based  interac6on   16  
17. PuReWidgets  library   •  Provides  an  object-­‐oriented  library  of  widgets   •  Hides  the  communica6on  details  with  the   PuReWidgets  service   •  Provides  other  features,  unrelated  to  interac6on,   such  as   –  Applica6on  “place-­‐aware”  storage   •  PuReWidgets  provides  a  name-­‐value  server  datastore  for   easy  storage  of  place-­‐specific  seqngs/state   –  Skeleton  admin  interface  for  place-­‐owners   •  To  configure  place-­‐specific  seqngs   17  
18. Local  applica6on  model   18  
19. Local  applica6on  model   •  Applica6on  logic  on  the  public  display  (player)   – Subject  to  the  schedule  6mings   •  May  not  be  able  to  react  (or  update  data   structures)  immediately  when  user  interacts   – But  will  eventually  (when  it  is  displayed  again)   •  When  displayed,  the  toolkit  periodically  asks   the  service  for  input  events   – And  triggers  the  appropriate  widget  event   19  
20. Remote  applica6on  model   20  
21. Remote  applica6on  model   •  Applica6on  logic  on  the  server   •  Can  react  (or  update  data  structures)   immediately  to  user  input   – Even  if  not  on  the  public  display,  the  reac6on  can   be  visible  on  other  channels   •  The  toolkit  periodically  asks  the  service  for   input  events   – And  triggers  the  appropriate  widget  event   21  
22. PuReWidgets  Implementa6on   •  Google  Appengine  (server)   •  Google  Web  Toolkit  –  GWT   (client)   •  Overview  of  the  library   22  
23. Using  PuReWidgets   •  Hello  World  applica6on  (local  model)   •  Video   23  
24. Ini6al  Evalua6on/Development   process   •  Two  applica6ons  developed   so  far  (ignore  the  “graphic  design”,  please)   –  Youtube  player,  media   intensive,  complex  graphical   components  (several  panels   that  change  over  6me),  local   applica6on  model   –  Everybody  votes,  based  on   display  owner  created  content,   simpler  graphical  component,   combines  remote  and  local   applica6on  models   24  
25. Ini6al  Evalua6on/Development   process   •  Con6nuous  refinement   – Develop  interac6ve  public  display  applica6ons   – Gain  more  insight  about  the  difficul6es       – Refine  the  toolkit  to  address  the  iden6fied   problems   – Refactor  the  applica6ons  to  include  the  toolkit   changes   25  
26. Summary   •  PuReWidgets  facilitates  applica6on   development   – Provides  ready-­‐to-­‐use  interac6ve  controls   – Integrates  various  input  mechanisms  via  an  I/O   infrastructure   – Generates  applica6on  interfaces  for  desktop,   mobile,  and  QR  codes   – Provides  client  and  server  applica6on   development  models   26  
27. The  end   27