Data Hacking and The City: UCL CASA Conference 2013 Workshop

These are the slides from the UCL CASA Conference 2013 Workshop where we collected data via the Twitter API and mapped them on Google Maps Engine. Full Tutorial and scripts to download are referenced inside presentation so reader can start collecting Twitter Data quickly and easily!

Transcript

1.                                                                                           Data Hacking and the City CASA Workshop Steven Gray Thursday 26th September 2013 - 9:30 - 12:30
2. Order  of  the  Morning 9:30  -­‐  10:00  -­‐  Showcase  of  what  we  do 10:00  -­‐  11:00  -­‐  Coding  -­‐-­‐  Collec8ng  the  Data 11:00  -­‐  12:00  -­‐  More  Coding  -­‐  Visualising  Data   12:00  -­‐  12:30  -­‐  Ques8on  and  Wrap  up
3. Lets  start  coding! But  first  a  few  things  we  need  to  sort  out Separate  into  Groups          -­‐-­‐  Social  Data  (Best  for  Beginners  to  coding)          -­‐-­‐  Transport  Data          -­‐-­‐  Environmental  Data Rules -­‐-­‐  Don’t  be  afraid  to  ask  ques8ons  if  you  don’t  understand -­‐-­‐  Have  fun  and  Enjoy  yourself
4. Quick  Showcase  of  my  work
12. Lets  start  coding!
13. Mac -­‐-­‐  Install  Xcode  Command  line  tools  for  your  Mac  OS  Version -­‐-­‐  h$ps://developer.apple.com/downloads/ -­‐-­‐  Install  Homebrew  (hVp://brew.sh/) -­‐-­‐  Run  on  terminal:   ruby  -­‐e  "$(curl  -­‐fsSL  hVps://raw.github.com/mxcl/homebrew/go)" -­‐-­‐  Follow  hVp://madebyhoundstooth.com/blog/install-­‐node-­‐with-­‐homebrew-­‐on-­‐os-­‐x/ -­‐-­‐  brew  install  node NPM  will  be  installed  at  the  same  8me  as  node! Install  NodeJS  +  NPM
14. Linux  (Ubuntu) -­‐-­‐    Follow  hVps://github.com/joyent/node/wiki/Installing-­‐Node.js-­‐via-­‐package-­‐manager Windows   -­‐-­‐  Download  and  follow  instruc8ons Install  NodeJS  +  NPM hVp://nodejs.org/download/ Open  Node.js  from  Start  Menu  -­‐  Programs  -­‐  Node.js  -­‐  Node.js  Command  Prompt Or  search  for  “Node.js  Command  Prompt”  on  your  system  (Windows  8,  8.1)  
15. Install  Modules  via  NPM    (All  Systems) Use  NPM  to  install  the  following  modules  used  by  the  collector.   Check  that  you  have  npm  installed  by  running  “npm  -­‐v”  (it  should  report  1.3.*)   Run  the  following  to  install npm  install  oauth  colors
16. CollecIng  Social  Media  Data  (TwiLer)  -­‐  A  Recipe What  you  will  need: 1  x  TwiVer  Account 1  x  Developer  Key  from  hLps://dev.twiLer.com/ Sprinkle  with  some  pa8ence  and  8me  and  in  8me  you’ll  be  collec8ng   data  from  TwiVer. Instruc8ons  coming  up  soon  aier  a  small  explanta8on.
17. Understanding  what’s  happening http://dev.twitter.com/doc User Pro!les Tweet Database
18. HTTP GET Request Results {"results":[ {"text":"@twitterapi is great!", "to_user_id":396524, "to_user":"TwitterAPI", "from_user":"jkoum", "metadata": { "result_type":"popular", "recent_retweets": 100 }, "id":1478555574, "from_user_id":1833773, "iso_language_code":"nl", ....} } Returns tweets that: • match speci!ed query • located in Lat/Lon box • Historical Tweets • User Pro!le Information • Return Public Timeline Service is rate limited JSON or Atom Making  Requests  to  TwiLer
19. AuthenIcaIon 2  Methods  of  Authen8ca8on -­‐-­‐  Basic  HTTP  Authen8ca8on -­‐-­‐  oAuth    (More  Complex Most  modern  day  web  services  use  oAuth  (including  TwiVer  and  Google  Services) Keeps  informa8on  safe  from  the  wrong  people  but  also  allows  providers  to  ban  people   if  they  are  using  too  many  resources  or  making.
20. So  how  does  oAuth  work Server Applica8on
21. So  how  does  oAuth  work Server Applica8on Request  a  Session  Token Pass  a  ID  and  a  Secret
22. So  how  does  oAuth  work Server Applica8on Request  a  Session  Token Pass  a  ID  and  a  Secret Asks  Login  with  ID   Server  gives  a  Pin  number  for  session
23. So  how  does  oAuth  work Server Applica8on
24. So  how  does  oAuth  work Server Applica8on Login  is  successful :)
25. So  how  does  oAuth  work Server Applica8on Login  is  successful :) Sends  App  Tokens These  need  to  be  saved!
26. So  how  does  oAuth  work Server Applica8on
27. So  how  does  oAuth  work Server Applica8on Request  using  tokens Server  checks  if  it’s  valid
28. So  how  does  oAuth  work Server Applica8on Request  using  tokens Server  checks  if  it’s  valid Data  Returned :)
29. Go  to  hLps://dev.twiLer.com/ Login  using  your  current  TwiVer  account Click  your  profile  picture  (top  right  corner)  and  click  My  ApplicaIons Click  Create  New  ApplicaIon Fill  in  your  details  for  the  app You’ll  be  presented  with  your  Consumer  Key  and  Consumer  Secret Save  these  keys  for  the  next  step CollecIng  Social  Media  Data  (TwiLer) CreaIng  your  TwiLer  ApplicaIon  Keys
30. Lets  Collect  some  data Download  the  Collector hVps://gist.github.com/sjg/6712144
31. Lets  Collect  some  data Aier  downloading  the  script  s8ck  in  your  keys  and  secret  in  the  script  between  the   quota8on  marks  and  save  the  file Consumer  Key Consumer  Secret
32. Lets  Collect  some  data Copy  the  script  into  a  folder  that  you  can  run  the  script  from  and  navigate  to  that   folder  in  your  system  terminal. You  can  edit  the  loca8on  of  the  collector  by  changing  these  values  in  the  file: To  run  the  collector  type:   node  twiLer_collect.js “SearchTag”,  “La8tude”,  “Longitude”,  “Radius”  -­‐  NB:    “”  will  return  all  tweets  in  that  area
33. Lets  Collect  some  data If  all  goes  well  and  everything  is  working  you  should  see  the  following  screen Go  to  the  link  in  a  browser,  login  to  twiVer  and  type  the  pin  number  in.    Press  return   and  tweets  will  start  collec8ng  in  a  file  called  “tweets.csv”  in  the  same  folder.
34. Visualising  the  output
35. Go  to  hLps://mapsengine.google.com Click  on  New  Map Name  your  map  by  clicking  on  “UnItled  Map” Click  “Import”  and  upload  your  CSV  file Click  Base  Map  to  change  the  map  style CreaIng  your  first  VisualisaIon  with  the  Data  Collected  
36. CreaIng  your  first  VisualisaIon  with  the  Data  Collected   Once  the  file  has  uploaded,  choose  the  correct  La8tude  and  Longitude  columns  and   then  click  Con8nue.    Click  the  ID  field  (lei  over)  as  the  marker  8tle  and  press  finish. The  updated  output  file  will,  hopefully,  auto  detect  the  lat/lng  fields.
37. CreaIng  your  first  VisualisaIon  with  the  Data  Collected   Style  your  markers  by  pressing  the  paint  bucket  (highlighted  in  red)  and  select  the   circle  shape  and  colour.    Experiment  with  styling  your  map.
38. CreaIng  your  first  VisualisaIon  with  the  Data  Collected   If  all  went  well  your  map  will  look  something  like  this.    Why  not  share  it  with  the   world!
39. CreaIng  your  first  VisualisaIon  with  the  Data  Collected   And  you  can  click  the  markers  and  see  the  tweets
40. Different  Sources  you  can  use  to  Visualise  your  data Google  Fusion  Tables -­‐  hVp://www.google.com/drive/apps.html#fusiontables Google  Map  Engine -­‐  hVps://mapsengine.google.com Google  Earth -­‐  hVp://www.google.com/earth/index.html MapTube -­‐  hVp://www.maptube.org
41. hVp://download.bigdatatoolkit.org
42.                                                                                           Thanks Steven Gray steven.gray@ucl.ac.uk @frogo    +Steven  Gray