Data Hacking and The City: UCL CASA Conference 2013 Workshop

These are the slides from the UCL CASA Conference 2013 Workshop where we collected data via the Twitter API and mapped them on Google Maps Engine. Full Tutorial and scripts to download are referenced inside presentation so reader can start collecting Twitter Data quickly and easily!

Transcript

1. 
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   Data Hacking and the City
   CASA Workshop
   Steven Gray
   Thursday 26th September 2013 - 9:30 - 12:30
   

   View Slide

2. Order  of  the  Morning
   9:30  -­‐  10:00  -­‐  Showcase  of  what  we  do
   10:00  -­‐  11:00  -­‐  Coding  -­‐-­‐  Collec8ng  the  Data
   11:00  -­‐  12:00  -­‐  More  Coding  -­‐  Visualising  Data  
   12:00  -­‐  12:30  -­‐  Ques8on  and  Wrap  up
   

   View Slide

3. Lets  start  coding!
   But  first  a  few  things  we  need  to  sort  out
   Separate  into  Groups
            -­‐-­‐  Social  Data  (Best  for  Beginners  to  coding)
            -­‐-­‐  Transport  Data
            -­‐-­‐  Environmental  Data
   Rules
   -­‐-­‐  Don’t  be  afraid  to  ask  ques8ons  if  you  don’t  understand
   -­‐-­‐  Have  fun  and  Enjoy  yourself
   

   View Slide

4. Quick  Showcase  of  my  work
   

   View Slide

5. View Slide

6. View Slide

7. View Slide

8. View Slide

9. View Slide

10. View Slide

11. View Slide

12. Lets  start  coding!
    

    View Slide

13. Mac
    -­‐-­‐  Install  Xcode  Command  line  tools  for  your  Mac  OS  Version
    -­‐-­‐  h$ps://developer.apple.com/downloads/
    -­‐-­‐  Install  Homebrew  (hVp://brew.sh/)
    -­‐-­‐  Run  on  terminal:  
    ruby  -­‐e  "$(curl  -­‐fsSL  hVps://raw.github.com/mxcl/homebrew/go)"
    -­‐-­‐  Follow  hVp://madebyhoundstooth.com/blog/install-­‐node-­‐with-­‐homebrew-­‐on-­‐os-­‐x/
    -­‐-­‐  brew  install  node
    NPM  will  be  installed  at  the  same  8me  as  node!
    Install  NodeJS  +  NPM
    

    View Slide

14. Linux  (Ubuntu)
    -­‐-­‐    Follow  hVps://github.com/joyent/node/wiki/Installing-­‐Node.js-­‐via-­‐package-­‐manager
    Windows  
    -­‐-­‐  Download  and  follow  instruc8ons
    Install  NodeJS  +  NPM
    hVp://nodejs.org/download/
    Open  Node.js  from  Start  Menu  -­‐  Programs  -­‐  Node.js  -­‐  Node.js  Command  Prompt
    Or  search  for  “Node.js  Command  Prompt”  on  your  system  (Windows  8,  8.1)  
    

    View Slide

15. Install  Modules  via  NPM    (All  Systems)
    Use  NPM  to  install  the  following  modules  used  by  the  collector.  
    Check  that  you  have  npm  installed  by  running  “npm  -­‐v”  (it  should  report  1.3.*)  
    Run  the  following  to  install
    npm  install  oauth  colors
    

    View Slide

16. CollecIng  Social  Media  Data  (TwiLer)  -­‐  A  Recipe
    What  you  will  need:
    1  x  TwiVer  Account
    1  x  Developer  Key  from  hLps://dev.twiLer.com/
    Sprinkle  with  some  pa8ence  and  8me  and  in  8me  you’ll  be  collec8ng  
    data  from  TwiVer.
    Instruc8ons  coming  up  soon  aier  a  small  explanta8on.
    

    View Slide

17. Understanding  what’s  happening
    http://dev.twitter.com/doc
    User Pro!les Tweet Database
    

    View Slide

18. HTTP
    GET
    Request
    Results
    {"results":[
    {"text":"@twitterapi is great!",
    "to_user_id":396524,
    "to_user":"TwitterAPI",
    "from_user":"jkoum",
    "metadata":
    {
    "result_type":"popular",
    "recent_retweets": 100
    },
    "id":1478555574,
    "from_user_id":1833773,
    "iso_language_code":"nl",
    ....}
    }
    Returns tweets that:
    • match speci!ed query
    • located in Lat/Lon box
    • Historical Tweets
    • User Pro!le Information
    • Return Public Timeline
    Service is rate limited
    JSON or Atom
    Making  Requests  to  TwiLer
    

    View Slide

19. AuthenIcaIon
    2  Methods  of  Authen8ca8on
    -­‐-­‐  Basic  HTTP  Authen8ca8on
    -­‐-­‐  oAuth    (More  Complex
    Most  modern  day  web  services  use  oAuth  (including  TwiVer  and  Google  Services)
    Keeps  informa8on  safe  from  the  wrong  people  but  also  allows  providers  to  ban  people  
    if  they  are  using  too  many  resources  or  making.
    

    View Slide

20. So  how  does  oAuth  work
    Server
    Applica8on
    

    View Slide

21. So  how  does  oAuth  work
    Server
    Applica8on
    Request  a  Session  Token
    Pass  a  ID  and  a  Secret
    

    View Slide

22. So  how  does  oAuth  work
    Server
    Applica8on
    Request  a  Session  Token
    Pass  a  ID  and  a  Secret
    Asks  Login  with  ID  
    Server  gives  a  Pin  number  for  session
    

    View Slide

23. So  how  does  oAuth  work
    Server
    Applica8on
    

    View Slide

24. So  how  does  oAuth  work
    Server
    Applica8on
    Login  is  successful
    :)
    

    View Slide

25. So  how  does  oAuth  work
    Server
    Applica8on
    Login  is  successful
    :)
    Sends  App  Tokens
    These  need  to  be  saved!
    

    View Slide

26. So  how  does  oAuth  work
    Server
    Applica8on
    

    View Slide

27. So  how  does  oAuth  work
    Server
    Applica8on
    Request  using  tokens
    Server  checks  if  it’s  valid
    

    View Slide

28. So  how  does  oAuth  work
    Server
    Applica8on
    Request  using  tokens
    Server  checks  if  it’s  valid
    Data  Returned
    :)
    

    View Slide

29. Go  to  hLps://dev.twiLer.com/
    Login  using  your  current  TwiVer  account
    Click  your  profile  picture  (top  right  corner)  and  click  My  ApplicaIons
    Click  Create  New  ApplicaIon
    Fill  in  your  details  for  the  app
    You’ll  be  presented  with  your  Consumer  Key  and  Consumer  Secret
    Save  these  keys  for  the  next  step
    CollecIng  Social  Media  Data  (TwiLer)
    CreaIng  your  TwiLer  ApplicaIon  Keys
    

    View Slide

30. Lets  Collect  some  data
    Download  the  Collector
    hVps://gist.github.com/sjg/6712144
    

    View Slide

31. Lets  Collect  some  data
    Aier  downloading  the  script  s8ck  in  your  keys  and  secret  in  the  script  between  the  
    quota8on  marks  and  save  the  file
    Consumer  Key
    Consumer  Secret
    

    View Slide

32. Lets  Collect  some  data
    Copy  the  script  into  a  folder  that  you  can  run  the  script  from  and  navigate  to  that  
    folder  in  your  system  terminal.
    You  can  edit  the  loca8on  of  the  collector  by  changing  these  values  in  the  file:
    To  run  the  collector  type:  
    node  twiLer_collect.js
    “SearchTag”,  “La8tude”,  “Longitude”,  “Radius”
     -­‐  NB:    “”  will  return  all  tweets  in  that  area
    

    View Slide

33. Lets  Collect  some  data
    If  all  goes  well  and  everything  is  working  you  should  see  the  following  screen
    Go  to  the  link  in  a  browser,  login  to  twiVer  and  type  the  pin  number  in.    Press  return  
    and  tweets  will  start  collec8ng  in  a  file  called  “tweets.csv”  in  the  same  folder.
    

    View Slide

34. Visualising  the  output
    

    View Slide

35. Go  to  hLps://mapsengine.google.com
    Click  on  New  Map
    Name  your  map  by  clicking  on  “UnItled  Map”
    Click  “Import”  and  upload  your  CSV  file
    Click  Base  Map  to  change  the  map  style
    CreaIng  your  first  VisualisaIon  with  the  Data  Collected  
    

    View Slide

36. CreaIng  your  first  VisualisaIon  with  the  Data  Collected  
    Once  the  file  has  uploaded,  choose  the  correct  La8tude  and  Longitude  columns  and  
    then  click  Con8nue.    Click  the  ID  field  (lei  over)  as  the  marker  8tle  and  press  finish.
    The  updated  output  file  will,  hopefully,  auto  detect  the  lat/lng  fields.
    

    View Slide

37. CreaIng  your  first  VisualisaIon  with  the  Data  Collected  
    Style  your  markers  by  pressing  the  paint  bucket  (highlighted  in  red)  and  select  the  
    circle  shape  and  colour.    Experiment  with  styling  your  map.
    

    View Slide

38. CreaIng  your  first  VisualisaIon  with  the  Data  Collected  
    If  all  went  well  your  map  will  look  something  like  this.    Why  not  share  it  with  the  
    world!
    

    View Slide

39. CreaIng  your  first  VisualisaIon  with  the  Data  Collected  
    And  you  can  click  the  markers  and  see  the  tweets
    

    View Slide

40. Different  Sources  you  can  use  to  Visualise  your  data
    Google  Fusion  Tables
    -­‐  hVp://www.google.com/drive/apps.html#fusiontables
    Google  Map  Engine
    -­‐  hVps://mapsengine.google.com
    Google  Earth
    -­‐  hVp://www.google.com/earth/index.html
    MapTube
    -­‐  hVp://www.maptube.org
    

    View Slide

41. hVp://download.bigdatatoolkit.org
    

    View Slide

42. 
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    Thanks
    Steven Gray
    steven.gray@ucl.ac.uk
    @frogo    +Steven  Gray
    

    View Slide