Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Data Hacking and The City: UCL CASA Conference 2013 Workshop

Steven Gray
September 26, 2013

Data Hacking and The City: UCL CASA Conference 2013 Workshop

These are the slides from the UCL CASA Conference 2013 Workshop where we collected data via the Twitter API and mapped them on Google Maps Engine. Full Tutorial and scripts to download are referenced inside presentation so reader can start collecting Twitter Data quickly and easily!

Steven Gray

September 26, 2013
Tweet

More Decks by Steven Gray

Other Decks in Programming

Transcript





























































































  1. Data Hacking and the City
    CASA Workshop
    Steven Gray
    Thursday 26th September 2013 - 9:30 - 12:30

    View Slide

  2. Order  of  the  Morning
    9:30  -­‐  10:00  -­‐  Showcase  of  what  we  do
    10:00  -­‐  11:00  -­‐  Coding  -­‐-­‐  Collec8ng  the  Data
    11:00  -­‐  12:00  -­‐  More  Coding  -­‐  Visualising  Data  
    12:00  -­‐  12:30  -­‐  Ques8on  and  Wrap  up

    View Slide

  3. Lets  start  coding!
    But  first  a  few  things  we  need  to  sort  out
    Separate  into  Groups
             -­‐-­‐  Social  Data  (Best  for  Beginners  to  coding)
             -­‐-­‐  Transport  Data
             -­‐-­‐  Environmental  Data
    Rules
    -­‐-­‐  Don’t  be  afraid  to  ask  ques8ons  if  you  don’t  understand
    -­‐-­‐  Have  fun  and  Enjoy  yourself

    View Slide

  4. Quick  Showcase  of  my  work

    View Slide

  5. View Slide

  6. View Slide

  7. View Slide

  8. View Slide

  9. View Slide

  10. View Slide

  11. View Slide

  12. Lets  start  coding!

    View Slide

  13. Mac
    -­‐-­‐  Install  Xcode  Command  line  tools  for  your  Mac  OS  Version
    -­‐-­‐  h$ps://developer.apple.com/downloads/
    -­‐-­‐  Install  Homebrew  (hVp://brew.sh/)
    -­‐-­‐  Run  on  terminal:  
    ruby  -­‐e  "$(curl  -­‐fsSL  hVps://raw.github.com/mxcl/homebrew/go)"
    -­‐-­‐  Follow  hVp://madebyhoundstooth.com/blog/install-­‐node-­‐with-­‐homebrew-­‐on-­‐os-­‐x/
    -­‐-­‐  brew  install  node
    NPM  will  be  installed  at  the  same  8me  as  node!
    Install  NodeJS  +  NPM

    View Slide

  14. Linux  (Ubuntu)
    -­‐-­‐    Follow  hVps://github.com/joyent/node/wiki/Installing-­‐Node.js-­‐via-­‐package-­‐manager
    Windows  
    -­‐-­‐  Download  and  follow  instruc8ons
    Install  NodeJS  +  NPM
    hVp://nodejs.org/download/
    Open  Node.js  from  Start  Menu  -­‐  Programs  -­‐  Node.js  -­‐  Node.js  Command  Prompt
    Or  search  for  “Node.js  Command  Prompt”  on  your  system  (Windows  8,  8.1)  

    View Slide

  15. Install  Modules  via  NPM    (All  Systems)
    Use  NPM  to  install  the  following  modules  used  by  the  collector.  
    Check  that  you  have  npm  installed  by  running  “npm  -­‐v”  (it  should  report  1.3.*)  
    Run  the  following  to  install
    npm  install  oauth  colors

    View Slide

  16. CollecIng  Social  Media  Data  (TwiLer)  -­‐  A  Recipe
    What  you  will  need:
    1  x  TwiVer  Account
    1  x  Developer  Key  from  hLps://dev.twiLer.com/
    Sprinkle  with  some  pa8ence  and  8me  and  in  8me  you’ll  be  collec8ng  
    data  from  TwiVer.
    Instruc8ons  coming  up  soon  aier  a  small  explanta8on.

    View Slide

  17. Understanding  what’s  happening
    http://dev.twitter.com/doc
    User Pro!les Tweet Database

    View Slide

  18. HTTP
    GET
    Request
    Results
    {"results":[
    {"text":"@twitterapi is great!",
    "to_user_id":396524,
    "to_user":"TwitterAPI",
    "from_user":"jkoum",
    "metadata":
    {
    "result_type":"popular",
    "recent_retweets": 100
    },
    "id":1478555574,
    "from_user_id":1833773,
    "iso_language_code":"nl",
    ....}
    }
    Returns tweets that:
    • match speci!ed query
    • located in Lat/Lon box
    • Historical Tweets
    • User Pro!le Information
    • Return Public Timeline
    Service is rate limited
    JSON or Atom
    Making  Requests  to  TwiLer

    View Slide

  19. AuthenIcaIon
    2  Methods  of  Authen8ca8on
    -­‐-­‐  Basic  HTTP  Authen8ca8on
    -­‐-­‐  oAuth    (More  Complex
    Most  modern  day  web  services  use  oAuth  (including  TwiVer  and  Google  Services)
    Keeps  informa8on  safe  from  the  wrong  people  but  also  allows  providers  to  ban  people  
    if  they  are  using  too  many  resources  or  making.

    View Slide

  20. So  how  does  oAuth  work
    Server
    Applica8on

    View Slide

  21. So  how  does  oAuth  work
    Server
    Applica8on
    Request  a  Session  Token
    Pass  a  ID  and  a  Secret

    View Slide

  22. So  how  does  oAuth  work
    Server
    Applica8on
    Request  a  Session  Token
    Pass  a  ID  and  a  Secret
    Asks  Login  with  ID  
    Server  gives  a  Pin  number  for  session

    View Slide

  23. So  how  does  oAuth  work
    Server
    Applica8on

    View Slide

  24. So  how  does  oAuth  work
    Server
    Applica8on
    Login  is  successful
    :)

    View Slide

  25. So  how  does  oAuth  work
    Server
    Applica8on
    Login  is  successful
    :)
    Sends  App  Tokens
    These  need  to  be  saved!

    View Slide

  26. So  how  does  oAuth  work
    Server
    Applica8on

    View Slide

  27. So  how  does  oAuth  work
    Server
    Applica8on
    Request  using  tokens
    Server  checks  if  it’s  valid

    View Slide

  28. So  how  does  oAuth  work
    Server
    Applica8on
    Request  using  tokens
    Server  checks  if  it’s  valid
    Data  Returned
    :)

    View Slide

  29. Go  to  hLps://dev.twiLer.com/
    Login  using  your  current  TwiVer  account
    Click  your  profile  picture  (top  right  corner)  and  click  My  ApplicaIons
    Click  Create  New  ApplicaIon
    Fill  in  your  details  for  the  app
    You’ll  be  presented  with  your  Consumer  Key  and  Consumer  Secret
    Save  these  keys  for  the  next  step
    CollecIng  Social  Media  Data  (TwiLer)
    CreaIng  your  TwiLer  ApplicaIon  Keys

    View Slide

  30. Lets  Collect  some  data
    Download  the  Collector
    hVps://gist.github.com/sjg/6712144

    View Slide

  31. Lets  Collect  some  data
    Aier  downloading  the  script  s8ck  in  your  keys  and  secret  in  the  script  between  the  
    quota8on  marks  and  save  the  file
    Consumer  Key
    Consumer  Secret

    View Slide

  32. Lets  Collect  some  data
    Copy  the  script  into  a  folder  that  you  can  run  the  script  from  and  navigate  to  that  
    folder  in  your  system  terminal.
    You  can  edit  the  loca8on  of  the  collector  by  changing  these  values  in  the  file:
    To  run  the  collector  type:  
    node  twiLer_collect.js
    “SearchTag”,  “La8tude”,  “Longitude”,  “Radius”
     -­‐  NB:    “”  will  return  all  tweets  in  that  area

    View Slide

  33. Lets  Collect  some  data
    If  all  goes  well  and  everything  is  working  you  should  see  the  following  screen
    Go  to  the  link  in  a  browser,  login  to  twiVer  and  type  the  pin  number  in.    Press  return  
    and  tweets  will  start  collec8ng  in  a  file  called  “tweets.csv”  in  the  same  folder.

    View Slide

  34. Visualising  the  output

    View Slide

  35. Go  to  hLps://mapsengine.google.com
    Click  on  New  Map
    Name  your  map  by  clicking  on  “UnItled  Map”
    Click  “Import”  and  upload  your  CSV  file
    Click  Base  Map  to  change  the  map  style
    CreaIng  your  first  VisualisaIon  with  the  Data  Collected  

    View Slide

  36. CreaIng  your  first  VisualisaIon  with  the  Data  Collected  
    Once  the  file  has  uploaded,  choose  the  correct  La8tude  and  Longitude  columns  and  
    then  click  Con8nue.    Click  the  ID  field  (lei  over)  as  the  marker  8tle  and  press  finish.
    The  updated  output  file  will,  hopefully,  auto  detect  the  lat/lng  fields.

    View Slide

  37. CreaIng  your  first  VisualisaIon  with  the  Data  Collected  
    Style  your  markers  by  pressing  the  paint  bucket  (highlighted  in  red)  and  select  the  
    circle  shape  and  colour.    Experiment  with  styling  your  map.

    View Slide

  38. CreaIng  your  first  VisualisaIon  with  the  Data  Collected  
    If  all  went  well  your  map  will  look  something  like  this.    Why  not  share  it  with  the  
    world!

    View Slide

  39. CreaIng  your  first  VisualisaIon  with  the  Data  Collected  
    And  you  can  click  the  markers  and  see  the  tweets

    View Slide

  40. Different  Sources  you  can  use  to  Visualise  your  data
    Google  Fusion  Tables
    -­‐  hVp://www.google.com/drive/apps.html#fusiontables
    Google  Map  Engine
    -­‐  hVps://mapsengine.google.com
    Google  Earth
    -­‐  hVp://www.google.com/earth/index.html
    MapTube
    -­‐  hVp://www.maptube.org

    View Slide

  41. hVp://download.bigdatatoolkit.org

    View Slide





























































































  42. Thanks
    Steven Gray
    [email protected]
    @frogo    +Steven  Gray

    View Slide