Mapping time

C010992dff8a2b7342936d312216ccab?s=47 tokumine
October 30, 2012

Mapping time

Using CartoDB to enable interactive rendering of big temporal data. Describes the technique used in our visualization of OldWeather datapoints: http://cartodb.github.com/oldweather_wwi/

C010992dff8a2b7342936d312216ccab?s=128

tokumine

October 30, 2012
Tweet

Transcript

  1. Mapping time using CartoDB to enable bigdata(™) on the client

    Andrew Hill & Simon Tokumine Friday, 28 September 12
  2. Generalist Vizzuality Friday, 28 September 12

  3. Working to improve our Working to improve our world, one

    project at a time. world, one project at a time. From data visualization to citizen science, it is all about stories that matter From data visualization to citizen science, it is all about stories that matter RECENT WORK Evolution Of Web 2.0 Google Endangered Languages Google GFW 2.0 Prototype World Resources Institute Irekia 2.0 Basque Government Carbon Calculator UNEP-WCMC OldWeather Zooniverse Friday, 28 September 12 member of Vizzuality. Small agency. Madrid and NYC. Data visualization. Working on stories that matter. Here to talk about CartoDB and some of the cool things you can do with it.
  4. Same old things Server config Geospatial Database (backups, optimal config)

    Import/export Admin UI and config Tiler + interactivity API Data API Caching Authentication ... Friday, 28 September 12 hosted geospatial database (postgis) + nice gui to take the grunt work out of our geospatial projects
  5. Friday, 28 September 12 hosted geospatial database (postgis) + nice

    gui to take the grunt work out of our geospatial projects
  6. Friday, 28 September 12

  7. Friday, 28 September 12

  8. Friday, 28 September 12

  9. CartoDB APIs Maps API SQL API Friday, 28 September 12

  10. SQL API HTTP data API Full postgis 2.0 R/W via

    API keys Public access Read only cache layer CDN'able Friday, 28 September 12
  11. SQL API “select * from quake” https://viz2.cartodb.com/api/v1/sql/? q=select%20*%20from %20quake&format=geojson Friday,

    28 September 12
  12. SQL API Friday, 28 September 12

  13. _I want to see ‘X’ over time using the SQL

    API with big data Friday, 28 September 12
  14. _Traditional approach Download the data to the client and render

    it. http://goo.gl/c6KxN - Recent earthquakes http://goo.gl/B7NK6 - Syria casualties Friday, 28 September 12
  15. Friday, 28 September 12

  16. _n-GB? n-Million rows? the problem Friday, 28 September 12

  17. Friday, 28 September 12

  18. “The main limiting factor we've noticed with webgl is the

    bottleneck in bandwidth to get all the data over.” Josh Livni & Chris Broadfoot - Google. SXSW 2012. http://www.youtube.com/watch? v=zH5bJSG0DZk&feature=youtu.be&t=2h 44m20s Friday, 28 September 12
  19. “Even though you can bring big data into the client

    you might not want to take all of it. There's definitely a case for taking slices of it” http://www.youtube.com/watch? v=zH5bJSG0DZk&feature=youtu.be&t=2h 44m20s Josh Livni & Chris Broadfoot - Google. SXSW 2012. Friday, 28 September 12
  20. _n-GB? n-Million rows? redefining the problem Friday, 28 September 12

  21. _Acceptable granularity? redefining the problem Friday, 28 September 12 Maybe

    Josh and Chris were talking about windowing/streaming, but maybe they were talking about datacubes.
  22. Datacubes Friday, 28 September 12

  23. Datacubes •BI/Finance analysis •multidimensional •summary data •binned variables Friday, 28

    September 12 Aka. Hypercubes/OLAP cubes (Make city people sound like they're lawnmowerman)
  24. Datacubes Cons •can be lossy •take time to generate Pros

    •fast access •lightweight •cachable Friday, 28 September 12 Aka. Hypercubes/OLAP cubes (Make city people sound like they're lawnmowerman)
  25. _Can we cube the data? rather than tile it? Friday,

    28 September 12 Real world usage. Can we replace tiles with these?
  26. Friday, 28 September 12

  27. WITH  hgrid            AS  (SELECT  Cdb_rectanglegrid(Cdb_xyz_extent(8,

     12,  5),                                  Cdb_xyz_resolution(5)  *  4,                                                        Cdb_xyz_resolution(5)  *  4)  AS  cell)   SELECT  x,                y,                Array_agg(c)  vals,                Array_agg(d)  dates   FROM      (SELECT  St_xmax(hgrid.cell)                                                                                        x,                                St_ymax(hgrid.cell)                                                                                        y,                                Count(i.cartodb_id)                                                                                        c,                                Floor((  Date_part('epoch',  built)  -­‐  -­‐10418716800  )  /  32837875)  d                  FROM      hgrid,                                us_po_offices  i                  WHERE    St_intersects(i.the_geom_webmercator,  hgrid.cell)                  GROUP    BY  hgrid.cell,                                      Floor((  Date_part('epoch',  built)  -­‐  -­‐10418716800  )  /  32837875)                )  f   GROUP    BY  x,                      y Friday, 28 September 12
  28. {    rows:  [    {        x:

     0,        y:  0,        vals:  [2],        dates:  [457]    },    {        x:  1,        y:  0,        vals:  [1,1,4],        dates:  [2,3,4]        }    ] } Friday, 28 September 12
  29. 1 10 100 1000 3mb 70mb 300mb 1.5 2 1.2

    300 70 3 Raw Datacube Payload sizes Friday, 28 September 12
  30. _Thanks! Checkout www.cartodb.com for more! Friday, 28 September 12