The Power Of Visualizing Deforestation Data

35f4d000a88cdbcf6392dfb206ebd5e2?s=47 Andrew W Hill
November 11, 2013

The Power Of Visualizing Deforestation Data

The power of visualising big data time-series that are derived from remote sensing products processed on Hadoop can not be overestimated. Visualization can give scientists, policy makers, journalists, and the public immediate insights into how the environment is changing over time, leading to quicker understanding and action. Effectively putting big data time-series on maps remains difficult. With the advent of Hadoop, CartoDB, and HTML5 APIs, our ability to create interactive maps in highly performant ways has greatly improved. New problems with the scale and complexity of near real-time data keep data visualization interesting and challenging. Here, we present our work to develop fast mapping solutions for 500 meter resolution deforestation data produced 16-days by the FORMA algorithm for the Global Forest Watch 2.0 web portal.

Deforestation data contain rich temporal information that is often lost when visualized using static map tiles. To ensure that these data characteristics are effectively surfaced in the Global Forest Watch portal, we developed new methods for big data storage, query, transfer, and map-based visualization. The Forest Monitoring for Action (FORMA) project provides free and open forest clearing alert data derived from MODIS satellite imagery every 16 days beginning in December 2005. FORMA is written in the Clojure programming language and rides on Cascading and Cascalog APIs for processing big spatial data on top of Hadoop using MapReduce. Here we will focus on the high level FORMA algorithm and data workflow, with a particular emphasis on the visualization mechanisms for these data.

At a high level, deforestation events are converted from raster products to JSON data objects. Each JSON data object efficiently stores an index of the date and pixel locations of deforestation on quadtree map-tiles. On the client, these three-dimensional JSON objects are unpacked and used to render HTML5 canvas objects that are displayed on the map. In combination with user-interface controls, users can interact with the history of deforestation on the map.

The methods developed for the Global Forest Watch website have been further generalized in an open-source library called, Torque (http://github.com/CartoDB/Torque). A generalized SQL statement to compress temporal-geospatial data to tile-based JSON objects and the HTML5 canvas rendering functions will be expanded in the future to visualize the motion of multiple agents and ordered, non-temporal, data. In this presentation we will describe in-depth the analysis of deforestation data, the efficiency of the temporal JSON data schema, and finally the challenges and rewards of visualizing temporal data on the web.

35f4d000a88cdbcf6392dfb206ebd5e2?s=128

Andrew W Hill

November 11, 2013
Tweet

Transcript

  1. The Power of Visualizing ! Deforestation Data @andrewxhill @robinkraft @jatorre

  2. Senior Scientist! Vizzuality / CartoDB Andrew Hill Data Architect! WRI

    Data Lab Robin Kraft
  3. None
  4. None
  5. None
  6. All meteorites fallen on earth

  7. None
  8. But… this talk is about more using data visualization to

    tell stories
  9. None
  10. This talk is really about storytelling using data visualization to

    tell stories
  11. Why storytelling? thoughts and reflection

  12. Human minds yield helplessly to the suction of story. Jonathan

    Gottschall
  13. None
  14. None
  15. We are primed to use stories. Part of our survival

    as a species depended upon listening to the stories of our tribal elders... Philippa Perry
  16. None
  17. Visual storytelling why maps and data visualization

  18. None
  19. None
  20. Storytelling is powerful The art of storytelling with maps

  21. Let me show you using data visualization to tell a

    story
  22. Two non-forest examples using data visualization to tell a story

  23. Tax lots are interesting! 1

  24. NYC OPEN DATA

  25. None
  26. c4a.me/1dYHKv9/ PLUTO IS FREE

  27. None
  28. bit.ly/19fjB6R

  29. bit.ly/19fjB6R

  30. bit.ly/19fjB6R

  31. None
  32. Visualizing NYC Open Data

  33. There is science in there! 2

  34. None
  35. None
  36. None
  37. None
  38. None
  39. None
  40. www.nychenge.com

  41. None
  42. None
  43. None
  44. When we can quickly see the effects on something over

    time... there’s an instant “Aha!” David Byrne
  45. When we make maps, we try to tell stories. the

    power of data visualization
  46. Let’s tell the story of deforestation using data visualization to

    tell it different
  47. Lead Data Architect, Data Lab! World Resources Institute! ! @robinkraft

    Robin Kraft
  48. Global Forest Watch

  49. It’s a website GFW

  50. Big data analysis, visualization,! and communication GFW

  51. Data exploration GFW

  52. Temporal data! GFW

  53. How do we create temporal! deforestation data? FORMA - Forest

    Monitoring for Action
  54. Scan complete MODIS Every 16 days FORMA - Forest Monitoring

    for Action
  55. A ton of data All the rain forests in ~90

    countries FORMA - Forest Monitoring for Action
  56. A ton of data 300 time periods (every 16 days

    over 13 years) FORMA - Forest Monitoring for Action
  57. High resolution 500m, soon 250m, pixels FORMA - Forest Monitoring

    for Action
  58. A ton of data 60 million forested pixels! 4-5 million

    w/ deforestation alerts FORMA - Forest Monitoring for Action
  59. Lots of challenges Scale, speed, clouds etc. FORMA - Forest

    Monitoring for Action
  60. So how do we do it? Besides coffee FORMA -

    Forest Monitoring for Action
  61. OSS & the cloud Machine learning + Satellite data

  62. Machine learning!

  63. Outputs dumped into CartoDB Time to see it online!

  64. This is what it looks like… http://www.gfw-beta.org! ! PM me

    for the password: @robinkraft
  65. None
  66. None
  67. None
  68. None
  69. None
  70. None
  71. None
  72. None
  73. And these are the things ! we can find… GFW

  74. None
  75. None
  76. None
  77. None
  78. None
  79. None
  80. None
  81. None
  82. None
  83. None
  84. http://en.wikipedia.org/wiki/Bakun_Dam

  85. Senior Scientist! Vizzuality / CartoDB Andrew Hill

  86. Allowing people to see ! stories of deforestation Putting time

    on the map
  87. GFW 2.0 How are we doing this on the web?

  88. None
  89. WITH  hgrid              AS  (SELECT

     Cdb_rectanglegrid(Cdb_xyz_extent(8,  12,  5),                                    Cdb_xyz_resolution(5)  *  4,                                                          Cdb_xyz_resolution(5)  *  4)  AS  cell)     SELECT  x,                  y,                  Array_agg(c)  vals,                  Array_agg(d)  dates     FROM      (SELECT  St_xmax(hgrid.cell)                                                                                        x,                                  St_ymax(hgrid.cell)                                                                                        y,                                  Count(i.cartodb_id)                                                                                        c,                                  Floor((  Date_part('epoch',  built)  -­‐  -­‐10418716800  )  /  32837875)  d                    FROM      hgrid,                                  us_po_offices  i                    WHERE    St_intersects(i.the_geom_webmercator,  hgrid.cell)                    GROUP    BY  hgrid.cell,                                        Floor((  Date_part('epoch',  built)  -­‐  -­‐10418716800  )  /  32837875)                  )  f     GROUP    BY  x,                        y
  90. {
    rows:  [
    {
        x:

     0,
        y:  0,
        vals:  [2],
        dates:  [457]
    },
    {
        x:  1,
        y:  0,
        vals:  [1,1,4],
        dates:  [2,3,4]
        }
    ]
 }
  91. Data payload 1 10 100 1000 3mb 70mb 300mb 1.5

    2 1.2 300 70 3 Raw Datacube
  92. This is powerful! GFW 2.0

  93. Temporal visualization is powerful GFW 2.0

  94. Mapping is powerful GFW 2.0

  95. Animated city traffic maps

  96. Mobile ready.

  97. Challenges and solution Big data + Dynamic data in the

    browser
  98. Big data without visualization is a failure We are trying

    to help fix that
  99. Maps are one of the most popular types High literacy,

    popular appeal, personal context
  100. Ugly map! But making a good map. A storytelling map

    can be hard
  101. None
  102. Wall Street Journal US election maps

  103. Big data analysis and reporting tool - UNEP Carbon calculator

  104. Putting time on the map Telling stories about change

  105. Announcing Torque! Making maps to improve the world

  106. CEO! Vizzuality / CartoDB Javier de la Torre

  107. Demo Torque + CartoDB

  108. Making maps to save the forest Making maps to improve

    the world
  109. Making maps to tell stories Making maps to improve the

    world
  110. You can do it too! here are some ways

  111. Come see me at Office Hours Today - 2:47 pm

  112. None
  113. None
  114. www.cartodb.com/academy spread the word!

  115. None
  116. @andrewxhill @robinkraft @jatorre Let us know how you are telling

    ! stories with data
  117. @andrewxhill @robinkraft @jatorre thanks!