The Power Of Visualizing Deforestation Data

The power of visualising big data time-series that are derived from remote sensing products processed on Hadoop can not be overestimated. Visualization can give scientists, policy makers, journalists, and the public immediate insights into how the environment is changing over time, leading to quicker understanding and action. Effectively putting big data time-series on maps remains difficult. With the advent of Hadoop, CartoDB, and HTML5 APIs, our ability to create interactive maps in highly performant ways has greatly improved. New problems with the scale and complexity of near real-time data keep data visualization interesting and challenging. Here, we present our work to develop fast mapping solutions for 500 meter resolution deforestation data produced 16-days by the FORMA algorithm for the Global Forest Watch 2.0 web portal.

Deforestation data contain rich temporal information that is often lost when visualized using static map tiles. To ensure that these data characteristics are effectively surfaced in the Global Forest Watch portal, we developed new methods for big data storage, query, transfer, and map-based visualization. The Forest Monitoring for Action (FORMA) project provides free and open forest clearing alert data derived from MODIS satellite imagery every 16 days beginning in December 2005. FORMA is written in the Clojure programming language and rides on Cascading and Cascalog APIs for processing big spatial data on top of Hadoop using MapReduce. Here we will focus on the high level FORMA algorithm and data workflow, with a particular emphasis on the visualization mechanisms for these data.

At a high level, deforestation events are converted from raster products to JSON data objects. Each JSON data object efficiently stores an index of the date and pixel locations of deforestation on quadtree map-tiles. On the client, these three-dimensional JSON objects are unpacked and used to render HTML5 canvas objects that are displayed on the map. In combination with user-interface controls, users can interact with the history of deforestation on the map.

The methods developed for the Global Forest Watch website have been further generalized in an open-source library called, Torque (http://github.com/CartoDB/Torque). A generalized SQL statement to compress temporal-geospatial data to tile-based JSON objects and the HTML5 canvas rendering functions will be expanded in the future to visualize the motion of multiple agents and ordered, non-temporal, data. In this presentation we will describe in-depth the analysis of deforestation data, the efficiency of the temporal JSON data schema, and finally the challenges and rewards of visualizing temporal data on the web.

Transcript

1. The Power of Visualizing ! Deforestation Data @andrewxhill @robinkraft @jatorre
2. Senior Scientist! Vizzuality / CartoDB Andrew Hill Data Architect! WRI Data Lab Robin Kraft
6. All meteorites fallen on earth
8. But… this talk is about more using data visualization to tell stories
10. This talk is really about storytelling using data visualization to tell stories
11. Why storytelling? thoughts and reflection
12. Human minds yield helplessly to the suction of story. Jonathan Gottschall
15. We are primed to use stories. Part of our survival as a species depended upon listening to the stories of our tribal elders... Philippa Perry
17. Visual storytelling why maps and data visualization
20. Storytelling is powerful The art of storytelling with maps
21. Let me show you using data visualization to tell a story
22. Two non-forest examples using data visualization to tell a story
23. Tax lots are interesting! 1
24. NYC OPEN DATA
26. c4a.me/1dYHKv9/ PLUTO IS FREE
28. bit.ly/19fjB6R
29. bit.ly/19fjB6R
30. bit.ly/19fjB6R
32. Visualizing NYC Open Data
33. There is science in there! 2
40. www.nychenge.com
44. When we can quickly see the effects on something over time... there’s an instant “Aha!” David Byrne
45. When we make maps, we try to tell stories. the power of data visualization
46. Let’s tell the story of deforestation using data visualization to tell it different
47. Lead Data Architect, Data Lab! World Resources Institute! ! @robinkraft Robin Kraft
48. Global Forest Watch
49. It’s a website GFW
50. Big data analysis, visualization,! and communication GFW
51. Data exploration GFW
52. Temporal data! GFW
53. How do we create temporal! deforestation data? FORMA - Forest Monitoring for Action
54. Scan complete MODIS Every 16 days FORMA - Forest Monitoring for Action
55. A ton of data All the rain forests in ~90 countries FORMA - Forest Monitoring for Action
56. A ton of data 300 time periods (every 16 days over 13 years) FORMA - Forest Monitoring for Action
57. High resolution 500m, soon 250m, pixels FORMA - Forest Monitoring for Action
58. A ton of data 60 million forested pixels! 4-5 million w/ deforestation alerts FORMA - Forest Monitoring for Action
59. Lots of challenges Scale, speed, clouds etc. FORMA - Forest Monitoring for Action
60. So how do we do it? Besides coffee FORMA - Forest Monitoring for Action
61. OSS & the cloud Machine learning + Satellite data
62. Machine learning!
63. Outputs dumped into CartoDB Time to see it online!
64. This is what it looks like… http://www.gfw-beta.org! ! PM me for the password: @robinkraft
73. And these are the things ! we can find… GFW
84. http://en.wikipedia.org/wiki/Bakun_Dam
85. Senior Scientist! Vizzuality / CartoDB Andrew Hill
86. Allowing people to see ! stories of deforestation Putting time on the map
87. GFW 2.0 How are we doing this on the web?
89. WITH  hgrid              AS  (SELECT  Cdb_rectanglegrid(Cdb_xyz_extent(8,  12,  5),                                    Cdb_xyz_resolution(5)  *  4,                                                          Cdb_xyz_resolution(5)  *  4)  AS  cell)     SELECT  x,                  y,                  Array_agg(c)  vals,                  Array_agg(d)  dates     FROM      (SELECT  St_xmax(hgrid.cell)                                                                                        x,                                  St_ymax(hgrid.cell)                                                                                        y,                                  Count(i.cartodb_id)                                                                                        c,                                  Floor((  Date_part('epoch',  built)  -­‐  -­‐10418716800  )  /  32837875)  d                    FROM      hgrid,                                  us_po_offices  i                    WHERE    St_intersects(i.the_geom_webmercator,  hgrid.cell)                    GROUP    BY  hgrid.cell,                                        Floor((  Date_part('epoch',  built)  -­‐  -­‐10418716800  )  /  32837875)                  )  f     GROUP    BY  x,                        y
90. {
    rows:  [
    {
        x:  0,
        y:  0,
        vals:  [2],
        dates:  [457]
    },
    {
        x:  1,
        y:  0,
        vals:  [1,1,4],
        dates:  [2,3,4]
        }
    ]
 }
91. Data payload 1 10 100 1000 3mb 70mb 300mb 1.5 2 1.2 300 70 3 Raw Datacube
92. This is powerful! GFW 2.0
93. Temporal visualization is powerful GFW 2.0
94. Mapping is powerful GFW 2.0
95. Animated city traffic maps
96. Mobile ready.
97. Challenges and solution Big data + Dynamic data in the browser
98. Big data without visualization is a failure We are trying to help fix that
99. Maps are one of the most popular types High literacy, popular appeal, personal context
100. Ugly map! But making a good map. A storytelling map can be hard
102. Wall Street Journal US election maps
103. Big data analysis and reporting tool - UNEP Carbon calculator
104. Putting time on the map Telling stories about change
105. Announcing Torque! Making maps to improve the world
106. CEO! Vizzuality / CartoDB Javier de la Torre
107. Demo Torque + CartoDB
108. Making maps to save the forest Making maps to improve the world
109. Making maps to tell stories Making maps to improve the world
110. You can do it too! here are some ways
111. Come see me at Office Hours Today - 2:47 pm
114. www.cartodb.com/academy spread the word!
116. @andrewxhill @robinkraft @jatorre Let us know how you are telling ! stories with data
117. @andrewxhill @robinkraft @jatorre thanks!