K E C U S T O M M A P S . W E N E E D E D A G O O D S A T E L L I T E B A S E M A P. Satellite Team: Chris Herwig (@hrwgc) – [email protected] Charlie Loyd (@vruba) – [email protected] Bruno Sánchez-Andrade Nuño (@brunosan) – [email protected]
and general accuracy both vital • Blue Marble was a big inspiration but not our goal • Goal: Avoid spatial interpolation • Goal: Show peak growth everywhere at once • MODIS was the clear choice
function is the core of this whole process • Algorithm assigns pixels scores based on how much they look like ground cover (as opposed to clouds, smoke, errors, missing data, etc.) • Since it looks at every single input pixel (~5e12 pixels), it has to be very, very simple
edge and reaching across to the right edge are valid in this space, because we’ve defined saturation as the brightest RGB channel minus the dimmest. • Under this definition, there are no highly saturated black or white pixels.
E RY • Total desired source images: > 400,000 • Timeline was ASAP • We talked with Ryan Boller (NASA GIBS) to figure out best way to handle data acquisition
N • Lance MODIS was good • GIBS (Global Imagery Browse Service) was better in terms of bulk download speeds • GIBS only had MODIS data back to 5/2012 • Lance MODIS offered pre-cut subsets; GIBS data was via global WMS
A N • Rows 0–7: days of the year 1–80, 265–366 • Rows 8–11: all days of the year • Rows 12+: days of the year 81–264 • All 2011 imagery from Lance • 2012-001–2012-137 from Lance • 2012-138+ from GIBS
E RY • Distributed downloads across 30–45 EC2s, each running 5–15 download processes • 10 second wait time between successive downloads • Random 0–3 second wait time between concurrent downloads (across machines) • Amazon SQS: Task queuing system to ensure we downloaded each file exactly once across our cloud
http://lance-modis.eosdis.nasa.gov/imagery/ subsets/RRGlobal_$LANCE_ID/$DATE/RRGlobal_$LANCE_ID. $DATE.$SATELLITE.250m.jpg L A N C E - M O D I S D O W N L O A D
$BBOX \ -co QUALITY=80 \ -a_srs EPSG:4326 \ "<GDAL_WMS> <Service name=\"TiledWMS\"> <ServerUrl>http://map1.vis.earthdata.nasa.gov/twms-geo/ twms.cgi?</ServerUrl> <TiledGroupName>MODIS ${SATELLITE} tileset</ TiledGroupName> <Change key=\"\${time}\">$DATE</Change> </Service> </GDAL_WMS>" \ "RRGlobal_${LANCE_ID}.${DOY_DATE}.${SATELLITE}.250m.jpg" G I B S D O W N L O A D
Declouding for all 374 tiles: 639 computing hours ! 40 Amazon EC2 spot instances (M2.2XL) for 16 hours ! Besides a few tests and re-dos, the bulk of the processing happened over a single weekend.
I S Landsat’s resolution is great compared to MODIS. But it comes with relative difficulties: • Frequency (≤ 1/16 day v. ≤ 2/day) • No atmospheric correction • Tiling and SLC-Off • Archival availability • Raw file size
improve L1 images substantially • Not enough documentation easily accessible for use of software/underlying algorithm • No Landsat 8 support (yet) • Compiling is a chore -> AWS Cloudformation + AMIs
W S Cloud shadows don’t really matter until you’re at about 100 m/px or sharper. ! They trick a naïve algorithm because they’re darker, which usually means clearer, than other pixels. ! So you have to get into fine-tuning the band math, which has other effects too!