Upgrade to Pro — share decks privately, control downloads, hide ads and more …

GINA_Data_Zoo.pdf

 GINA_Data_Zoo.pdf

This talk was presented to the CUGOS Fall fling event on Oct. 16th

Will Fisher

October 16, 2013
Tweet

Other Decks in Programming

Transcript

  1. The GINA Data Zoo
    Puffins, and lynx, and hamsters! Oh my...
    October 16th, 2013
    Will Fisher
    Geographic Information Network of Alaska
    International Arctic Research Center
    University of Alaska Fairbanks
    Friday, October 18, 13

    View full-size slide

  2. Friday, October 18, 13

    View full-size slide

  3. • transfer data?
    • manage our processing systems?
    • let people consume data?
    How do we...
    Friday, October 18, 13

    View full-size slide

  4. What does GINA do?
    Friday, October 18, 13

    View full-size slide

  5. We get data and figure out
    how to make it usable for
    the people that need it.
    Friday, October 18, 13

    View full-size slide

  6. Or to put it another way, we feed the
    animals
    Nom nom nom
    Friday, October 18, 13

    View full-size slide

  7. Direct Reception
    Friday, October 18, 13

    View full-size slide

  8. Partner with
    NOAA/NESDIS FCDAS
    Friday, October 18, 13

    View full-size slide

  9. MODIS Truecolor - 2013.06.17
    h p://feeder.gina.alaska.edu/modis-gina-alaska-naturalcolor-images/2013_06_17_21_32_jd168
    Friday, October 18, 13

    View full-size slide

  10. SUOMI NPP Day-night-band- 2013.01.18
    h p://feeder.gina.alaska.edu/modis-gina-alaska-naturalcolor-images/2013_06_17_21_32_jd168
    Friday, October 18, 13

    View full-size slide

  11. Source data ~7.6 times the area of Washington state
    Ortho tile data ~4.3 times the area of Washington state
    Friday, October 18, 13

    View full-size slide

  12. Friday, October 18, 13

    View full-size slide

  13. Friday, October 18, 13

    View full-size slide

  14. Satellite'Remote'Sensing'
    •  Statewide(Imagery(
    •  High(resolu4on(
    •  Historical(and(4me(series(
    Aerial'Remote'Sensing'
    •  LiDAR(–(eleva4on(
    •  Allows(hydro=modeling(
    •  Imagery(=(Historical(
    Water'Sensors'
    •  USGS(In=stream(Data(Loggers(
    In2situ'Sensors'
    •  MET(Sensors(
    Friday, October 18, 13

    View full-size slide

  15. Transferring data
    Image(c) Paramount Pictures
    Friday, October 18, 13

    View full-size slide

  16. waitp.perl
    •Based on waitd made by SeaSpace for
    Terascan
    •Simple, but...
    •Serial, no logging, no notifications
    Friday, October 18, 13

    View full-size slide

  17. Conveyor
    http://www.flickr.com/photos/captainmcdan/4182287774/
    Friday, October 18, 13

    View full-size slide

  18. The basics
    • Foreman
    • Blueprints
    • Workers
    • Tools
    • Packages
    • :shipit:
    Friday, October 18, 13

    View full-size slide

  19. RUBY IS FAST
    I LOVE IT
    SO MUCH
    Friday, October 18, 13

    View full-size slide

  20. Libraries
    • Celluloid - celluloid.io
    • Hamster - github.com/harukizaemon/
    hamster
    • Listen - github.com/guard/listen
    • Thor - github.com/erikhuda/thor
    • StatsD - github.com/etsy/statsd/
    Friday, October 18, 13

    View full-size slide

  21. Version 0.1
    •Native FS events (Polling as a fallback)
    •Parallel processing
    •Logging/Notifications
    •http://github.com/gina-alaska/conveyor
    Friday, October 18, 13

    View full-size slide

  22. Version 2.0
    • Concurrency
    • Metrics
    • More robust error handling / job tracking
    Friday, October 18, 13

    View full-size slide

  23. Down the road
    • Object store support (S3 compatible)
    • Message queues to trigger other processing
    jobs (Redis/0MQ?)
    Friday, October 18, 13

    View full-size slide

  24. Processing Data
    Friday, October 18, 13

    View full-size slide

  25. Image Processing
    Friday, October 18, 13

    View full-size slide

  26. Configuration management
    Friday, October 18, 13

    View full-size slide

  27. Use computers to do
    things
    Friday, October 18, 13

    View full-size slide

  28. Use humans to drink
    with
    Friday, October 18, 13

    View full-size slide

  29. Use computers to do
    things
    Friday, October 18, 13

    View full-size slide

  30. Configuration
    management
    • Chef - www.opscode.com/chef/
    • Cookbooks available at
    github.com/gina-alaska
    Friday, October 18, 13

    View full-size slide

  31. Consuming data
    Friday, October 18, 13

    View full-size slide

  32. Puffin Feeder
    h p://feeder.gina.alaska.edu
    http://www.flickr.com/photos/alessiomesiano/6227234098/
    Friday, October 18, 13

    View full-size slide

  33. Website: feeder.gina.alaska.edu
    Android app: GINA Puffin Feeder
    Friday, October 18, 13

    View full-size slide

  34. Puffin Feeder - Origins
    Friday, October 18, 13

    View full-size slide

  35. Puffin Feeder - Origins
    Friday, October 18, 13

    View full-size slide

  36. Images
    • 12 Feeds
    o 5 Modis
    o 4 NPP
    o 2 Radar
    o 1 Webcam
    • ~3TB Archive
    • ~10GB/day
    Friday, October 18, 13

    View full-size slide

  37. Movies
    Friday, October 18, 13

    View full-size slide

  38. Friday, October 18, 13

    View full-size slide

  39. Friday, October 18, 13

    View full-size slide

  40. Puffin Feeder - In Use
    Friday, October 18, 13

    View full-size slide

  41. Friday, October 18, 13

    View full-size slide

  42. Friday, October 18, 13

    View full-size slide

  43. gLynx - Catalog and data portals
    h p://alaska.portal.gina.alaska.edu
    http://www.flickr.com/photos/luvi/3963857274/
    Friday, October 18, 13

    View full-size slide

  44. gLynx - Origins
    Friday, October 18, 13

    View full-size slide

  45. Friday, October 18, 13

    View full-size slide

  46. Yo dawg i herd you like portals, so we put a portal
    in your portal, so you can search all the portals
    Friday, October 18, 13

    View full-size slide

  47. Friday, October 18, 13

    View full-size slide

  48. What is gLynx?
    • A single database of data
    records
    • But with multiple front
    ends (portals)
    • Separate search indexes
    for each portal
    • Each portal can manage
    its own look, data records
    and users
    Friday, October 18, 13

    View full-size slide

  49. Why?
    • Automatic sharing of data records
    • Shared collection of organizations and
    contact information
    • Giving the data owners control
    Friday, October 18, 13

    View full-size slide

  50. Yo dawg i herd you like portals, so we put a portal
    in your portal, so you can search all the portals
    Friday, October 18, 13

    View full-size slide

  51. Questions?
    https://twitter.com/AlexJamesFitz/status/388310922005336064/photo/1
    • Will Fisher
    • www.gina.alaska.edu
    • github.com/gina-alaska
    • uafgina.tumblr.com
    • @uafgina
    Friday, October 18, 13

    View full-size slide