GINA_Data_Zoo.pdf

This talk was presented to the CUGOS Fall fling event on Oct. 16th

Transcript

1. The GINA Data Zoo Puffins, and lynx, and hamsters! Oh my... October 16th, 2013 Will Fisher Geographic Information Network of Alaska International Arctic Research Center University of Alaska Fairbanks Friday, October 18, 13
2. Friday, October 18, 13
3. • transfer data? • manage our processing systems? • let people consume data? How do we... Friday, October 18, 13
4. What does GINA do? Friday, October 18, 13
5. We get data and figure out how to make it usable for the people that need it. Friday, October 18, 13
6. Or to put it another way, we feed the animals Nom nom nom Friday, October 18, 13
7. Direct Reception Friday, October 18, 13
8. Partner with NOAA/NESDIS FCDAS Friday, October 18, 13
9. MODIS Truecolor - 2013.06.17 h p://feeder.gina.alaska.edu/modis-gina-alaska-naturalcolor-images/2013_06_17_21_32_jd168 Friday, October 18, 13
10. SUOMI NPP Day-night-band- 2013.01.18 h p://feeder.gina.alaska.edu/modis-gina-alaska-naturalcolor-images/2013_06_17_21_32_jd168 Friday, October 18, 13
11. Source data ~7.6 times the area of Washington state Ortho tile data ~4.3 times the area of Washington state Friday, October 18, 13
12. Friday, October 18, 13
13. Friday, October 18, 13
14. Satellite'Remote'Sensing' •  Statewide(Imagery( •  High(resolu4on( •  Historical(and(4me(series( Aerial'Remote'Sensing' •  LiDAR(–(eleva4on( •  Allows(hydro=modeling( •  Imagery(=(Historical( Water'Sensors' •  USGS(In=stream(Data(Loggers( In2situ'Sensors' •  MET(Sensors( Friday, October 18, 13
15. Transferring data Image(c) Paramount Pictures Friday, October 18, 13
16. waitp.perl •Based on waitd made by SeaSpace for Terascan •Simple, but... •Serial, no logging, no notifications Friday, October 18, 13
17. Conveyor http://www.flickr.com/photos/captainmcdan/4182287774/ Friday, October 18, 13
18. The basics • Foreman • Blueprints • Workers • Tools • Packages • :shipit: Friday, October 18, 13
19. RUBY IS FAST I LOVE IT SO MUCH Friday, October 18, 13
20. Libraries • Celluloid - celluloid.io • Hamster - github.com/harukizaemon/ hamster • Listen - github.com/guard/listen • Thor - github.com/erikhuda/thor • StatsD - github.com/etsy/statsd/ Friday, October 18, 13
21. Version 0.1 •Native FS events (Polling as a fallback) •Parallel processing •Logging/Notifications •http://github.com/gina-alaska/conveyor Friday, October 18, 13
22. Version 2.0 • Concurrency • Metrics • More robust error handling / job tracking Friday, October 18, 13
23. Down the road • Object store support (S3 compatible) • Message queues to trigger other processing jobs (Redis/0MQ?) Friday, October 18, 13
24. Processing Data Friday, October 18, 13
25. Image Processing Friday, October 18, 13
26. Configuration management Friday, October 18, 13
27. Use computers to do things Friday, October 18, 13
28. Use humans to drink with Friday, October 18, 13
29. Use computers to do things Friday, October 18, 13
30. Configuration management • Chef - www.opscode.com/chef/ • Cookbooks available at github.com/gina-alaska Friday, October 18, 13
31. Consuming data Friday, October 18, 13
32. Puffin Feeder h p://feeder.gina.alaska.edu http://www.flickr.com/photos/alessiomesiano/6227234098/ Friday, October 18, 13
33. Website: feeder.gina.alaska.edu Android app: GINA Puffin Feeder Friday, October 18, 13
34. Puffin Feeder - Origins Friday, October 18, 13
35. Puffin Feeder - Origins Friday, October 18, 13
36. Images • 12 Feeds o 5 Modis o 4 NPP o 2 Radar o 1 Webcam • ~3TB Archive • ~10GB/day Friday, October 18, 13
37. Movies Friday, October 18, 13
38. Friday, October 18, 13
39. Friday, October 18, 13
40. Puffin Feeder - In Use Friday, October 18, 13
41. Friday, October 18, 13
42. Friday, October 18, 13
43. gLynx - Catalog and data portals h p://alaska.portal.gina.alaska.edu http://www.flickr.com/photos/luvi/3963857274/ Friday, October 18, 13
44. gLynx - Origins Friday, October 18, 13
45. Friday, October 18, 13
46. Yo dawg i herd you like portals, so we put a portal in your portal, so you can search all the portals Friday, October 18, 13
47. Friday, October 18, 13
48. What is gLynx? • A single database of data records • But with multiple front ends (portals) • Separate search indexes for each portal • Each portal can manage its own look, data records and users Friday, October 18, 13
49. Why? • Automatic sharing of data records • Shared collection of organizations and contact information • Giving the data owners control Friday, October 18, 13
50. Yo dawg i herd you like portals, so we put a portal in your portal, so you can search all the portals Friday, October 18, 13
51. Questions? https://twitter.com/AlexJamesFitz/status/388310922005336064/photo/1 • Will Fisher • www.gina.alaska.edu • github.com/gina-alaska • uafgina.tumblr.com • @uafgina Friday, October 18, 13