GINA_Data_Zoo.pdf - Speaker Deck

Slide 1

Slide 1 text

The GINA Data Zoo Puﬃns, and lynx, and hamsters! Oh my... October 16th, 2013 Will Fisher Geographic Information Network of Alaska International Arctic Research Center University of Alaska Fairbanks Friday, October 18, 13

Slide 2

Slide 2 text

Friday, October 18, 13

Slide 3

Slide 3 text

• transfer data? • manage our processing systems? • let people consume data? How do we... Friday, October 18, 13

Slide 4

Slide 4 text

What does GINA do? Friday, October 18, 13

Slide 5

Slide 5 text

We get data and ﬁgure out how to make it usable for the people that need it. Friday, October 18, 13

Slide 6

Slide 6 text

Or to put it another way, we feed the animals Nom nom nom Friday, October 18, 13

Slide 7

Slide 7 text

Direct Reception Friday, October 18, 13

Slide 8

Slide 8 text

Partner with NOAA/NESDIS FCDAS Friday, October 18, 13

Slide 9

Slide 9 text

MODIS Truecolor - 2013.06.17 h p://feeder.gina.alaska.edu/modis-gina-alaska-naturalcolor-images/2013_06_17_21_32_jd168 Friday, October 18, 13

Slide 10

Slide 10 text

SUOMI NPP Day-night-band- 2013.01.18 h p://feeder.gina.alaska.edu/modis-gina-alaska-naturalcolor-images/2013_06_17_21_32_jd168 Friday, October 18, 13

Slide 11

Slide 11 text

Source data ~7.6 times the area of Washington state Ortho tile data ~4.3 times the area of Washington state Friday, October 18, 13

Slide 12

Slide 12 text

Friday, October 18, 13

Slide 13

Slide 13 text

Friday, October 18, 13

Slide 14

Slide 14 text

Satellite'Remote'Sensing' •  Statewide(Imagery( •  High(resolu4on( •  Historical(and(4me(series( Aerial'Remote'Sensing' •  LiDAR(–(eleva4on( •  Allows(hydro=modeling( •  Imagery(=(Historical( Water'Sensors' •  USGS(In=stream(Data(Loggers( In2situ'Sensors' •  MET(Sensors( Friday, October 18, 13

Slide 15

Slide 15 text

Transferring data Image(c) Paramount Pictures Friday, October 18, 13

Slide 16

Slide 16 text

waitp.perl •Based on waitd made by SeaSpace for Terascan •Simple, but... •Serial, no logging, no notifications Friday, October 18, 13

Slide 17

Slide 17 text

Conveyor http://www.ﬂickr.com/photos/captainmcdan/4182287774/ Friday, October 18, 13

Slide 18

Slide 18 text

The basics • Foreman • Blueprints • Workers • Tools • Packages • :shipit: Friday, October 18, 13

Slide 19

Slide 19 text

RUBY IS FAST I LOVE IT SO MUCH Friday, October 18, 13

Slide 20

Slide 20 text

Libraries • Celluloid - celluloid.io • Hamster - github.com/harukizaemon/ hamster • Listen - github.com/guard/listen • Thor - github.com/erikhuda/thor • StatsD - github.com/etsy/statsd/ Friday, October 18, 13

Slide 21

Slide 21 text

Version 0.1 •Native FS events (Polling as a fallback) •Parallel processing •Logging/Notifications •http://github.com/gina-alaska/conveyor Friday, October 18, 13

Slide 22

Slide 22 text

Version 2.0 • Concurrency • Metrics • More robust error handling / job tracking Friday, October 18, 13

Slide 23

Slide 23 text

Down the road • Object store support (S3 compatible) • Message queues to trigger other processing jobs (Redis/0MQ?) Friday, October 18, 13

Slide 24

Slide 24 text

Processing Data Friday, October 18, 13

Slide 25

Slide 25 text

Image Processing Friday, October 18, 13

Slide 26

Slide 26 text

Conﬁguration management Friday, October 18, 13

Slide 27

Slide 27 text

Use computers to do things Friday, October 18, 13

Slide 28

Slide 28 text

Use humans to drink with Friday, October 18, 13

Slide 29

Slide 29 text

Use computers to do things Friday, October 18, 13

Slide 30

Slide 30 text

Conﬁguration management • Chef - www.opscode.com/chef/ • Cookbooks available at github.com/gina-alaska Friday, October 18, 13

Slide 31

Slide 31 text

Consuming data Friday, October 18, 13

Slide 32

Slide 32 text

Puﬃn Feeder h p://feeder.gina.alaska.edu http://www.ﬂickr.com/photos/alessiomesiano/6227234098/ Friday, October 18, 13

Slide 33

Slide 33 text

Website: feeder.gina.alaska.edu Android app: GINA Puﬃn Feeder Friday, October 18, 13

Slide 34

Slide 34 text

Puﬃn Feeder - Origins Friday, October 18, 13

Slide 35

Slide 35 text

Puﬃn Feeder - Origins Friday, October 18, 13

Slide 36

Slide 36 text

Images • 12 Feeds o 5 Modis o 4 NPP o 2 Radar o 1 Webcam • ~3TB Archive • ~10GB/day Friday, October 18, 13

Slide 37

Slide 37 text

Movies Friday, October 18, 13

Slide 38

Slide 38 text

Friday, October 18, 13

Slide 39

Slide 39 text

Friday, October 18, 13

Slide 40

Slide 40 text

Puﬃn Feeder - In Use Friday, October 18, 13

Slide 41

Slide 41 text

Friday, October 18, 13

Slide 42

Slide 42 text

Friday, October 18, 13

Slide 43

Slide 43 text

gLynx - Catalog and data portals h p://alaska.portal.gina.alaska.edu http://www.ﬂickr.com/photos/luvi/3963857274/ Friday, October 18, 13

Slide 44

Slide 44 text

gLynx - Origins Friday, October 18, 13

Slide 45

Slide 45 text

Friday, October 18, 13

Slide 46

Slide 46 text

Yo dawg i herd you like portals, so we put a portal in your portal, so you can search all the portals Friday, October 18, 13

Slide 47

Slide 47 text

Friday, October 18, 13

Slide 48

Slide 48 text

What is gLynx? • A single database of data records • But with multiple front ends (portals) • Separate search indexes for each portal • Each portal can manage its own look, data records and users Friday, October 18, 13

Slide 49

Slide 49 text

Why? • Automatic sharing of data records • Shared collection of organizations and contact information • Giving the data owners control Friday, October 18, 13

Slide 50

Slide 50 text

Yo dawg i herd you like portals, so we put a portal in your portal, so you can search all the portals Friday, October 18, 13

Slide 51

Slide 51 text

Questions? https://twitter.com/AlexJamesFitz/status/388310922005336064/photo/1 • Will Fisher • www.gina.alaska.edu • github.com/gina-alaska • uafgina.tumblr.com • @uafgina Friday, October 18, 13