Upgrade to Pro — share decks privately, control downloads, hide ads and more …

GeoIQ Stack - Where 2012

GeoIQ Stack - Where 2012

Overview of the technology stack behind GeoIQ and how it enables open collaboration around data, visualization and analysis.

Andrew Turner

April 13, 2012
Tweet

Other Decks in Technology

Transcript

  1. Hurricanes Katrina and Rita Geospatial Analysis with Long Production Cycles

    flickr: gregorio in 2005 we were active in the response to Hurricanes Katrina and Rita. Back then the request came to perform a flood analysis. Analysts worked in DC for a few days and sent their results in powerpoint slides to the operations center. They then asked “where are our shelters and response centers?” Churn away a few days and send that back down. By then they were knee deep in water. How do you change the paradigm to give the tools to access data and ask questions into the hands of the people that need it. Go from question to answer in under 5 minutes.
  2. Crisis Reporting Realtime emergent information sources at the same time

    we saw the emergence of new forms of citizen communication and reporting. No voice and no data meant people sitting on rooftops would text family in Michigan to then call the Red Cross back in Louisiana. The typical organizations had little capability to quickly adapt to these new streams and incorporate it into their response.
  3. Our Mission: On-demand realtime interactive maps in the hands of

    everyone, exploration and collaboration Today this is dramatically different. We’ve spent the past 6 years building tools and capability for anyone to quickly and effectively access their data and devise actionable information. By leveraging the web and modern, emergent technology and an open-architecture it’s now possible to provide advanced tools to everyone - opening the conversation to easily enable collaboration when it’s needed most.
  4. Access Visualize Analyze we focus on three overall capabilities Data:

    networked, unbounded, standards-based, community driven, open data. Visualization: We need ways to get through this data. What do I have? What is the shape and idea of it? How is it changing? What can I do with it? Analysis: In the end, I want to make decisions. To do that, I need to ask questions. I need tools to help me easily ask questions and find answers without having to worry about the “how” of the question asking. Just let me explore.
  5. a few GeoIQ features... • Data upload • Data Parsing

    • Web Feeds • Dataset Aggregation • Geocoding • Geojoin • Data Permissions • Group Management • Data Cleanup • Metadata management • Tagging • Data export • Dataset caching • External Relational Database integration • User Authentication • LDAP Integration • Thematic Visualization • Spatial, Tabular, Statistical Analysis • User-defined analysis • Stream updating • Mobile web interface • Native mobile applications • Drawing and annotation • Geographic Maps • User defined basemaps • Charts (Pie, Line, etc.) • REST and Javascript API Integration • Facebook integration • OGC Web Services • PNG and PDF Map Export • Multisite Catalog Federation • White Label Customization • Featured Maps • External Analysis Integration • Stats and Tracking • Graceful Map Degradation “GIS” in all but name
  6. http://geocommons.com 15m+ maps served 100k+ unique datasets billions of features.

    4m+ downloads 1m+ geographic attributes Start now by joining our GeoCommons community of more than 25k GeoIQ users. You can readily find, visualize, and very soon analyze the public repository of 1m+ geographic attributes, billions of features.
  7. Streaming Services Streaming Analysis GeoiQ Social REST and Javascript API

    GeoiQ Explorer GeoiQ Mobile Tabular Spatial Image Custom Analysis Workers Spatial Data ETL Spatial Data Management User Management Distributed Object Store GeoiQ Connect MySQL PostgreSQL Oracle Mongo HBase Geospatial Search Index Hadoop Distributed Job Queues External Analysis HDFS Federated Data Network OpenStreetMap Map Rendering Engine Open-Source Stack
  8. Streaming Services Streaming Analysis GeoiQ Social REST and Javascript API

    GeoiQ Explorer GeoiQ Mobile Tabular Spatial Image Custom Analysis Workers Spatial Data ETL Spatial Data Management User Management Distributed Object Store GeoiQ Connect MySQL PostgreSQL Oracle Mongo HBase Geospatial Search Index Hadoop Distributed Job Queues External Analysis HDFS Federated Data Network OpenStreetMap Map Rendering Engine ModestMaps)) &)Polymaps) jQuery'&'' Backbone.js' Ac#onScript+ &+Appcelerator+ AMQP% &%Redis% WebSockets* Socket.io* Ruby%on% Rails% Lucene& &&Solr& Mapnik' Ruby%&% OSGeo% HBase& PostgreSQL+ Node.js(&( MongoDB( Open-Source Stack
  9. Open-Architecture Open-Standards •REST, GeoJSON •OpenSearch-Geo •GeoRSS, Atom, KML •OAuth, LDAP

    •OGC WFS, WMS, CSW •Usable Interface •Web Services •Download button flickr: dolinski
  10. We worked with the FCC to make this even easier.

    Enable Staffers and anyone to go from a Spreadsheet to a Map as quickly as possible. Then download or publish it for use in meeting, publications, and social networks.
  11. We worked with the FCC to make this even easier.

    Enable Staffers and anyone to go from a Spreadsheet to a Map as quickly as possible. Then download or publish it for use in meeting, publications, and social networks.
  12. We worked with the FCC to make this even easier.

    Enable Staffers and anyone to go from a Spreadsheet to a Map as quickly as possible. Then download or publish it for use in meeting, publications, and social networks.
  13. issuemap.org github.com/geoiq/issuemap We worked with the FCC to make this

    even easier. Enable Staffers and anyone to go from a Spreadsheet to a Map as quickly as possible. Then download or publish it for use in meeting, publications, and social networks.
  14. Open Visualization Modest maps Polymaps Full user choice for quickly

    growing number of basemaps and providers. Each vary by cartographic style, data quality, language.
  15. Mapnik OpenStreetMap TileStache developer.geoiq.com/tools/acetate/ We wanted to provide a basemap

    uniquely capable of visually attractive and understandable quantitative thematic visualizations. We also wanted to bring back a traditional concept of cartography by separating the layers to “Sandwich” it with the data. Together users can see the data and also the context of roads, places and other geographic data. Our UI makes this easy without having to think about it.
  16. Mapnik OpenStreetMap TileStache developer.geoiq.com/tools/acetate/ terrain areas roads points labels We

    wanted to provide a basemap uniquely capable of visually attractive and understandable quantitative thematic visualizations. We also wanted to bring back a traditional concept of cartography by separating the layers to “Sandwich” it with the data. Together users can see the data and also the context of roads, places and other geographic data. Our UI makes this easy without having to think about it.
  17. Spatial Operations Buffer, Intersect, Aggregation Correlation, Prediction, Custom Tools: geos,

    JTS, PostGIS We want to go beyond simple visualization and allow people to ask deep questions of their data. So we built a number of common tabular, statistical, and geospatial analysis modules that are easy to use with your data.
  18. Realtime Social Media Indicators http://geocommons.com/maps/51058 And we’re working on more

    than just traditional, static, analysis. We can analyze emergent and realtime streams of data.
  19. Deploy to the Edge flickr: oddboy The “cloud” offers amazing

    capability to publish and share across the web. But many users need to work in remote and disconnected locations. Because we use open-source stack and our own technology we can easily deploy to any online or offline systems. We’ve deployed servers to work offline in Jalalabad, Afghanistan as part of the #beer4data program that enables field collaboration of data sharing. The full offline stack can geocode, visualize and analyze data locally and optionally push the data up to the public web when
  20. Open-Source Deployment Workers Data Store Deployed as a Single VM

    App Servers Data Stores Clustered Cloud Deployment We use the same installers and virtual machine to on-demand deploy a new front-end server, or back-end analysis worker or data server.
  21. Publish(to(mobile( Make(a(map(in(GeoIQ( Layer(Control( Filter((by(Loca;on( Field(Annota;ons( Track(Field(Annota;ons(( Real(;me(upda;ng( SelfCService(Mobile(App(Crea;on(in(Minutes( And beyond

    scaling horizontally, we can push to the real edge of mobile devices. Users can create and publish a map in minutes that enables mobile access to maps and even realtime mobile collaboration.
  22. Distributed Operations Search and Rescue https://vimeo.com/30435757 Combined together, these tools

    allow for on-demand access to emergent information, realtime streams, analysis, and field reporting.
  23. Organizations Research Media Developers Enterprise Citizens By providing a common

    open platform for collaboration, members of many communities can contribute data and insight to reach common consensus.
  24. Development Aid http://maps.worldbank.org The World Bank is sharing the locations

    and impact of their projects throughout the world. The data is explorable and downloadable.
  25. IADB MapAmericas http://www.iadb.org/mapamericas The Inter-American Development Bank has launched a

    similar initiative linking project funding with on the ground outputs, photos, videos, and stories.
  26. Climate Science WWF Climascope The information that has traditionally been

    limited to expert only access is now in the hands of anyone to ask detailed important questions of quality scientific data. Citizens can ask about the impact of global phenomenon on their local environments.
  27. Expert & Citizen Sourced Science Data http://marine.tokelau.grida.no/ Organizations such as

    the United Nations Environmental Programme Grid-Arendal are gathering more scientific data and opening the tools for visualization and analysis to the world.
  28. Social Governance Beyond traditional science, government and organizations can ask

    relevant questions on urban planning and society by combining social media information. For example, investigating the crowding and ‘popularity’ of metro transit locations compared with location checkins.
  29. Join the Community http://geocommons.com GeoCommons is the public repository to

    explore this data - or find out how to use GeoIQ within your organization to get better insight and information.
  30. Collaborative Open Access to Geospatial Analysis Andrew Turner Chief Technology

    Officer & Neogeographer @ajturner [email protected] slideshare.net/ajturner github.com/geoiq github.com/geocommons