Save 37% off PRO during our Black Friday Sale! »

Open geo data in The Netherlands (and beyond)

Open geo data in The Netherlands (and beyond)

A brief introduction to the Dutch open geo data scene.

Provides answers to:
- why it is perceived as being crap
- why it is not crap
- where's the data
- how to get it
- how to make a map using TileMill / CartoDB

Go to Google Drive for working links: https://docs.google.com/presentation/d/1lgoJ4R6CGEcBKrfzs2dn5eGswsjoSpbqgntU8TuywI0/edit#slide=id.p

5b0a23d8e4812322395495468807a9a1?s=128

Simeon Nedkov

April 02, 2013
Tweet

Transcript

  1. Open Geo Data in The Netherlands (and beyond)

  2. Bio and outlook @simeonnedkov Past: MSc Geomatics at TU Delft

    Current: PhD research at Vrije Universiteit on geo tech use in the city/people/government triad - redefine the smart city Today - Dutch/European open geo data and stores - View and get data - Display (in fancy ways) and host your own data - No D3 and OpenGeo Suite
  3. ... everything you ever wanted to know about geographical information,

    standards and services?
  4. 1. Why it sucks. 2. When it will get better.

    3. What's in it for me.
  5. 1. Out of sight, out of mind.. well, almost. 2.

    Gradually... 3. Easy access to pan-European data
  6. It sucks. Verbose Unpolished and obscure docs SHP? EPSG: 28992??

    GIS? WMS? WFS? Reference servers and implementations are mostly Java (and C++) Geoportals.. ugh! Lack of decent geo data sources and services Politics
  7. It needs to get better And it is! Better mapping

    tech: OpenLayers 3, MapBox, CartoDB, Shapely, Fiona More tutorials by non-geo developers Favourable legislation: Wet Basisregistraties Publieke Dienstverlening op de Kaart (PDOK) Nationaal Georegister 2... INSPIRE
  8. Why bother? Well established and powerful standards Extensive infrastructure Access

    to European open geo data Interoperability and ease of use Use it to make it better Encourage cities / open data initiatives to adopt standards You have no choice... :)
  9. http://www.slideshare.net/jatorre/siglibre-english-17036935 Javier de la Torre (@jatorre) - Vizzuality, creators of

    CartoDB
  10. 1. Demystify Dutch geo data stores and standards 2. Demystify

    some geo tools
  11. Let's dive in SDIs and geoportals

  12. Client Data Interact Spatial Data Infrastructures?

  13. Client Data Discover View Download Spatial Data Infrastructures?

  14. Client Data Discover View Download Spatial Data Infrastructures? w e

    b s i t e c a t a l o g machine readable geo services
  15. Client Data Discover View Download Spatial Data Infrastructures? w e

    b s i t e c a t a l o g Geoportal
  16. Client Data Discover View Download Spatial Data Infrastructures? w e

    b s i t e c a t a l o g Geoportal SDI
  17. Geo data in theory Client Metadata - location URL -

    format - extent - accuracy - temporal info - coordinate system - owner - access conditions Data store I Data store III Data store II Data store IV Catalog Discover data Location of data URL Query for data Download Data - georeferenced - vector (shp) - raster (geotiff) View
  18. Geo data in theory Client (Machine) searchable catalog NLR Universities

    Schools Governmental bodies Provinces Cities Research institutes Aerial images? NLR has them at this URL Geo extent json, svg .png, jpg
  19. Geo data in practice Client NLR Universities Schools Provinces Research

    institutes Data store V Provinciaal GeoRegister Governmental bodies Cities Most stores CKAN flat files
  20. Geo data in practice Client NLR Universities Schools Provinces Research

    institutes Data store V Provinciaal GeoRegister Governmental bodies Cities Most stores CKAN flat files Politics and organisational issues
  21. Geo data in practice Client NLR Universities Schools Provinces Research

    institutes Data store V Provinciaal GeoRegister Governmental bodies Cities Most stores CKAN flat files Politics and organisational issues The tech works!
  22. Forget it!

  23. But there is hope!

  24. http://geo.admin.ch

  25. nationaalgeoregister.nl Needs to grow... Back to reality pdok.nl national data

    sets robust up-to-date great use it! provinciaalgeoregister.nl provinces, no services KNMI / NLR
  26. Interesting datasets BAG Basisregistraties Adressen en Gebouwen

  27. Census data 2009, 2010, 2011

  28. AHN Actueel Hoogtebestand Nederland - 25 m resolution

  29. National cycling routes

  30. More data Local announcements (live!) Topographic maps Rail/Roads/Canals/Parks CBS data

    Aerial and satellite images
  31. PDOK.nl https://www.pdok.nl/en/products/pdok-services/overview-urls

  32. Even more data INSPIRE "... an EU initiative to establish

    an infrastructure for spatial information in Europe ..." 1. Data from all EU member states 2. More Dutch data
  33. Accessed and exchanged through geo services

  34. The geo services

  35. Vector vs. raster Space filling GeoTIFF Objects GeoJSON, SHP

  36. Geo services View Web Map Service (WMS) Raster: images Web

    Map Tile Service (WMTS) Tiles Download Web Feature Service (WFS) Vector: geometry and attributes Web Coverage Service (WCS) Raw raster data
  37. http://geo.serv.er/wms Server View vector and raster Download vector From data

    to services Tile Cache geometry sttyling http://geo.serv.er/wfs - (Spatial) filters - (Spatial) operations - Coordinate transformations vector raster values
  38. AHN http://geodata.nationaalgeoregister.nl/ahn25m/wms Anatomy of WMS

  39. http://geodata.nationaalgeoregister.nl/ahn25m/wms? request=GetCapabilities& service=WMS& version=1.1.3 Interacting with services Request: HTTP GET

    / POST Response: XML, .png, .jpg, .GML, .JSON, .svg
  40. Client GetCapabilities Server Capabilities document (XML) Point of entry: GetCapabilities

    request
  41. Client Server - Description of service - Layers - Requests

    - GetMap - GetFeatureInfo - GetLegendGraphic Client GetCapabilities Capabilities document (XML) Server Subsequent requests .png, .jpg and text Parse
  42. http://geodata.nationaalgeoregister.nl/ahn25m/wms? request=getMap& service=WMS& version=1.1.1& layers=ahn25m& srs=EPSG:28992& bbox=13014,306243,286599,623492& width=300& height=400& format=image/png

    GetMap - request
  43. GetMap - response http://geodata.nationaalgeoregister.nl/ahn25m/wms? request=getMap& service=WMS& version=1.1.1& layers=ahn25m& srs=EPSG:28992& bbox=13014,306243,286599,623492&

    width=300& height=400& format=image/png
  44. GetFeatureInfo - request http://geodata.nationaalgeoregister.nl/ahn25m/wms? request=GetFeatureInfo& service=WMS& version=1.1.1& query_layers=ahn25m& info_format=text/html& x=353&

    y=145& layers=ahn25m& srs=EPSG:28892& bbox=197896, 429027, 205562, 433818& format=image/png& width=780& height=330&
  45. GetFeatureInfo - response http://docs.geoserver.org/latest/en/user/services/wms/reference.html text/plain text/html

  46. View and map

  47. Viewing data: WMS PDOK kaart wizard kaart.pdok.nl QGIS - open

    source GIS OpenLayers 2 >> WMS made easy
  48. QGIS add WMS layer QGIS add WMS layer Add WMS

    layer button
  49. OpenLayers - WMS var map = new OpenLayers.Map('map', { maxExtent:

    new OpenLayers.Bounds(10000.0, 305000.0, 280000.0, 619000.0), projection: new OpenLayers.Projection("EPSG:28992") }); var wms_url = "http://geodata.nationaalgeoregister.nl/ahn25m/wms" var layer = new OpenLayers.Layer.WMS( "Actueel Hoogtebestand Nederland 25 m", wms_url, { layers: "ahn25m", format: "image/png", srs: "EPSG:28992", styles: "" }, { transitionEffect: 'resize' } ); map.addLayer(layer); map.setCenter(new OpenLayers.LonLat(445463, 156271), 1); http://bl.ocks.org/ndkv/
  50. OpenLayers - Feature info var getFeatureControl = new OpenLayers.Control.WMSGetFeatureInfo({ url:

    proxy + wms_url, layers: [layer], queryVisible: true, eventListeners: { getfeatureinfo: function (event) { var pop = new OpenLayers.Popup.FramedCloud( "ahn25m", map.getLonLatFromPixel(event.xy), null, event.text, null, true) pop.panMapIfOutOfView = false; map.addPopup(pop); } } });
  51. Waalweelde dashboard http://waalweelde.ndkv.nl

  52. WMS - a note on tiling WMS: single images, no

    tiling, no caching WMS-C: tiling, caching - same endpoint as "pure" WMS - invoke with parameter: tiled=true - OSM and Leaflet do this automagically TMS / WMTS - used for the Basic Registry Topography service - TMS docs: http://wiki.osgeo.org/wiki/Tile_Map_Service_Specification - WMTS docs: ...
  53. Anatomy of WFS Delivers geometry Geographical Markup Language GeoJSON Anatomy

    of WFS BAG http://geodata.nationaalgeoregister.nl/bagviewer/wfs?
  54. Client Server - Description of service - Filters - Functions

    - Feature types i.e. layers - Requests - GetFeature [ filter ] - DescribeFeatureType - ... Client GetCapabilities Capabilities document (XML) Server Subsequent requests .gml, .json, .shp Parse
  55. http://geodata.nationaalgeoregister.nl/bagviewer/wfs? request=getFeature& service=wfs& vesion=2.0.0& srs=EPSG:28992& bbox=70420.6,429242.6,70423.6,429243& typeName=bagviewer:pand& outputFormat=json GetFeature -

    request
  56. { "type": "Feature", "id": "pand.fid--2e3a32e5_13db0af9ceb_65e", "geometry": { "type": "Polygon", "coordinates":

    [ [ [ 70418.262, 429235.086 ], [ ... ] ] ] } "geometry_name": "geometrie", "properties": { "gid": 6571756, "identificatie": 530100002242340, "bouwjaar": 2002, "status": "Pand in gebruik", "gebruiksdoel": "woonfunctie", "oppervlakte_min": 165, "oppervlakte_max": 165, "aantal_verblijfsobjecten": 1 } } GetFeature - response
  57. http://geodata.nationaalgeoregister.nl/bagviewer/wfs? request=getFeature& service=wfs& version=2.0.0 srs=EPSG:28992& bbox=70420.6, 429242.6,70423.6,429243& typeName=bagviewer:pand& outputFormat=json& srsname=EPSG:4326

    Transform coordinates
  58. { "type": "Feature", "id": "pand.fid--2e3a32e5_13db0af9ceb_65e", "geometry": { "type": "Polygon", "coordinates":

    [ [ [ 4.159718956737263, 51.84527267018053 ], [ ... ] ] ] } "geometry_name": "geometrie", "properties": { "gid": 6571756, "identificatie": 530100002242340, "bouwjaar": 2002, "status": "Pand in gebruik", "gebruiksdoel": "woonfunctie", "oppervlakte_min": 165, "oppervlakte_max": 165, "aantal_verblijfsobjecten": 1 } } GetFeature in lon/lat
  59. Get, transform, save QGIS reads WFS QGIS transforms coordinates QGIS

    saves to disk QGIS handles many file types
  60. Add WFS layer Add WFS layer button

  61. Save WFS layer as lat/lon in GeoJSON Right-click on layer

    -> Save As...
  62. Images vs features WMS Data display Images only Predefined styling

    No analysis WFS Data download Access to geometries Not styled, up to you Analysis
  63. WMS + WFS - Serve tiles with WMS - Get

    object id through GetFeatureInfo - Get object geometry through WFS GetFeature http://example.com/geoserver/wfs? service=wfs& version=2.0.0& request=GetFeature& typeName=namespace:featuretype& featureID=feature http://bagviewer.geodan.nl/index.html
  64. Make me a map with TileMill (and CartoDB)

  65. Make me a map 1. Extract data from WFS endpoint

    2. Style with TileMill / CartoDB 3. Host on web through mapbox.com, own server and CartoDB
  66. TileMill + MapProxy open-source cartography toolbox by MapBox open-source proxy

    for geo data - Style GIS data with CartoCSS (revolution! Google "SLD") - Bake .mbtiles - pre-rendered tile cache in SQLite - Host on mapbox.com - Cache, accelerate and transform data from existing data sources: WMS, .mbtiles and others - Disseminate through WMS, WMTS, TMS and KML - View with OpenLayers MapProxy or TileStache or Django TileMill
  67. Data and tools CBS census data http://geodata.nationaalgeoregister.nl/wijkenbuurten2011/wfs - Municipalities >>

    inhabitants - Neighbourhoods >> inhabitants Road network http://geodata.nationaalgeoregister.nl/nwbwegen/wfs - Street geometries and names QGIS and gdal/ogr
  68. Data and tools CBS census data http://geodata.nationaalgeoregister.nl/wijkenbuurten2011/wfs - Municipalities >>

    inhabitants - Neighbourhoods >> inhabitants Road network http://geodata.nationaalgeoregister.nl/nwbwegen/wfs - Street geometries and names QGIS and gdal/ogr Somehow QGIS saves WFS numerical fields as text... ... GDAL/OGR to the rescue!
  69. gdal/ogr Swiss army knife for querying, transforming and manipulating GIS

    data shapefile, csv, gml, gpx, kml, geojson, sqlite, postgresql, mysql, svg, couchdb, wfs ogrinfo - investigate data source ogr2ogr - interact with data [-f format_name] - write output in specified format [-s_srs] - source coordinate system [-t_srs] - target coordinate system [-sql] - execute sql query
  70. ogrinfo investigate WFS endpoint ogrinfo -so WFS:"url" investigate gemeenten2011 layer

    ogrinfo -so WFS:"url" gemeenten2011 url = http://geodata.nationaalgeoregister.nl/wijkenbuurten2011/wfs
  71. ogr2ogr - get - ogr2ogr -f format_name destination source get

    gemeenten2011 layer in native SRS as GeoJSON ogr2ogr -f GeoJSON gemeenten2011.geojson WFS:"url" gemeenten2011 url = http://geodata.nationaalgeoregister.nl/wijkenbuurten2011/wfs
  72. ogr2ogr - get + transform - ogr2ogr -f format_name destination

    source target_srs get gemeenten2011 layer in lat/lng as GeoJSON ogr2ogr -f GeoJSON gemeenten2011.geojson WFS:"url" -t_srs EPSG: 4326 gemeenten2011 url = http://geodata.nationaalgeoregister.nl/wijkenbuurten2011/wfs
  73. ogr2ogr - sql ogr2ogr -f format_name destination source target_srs sql

    ogr2ogr -f GeoJSON gemeenten_2011_84.geojson WFS:" url" -sql "SELECT 'gemeentenaam', cast('aantal_inwoners' AS integer) AS 'aantal_inwoners' FROM 'wijkenbuurten2011:gemeenten2011'" -t_srs EPSG:4326 Load in TileMill
  74. CartoDB "Web platform for creating dynamic maps [...]" - Style

    GIS data with CartoCSS (revolution! Google "SLD") - Store data in PostGIS (revolution! Gazillions of spatial functions) - SQL queries directly from JS (revolution! NoSQL is dead) - Real-time visualisations (revolution!) - Hosting at cartodb.com is expensive... - ... but it's on github
  75. Prepare shp - ogr2ogr -f format_name destination source ogr2ogr -f

    "ESRI Shapefile" gemeenten2011.shp gemeenten2011.geojson - Zip gemeente2011.* and upload to CartoDB
  76. Neat Thanks for your time! (Code) examples at @simeonnedkov http://ndkv.nl

    bl.ocks.org/ndkv waalweelde.ndkv.nl