Tweet
Tweet

Slide 1

Slide 1 text

Open Geo Data in The Netherlands (and beyond)

Slide 2

Slide 2 text

Bio and outlook @simeonnedkov Past: MSc Geomatics at TU Delft Current: PhD research at Vrije Universiteit on geo tech use in the city/people/government triad - redefine the smart city Today - Dutch/European open geo data and stores - View and get data - Display (in fancy ways) and host your own data - No D3 and OpenGeo Suite

Slide 3

Slide 3 text

... everything you ever wanted to know about geographical information, standards and services?

Slide 4

Slide 4 text

1. Why it sucks. 2. When it will get better. 3. What's in it for me.

Slide 5

Slide 5 text

1. Out of sight, out of mind.. well, almost. 2. Gradually... 3. Easy access to pan-European data

Slide 6

Slide 6 text

It sucks. Verbose Unpolished and obscure docs SHP? EPSG: 28992?? GIS? WMS? WFS? Reference servers and implementations are mostly Java (and C++) Geoportals.. ugh! Lack of decent geo data sources and services Politics

Slide 7

Slide 7 text

It needs to get better And it is! Better mapping tech: OpenLayers 3, MapBox, CartoDB, Shapely, Fiona More tutorials by non-geo developers Favourable legislation: Wet Basisregistraties Publieke Dienstverlening op de Kaart (PDOK) Nationaal Georegister 2... INSPIRE

Slide 8

Slide 8 text

Why bother? Well established and powerful standards Extensive infrastructure Access to European open geo data Interoperability and ease of use Use it to make it better Encourage cities / open data initiatives to adopt standards You have no choice... :)

Slide 9

Slide 9 text

http://www.slideshare.net/jatorre/siglibre-english-17036935 Javier de la Torre (@jatorre) - Vizzuality, creators of CartoDB

Slide 10

Slide 10 text

1. Demystify Dutch geo data stores and standards 2. Demystify some geo tools

Slide 11

Slide 11 text

Let's dive in SDIs and geoportals

Slide 12

Slide 12 text

Client Data Interact Spatial Data Infrastructures?

Slide 13

Slide 13 text

Client Data Discover View Download Spatial Data Infrastructures?

Slide 14

Slide 14 text

Client Data Discover View Download Spatial Data Infrastructures? w e b s i t e c a t a l o g machine readable geo services

Slide 15

Slide 15 text

Client Data Discover View Download Spatial Data Infrastructures? w e b s i t e c a t a l o g Geoportal

Slide 16

Slide 16 text

Client Data Discover View Download Spatial Data Infrastructures? w e b s i t e c a t a l o g Geoportal SDI

Slide 17

Slide 17 text

Geo data in theory Client Metadata - location URL - format - extent - accuracy - temporal info - coordinate system - owner - access conditions Data store I Data store III Data store II Data store IV Catalog Discover data Location of data URL Query for data Download Data - georeferenced - vector (shp) - raster (geotiff) View

Slide 18

Slide 18 text

Geo data in theory Client (Machine) searchable catalog NLR Universities Schools Governmental bodies Provinces Cities Research institutes Aerial images? NLR has them at this URL Geo extent json, svg .png, jpg

Slide 19

Slide 19 text

Geo data in practice Client NLR Universities Schools Provinces Research institutes Data store V Provinciaal GeoRegister Governmental bodies Cities Most stores CKAN flat files

Slide 20

Slide 20 text

Geo data in practice Client NLR Universities Schools Provinces Research institutes Data store V Provinciaal GeoRegister Governmental bodies Cities Most stores CKAN flat files Politics and organisational issues

Slide 21

Slide 21 text

Geo data in practice Client NLR Universities Schools Provinces Research institutes Data store V Provinciaal GeoRegister Governmental bodies Cities Most stores CKAN flat files Politics and organisational issues The tech works!

Slide 22

Slide 22 text

Forget it!

Slide 23

Slide 23 text

But there is hope!

Slide 24

Slide 24 text

http://geo.admin.ch

Slide 25

Slide 25 text

nationaalgeoregister.nl Needs to grow... Back to reality pdok.nl national data sets robust up-to-date great use it! provinciaalgeoregister.nl provinces, no services KNMI / NLR

Slide 26

Slide 26 text

Interesting datasets BAG Basisregistraties Adressen en Gebouwen

Slide 27

Slide 27 text

Census data 2009, 2010, 2011

Slide 28

Slide 28 text

AHN Actueel Hoogtebestand Nederland - 25 m resolution

Slide 29

Slide 29 text

National cycling routes

Slide 30

Slide 30 text

More data Local announcements (live!) Topographic maps Rail/Roads/Canals/Parks CBS data Aerial and satellite images

Slide 31

Slide 31 text

PDOK.nl https://www.pdok.nl/en/products/pdok-services/overview-urls

Slide 32

Slide 32 text

Even more data INSPIRE "... an EU initiative to establish an infrastructure for spatial information in Europe ..." 1. Data from all EU member states 2. More Dutch data

Slide 33

Slide 33 text

Accessed and exchanged through geo services

Slide 34

Slide 34 text

The geo services

Slide 35

Slide 35 text

Vector vs. raster Space filling GeoTIFF Objects GeoJSON, SHP

Slide 36

Slide 36 text

Geo services View Web Map Service (WMS) Raster: images Web Map Tile Service (WMTS) Tiles Download Web Feature Service (WFS) Vector: geometry and attributes Web Coverage Service (WCS) Raw raster data

Slide 37

Slide 37 text

http://geo.serv.er/wms Server View vector and raster Download vector From data to services Tile Cache geometry sttyling http://geo.serv.er/wfs - (Spatial) filters - (Spatial) operations - Coordinate transformations vector raster values

Slide 38

Slide 38 text

AHN http://geodata.nationaalgeoregister.nl/ahn25m/wms Anatomy of WMS

Slide 39

Slide 39 text

http://geodata.nationaalgeoregister.nl/ahn25m/wms? request=GetCapabilities& service=WMS& version=1.1.3 Interacting with services Request: HTTP GET / POST Response: XML, .png, .jpg, .GML, .JSON, .svg

Slide 40

Slide 40 text

Client GetCapabilities Server Capabilities document (XML) Point of entry: GetCapabilities request

Slide 41

Slide 41 text

Client Server - Description of service - Layers - Requests - GetMap - GetFeatureInfo - GetLegendGraphic Client GetCapabilities Capabilities document (XML) Server Subsequent requests .png, .jpg and text Parse

Slide 42

Slide 42 text

http://geodata.nationaalgeoregister.nl/ahn25m/wms? request=getMap& service=WMS& version=1.1.1& layers=ahn25m& srs=EPSG:28992& bbox=13014,306243,286599,623492& width=300& height=400& format=image/png GetMap - request

Slide 43

Slide 43 text

GetMap - response http://geodata.nationaalgeoregister.nl/ahn25m/wms? request=getMap& service=WMS& version=1.1.1& layers=ahn25m& srs=EPSG:28992& bbox=13014,306243,286599,623492& width=300& height=400& format=image/png

Slide 44

Slide 44 text

GetFeatureInfo - request http://geodata.nationaalgeoregister.nl/ahn25m/wms? request=GetFeatureInfo& service=WMS& version=1.1.1& query_layers=ahn25m& info_format=text/html& x=353& y=145& layers=ahn25m& srs=EPSG:28892& bbox=197896, 429027, 205562, 433818& format=image/png& width=780& height=330&

Slide 45

Slide 45 text

GetFeatureInfo - response http://docs.geoserver.org/latest/en/user/services/wms/reference.html text/plain text/html

Slide 46

Slide 46 text

View and map

Slide 47

Slide 47 text

Viewing data: WMS PDOK kaart wizard kaart.pdok.nl QGIS - open source GIS OpenLayers 2 >> WMS made easy

Slide 48

Slide 48 text

QGIS add WMS layer QGIS add WMS layer Add WMS layer button

Slide 49

Slide 49 text

OpenLayers - WMS var map = new OpenLayers.Map('map', { maxExtent: new OpenLayers.Bounds(10000.0, 305000.0, 280000.0, 619000.0), projection: new OpenLayers.Projection("EPSG:28992") }); var wms_url = "http://geodata.nationaalgeoregister.nl/ahn25m/wms" var layer = new OpenLayers.Layer.WMS( "Actueel Hoogtebestand Nederland 25 m", wms_url, { layers: "ahn25m", format: "image/png", srs: "EPSG:28992", styles: "" }, { transitionEffect: 'resize' } ); map.addLayer(layer); map.setCenter(new OpenLayers.LonLat(445463, 156271), 1); http://bl.ocks.org/ndkv/

Slide 50

Slide 50 text

OpenLayers - Feature info var getFeatureControl = new OpenLayers.Control.WMSGetFeatureInfo({ url: proxy + wms_url, layers: [layer], queryVisible: true, eventListeners: { getfeatureinfo: function (event) { var pop = new OpenLayers.Popup.FramedCloud( "ahn25m", map.getLonLatFromPixel(event.xy), null, event.text, null, true) pop.panMapIfOutOfView = false; map.addPopup(pop); } } });

Slide 51

Slide 51 text

Waalweelde dashboard http://waalweelde.ndkv.nl

Slide 52

Slide 52 text

WMS - a note on tiling WMS: single images, no tiling, no caching WMS-C: tiling, caching - same endpoint as "pure" WMS - invoke with parameter: tiled=true - OSM and Leaflet do this automagically TMS / WMTS - used for the Basic Registry Topography service - TMS docs: http://wiki.osgeo.org/wiki/Tile_Map_Service_Specification - WMTS docs: ...

Slide 53

Slide 53 text

Anatomy of WFS Delivers geometry Geographical Markup Language GeoJSON Anatomy of WFS BAG http://geodata.nationaalgeoregister.nl/bagviewer/wfs?

Slide 54

Slide 54 text

Client Server - Description of service - Filters - Functions - Feature types i.e. layers - Requests - GetFeature [ filter ] - DescribeFeatureType - ... Client GetCapabilities Capabilities document (XML) Server Subsequent requests .gml, .json, .shp Parse

Slide 55

Slide 55 text

http://geodata.nationaalgeoregister.nl/bagviewer/wfs? request=getFeature& service=wfs& vesion=2.0.0& srs=EPSG:28992& bbox=70420.6,429242.6,70423.6,429243& typeName=bagviewer:pand& outputFormat=json GetFeature - request

Slide 56

Slide 56 text

{ "type": "Feature", "id": "pand.fid--2e3a32e5_13db0af9ceb_65e", "geometry": { "type": "Polygon", "coordinates": [ [ [ 70418.262, 429235.086 ], [ ... ] ] ] } "geometry_name": "geometrie", "properties": { "gid": 6571756, "identificatie": 530100002242340, "bouwjaar": 2002, "status": "Pand in gebruik", "gebruiksdoel": "woonfunctie", "oppervlakte_min": 165, "oppervlakte_max": 165, "aantal_verblijfsobjecten": 1 } } GetFeature - response

Slide 57

Slide 57 text

http://geodata.nationaalgeoregister.nl/bagviewer/wfs? request=getFeature& service=wfs& version=2.0.0 srs=EPSG:28992& bbox=70420.6, 429242.6,70423.6,429243& typeName=bagviewer:pand& outputFormat=json& srsname=EPSG:4326 Transform coordinates

Slide 58

Slide 58 text

{ "type": "Feature", "id": "pand.fid--2e3a32e5_13db0af9ceb_65e", "geometry": { "type": "Polygon", "coordinates": [ [ [ 4.159718956737263, 51.84527267018053 ], [ ... ] ] ] } "geometry_name": "geometrie", "properties": { "gid": 6571756, "identificatie": 530100002242340, "bouwjaar": 2002, "status": "Pand in gebruik", "gebruiksdoel": "woonfunctie", "oppervlakte_min": 165, "oppervlakte_max": 165, "aantal_verblijfsobjecten": 1 } } GetFeature in lon/lat

Slide 59

Slide 59 text

Get, transform, save QGIS reads WFS QGIS transforms coordinates QGIS saves to disk QGIS handles many file types

Slide 60

Slide 60 text

Add WFS layer Add WFS layer button

Slide 61

Slide 61 text

Save WFS layer as lat/lon in GeoJSON Right-click on layer -> Save As...

Slide 62

Slide 62 text

Images vs features WMS Data display Images only Predefined styling No analysis WFS Data download Access to geometries Not styled, up to you Analysis

Slide 63

Slide 63 text

WMS + WFS - Serve tiles with WMS - Get object id through GetFeatureInfo - Get object geometry through WFS GetFeature http://example.com/geoserver/wfs? service=wfs& version=2.0.0& request=GetFeature& typeName=namespace:featuretype& featureID=feature http://bagviewer.geodan.nl/index.html

Slide 64

Slide 64 text

Make me a map with TileMill (and CartoDB)

Slide 65

Slide 65 text

Make me a map 1. Extract data from WFS endpoint 2. Style with TileMill / CartoDB 3. Host on web through mapbox.com, own server and CartoDB

Slide 66

Slide 66 text

TileMill + MapProxy open-source cartography toolbox by MapBox open-source proxy for geo data - Style GIS data with CartoCSS (revolution! Google "SLD") - Bake .mbtiles - pre-rendered tile cache in SQLite - Host on mapbox.com - Cache, accelerate and transform data from existing data sources: WMS, .mbtiles and others - Disseminate through WMS, WMTS, TMS and KML - View with OpenLayers MapProxy or TileStache or Django TileMill

Slide 67

Slide 67 text

Data and tools CBS census data http://geodata.nationaalgeoregister.nl/wijkenbuurten2011/wfs - Municipalities >> inhabitants - Neighbourhoods >> inhabitants Road network http://geodata.nationaalgeoregister.nl/nwbwegen/wfs - Street geometries and names QGIS and gdal/ogr

Slide 68

Slide 68 text

Data and tools CBS census data http://geodata.nationaalgeoregister.nl/wijkenbuurten2011/wfs - Municipalities >> inhabitants - Neighbourhoods >> inhabitants Road network http://geodata.nationaalgeoregister.nl/nwbwegen/wfs - Street geometries and names QGIS and gdal/ogr Somehow QGIS saves WFS numerical fields as text... ... GDAL/OGR to the rescue!

Slide 69

Slide 69 text

gdal/ogr Swiss army knife for querying, transforming and manipulating GIS data shapefile, csv, gml, gpx, kml, geojson, sqlite, postgresql, mysql, svg, couchdb, wfs ogrinfo - investigate data source ogr2ogr - interact with data [-f format_name] - write output in specified format [-s_srs] - source coordinate system [-t_srs] - target coordinate system [-sql] - execute sql query

Slide 70

Slide 70 text

ogrinfo investigate WFS endpoint ogrinfo -so WFS:"url" investigate gemeenten2011 layer ogrinfo -so WFS:"url" gemeenten2011 url = http://geodata.nationaalgeoregister.nl/wijkenbuurten2011/wfs

Slide 71

Slide 71 text

ogr2ogr - get - ogr2ogr -f format_name destination source get gemeenten2011 layer in native SRS as GeoJSON ogr2ogr -f GeoJSON gemeenten2011.geojson WFS:"url" gemeenten2011 url = http://geodata.nationaalgeoregister.nl/wijkenbuurten2011/wfs

Slide 72

Slide 72 text

ogr2ogr - get + transform - ogr2ogr -f format_name destination source target_srs get gemeenten2011 layer in lat/lng as GeoJSON ogr2ogr -f GeoJSON gemeenten2011.geojson WFS:"url" -t_srs EPSG: 4326 gemeenten2011 url = http://geodata.nationaalgeoregister.nl/wijkenbuurten2011/wfs

Slide 73

Slide 73 text

ogr2ogr - sql ogr2ogr -f format_name destination source target_srs sql ogr2ogr -f GeoJSON gemeenten_2011_84.geojson WFS:" url" -sql "SELECT 'gemeentenaam', cast('aantal_inwoners' AS integer) AS 'aantal_inwoners' FROM 'wijkenbuurten2011:gemeenten2011'" -t_srs EPSG:4326 Load in TileMill

Slide 74

Slide 74 text

CartoDB "Web platform for creating dynamic maps [...]" - Style GIS data with CartoCSS (revolution! Google "SLD") - Store data in PostGIS (revolution! Gazillions of spatial functions) - SQL queries directly from JS (revolution! NoSQL is dead) - Real-time visualisations (revolution!) - Hosting at cartodb.com is expensive... - ... but it's on github

Slide 75

Slide 75 text

Prepare shp - ogr2ogr -f format_name destination source ogr2ogr -f "ESRI Shapefile" gemeenten2011.shp gemeenten2011.geojson - Zip gemeente2011.* and upload to CartoDB

Slide 76

Slide 76 text

Neat Thanks for your time! (Code) examples at @simeonnedkov http://ndkv.nl bl.ocks.org/ndkv waalweelde.ndkv.nl