Upgrade to Pro — share decks privately, control downloads, hide ads and more …

OpenStreetMap for GIS Pros - 2014

OpenStreetMap for GIS Pros - 2014

Workshop given in UCC, 29th Jan 2014.


Richard Cantwell

January 29, 2014


  1. An Introduction to OpenStreetMap for GIS Professionals Richard Cantwell @ManAboutCouch

    richard.cantwell@gamma.ie UCC 29th January 2014
  2. Agenda: Introduction to OpenStreetMap: The origin of OSM Project Structure

    / Management Data Licensing How OSM represents data in GIS Terms: Nodes / Ways / Relations Key / Value pairs TagInfo Contributing: GIS Based Editors Consuming: Tiles, Files and API’s: Tileservers – your own and/or others Files – Downloads, Databases & QGIS API’s – OverPass Turbo TileMill Spatialite Wider Issues: Comparing OSM and Commercial Data What now for Commercial Mapping? The future of Tiles Close: Contacting OSM 10 'Takeaways' Q & A
  3. Introduction to OpenStreetMap: OSM in 2 minutes: The origin of

    OSM How OSM is organised Data Licensing
  4. None
  5. None
  6. Rapid and continuing growth wiki.openstreetmap.org/wiki/Stats

  7. Project Structure • “Rather Light” • “Do-ocracy” / “Meritocracy” •

    Somebody has to run the servers etc.: OSM Foundation (OSMF) • Contributors are a broad church
  8. www.openstreetmap.org/copyright

  9. opendatacommons.org/licenses/odbl/summary/

  10. How OSM represents data in GIS Terms: Nodes / Ways

    / Relations Key / Value pairs TagInfo
  11. Geographic Information is represented as: Nodes Point Ways Line Closed

    Line Area (Polygon) eg. Roundabouts Relations Multi (Point / Line / Polygon) eg. Bus Routes / Inner Rings
  12. Tagging: Key / Value Pairs

  13. Tagging: Key / Value Pairs

  14. Tagging: Oversight, not moderation

  15. Tagging: Oversight, not moderation

  16. Tagging: Oversight, not moderation

  17. Tagging: Oversight, not moderation

  18. Contributing: Wide Range of Editors: • In Browser • In

    Desktop GIS • Mobile • Paper (!)
  19. Contributing to OpenStreetMap Rule 1: Respect Copyright

  20. wiki.openstreetmap.org/wiki/Beginners%27_guide Contributing to OpenStreetMap

  21. ID Editor

  22. http://github.com/Esri/arcgis-osm-editor ArcGIS Editor

  23. QGIS OpenStreetMap Plugin – Currently Broken, replacement due in ‘months’

  24. fieldpapers.org GPS Mapping – on Paper

  25. fieldpapers.org GPS Mapping – on Paper

  26. Consuming: Tiles, Files and API’s: The OSM ‘Stack’ Tiles –

    your own and/or others Files – Downloads, Databases, QGIS API’s – OverPass Turbo TileMill – quick overview Spatialite – SQL without the overhead
  27. Edit Store Render Consume Store Analyse

  28. Tile Data: .../{z}/{x}/{y}.png mike.teczno.com

  29. Tile Data Pyramid tile.openstreetmap.org/0/0/0.png tile.openstreetmap.org/10/494/331.png tile.openstreetmap.org/18/126521/84975.png

  30. OpenStreetMap – Tile Usage wiki.openstreetmap.org/wiki/Tile_Disk_Usage OSM does NOT pre-render every

    tile. Pre-rendering all tiles would use around 54,000 GB of storage. The majority of tiles are never viewed. In fact just 1.79% have ever been viewed. This is because the majority of tiles are at zoom level 18 and the vast majority of those contain nothing of interest (the sea, for example) By following an on-the-fly rendering approach we can avoid rendering these tiles unnecessarily. The tile view count column shows how many tiles have been produced on the OSM Tile server. Tile server disk usage: 1,272 GB used. (6 Jan 2012). (z0 to z15: 252 GB, z16 to z18: 1,020 GB)
  31. wiki.openstreetmap.org/wiki/Tile_usage_policy OSM Tile Usage Policy

  32. QGIS – OpenLayers plugin

  33. maps.stamen.com/watercolor

  34. Using OSM data in GIS

  35. www.thunderforest.com

  36. MapInfo - .xml and .TAB files

  37. maps.cloudmade.com/editor

  38. Why make your own? Tile Usage Policy You’re in control:

    Design and Content Need tiles in EPSG:27700? 29903? Having OSM Data in a Database enables a lot of things You will learn a lot
  39. www.switch2osm.org

  40. www.switch2osm.org

  41. www.switch2osm.org Tileserver Software Stack • Software Stack: • mod_tile: apache

    module, serves and expires tiles • renderd: priority queuing system for rendering requests • mapnik: does the actual rendering • postgis: Spatially enabled PostgreSQL database • osm2pgsql: loads osm data into the postgis database
  42. Steps for setting up your own Tileserver • Download and

    install libraries (apt-get) • Install PostgreSQL / Postgis • Install osm2pgsql and Mapnik (python) • Install mod_tile and renderd • Install & configure mapnik style-sheet • Configure renderd and mod_tile • Tune PostgreSQL • Download Planet and Coastline data • Import into Postgis • Start tileserver www.switch2osm.org
  43. Phew! Is there an easier way? • Postgis and Tilemill

    • maperitive.net – Reads .osm / .pbf, exports bitmaps & SVG • tiledrawer.com – Amazon EC2 based rendering / hosting
  44. www.mapbox.com

  45. www.mapbox.com/tilemill

  46. www.mapbox.com/tilemill

  47. www.mapbox.com/tilemill

  48. www.mapbox.com/blog/interactive-woodcut-map

  49. www.mapbox.com/blog/vector-tiles

  50. www.mapbox.com/blog/vector-tiles

  51. www.mapbox.com/blog/vector-tiles

  52. www.mapbox.com/blog/vector-tiles

  53. Entire Planet downloadable (~500GB uncompressed) planet.openstreetmap.org

  54. Processed data downloads & services www.geofabrik.de

  55. download.geofabrik.de Including pre-configured Shapefiles (reduced featureset)

  56. None
  57. None
  58. www.postgis.org

  59. osm2pgsql

  60. osm2pgsql Approx 70 Minutes for a small extract (Ireland .pbf

    = 69MB, Planet is 23GB) Would be faster on a machine with more RAM and SSD disks Whole Planet can take days osm2pgsql -s -U username -d gis -C 100 /path/to/downloaded/osm/data -s = Slim mode -U = postgres username (owner of gis database) -d = database name (usually gis) -C = amount of memory in MB, usually 800, but my PC is a bit old
  61. None
  62. None
  63. None
  64. None
  65. Using the Data API “The Overpass API (or OSM3S) is

    a read-only API that serves up custom selected parts of the OSM map data. It acts as a database over the web: the client sends a query to the API and gets back the data set that corresponds to the query.” wiki.openstreetmap.org/wiki/Overpass_API overpass-turbo.eu
  66. None
  67. None
  68. None
  69. None
  70. None
  71. None
  72. Extract Raw Data – Embedded in a map <div> on

    a web page map.addLayers([ make_layer("http://overpass-api.de/api/interpreter?data= node[amenity=pub](bbox);out+skel; (way[amenity=pub](bbox);node(w););out+skel;", “Blue") ]);
  73. harrywood.co.uk/maps/uixapi/xapi.html

  74. osmosis – java based command line OSM data processing tool

    Some examples of the things it can currently do are: • Generate planet dumps from a database • Load planet dumps into a database • Produce change sets using database history tables • Apply change sets to a local database • Compare two planet dump files and produce a change set • Re-sort the data contained in planet dump files • Extract data inside a bounding box or polygon wiki.openstreetmap.org/wiki/Osmosis
  75. osmosis – java based command line OSM data processing tool

  76. None
  77. Spatialite – SQLite Spatial www.gaia-gis.it/gaia-sins/

  78. Spatialite – SQLite Spatial www.gaia-gis.it/fossil/spatialite_tools/index

  79. Spatialite – SQLite Spatial www.gaia-gis.it/fossil/spatialite_tools/index spatialite_osm_map – converts .osm to

    a sqlite format usable in a GIS spatialite_osm_raw – converts .osm to a sqlite format usable for stats spatialite_osm_filter – generates an OSM.XML file masked by an input polygon spatialite_osm_net – generates ‘routable’ data from roads
  80. spatialite_osm_map / raw – importing .pbf file www.gaia-gis.it/fossil/spatialite_tools/index spatialite_osm_map –o

    input.pbf –d output.sqlite Took about 30 mins (compared to 70 for osm2pgsql) Resulting .sqlite file was 300MB for the map, 1.5GB for raw
  81. Spatialite – SQLite Spatial www.gaia-gis.it/fossil/spatialite_gui/index

  82. spatialite_gui www.gaia-gis.it/fossil/spatialite_gui/index

  83. spatialite_gui www.gaia-gis.it/fossil/spatialite_gui/index

  84. Spatialite in QGIS – use ‘Add Vector Layer’

  85. Left as an exercise: Other services that you can run

    / setup: Nominatim – OSM’s Geocoder. pgRouting – PostgreSQL based routing OSRM – Open Source Routing Machine, closely tied to OSM Using Osmosis to keep your tileserver up to date pgRouting.org project-osrm.org
  86. Wider Issues: Comparing OSM and Commercial Data What now for

    Commercial Mapping? Privacy & Data Protection The future of tiles
  87. Map contains data Copyright NavTeq Map contains data Copyright NavTeq

  88. Map contains data Copyright NavTeq Map contains data Copyright NavTeq

  89. Map contains data Copyright NavTeq Map contains data Copyright NavTeq

  90. But! © Stan

  91. None
  92. 4,500 Done, 56,500 to go..

  93. Vs creativecommons.org

  94. VGI: OpenStreetMap • Limited to what is ‘visible’ • API’s

    limited, aimed at data editors • ‘Viral’ nature of data licence • Focus on map data • ‘Warm Geography’ • Aims to be of Public Interest • Instant data updates, Sporadic • ‘Depth’ Issues (coverage, completeness) • Weak Metadata / Standards Commercial GeoData Providers • Wide source of inputs • API’s extensive, aimed at data consumers • Derived data issues • Focus on services • ‘Cold Geography’ • Aims to be Authoritive • Moderated update process, Consistent • ‘Width’ issues (limited featureset) • Expensive
  95. The future of commercial online mapping? Logorama: vimeo.com/10149605

  96. www.citymaps.com The future of commercial online mapping?

  97. maps.google.ie The future of commercial online mapping?

  98. www.jasondavies.com/maps/raster/ Reprojected Raster Tiles

  99. www.mapbox.com/blog/vector-tiles wiki.openstreetmap.org/wiki/Vector_tiles Vector Tiles /{z}/{x}/{y}.png .geojson

  100. Close: Contacting OSM 10 'Takeaways' Q & A

  101. OSM Wiki: wiki.openstreetmap.org OSM Help Forums: help.openstreetmap.org OSM-F Blog: blog.osmfoundation.org

    OpenGeoData Blog (OSM News): opengeodata.org IRC (Internet Relay Chat): irc://irc.oftc.net #osm Mailing Lists: lists.openstreetmap.org/listinfo Contacting OSM
  102. 10 'Takeaways' License Check obligations if mixing data Data Model

    Area / Multipolygon handling is complex Usage Policies Desktop prob not an issue. Server, web different Build Your Own No apt-get install tileserver Yet Tilemill Great for designers, self hosting options exist Shapefiles Easy, not complete Postgis Starting point for many possibilities Data APIs Great for snapshots in geojson or POI webpages Spatialite Spatial Database without the Database Data vs Services Is OSM commoditising data? Is that bad?
  103. This slide deck is available at: speakerdeck.com/manaboutcouch Thank you. Any

    Questions? Richard Cantwell @ManAboutCouch richard.cantwell@gamma.ie