$30 off During Our Annual Pro Sale. View Details »

OpenStreetMap for GIS Pros

OpenStreetMap for GIS Pros

At the 2013 State of the Map conference I gave a one hour workshop aimed at introducing OpenStreetMap's inner workings to those of us who work with GIS.

Richard Cantwell

September 06, 2013
Tweet

More Decks by Richard Cantwell

Other Decks in Technology

Transcript

  1. An Introduction to
    OpenStreetMap for
    GIS Professionals
    Richard Cantwell
    @ManAboutCouch [email protected]

    View Slide

  2. Agenda:
    Introduction to OpenStreetMap:
    The origin of OSM
    Project Structure / Management
    Data Licensing
    How OSM represents data in GIS Terms:
    Nodes / Ways / Relations
    Key / Value pairs
    TagInfo
    Contributing:
    GIS Based Editors
    Consuming: Tiles, Files and API’s:
    Tileservers – your own and/or others
    Files – Downloads, Databases & QGIS
    API’s – OverPass Turbo
    TileMill
    Spatialite
    Wider Issues:
    Comparing OSM and Commercial Data
    What now for Commercial Mapping?
    Privacy & Data Protection
    The future of Tiles
    Close:
    Contacting OSM
    Takeaways from this session
    Q & A

    View Slide

  3. Introduction to OpenStreetMap:
    OSM in 2 minutes:
    The origin of OSM
    How OSM is organised
    Data Licensing

    View Slide

  4. View Slide

  5. View Slide

  6. Rapid and continuing growth
    wiki.openstreetmap.org/wiki/Stats

    View Slide

  7. Project Structure
    • “Rather Light”
    • “Do-ocracy” / “Meritocracy”
    • Somebody has to run the servers etc.:
    OSM Foundation (OSMF)
    • Contributors are a broad church

    View Slide

  8. www.openstreetmap.org/copyright

    View Slide

  9. opendatacommons.org/licenses/odbl/summary/

    View Slide

  10. How OSM represents data in GIS Terms:
    Nodes / Ways / Relations
    Key / Value pairs
    TagInfo

    View Slide

  11. Geographic Information is represented as:
    Nodes
    Point
    Ways
    Line Closed Line Area (Polygon)
    eg. Roundabouts
    Relations
    Multi (Point / Line / Polygon)
    eg. Bus Routes / Inner Rings

    View Slide

  12. Tagging: Key / Value Pairs

    View Slide

  13. Tagging: Key / Value Pairs

    View Slide

  14. Tagging: Oversight, not moderation

    View Slide

  15. Tagging: Oversight, not moderation

    View Slide

  16. Tagging: Oversight, not moderation

    View Slide

  17. Tagging: Oversight, not moderation

    View Slide

  18. Contributing:
    Wide Range of Editors

    View Slide

  19. Contributing to OpenStreetMap
    Rule 1: Respect Copyright

    View Slide

  20. http://github.com/Esri/arcgis-osm-editor
    ArcGIS Editor

    View Slide

  21. QGIS OpenStreetMap Plugin – Currently Broken, replacement due in ‘months’

    View Slide

  22. Consuming: Tiles, Files and API’s:
    The OSM ‘Stack’
    Tiles – your own and/or others
    Files – Downloads, Databases, QGIS
    API’s – OverPass Turbo
    TileMill – quick overview
    Spatialite – SQL without the overhead

    View Slide

  23. Edit Store Render Consume
    Store Analyse

    View Slide

  24. Tile Data: .../{z}/{x}/{y}.png
    mike.teczno.com

    View Slide

  25. Tile Data Pyramid
    tile.openstreetmap.org/0/0/0.png
    tile.openstreetmap.org/10/494/331.png
    tile.openstreetmap.org/18/126521/84975.png

    View Slide

  26. OpenStreetMap – Tile Usage
    wiki.openstreetmap.org/wiki/Tile_Disk_Usage
    OSM does NOT pre-render every tile.
    Pre-rendering all tiles would use around 54,000 GB of
    storage.
    The majority of tiles are never viewed. In fact just
    1.79% have ever been viewed.
    This is because the majority of tiles are at zoom
    level 18 and the vast majority of those contain nothing
    of interest (the sea, for example)
    By following an on-the-fly rendering approach we can
    avoid rendering these tiles unnecessarily.
    The tile view count column shows how many tiles have
    been produced on the OSM Tile server.
    Tile server disk usage: 1,272 GB used. (6 Jan 2012).
    (z0 to z15: 252 GB, z16 to z18: 1,020 GB)

    View Slide

  27. wiki.openstreetmap.org/wiki/Tile_usage_policy
    OSM Tile Usage Policy

    View Slide

  28. QGIS – OpenLayers plugin

    View Slide

  29. maps.stamen.com/watercolor

    View Slide

  30. Using OSM data in GIS

    View Slide

  31. www.thunderforest.com

    View Slide

  32. MapInfo - .xml and .TAB files

    View Slide

  33. maps.cloudmade.com/editor

    View Slide

  34. Why make your own?
    Tile Usage Policy
    You’re in control: Design and Content
    Need tiles in EPSG:27700? 29903?
    Having OSM Data in a Database enables a lot of things
    You will learn a lot

    View Slide

  35. www.switch2osm.org

    View Slide

  36. www.switch2osm.org

    View Slide

  37. www.switch2osm.org
    Tileserver Software Stack
    • Software Stack:
    • mod_tile: apache module, serves and expires tiles
    • renderd: priority queuing system for rendering
    requests
    • mapnik: does the actual rendering
    • postgis: Spatially enabled PostgreSQL database
    • osm2pgsql: loads osm data into the postgis
    database

    View Slide

  38. Steps for setting up your own Tileserver
    • Download and install libraries (apt-get)
    • Install PostgreSQL / Postgis
    • Install osm2pgsql and Mapnik (python)
    • Install mod_tile and renderd
    • Install & configure mapnik style-sheet
    • Configure renderd and mod_tile
    • Tune PostgreSQL
    • Download Planet and Coastline data
    • Import into Postgis
    • Start tileserver
    www.switch2osm.org

    View Slide

  39. Phew!
    Is there an easier way?
    • Postgis and Tilemill
    • maperitive.net – Reads .osm / .pbf, exports bitmaps & SVG
    • tiledrawer.com – Amazon EC2 based rendering / hosting

    View Slide

  40. www.mapbox.com

    View Slide

  41. www.mapbox.com/tilemill

    View Slide

  42. www.mapbox.com/tilemill

    View Slide

  43. www.mapbox.com/tilemill

    View Slide

  44. Entire Planet downloadable (~500GB uncompressed)
    planet.openstreetmap.org

    View Slide

  45. Processed data downloads & services
    www.geofabrik.de

    View Slide

  46. download.geofabrik.de
    Including pre-configured Shapefiles (reduced featureset)

    View Slide

  47. View Slide

  48. View Slide

  49. www.postgis.org

    View Slide

  50. osm2pgsql

    View Slide

  51. osm2pgsql
    Approx 70 Minutes for a small extract (Ireland .pbf = 62MB, Planet is 21GB)
    Would be faster on a machine with more RAM and SSD disks
    Whole Planet can take days
    osm2pgsql -s -U username -d gis -C 100
    /path/to/downloaded/osm/data
    -s = Slim mode
    -U = postgres username (owner of gis database)
    -d = database name (usually gis)
    -C = amount of memory in MB, usually 800, but my PC is a bit old

    View Slide

  52. View Slide

  53. View Slide

  54. View Slide

  55. View Slide

  56. Using the Data API
    “The Overpass API (or OSM3S) is a read-only API that serves up custom
    selected parts of the OSM map data. It acts as a database over the web: the
    client sends a query to the API and gets back the data set that corresponds to
    the query.”
    wiki.openstreetmap.org/wiki/Overpass_API overpass-turbo.eu

    View Slide

  57. View Slide

  58. View Slide

  59. View Slide

  60. View Slide

  61. View Slide

  62. View Slide

  63. Extract Raw Data – Embedded in a map on a web page
    map.addLayers([
    make_layer("http://overpass-api.de/api/interpreter?data=
    node[amenity=pub](bbox);out+skel;
    (way[amenity=pub](bbox);node(w););out+skel;", “Blue")
    ]);

    View Slide

  64. harrywood.co.uk/maps/uixapi/xapi.html

    View Slide

  65. osmosis – java based command line OSM data processing tool
    Some examples of the things it can currently do are:
    • Generate planet dumps from a database
    • Load planet dumps into a database
    • Produce change sets using database history tables
    • Apply change sets to a local database
    • Compare two planet dump files and produce a change set
    • Re-sort the data contained in planet dump files
    • Extract data inside a bounding box or polygon
    wiki.openstreetmap.org/wiki/Osmosis

    View Slide

  66. osmosis – java based command line OSM data processing tool

    View Slide

  67. View Slide

  68. Spatialite – SQLite Spatial
    www.gaia-gis.it/gaia-sins/

    View Slide

  69. Spatialite – SQLite Spatial
    www.gaia-gis.it/fossil/spatialite_tools/index

    View Slide

  70. Spatialite – SQLite Spatial
    www.gaia-gis.it/fossil/spatialite_tools/index
    spatialite_osm_map – converts .osm to a sqlite format usable in a GIS
    spatialite_osm_raw – converts .osm to a sqlite format usable for stats
    spatialite_osm_filter – generates an OSM.XML file masked by an input polygon
    spatialite_osm_net – generates ‘routable’ data from roads

    View Slide

  71. spatialite_osm_map / raw – importing .pbf file
    www.gaia-gis.it/fossil/spatialite_tools/index
    spatialite_osm_map –o input.pbf –d output.sqlite
    Took about 30 mins (compared to 70 for osm2pgsql)
    Resulting .sqlite file was 300MB for the map, 1.5GB for raw

    View Slide

  72. Spatialite – SQLite Spatial
    www.gaia-gis.it/fossil/spatialite_gui/index

    View Slide

  73. spatialite_gui
    www.gaia-gis.it/fossil/spatialite_gui/index

    View Slide

  74. spatialite_gui
    www.gaia-gis.it/fossil/spatialite_gui/index

    View Slide

  75. Spatialite in QGIS – use ‘Add Vector Layer’

    View Slide

  76. Left as an exercise:
    Other services that you can run / setup:
    Nominatim – OSM’s Geocoder.
    pgRouting – PostgreSQL based routing
    OSRM – Open Source Routing Machine, closely tied to OSM
    Using Osmosis to keep your tileserver up to date
    pgRouting.org
    project-osrm.org

    View Slide

  77. Wider Issues:
    Comparing OSM and Commercial Data
    What now for Commercial Mapping?
    Privacy & Data Protection
    The future of tiles

    View Slide

  78. Map contains data Copyright NavTeq
    Map contains data Copyright NavTeq

    View Slide

  79. Map contains data Copyright NavTeq
    Map contains data Copyright NavTeq

    View Slide

  80. Map contains data Copyright NavTeq
    Map contains data Copyright NavTeq

    View Slide

  81. Vs
    creativecommons.org

    View Slide

  82. VGI: OpenStreetMap
    • Limited to what is ‘visible’
    • API’s limited, aimed at data editors
    • ‘Viral’ nature of data licence
    • Focus on map data
    • ‘Warm Geography’
    • Aims to be of Public Interest
    • Instant data updates, Sporadic
    • ‘Depth’ Issues (coverage, completeness)
    • Weak Metadata / Standards
    Commercial GeoData Providers
    • Wide source of inputs
    • API’s extensive, aimed at data consumers
    • Derived data issues
    • Focus on services
    • ‘Cold Geography’
    • Aims to be Authoritive
    • Moderated update process, Consistent
    • ‘Width’ issues (limited featureset)
    • Expensive

    View Slide

  83. The future of commercial online mapping?
    Logorama: vimeo.com/10149605

    View Slide

  84. www.citymaps.com
    The future of commercial online mapping?

    View Slide

  85. maps.google.ie
    The future of commercial online mapping?

    View Slide

  86. Crowdsourcing: Active vs Passive

    View Slide

  87. Privacy / Data Protection
    petewarden.github.com/iPhoneTracker

    View Slide

  88. www.jasondavies.com/maps/raster/
    Reprojected Raster Tiles

    View Slide

  89. www.mapbox.com/blog/vector-tiles wiki.openstreetmap.org/wiki/Vector_tiles
    Vector Tiles
    /{z}/{x}/{y}.png
    .geojson

    View Slide

  90. Close:
    Contacting OSM
    10 “Takeaway's”
    Q & A

    View Slide

  91. OSM Wiki:
    wiki.openstreetmap.org
    OSM Help Forums:
    help.openstreetmap.org
    OSM-F Blog:
    blog.osmfoundation.org
    OpenGeoData Blog (OSM News):
    opengeodata.org
    IRC (Internet Relay Chat):
    irc://irc.oftc.net #osm
    Mailing Lists:
    lists.openstreetmap.org/listinfo
    Contacting OSM

    View Slide

  92. 10 'Takeaways'
    License Check obligations if mixing data
    Data Model Area / Multipolygon handling is complex
    Usage Policies Desktop probably not a problem. Server, web different
    Build Your Own No apt-get install tileserver, yet
    Tilemill Great for designers, self hosting options exist
    Shapefiles Easy, but not complete
    Postgis Starting point for many possibilities
    Data APIs Great for snapshots in geoJSON or POI webpages
    Spatialite Spatial Database without the overhead of Postgis
    Data vs Services Is OSM commoditising data? Is that bad?

    View Slide

  93. This slide deck is available at:
    speakerdeck.com/manaboutcouch
    Thank you.
    Any Questions?
    Richard Cantwell
    @ManAboutCouch [email protected]

    View Slide