At the 2013 State of the Map conference I gave a one hour workshop aimed at introducing OpenStreetMap's inner workings to those of us who work with GIS.
Agenda: Introduction to OpenStreetMap: The origin of OSM Project Structure / Management Data Licensing How OSM represents data in GIS Terms: Nodes / Ways / Relations Key / Value pairs TagInfo Contributing: GIS Based Editors Consuming: Tiles, Files and API’s: Tileservers – your own and/or others Files – Downloads, Databases & QGIS API’s – OverPass Turbo TileMill Spatialite Wider Issues: Comparing OSM and Commercial Data What now for Commercial Mapping? Privacy & Data Protection The future of Tiles Close: Contacting OSM Takeaways from this session Q & A
Project Structure • “Rather Light” • “Do-ocracy” / “Meritocracy” • Somebody has to run the servers etc.: OSM Foundation (OSMF) • Contributors are a broad church
Geographic Information is represented as: Nodes Point Ways Line Closed Line Area (Polygon) eg. Roundabouts Relations Multi (Point / Line / Polygon) eg. Bus Routes / Inner Rings
Consuming: Tiles, Files and API’s: The OSM ‘Stack’ Tiles – your own and/or others Files – Downloads, Databases, QGIS API’s – OverPass Turbo TileMill – quick overview Spatialite – SQL without the overhead
OpenStreetMap – Tile Usage wiki.openstreetmap.org/wiki/Tile_Disk_Usage OSM does NOT pre-render every tile. Pre-rendering all tiles would use around 54,000 GB of storage. The majority of tiles are never viewed. In fact just 1.79% have ever been viewed. This is because the majority of tiles are at zoom level 18 and the vast majority of those contain nothing of interest (the sea, for example) By following an on-the-fly rendering approach we can avoid rendering these tiles unnecessarily. The tile view count column shows how many tiles have been produced on the OSM Tile server. Tile server disk usage: 1,272 GB used. (6 Jan 2012). (z0 to z15: 252 GB, z16 to z18: 1,020 GB)
Why make your own? Tile Usage Policy You’re in control: Design and Content Need tiles in EPSG:27700? 29903? Having OSM Data in a Database enables a lot of things You will learn a lot
www.switch2osm.org Tileserver Software Stack • Software Stack: • mod_tile: apache module, serves and expires tiles • renderd: priority queuing system for rendering requests • mapnik: does the actual rendering • postgis: Spatially enabled PostgreSQL database • osm2pgsql: loads osm data into the postgis database
osm2pgsql Approx 70 Minutes for a small extract (Ireland .pbf = 62MB, Planet is 21GB) Would be faster on a machine with more RAM and SSD disks Whole Planet can take days osm2pgsql -s -U username -d gis -C 100 /path/to/downloaded/osm/data -s = Slim mode -U = postgres username (owner of gis database) -d = database name (usually gis) -C = amount of memory in MB, usually 800, but my PC is a bit old
Using the Data API “The Overpass API (or OSM3S) is a read-only API that serves up custom selected parts of the OSM map data. It acts as a database over the web: the client sends a query to the API and gets back the data set that corresponds to the query.” wiki.openstreetmap.org/wiki/Overpass_API overpass-turbo.eu
Extract Raw Data – Embedded in a map on a web page map.addLayers([ make_layer("http://overpass-api.de/api/interpreter?data= node[amenity=pub](bbox);out+skel; (way[amenity=pub](bbox);node(w););out+skel;", “Blue") ]);
osmosis – java based command line OSM data processing tool Some examples of the things it can currently do are: • Generate planet dumps from a database • Load planet dumps into a database • Produce change sets using database history tables • Apply change sets to a local database • Compare two planet dump files and produce a change set • Re-sort the data contained in planet dump files • Extract data inside a bounding box or polygon wiki.openstreetmap.org/wiki/Osmosis
Spatialite – SQLite Spatial www.gaia-gis.it/fossil/spatialite_tools/index spatialite_osm_map – converts .osm to a sqlite format usable in a GIS spatialite_osm_raw – converts .osm to a sqlite format usable for stats spatialite_osm_filter – generates an OSM.XML file masked by an input polygon spatialite_osm_net – generates ‘routable’ data from roads
spatialite_osm_map / raw – importing .pbf file www.gaia-gis.it/fossil/spatialite_tools/index spatialite_osm_map –o input.pbf –d output.sqlite Took about 30 mins (compared to 70 for osm2pgsql) Resulting .sqlite file was 300MB for the map, 1.5GB for raw
Left as an exercise: Other services that you can run / setup: Nominatim – OSM’s Geocoder. pgRouting – PostgreSQL based routing OSRM – Open Source Routing Machine, closely tied to OSM Using Osmosis to keep your tileserver up to date pgRouting.org project-osrm.org
VGI: OpenStreetMap • Limited to what is ‘visible’ • API’s limited, aimed at data editors • ‘Viral’ nature of data licence • Focus on map data • ‘Warm Geography’ • Aims to be of Public Interest • Instant data updates, Sporadic • ‘Depth’ Issues (coverage, completeness) • Weak Metadata / Standards Commercial GeoData Providers • Wide source of inputs • API’s extensive, aimed at data consumers • Derived data issues • Focus on services • ‘Cold Geography’ • Aims to be Authoritive • Moderated update process, Consistent • ‘Width’ issues (limited featureset) • Expensive
10 'Takeaways' License Check obligations if mixing data Data Model Area / Multipolygon handling is complex Usage Policies Desktop probably not a problem. Server, web different Build Your Own No apt-get install tileserver, yet Tilemill Great for designers, self hosting options exist Shapefiles Easy, but not complete Postgis Starting point for many possibilities Data APIs Great for snapshots in geoJSON or POI webpages Spatialite Spatial Database without the overhead of Postgis Data vs Services Is OSM commoditising data? Is that bad?
manaboutcouch
renjikari
se_o_chan
piacerex
dena_tech
line_developers
iwashi86
takakuni
teckl
mapbox_jp
yhana
sbtechnight
cromwellryan
sferik
pauljervisheath
jacobian
holman
philnash
addyosmani
hannesfritz
erikaheidi
shpigford
lara
zakiwarfel