Building a National Tile Server

How Astun Technology (http://astuntechnology.com) built a national (Great Britain) web map tile server and WMS using PostgreSQL, PostGIS, MapServer and MapProxy all running on AWS. Presented at OSGIS 2012.(http://www.nottingham.ac.uk/osgis/)

1. Building a National Tile Server Matt Walker mattwalker@astuntechnology.com @_walkermatt
2. What is it? Great Britain base map tile service (TMS) and WMS OS Open (Strategy, Meridian 2, VectorMap District) OS Premium (OS Open + VectorMap Local & OS MasterMap) Smaller areas of Aerial Photography and other base mapping
3. © crown copyright 2012
4. Who's using it? Exactrak Vehicle Tracking and Fleet Management Mendip District Council Public Website Windsor and Maidenhead Public Website Surrey County Council Public Website South Norfolk Council Public Website
5. Who's using it? Stockport MBC Public Website Surrey Heath Borough Council Public Website / Intranet GIS Chichester District Council Public Website
6. Software
7. Infrastructure
8. Loader • Python script for loading GML and KML data using OGR • Originally for loading OS GML data (but has been used with other GML and KML data) • Allows the source GML to be prepared and enhanced during loading via Python
9. Loader • Fairly fast (national cover OS MasterMap in 2 days) • Concurrent loading via 4 processes into PostgreSQL via PGDump with COPY statements, create indexes once when all data loaded
10. PostgreSQL/PostGIS • PostgreSQL v9 with PostGIS v1.5 • 210GB of OS MasterMap and VectorMap Local data and indexes • 10GB for Meridian, Strategi etc. • Separate tablespace for OSMM data; OSMM index; VML data; VML index
11. PostgreSQL/PostGIS • During load – fsync = off to avoid flushing to disk too frequently – work_mem = 1GB when building indexes • Effectively read-only once in production so configured for read: – shared_buffers = 2GB – effecitive_cache_size = 4GB
12. MapServer • Fast and reliable • Simple configuration • Mature feature set • All data read direct from PostgreSQL
13. MapServer • PROCESSING "CLOSE_CONNECTION=DEFER" to share connections • OS MasterMap and VectorMap Local generally styled based on featurecode • Using some attributes derived by Loader for CartoText such as orientdeg and anchorposition for better text placement
14. MapProxy • Provides caching of map tiles for performance • Exposes WMS, TMS services for your application, serving WMS direct from the cache
15. MapProxy
16. MapProxy • Nice features like only storing one tile for single colour tiles and mixed format caches • Sophisticated cache seeding and management tools • We seed popular areas (urban areas) for our customers and cache is only about 20GB compared to hundreds of GB if pre-seeded
17. Infrastructure
18. • ~8 EC2 instances all running Ubuntu • Seperate EBS Volumes for PostgreSQL Data, Indexes and Tile Cache • One Load Balancer in front of the production servers • AWS monitoring for health as well as seperate web monitoring Infrastructure
19. • Shared, High Performance, Redundant Cache, possibly S3 SimpleDB or similar • Improved cartography, possibly Mapnik • Co-located hosting for imporved redundency What's next?
20. astuntechnology.com AstunTechnology walkermatt @isharemaps @_walkermatt Thanks