2021-05-28_Vector Tile conversion with Doker (unvt nanban)

materials for the series of exercise.

Transcript

1. How to make a Vector Tile - Simple conversion practice using Docker(unvt/nanban) - Taro Ubukawa Senior Geospatial Expert, UN Geospatial Information Section 28 May 2021
2. Contents 1. Prerequisite 2. Introduction (format, workflow overview) 3. Preparing the source data 4. Conversion work • Start Docker • Convert source data into GeoJSON • Structure of your vector tile • Data Conversion using Tippecanoe • Result 5. Conclusion and next step
3. 1. Prerequisite • Installation of Docker in windows. Windows 10 64-bit:Pro, Enterprise, or Education For detail: https://docs.docker.com/docker-for-windows/install/ • I used Docker Desktop for Windows (Hyper-V backend and windows containers). For the user of Windows 10 Home, WSL 2 backend would be used. (but I have not confirmed if this works.) This practice is specially for windows users because Mac users may be able to use data conversion tool “Tippecanoe” in Terminal, while Windows (powershell or cmd) cannot directly use Tippecanoe.
4. 2. Introduction (1): Revisit to typical formats of Vector Tiles Mbtiles: One mbtiles file stores several tiles inside. It is a SQLite database. If you have a lot of tiles, this format is preferable. But you may need to have a hosting server to deliver this format as pbf. PBF: Binary tiled data. Generally stored in the folder (Z/X/Y). Same with the mapbox vector tile (.mvt). The number of files/folders can be big. I converted into the both formats, but the pbf is good enough for this practice because the number of files and their total size are not so large.
5. 2. Introduction (2): VT Production Workflow overview • Vector tile (VT) is developed from GeoJSON files using a tool named “Tippecanoe”. • Tippecanoe: https://github.com/mapbox/tippecanoe • This time, we will start from shapefiles as the following flow. Vector Tiles shapefile GeoJSON shapefile GeoJSON 2. Convert the source into GeoJSON. (It they are not.) 1. Prepare source data 3. Make vector tile with Tippecanoe This is the work flow for this practice. Although it is one of typical procedures, you can choose other ways (e.g. directly from PostGIS).
6. 3. Preparing the source data You can use your own data. If you have source data, please skip this page. This practice will use data from Natural Earth at naturalearthdata.com. (1:110m) • Coastline • Land • Admin 0 – Boundary Lines • Populated Places Here, NaturalEarth (free data) is used just for practice purpose. The boundaries in the map do not imply official endorsement/acceptance by the UN or any agencies.
7. 4. Conversion (1): Start Docker • Start command prompt (I used Windows Powershell), and move to the working directory. • Then, start Docker (unvt/nanban) > docker pull unvt/nanban > Docker run -it --rm -v ${PWD}:/data unvt/nanban By using “-v” option, please connect your local working directory to “/data” directory. Here, I do not explain the preparation of the Docker environment. Please see https://docs.docker.com/docker-for- windows/install/
8. (FYI) About unvt/nanban • This is a developer friendly UNVT Dockerfile. Developed by Mr. Fujimura in July 2020. I think this tool helps windows users to easily use UNVT. • Features: • based on Ubuntu/Intel → This is the base of the various tools • Tippecanoe is contained. → It can be used for vector tile data conversion. • gdal is contained → It can be used for basic GIS data processing. • HOCON Parse is also available → It can be used for style editing. • The name nanban is from Tamil language. It means friend. You can find detailed information at https://github.com/unvt/nanban
9. 4. Conversion (2): Convert the source data into GeoJSON • Move to the working directory (it is associated to “/data”). • Convert shapefiles into geojson using ogr2ogr > cd /data > ogr2ogr –f GeoJSON output.geojson source.shp > … Gdal (ogr2ogr) is included. You can simple use it to prepare GeoJSON.
10. (FYI) GeoJSON conversion with QGIS If you are not familiar with gdal, you can use QGIS to prepare GeoJSON. • Export the files to GeoJSON • Right clicking layer name. -> Export -> Save feature as The map data in this figure is from another source. This figure is just for practice.
11. 4. Conversion (3): The structure of your vector tile • You need to think about the structure of your Vector Tile based on your source data. This practice will use the following settings. • Max Zoom: 3 • Min Zoom: 0 • Layers: Theme GeoJSON name VT layer name (fyi) key Attributes Coastline coastl.geojson coastl featurecla Land landmass.geojson landmass featurecla Admin 0 Boundary (National level) bndl.geojson bndl featurecla Populated place popp.geojson popp featurecla, name Attributes comes from original data. You do not have to work on it, but please check their value if necessary.
12. 4. Conversion (4): Data Conversion using Tippecanoe • Continue your work on Docker, and run the following: > tippecanoe -e /data/ne0530 --no-tile-compression --no-feature-limit --no-tile-size-limit --drop-rate=1 -Z0 -z3 -L coastl:coastl.geojson -L landmass:landmass.geojson -L bndl:bndl.geojson -L popp:popp.geojson Easy Explanation of options: • -e /data/ne0530 → export file. Option “e” for pbf format, “o” for mbtiles format. • --no-tile-compression → compressed pbf cannot be read by some APP. Please add this option. • --no-feature-limit → eliminate the limit of the number of the tile per tile. • --no-tile-size-limit → eliminate the limit of the data size per tile. • --drop-rate → Points will be dropped with this rate at smaller zoom, based on its base-scale (in general, largest zoom of the tile). Keep this as 1 if you do not want to drop any point. • -z, -Z → Specifys min and max zoom of the tile. • -L → defines each layer. Each layer is defined by -L Layername:source.geojson See https://github.com/mapbox/tippecanoe for details
13. 4. Conversion (5): Result (Produced vector tile) • Please check your local folder. You will have pbf files in a folder. → name/{z}/{x}/{y}.pbf structure You can stop your docker.
14. 5. Conclusion and Next step With this practice, we learned how to make VT with Docker. Here are next steps of your vector tiles. We will practice the following steps in another exercise. • Vector tile hosing • You can just simply upload the folder to the web server. • Then, you can refer to your vector tile. • Styling • You can add your style. Uploading the file to GitHub page, then you can easily see your VT with Maputnik.
15. References • Matsuzawa, T. (2019): Create/distribute vector tile, https://speakerdeck.com/smellman/distrubute-vector-tile • Install Docker Desktop on Windows: https://docs.docker.com/docker-for-windows/install/ • Natural Earth: https://www.naturalearthdata.com/ • Tippecanoe: https://github.com/mapbox/tippecanoe • Maputnik editor: https://maputnik.github.io/editor/ • UNVT workshop (in Japanese): https://github.com/unvt/512
16. More practice ? For more practice, please visit https://github.com/unvt/ezPrac