2021-06-17_UNVT exercise day02

Material for a casual vector tile exercise (Day 02). Exercise of Vector tile production using docker (unvt/nanban) and web map hosting.

Transcript

1. UNVT Exercise Day 2 17 June 2021 By Taro Ubukawa (Senior Geospatial Expert at UN Geospatial Information Section)
2. Whole schedule
3. Today’s contents • Exercise Part 1: Vector Tile Production in Docker (Data conversion) • Exercise Part 2: Hosting web map app (Styling and hosting) • Style • html preparation • hosting
4. Vector tile - revisit • VT Production (Conversion) • VT Hosting • VT Styling • VT Optimizing Produce Host Style Optimize Import UNVT workflow of developing vector tile for base map Supported by UNVT, open source scripts Source DB Web APP Users
5. Exercise Part 1: Vector Tile Production in Docker (Data conversion)
6. Introduction (1): Revisit to typical formats of Vector Tiles PBF: Binary tiled data. Generally stored in the folder (Z/X/Y). Same with the mapbox vector tile (.mvt). The number of files/folders can be big. Mbtiles: One mbtiles file stores several tiles inside. It is a SQLite database. If you have a lot of tiles, this format is preferable. But you may need to have a hosting server to deliver this format as pbf. The pbf is good enough for this practice because the number of files and their total size are not so large. In addition, we can simply host vector tile by uploading pbf tile to a web server. Forget about mbtiles at today’s practice!
7. Introduction (2): VT Production Workflow overview • This time, we will start from shapefiles as the following flow. • Vector tile (VT) is developed from GeoJSON files using a tool named “Tippecanoe”. • Tippecanoe: https://github.com/mapbox/tippecanoe Vector Tiles shapefile GeoJSON shapefile GeoJSON 2. Convert the source into GeoJSON. (It they are not.) 1. Prepare source data 3. Make vector tile with Tippecanoe This is the workflow for this practice. Although it is one of typical procedures, you can choose other ways (e.g. directly from PostGIS). gdal Tippecanoe
8. Introduction (3): A core tool named Tippecanoe Tippecanoe (by mapbox): https://github.com/mapbox/tippecanoe • A tool to build vector tile • BSD 2-Clause "Simplified" License • Runs on LINUX environment (or OSX with homebrew) LINUX environment is the one of the big challenges for many people who use Windows environment. You can run it on Docker in the Windows environment. This is the reason why we use Docker today. (I have not yet test, but WSL would work, too. Raspberry Pi is also good for practice.)
9. Preparation: Docker Install • Enable Hyper-V • Get started with Docker at https://www.docker.com/ You can run it with Windows Powershell • Installation of Docker in windows. Windows 10 64-bit:Pro, Enterprise, or Education For detail: https://www.docker.com/products/docker-desktop I used Docker Desktop for Windows (Hyper-V backend and windows containers). For the user of Windows 10 Home, WSL 2 backend would be used. This practice is mainly for windows users because Mac users may be able to use “Tippecanoe” in Terminal, while Windows (powershell or cmd) cannot directly use Tippecanoe.
10. Preparation: the source data You can use your own data. If you have source data, please skip this page. This practice will use data from Natural Earth at naturalearthdata.com. (1:110m) • Coastline • Land • Admin 0 – Boundary Lines • Populated Places Here, NaturalEarth (free data) is used just for practice purpose. The boundaries in the map do not imply official endorsement/acceptance by the UN or any agencies. https://www.naturalearthdata.com/
11. Preparation: Additional source for practice • I have created additional data source for your practice. • location.shp (point) • Locations of UNGIS, UNGSC and UNMISS. (Only 3 for testing purpose.) • It has “name” attribution. • Projection is not the same with other (World Mercator projection)
12. Preparation: Source data for practice • For this test, we use the following data set (see data/0_source): • ne_110m_admin_0_boundary_lines_land.shp (will be “bndl” layer), • ne_110m_land.shp (will be “landmass” layer), • ne_110m_coastline.shp (will be “coastl” layer), • ne_110m_populated_places.shp (will be “popp” layer) • location.shp (will be “loc” layer) If you prepare your own data: About format, both shapefile and GeoJSON are OK. We can easily convert shapefile into GeoJSON. (Any format is OK as long as it can be converted to geojson format.)
13. Conversion (1): Start Docker • Copy data from the MS Teams channel. • Start command prompt (I used Windows Powershell), and move to the working directory (data). • Then, start Docker (unvt/nanban) > docker pull unvt/nanban > docker run -it --rm -v ${PWD}:/data unvt/nanban Please check your location and move to /data directory (cd /data). By using “-v” option, please connect your local working directory to “/data” directory. Here, I do not explain the preparation of the Docker environment. Please see https://docs.docker.com/docker-for- windows/install/
14. (FYI) About unvt/nanban • One of the UN Vector Tile Toolkit. This is a developer friendly UNVT Dockerfile. Developed by Mr. Fujimura in July 2020. I think this tool helps windows users to easily use UNVT. • Features: • based on Ubuntu/Intel → This is the base of the various tools • Tippecanoe is contained. → It can be used for vector tile data conversion. • gdal is included → It can be used for basic GIS data processing. • HOCON Parse is also available → It can be used for style editing (advanced). • The name nanban is from Tamil language. It means friend. You can find detailed information at https://github.com/unvt/nanban
15. Conversion (2): Conversion into GeoJSON • Convert shapefiles into geojson using ogr2ogr • Type the following command (for data with WGS84 projection EPSG:4326). > ogr2ogr -f GeoJSON output.geojson source.shp > … • For “location.shp”, as its projection is different from EPSG:4326, it needs a little more adjustment..(change the projection to 4326 (or 3857) ) > ogr2ogr -f GeoJSON -s_srs EPSG:3395 -t_srs EPSG:4326 1_geojson/loc.geojson 0_source/location.shp Gdal (ogr2ogr) is included in unvt/nanban. You can simply use it to prepare GeoJSON. See next slide to confirm the command.
16. Result • The sources are in “0_source” directory and geojson will be stored in “1_geojson.” Then, now you have Geojson files.
17. (FYI) GeoJSON conversion with QGIS If you are not familiar with gdal, you can use QGIS to prepare GeoJSON. • Export the files to GeoJSON • Right clicking layer name. -> Export -> Save feature as The map data in this figure is from another source. This figure is just for practice.
18. Conversion (3): Think about the structure of your vector tile • Vector tile can have several layers in it. You need to think about the structure of your Vector Tile based on your source data. This practice will use the following settings. • Max Zoom: 3, Min Zoom: 0 • Layers: Theme GeoJSON name VT layer name (fyi) key Attributes Coastline coastl.geojson coastl featurecla Land landmass.geojson landmass featurecla Admin 0 Boundary (National level) bndl.geojson bndl featurecla Populated place popp.geojson popp featurecla, name Test Location.geojson loc name, location Attributes comes from original data. You do not have to work on it, but please check their value if necessary.
19. Conversion (4): Data Conversion using Tippecanoe • Confirm your location (“/data” is properly connected with your local folder? ) • Continue your work on Docker, and run the following: > tippecanoe -e 2_tile/myVT --no-tile-compression --no-feature-limit --no-tile-size-limit --drop-rate=1 -Z0 -z3 -L coastl:1_geojson/coastl.geojson -L landmass:1_geojson/landmass.geojson -L bndl:1_geojson/bndl.geojson -L popp:1_geojson/popp.geojson -L loc:1_geojson/loc.geojson
20. Conversion (4): Data Conversion using Tippecanoe Easy Explanation of options: • -e 2_tile/myVT → export file. Option “e” for pbf format, “o” for mbtiles format. • --no-tile-compression → compressed pbf cannot be read by some APP. Please add this option. • --no-feature-limit → eliminate the limit of the number of the tile per tile. • --no-tile-size-limit → eliminate the limit of the data size per tile. • --drop-rate → Points will be dropped with this rate at smaller zoom, based on its base-scale (in general, largest zoom of the tile). Keep this as 1 if you do not want to drop any point. • -z, -Z → Specifies min and max zoom of the tile. • -L → defines each layer. Each layer is defined by -L Layername:source.geojson, you can add more than one See https://github.com/mapbox/tippecanoe for details If you forget –no-tile- compression option, you may not be able to see the tile in your web map app.
21. Conversion (5): Result (Produced vector tile) • Please check your local folder. You will have pbf files in a folder. → name/{z}/{x}/{y}.pbf structure Then, please simply upload your vector tile to a web server. Fore your practice, I uploaded the tile in advance at the following URL: https://ubukawa.github.io/VT-test-0614/myVT/{z}/{x}/{y}.pbf
22. Part1 Conclusion and Next step With this practice, we learned how to make VT with Docker. Here are next steps of your vector tiles. We will practice the following steps in another exercise. • Vector tile hosing • You can just simply upload the folder to the web server. • Then, you can refer to your vector tile. • Styling • You can add your style. Uploading the file to GitHub page, then you can easily see your VT with Maputnik.
23. Part 2: Hosting Web Map
24. Part 2 • Hosting Web map • Editing Style • Preparing html file • Establish a web server for practice/development
25. Let’s use maputnik again to develop a style file. Please open it at https://maputnik.githu b.io/editor/ Styling with maputnik (0): Opening a maputnik If you do not want to follow each steps, just visit https://maputnik.github.io/editor?style=https://ubukawa.github.io/VT-test-0614/maps/mystyle.json .
26. Styling with maputnik (1): Opening a new style You might see your previous work. So, let’s open new one. • Open “Empty Style” • “Open” at top menue • Choose “Empty Style”
27. Styling with maputnik (2): Add your data source First, you need to add your vector tile as a source. • Click “Data Source” at top menu. • “Add New Source” • Source ID: any (“v” recommended) • Source Type: Vector (XYX URLs) • URL: https://ubukawa.github.io/VT-test- 0614/myVT/{z}/{x}/{y}.pbf • Min Zoom: 0 • Max Zoom: 3 If there is a typo, you will not see data. (in this example “myVY” should be “myVT”)
28. Styling with maputnik (3): Edit Style Setting Next, let’s edit style setting. Click “Style Setting” at top menu. Then, please edit style setting. • Name: change your style name as you like. Here, I named it as “MyStyle.” • Sprite URL: https://ubukawa.github.io/VT-test- 0614/sprite/sprite • Glyphs URL: https://ubukawa.github.io/VT-test- 0614/font/{fontstack}/{range}.pbf You will have better understanding of “sprite” and “glyphs,” but now, let’s use above value for our initial practice.
29. Styling with maputnik (4): Adding a layer: Background 1. Add layer 2. Name layer at “ID” 3. Select “Background” in “Type” 4. Click add layer
30. Styling with maputnik (4): Adding a layer: Background Set min/max zoom Adjust color etc.
31. Styling with maputnik (4): Adding a layer: landmass (fill) 1. Add layer 2. Name layer at “ID” 3. Select “Fill” 4. Specify the source (the name you decided) and choose the layer 5. Click “Add layer”
32. Styling with maputnik (4): Adding a layer: landmass (fill) Set min/max zoom Adjust color etc.
33. Styling with maputnik (4): Adding a layer: Coastl (line) 1. Add layer 2. Name layer at “ID” 3. Select “line” 4. Specify the source (the name you decided) and choose the layer 5. Click “Add layer” Practice “bndl” by yourself if you want.
34. Styling with maputnik (4): Adding a layer: loc (as symbol) 1. Add layer 2. Name layer at “ID” 3. Select “symbol” 4. Specify the source (the name you decided) and choose the layer 5. Click “Add layer” You may see no change at first. Click “Image” at “Icon layout properties” and choose any map symbol.
35. Styling with maputnik (4): Adding a layer: loc (as symbol) At its text layout properties, by specifying text field ( {name} in this case ) and text font (only “sans” will work for this glyph source), you can see label. Adjust the size and offset as you like. If you specify wrong font, there will no map. Offset (1,1)
36. Styling with maputnik (4): Adding a layer: popp (circle) 1. Add layer 2. Name layer at “ID” 3. Select “circle” 4. Specify the source (the name you decided) and choose the layer 5. Click “Add layer” Point data can be added as not only “symbol” but also “circle”.
37. Styling with maputnik (4): Export style • Once you are ready, click “Export” in the top menu and click “Download.”
38. Now, the Style file is ready I have uploaded sample below: https://ubukawa.github.io/VT-test-0614/maps/mystyle.json Please upload it somewhere in the web server.
39. Html(1): Let’s make a html file • Open “mywebmap.html” in the practice folder (data/9_host). • At first, you will see empty map. So, please edit the location of the style. (It will not work if style.json is referred in file system.) File is uploaded here: https://ubukawa.github.io/VT-test- 0614/maps/mywebmap.html
40. Html(2): Your web map • Can you see a web map? https://ubukawa.github.io/VT-test-0614/maps/mywebmap.html You may find that the map disappear at the zoom level 3. This is because it is so defined in the style file. You can edit its style to change it. (Even if the original tile is up to zoom level 3, with mapbox-gl-js, overzooming will be available up to zoom level 5 (+2). )
41. Html(3) Tips: What’s in html? (1) • The location of map library is specified. • In this case, we use mapbox-gl-js (ver 1.x) and css. *You could upload them to your own server. But this case, we refer to ones already uploaded to the Internet.
42. • You can specify parameters such as. • Style: Style location • Center: center of the initial map. • Zoom: zoom level of the initial map. Html(3) Tips: What’s in html? (2)
43. • User Interface can be also found. Try exploring these. Html(3) Tips: What’s in html? (3)
44. Hosting For practice (or development), what would be a good way of web hosting? • GitHub page • Establishing a local server • nodejs • budo (I do not have enough experience on it)
45. Demo: Create a GitHub Repository In order to establish a web page, here, we will use a GitHub repository. I will not explain. Introduced at the previous lesson.
46. Demo: Start GitHub page Setting > GitHub Page Confirm the root URL of your repository. I will not explain. Introduced at the previous lesson.
47. Then, upload the necessary files • Copy the file from the shared directory in MS Teams channel. • Make sure that you have adjust your file path in style.json and html. I will not explain. Introduced at the previous lesson.
48. Establish a local server using nodejs • Please go to the “x_localhost” frolder. (or access original content is from https://github.com/ubukawa/style-editing ) • Using powershell (with nodejs), > npm install > Node app.js (ctrl+c to stop) Access localhost at: http://localhost:8836/hello.html http://localhost:8836/index.html Then, replace “style.json” in htdocs folder. If you cannot use nodejs in your pc, try unvt/nanban See https://github.com/ubukawa/style-editing
49. Hosting – overview • We have covered all elements so far. Vector Tile Data Map Library Map Symbol Icons (sprite) Text font (Glyphs) HTML Hosting server(s) • Server itself → GitHub page / local host • Data → Your own / External tile • Map Symbol → Your own / External source • Font → Your own / External source • Library → External source / or upload • Style info → Your own / external source • Html → your own. Style info
50. Application example (1) You can create vector tile map of your area from OSM data Example: https://ubukawa.github.io/osm- neighbor/index.html#9.62/40.7676/-73.9925 VT development Method: Study at the following repository https://github.com/unvt/naru https://github.com/optgeo/kokoromi Example: https://ubukawa.github.io/osm-neighbor/story- test/test-v1.html Story map https://speakerdeck.com/ubukawa/2021- 06-07-storytelling-trial-report
51. Web Map can be inserted into html page by using the iframe tag. (e.g. Carto Tile) https://www.un.org/geospatial/mapsgeo/webservices https://un-geospatial.github.io/cartotile-plain- design/index.html If you are good at web page development, you can add various functions like pop-up and so forth. Application example (2)
52. Summary of part 2 • Though the part 2, • You can edit style.json • You can edit html file for web mapping. • You thought about how we can establish a web server for practice.
53. END of today’s practice • Thank you for your kind attention and patience!!! • Any question? • Please feel free to MS Teams channel to share your question, practice result, or your nice web map. • Next week: • I will introduce how we developed maps in https://dev-unvt.dfs.un.org/ from PostGIS data. • The upcoming lessons are only targeted for UNGSC colleagues initially, while first two session is open for all colleagues in UN. (Because the contents could be further detail and we will talk specific data base.) • Please let me know if you have any request to join. I will consult with my colleague.
54. References • Matsuzawa, T. (2019): Create/distribute vector tile, https://speakerdeck.com/smellman/distrubute-vector-tile • Install Docker Desktop on Windows: https://docs.docker.com/docker-for-windows/install/ • Natural Earth: https://www.naturalearthdata.com/ • Tippecanoe: https://github.com/mapbox/tippecanoe • Maputnik editor: https://maputnik.github.io/editor/ • UNVT workshop (in Japanese): https://github.com/unvt/512 • Fujimura, H. (2021): UNVT Workshop: Introduction and Application https://speakerdeck.com/hfu/unvt-workshop-introduction-and-application
55. More practice? • https://github.com/unvt/ezPrac