The OSGeoJP Workshop for the UN Vector Tile Toolkit

Transcript

1. OSGeo.JP Workshop for the UN Vector Tile Toolkit – Let’s make sustainable web maps Supported by OSGeo.JP 2018-12-02T13:30/17:00 @Moratuwa University
2. Today’s schedule 2 1 Introduction 13:30 2 Fully guided hands-on 14:00 3 More individual challenges 15:30 4 Comments and wrap-up 16:40
3. 3 Introduction
4. 4 By the end of the workshop, You create a web map as good as this. Or you can make it even better.
5. Questionnaire 5 Q1: What Operating System do you mainly use today? ü Windows 10? ü Windows 7? ü OS X? ü Linux? ü Others? Q2: What Browser do you use today? (choose all you want to use) ü Google Chrome? ü Firefox? ü Safari? ü Microsoft Edge? ü Others?
6. Step 0: network check 6 open http://vectortiles.xyz:8765/hands-on You will see this map; otherwise plese let us know.
7. 7 Natural Earth Natural Earth Global Map Survey Department, Sri Lanka OpenStreetMap OpenStreetMap contributors Integrated into a single vector tileset using the UN Vector Tile Toolkit
8. 8 And this is: 100% vector tiles. 100% open source. No access token required.
9. 9 Why Vector Tiles?
10. 10 Evolution of web maps
11. 11 Reference: https://tmcw.github.io/presentations/jsgeo 1 Untiled WMS Selection by BBOX 2 Image tiles Predictable & cached 3 Tiles and overlays Vector retrofit 4 Vector tiles Everything in vector Evolution of web maps for performance
12. 12 Reference: https://tmcw.github.io/presentations/jsgeo/#33 Image tiles are facets of a gigantic image Vector tiles are facets into datastructure, a straightforward way to vector geospatial data.
13. 13 Web maps are only for the bandwidth-rich? No.
14. 14 9.C Significantly increase access to information and communications technology and strive to provide universal and affordable access to the Internet in least developed countries by 2020 + Vector Tiles even saves the bandwidth use. Vector tiles are gradually getting ready for everyone.
15. UN Vector Tile Toolkit Objective To facilitate the production of vector tiles for basemaps by public organizations. Structure Import Produce Host Style Optimize Existing datasource e.g. PostGIS, Shapefile, GeoJSON Tippecanoe Maputnik Interoperable with both Open Source and other frameworks. Tailor-made Node.js scripts Openly maintained on GitHub. Maximum use of existing OSS while adding tailor-made scripts shared openly. vt-optimizer 15 You have accessed this in the step 0.
16. 16 Technical details Import data and assign layer names and zoom levels. Custom open source scripts. Create vector tile packages in mbtiles format. Tippecanoe Host modular vector tile packages in a simple and scalable manner. Custom open source scripts. Style hosted vector tiles. Maputnik Enhance the optimization of the size of individual vector tiles by visualizing them. vt-optimizer and other custom open source scripts. Import Produce Host Style Optimize
17. 17 The UN Vector Tile Toolkit is a collection of tools, rather than a monolithic DB/Server suite. Because: ü Inputs are diverse. ü Depolyment environments are diverse. Flexibility is the core value. And sometimes the server shall be isolated from the master database (for security and scarability).
18. 18 Hands-on
19. The stages 19 1. Get source data 2. Add design properties 3. Build vector tilesets 4. Host the vector tilesets 5. Style the hosted vector tiles
20. The stages 20 1. Get source data 2. Add design properties 3. Build vector tilesets 4. Host the vector tilesets 5. Style the hosted vector tiles
21. 21 spinel- produce tileserver- gl-light spinel Maputnik Four packages used
22. 22 [Please check your workspace] You probably need about 2GB for your work area. This material refers the path to your work area as ${your-workspace}
23. Step 1: setup scripts to produce vector tileset 23 $ cd ${your-workspace} $ git clone https://github.com/un-vector-tile- toolkit/spinel-produce $ cd spinel-produce Download and install the scripts for producing vector tiles. If this does not work… ü Install git. ü Setup git (next slide). [spinel-produce]$ npm install If this does not work… ü Setup Node.js (next to the next slide)
24. Setup git 24 https://help.github.com
25. Setup Node.js 25 ü Recommendation: sudo apt install nodejs Ø In Ubuntu, you may want to run the following command, too. curl -sL https://deb.nodesource.com/setup_10.x | sudo -E bash - For other Linux-based platforms: https://github.com/nodesource/distributions
26. Or you still have a problem with “git clone”… 26 You can download a ZIP file from https://github.com/{user}/{repo} In this case https://github.com/un-vector-tile-toolkit/spinel-produce
27. Step 2: get the source data 27 [spinel-produce]$ node download.js # If you have time… [spinel-produce]$ gzcat src/6-46-30.ndjson.gz [spinel-produce]$ gzcat src/gmlk20.ndjson.gz This may take time depending on the Internet bandwidth. These are Newline Delimited GeoJSON features. i.e. each line is a GeoJSON Feature. This format is also called GeoJSON Text Sequences (RFC 8142) with Record Separator (RS) omitted.
28. RFC 8142 https://tools.ietf.org/html/rfc8142 28
29. The stages 29 1. Get source data 2. Add design properties 3. Build vector tilesets 4. Host the vector tilesets 5. Style the hosted vector tiles
30. Step 3: Add design properties 30 [spinel-produce]$ node import.js | less
31. Step 3: Add design properties 31 # If you have time… [spinel-produce]$ less 6-46-30.ndjson [spinel-produce]$ node import.js > 6-46- 30.ndjson
32. Break Point A: definition of what3numbers, import.js, and modify.js 32 what3numbers import.js modify.js The number assigned to a tile (in slippy map tilenames) used as the extent of the vector tile package. e.g. 4-46-30 overwraps Sri Lanka. The code to read each features, apply “modify function” to add design properties (layer name, minimum zoom and maximum zoom), and store the resulting features to NDJSON. The code that contains a function to modify a feature by adding design properties and edit other properties of the feature. You can edit this to change the “vector tile schema”.
34. An example of modify.js 34
35. The stages 35 1. Get source data 2. Add design properties 3. Build vector tilesets 4. Host the vector tilesets 5. Style the hosted vector tiles
36. Step 4: build vector tilesets 36 [spinel-produce]$ tippecanoe -f --read-parallel --drop-densest-as-needed --simplification=2 -- minimum-zoom=6 --maximum-zoom=15 --base- zoom=15 --output=6-46-30.mbtiles 6-46- 30.ndjson This may take time. You’re rendering areas as big as Sri Lanka. If this does not work… ü Install Tippecanoe (next slide)
37. Troubleshooting: install Tippecanoe 37 $ cd ${your-workspace} $ git clone https://github.com/mapbox/tippecanoe $ cd tippecanoe [tippecanoe]$ make –j [tippecanoe]$ sudo make install # Or you can just use ./tippecanoe without make install [tippecanoe]$ tippecanoe -v If this still does not work, see the next slide.
38. Troubleshooting in installing Tippecanoe • If you see errors at “make -j”, posslble solutions are: • Install g++ • Install libsqlite3 38 $ sudo apt install build-essential $ sudo apt install libsqlite3-dev zlib1g-dev For details, see https://github.com/mapbox/tippecanoe#development
39. 39 This is a vector tile package that contains the module 6-46-30. Format: mbtiles-spec (CC-BY) 6-46-30.mbtiles
40. Step 5: Quick-check the content of vector tileset 40 $ sudo npm install –g tileserver-gl-light $ tileserver-gl-light 6-46-30.mbtiles $ open http://localhost:8080 Please see a workaround if you see errors (e.g. in Ubuntu) Don’t worry if tileserver-gl-light doesn’t work. You can simply skip this step.
41. tileserver-gl-light install error workaround 41 $ sudo npm install -g tileserver-gl-light --unsafe-perm=true --allow-root Source: https://github.com/npm/npm/issues/17268
43. On the console 43 Ctrl-C to stop. The vector tiles you created were sent to the browser.
44. The stages 44 1. Get source data 2. Add design properties 3. Build vector tilesets 4. Host the vector tilesets 5. Style the hosted vector tiles
45. Step 6: install the server script 45 $ cd {your-workspace} # probably “cd ..” $ git clone https://github.com/un-vector-tile- toolkit/spinel $ cd spinel [spinel]$ npm install [spinel]$ mv ../spinel-produce/6-46-30.mbtiles mbtiles/hands-on [spinel]$ node index.js spinel vector tile server starts on 8765 # After you see “spinel vector tile server starts on 8765” on the original console; $ open http://localhost:8765 $ open http://localhost:8765/hands-on Ctrl-C to stop.
46. 46 http://localhost:8765
47. 47 http://localhost:8765/hands-on
48. Step 7: a little bit more professional way to run the server 48 $ npm install -g pm2 $ pm2 start index.js -name spinel $ open http://localhost:8765/hands-on # when you want to monitor the server $ pm2 monit # when you want to stop and delete the server $ pm2 stop all $ pm2 delete all You don’t have to do it now. You will use the server in the next stage.
49. The stages 49 1. Get source data 2. Add design properties 3. Build vector tilesets 4. Host the vector tilesets 5. Style the hosted vector tiles
50. Step 8: install Maputnik 50 $ cd {your-workspace} # probably “cd ..” $ git clone https://github.com/maputnik/editor $ cd editor [editor]$ npm install $ open http://localhost:8888 $ open http://localhost:8888?style=http://localhost:8 765/hands-on/style.json [editor]$ npm start
51. http://localhost:8888 51
52. http://localhost:8888?style=http://localhost:8765 /hands-on/style.json 52
53. 53 More individual challenges
54. Individual challenges 54 1 Additional tutorials 2 Individual challenges
55. Additional tutorials 55 6. Evaluate the size of vector tiles 7. Convert Shapefile to NDJSON 8. Extract OpenStreetMap data
56. Additional hands-on exercise 56 6. Evaluate the size of vector tiles 7. Convert Shapefile to NDJSON 8. Extract OpenStreetMap data
57. Step 9: evaluate the size of vector tiles 57 $ cd {your-workspace} # probably “cd ..” $ git clone https://github.com/ibesora/vt- optimizer $ cd vt-optimizer $ npm install $ node index.js –m ../spinel/mbtiles/hands-on/6- 46-30.mbtiles
59. Additional technical introduction 59 6. Evaluate the size of vector tiles 7. Convert Shapefile to NDJSON 8. Extract OpenStreetMap data
60. Step 10: Convert Shapefile to NDJSON 60 Recommendation A: Use ver. 2.4.0dev of GDAL Challenge: RFC 8142 is still a new format for most software. Recommendation B: Use node-shapefile and script in Node.js Example: https://github.com/hfu/natural-coast Application code example: https://github.com/hfu/shapefile-pass-through https://github.com/mbostock/shapefile
61. Additional hands-on exercise 61 6. Evaluate the size of vector tiles 7. Convert Shapefile to NDJSON 8. Extract OpenStreetMap data
62. Step 10: Extract OpenStreetMap data 62 Source: https://github.com/hfu/extract-all/blob/master/index.js e.g. Use Osmium Extract Then use Osmium Export to convert OSM PBF to NDJSON
63. I have the following extracts for your testing purpose 63 Today, you can get these modules from http://first.vectortiles.xyz:8765/modules/{z}-{x}-{y}.pbf e.g. http://first.vectortiles.xyz:8765/modules/6-29-30.pbf
64. 64 Individual challenges suggestions
65. Example for individual challenges 65 # challenge difficulty 1 Change the style of the map EASIER 2 Change vector tile schema MODERATE 3 Add your own data HARD 4 Add HTTPS or SPDY for better interoperability and performance HARD
66. 66 Comments and wrap-up 16:40
67. 67 Group Photos, if we may?
68. 68 Notes for tutors
69. Setup procedure for the “lifeboat” 1. Set-up the workshop LAN. 2. Start a UNIX server inside the LAN. 3. [op] Check the IP address of the server and provide a domain name using a public DNS for the ease of access. 4. Prepare tickets that contains account information, e.g. [user1, some_fancy_password] and setup such temporal accounts on the server. 5. If a participant want a ”lifeboat”, provide the ticket and allow the participant to access the server. ü If the participant does not have an ssh client, the participant may want to install “Chrome Secure Shell Extension” that provide an ssh client on top of Chrome. 69
70. Providing a ssh server Set up a server ifconfig | less to find the local IP address, probably 192.168.0.{some} sudo service ssh start sudo adduser user1 Test a connection from the client ssh –l user1 192.168.0.{some} After the session sudo deluser user1 sudo rm –rf /home/user1 70 Similar on mac, but use Settings to start the server.