Upgrade to Pro — share decks privately, control downloads, hide ads and more …

The OSGeoJP Workshop for the UN Vector Tile Too...

The OSGeoJP Workshop for the UN Vector Tile Toolkit

Hidenori FUJIMURA

December 02, 2018
Tweet

More Decks by Hidenori FUJIMURA

Other Decks in Technology

Transcript

  1. OSGeo.JP Workshop for the UN Vector Tile Toolkit – Let’s

    make sustainable web maps Supported by OSGeo.JP 2018-12-02T13:30/17:00 @Moratuwa University
  2. Today’s schedule 2 1 Introduction 13:30 2 Fully guided hands-on

    14:00 3 More individual challenges 15:30 4 Comments and wrap-up 16:40
  3. 4 By the end of the workshop, You create a

    web map as good as this. Or you can make it even better.
  4. Questionnaire 5 Q1: What Operating System do you mainly use

    today? ü Windows 10? ü Windows 7? ü OS X? ü Linux? ü Others? Q2: What Browser do you use today? (choose all you want to use) ü Google Chrome? ü Firefox? ü Safari? ü Microsoft Edge? ü Others?
  5. 7 Natural Earth Natural Earth Global Map Survey Department, Sri

    Lanka OpenStreetMap OpenStreetMap contributors Integrated into a single vector tileset using the UN Vector Tile Toolkit
  6. 11 Reference: https://tmcw.github.io/presentations/jsgeo 1 Untiled WMS Selection by BBOX 2

    Image tiles Predictable & cached 3 Tiles and overlays Vector retrofit 4 Vector tiles Everything in vector Evolution of web maps for performance
  7. 12 Reference: https://tmcw.github.io/presentations/jsgeo/#33 Image tiles are facets of a gigantic

    image Vector tiles are facets into datastructure, a straightforward way to vector geospatial data.
  8. 14 9.C Significantly increase access to information and communications technology

    and strive to provide universal and affordable access to the Internet in least developed countries by 2020 + Vector Tiles even saves the bandwidth use. Vector tiles are gradually getting ready for everyone.
  9. UN Vector Tile Toolkit Objective To facilitate the production of

    vector tiles for basemaps by public organizations. Structure Import Produce Host Style Optimize Existing datasource e.g. PostGIS, Shapefile, GeoJSON Tippecanoe Maputnik Interoperable with both Open Source and other frameworks. Tailor-made Node.js scripts Openly maintained on GitHub. Maximum use of existing OSS while adding tailor-made scripts shared openly. vt-optimizer 15 You have accessed this in the step 0.
  10. 16 Technical details Import data and assign layer names and

    zoom levels. Custom open source scripts. Create vector tile packages in mbtiles format. Tippecanoe Host modular vector tile packages in a simple and scalable manner. Custom open source scripts. Style hosted vector tiles. Maputnik Enhance the optimization of the size of individual vector tiles by visualizing them. vt-optimizer and other custom open source scripts. Import Produce Host Style Optimize
  11. 17 The UN Vector Tile Toolkit is a collection of

    tools, rather than a monolithic DB/Server suite. Because: ü Inputs are diverse. ü Depolyment environments are diverse. Flexibility is the core value. And sometimes the server shall be isolated from the master database (for security and scarability).
  12. The stages 19 1. Get source data 2. Add design

    properties 3. Build vector tilesets 4. Host the vector tilesets 5. Style the hosted vector tiles
  13. The stages 20 1. Get source data 2. Add design

    properties 3. Build vector tilesets 4. Host the vector tilesets 5. Style the hosted vector tiles
  14. 22 [Please check your workspace] You probably need about 2GB

    for your work area. This material refers the path to your work area as ${your-workspace}
  15. Step 1: setup scripts to produce vector tileset 23 $

    cd ${your-workspace} $ git clone https://github.com/un-vector-tile- toolkit/spinel-produce $ cd spinel-produce Download and install the scripts for producing vector tiles. If this does not work… ü Install git. ü Setup git (next slide). [spinel-produce]$ npm install If this does not work… ü Setup Node.js (next to the next slide)
  16. Setup Node.js 25 ü Recommendation: sudo apt install nodejs Ø

    In Ubuntu, you may want to run the following command, too. curl -sL https://deb.nodesource.com/setup_10.x | sudo -E bash - For other Linux-based platforms: https://github.com/nodesource/distributions
  17. Or you still have a problem with “git clone”… 26

    You can download a ZIP file from https://github.com/{user}/{repo} In this case https://github.com/un-vector-tile-toolkit/spinel-produce
  18. Step 2: get the source data 27 [spinel-produce]$ node download.js

    # If you have time… [spinel-produce]$ gzcat src/6-46-30.ndjson.gz [spinel-produce]$ gzcat src/gmlk20.ndjson.gz This may take time depending on the Internet bandwidth. These are Newline Delimited GeoJSON features. i.e. each line is a GeoJSON Feature. This format is also called GeoJSON Text Sequences (RFC 8142) with Record Separator (RS) omitted.
  19. The stages 29 1. Get source data 2. Add design

    properties 3. Build vector tilesets 4. Host the vector tilesets 5. Style the hosted vector tiles
  20. Step 3: Add design properties 31 # If you have

    time… [spinel-produce]$ less 6-46-30.ndjson [spinel-produce]$ node import.js > 6-46- 30.ndjson
  21. Break Point A: definition of what3numbers, import.js, and modify.js 32

    what3numbers import.js modify.js The number assigned to a tile (in slippy map tilenames) used as the extent of the vector tile package. e.g. 4-46-30 overwraps Sri Lanka. The code to read each features, apply “modify function” to add design properties (layer name, minimum zoom and maximum zoom), and store the resulting features to NDJSON. The code that contains a function to modify a feature by adding design properties and edit other properties of the feature. You can edit this to change the “vector tile schema”.
  22. 33

  23. The stages 35 1. Get source data 2. Add design

    properties 3. Build vector tilesets 4. Host the vector tilesets 5. Style the hosted vector tiles
  24. Step 4: build vector tilesets 36 [spinel-produce]$ tippecanoe -f --read-parallel

    --drop-densest-as-needed --simplification=2 -- minimum-zoom=6 --maximum-zoom=15 --base- zoom=15 --output=6-46-30.mbtiles 6-46- 30.ndjson This may take time. You’re rendering areas as big as Sri Lanka. If this does not work… ü Install Tippecanoe (next slide)
  25. Troubleshooting: install Tippecanoe 37 $ cd ${your-workspace} $ git clone

    https://github.com/mapbox/tippecanoe $ cd tippecanoe [tippecanoe]$ make –j [tippecanoe]$ sudo make install # Or you can just use ./tippecanoe without make install [tippecanoe]$ tippecanoe -v If this still does not work, see the next slide.
  26. Troubleshooting in installing Tippecanoe • If you see errors at

    “make -j”, posslble solutions are: • Install g++ • Install libsqlite3 38 $ sudo apt install build-essential $ sudo apt install libsqlite3-dev zlib1g-dev For details, see https://github.com/mapbox/tippecanoe#development
  27. 39 This is a vector tile package that contains the

    module 6-46-30. Format: mbtiles-spec (CC-BY) 6-46-30.mbtiles
  28. Step 5: Quick-check the content of vector tileset 40 $

    sudo npm install –g tileserver-gl-light $ tileserver-gl-light 6-46-30.mbtiles $ open http://localhost:8080 Please see a workaround if you see errors (e.g. in Ubuntu) Don’t worry if tileserver-gl-light doesn’t work. You can simply skip this step.
  29. tileserver-gl-light install error workaround 41 $ sudo npm install -g

    tileserver-gl-light --unsafe-perm=true --allow-root Source: https://github.com/npm/npm/issues/17268
  30. 42

  31. On the console 43 Ctrl-C to stop. The vector tiles

    you created were sent to the browser.
  32. The stages 44 1. Get source data 2. Add design

    properties 3. Build vector tilesets 4. Host the vector tilesets 5. Style the hosted vector tiles
  33. Step 6: install the server script 45 $ cd {your-workspace}

    # probably “cd ..” $ git clone https://github.com/un-vector-tile- toolkit/spinel $ cd spinel [spinel]$ npm install [spinel]$ mv ../spinel-produce/6-46-30.mbtiles mbtiles/hands-on [spinel]$ node index.js spinel vector tile server starts on 8765 # After you see “spinel vector tile server starts on 8765” on the original console; $ open http://localhost:8765 $ open http://localhost:8765/hands-on Ctrl-C to stop.
  34. Step 7: a little bit more professional way to run

    the server 48 $ npm install -g pm2 $ pm2 start index.js -name spinel $ open http://localhost:8765/hands-on # when you want to monitor the server $ pm2 monit # when you want to stop and delete the server $ pm2 stop all $ pm2 delete all You don’t have to do it now. You will use the server in the next stage.
  35. The stages 49 1. Get source data 2. Add design

    properties 3. Build vector tilesets 4. Host the vector tilesets 5. Style the hosted vector tiles
  36. Step 8: install Maputnik 50 $ cd {your-workspace} # probably

    “cd ..” $ git clone https://github.com/maputnik/editor $ cd editor [editor]$ npm install $ open http://localhost:8888 $ open http://localhost:8888?style=http://localhost:8 765/hands-on/style.json [editor]$ npm start
  37. Additional tutorials 55 6. Evaluate the size of vector tiles

    7. Convert Shapefile to NDJSON 8. Extract OpenStreetMap data
  38. Additional hands-on exercise 56 6. Evaluate the size of vector

    tiles 7. Convert Shapefile to NDJSON 8. Extract OpenStreetMap data
  39. Step 9: evaluate the size of vector tiles 57 $

    cd {your-workspace} # probably “cd ..” $ git clone https://github.com/ibesora/vt- optimizer $ cd vt-optimizer $ npm install $ node index.js –m ../spinel/mbtiles/hands-on/6- 46-30.mbtiles
  40. 58

  41. Additional technical introduction 59 6. Evaluate the size of vector

    tiles 7. Convert Shapefile to NDJSON 8. Extract OpenStreetMap data
  42. Step 10: Convert Shapefile to NDJSON 60 Recommendation A: Use

    ver. 2.4.0dev of GDAL Challenge: RFC 8142 is still a new format for most software. Recommendation B: Use node-shapefile and script in Node.js Example: https://github.com/hfu/natural-coast Application code example: https://github.com/hfu/shapefile-pass-through https://github.com/mbostock/shapefile
  43. Additional hands-on exercise 61 6. Evaluate the size of vector

    tiles 7. Convert Shapefile to NDJSON 8. Extract OpenStreetMap data
  44. I have the following extracts for your testing purpose 63

    Today, you can get these modules from http://first.vectortiles.xyz:8765/modules/{z}-{x}-{y}.pbf e.g. http://first.vectortiles.xyz:8765/modules/6-29-30.pbf
  45. Example for individual challenges 65 # challenge difficulty 1 Change

    the style of the map EASIER 2 Change vector tile schema MODERATE 3 Add your own data HARD 4 Add HTTPS or SPDY for better interoperability and performance HARD
  46. Setup procedure for the “lifeboat” 1. Set-up the workshop LAN.

    2. Start a UNIX server inside the LAN. 3. [op] Check the IP address of the server and provide a domain name using a public DNS for the ease of access. 4. Prepare tickets that contains account information, e.g. [user1, some_fancy_password] and setup such temporal accounts on the server. 5. If a participant want a ”lifeboat”, provide the ticket and allow the participant to access the server. ü If the participant does not have an ssh client, the participant may want to install “Chrome Secure Shell Extension” that provide an ssh client on top of Chrome. 69
  47. Providing a ssh server Set up a server ifconfig |

    less to find the local IP address, probably 192.168.0.{some} sudo service ssh start sudo adduser user1 Test a connection from the client ssh –l user1 192.168.0.{some} After the session sudo deluser user1 sudo rm –rf /home/user1 70 Similar on mac, but use Settings to start the server.