2023-02-09_VectorTilePractice

Vector tile creation practice from PostGIS database. A meeting material with my colleagues.

Transcript

1. Vector tile creation - An example for small scale data Technical meeting 9 February 2023 Taro Ubukawa UN Geospatial Information Section Note: Some contents were removed to release this slid to the public. I am trying to be as much as accurate, but please understand that there is no warranty. Your use of this knowledge is solely at your own risk.
2. Index • Practice 1 • Practice 2 • test001.js • test002.js • test003.js and test004.js • test005.js • test006.js • Summary
3. Practice 1: Conversion in DEV server Trial with our server in DEV • Please access our server > ssh user@(hostserver) (enter password) Please move to “/home/vectortile/data_conversion/produce-gsc-6” The repository “produce-gsc-6” is already installed.
4. Please move to “/home/vectortile/data_conversion/produce-gsc-6” > cd data_conversion/produce-gsc-6 > node produce-clearmap/index-clear.js After about 71 seconds, you will obtain 0-0-0.mbtiles. Practice 1 (cont.)
5. Did you see it?? If you want to create it every day, try “crontab” 0-0-0 means tile xyz, in the order of z-x-y.
6. (Tips) What is mbtiles? • Mbtiles is a SQlite data base that stores several vector tiles. • (A single mbtiles file can compile several pbf vector tiles. • c.f. An ArcGIS vector tile server stores all pbf tiles as a bundle file. • From my experience, pbf tiles in directory are 3 to 4 times larger than mbtiles format. (But “pbf” is good for static hosting.) • Try following command to make pbf tiles from mbtiles. (tile-join is a part of tippecanoe) > cd produce-clearmap/clearmaptile > ls > tile-join -e test-zxy --no-tile-size-limit --no-tile-compression -f 0-0-0.mbtiles Mbtiles format Pbf format PMTiles format Many files No compression Static hosting Single file Compression Hosting script needed Single file Compression Static hosting is ok Tippecanoe v2.17 or later needed
7. (Tips) Adjustment You do not have to change it so often. But if you want to edit the vector tile structure or modify PG database access information: • Config parameters (list of input view/table) are in config/default.hjson • Adjust the input view/table • PostGIS database access information • Adjustment of attribution should be written in /home/vectortile/data_conversion/produce-gsc-6/produce- clearmap/modify.js • Vector tile layer name. • Editing the attribution, etc. FYI. Advanced content
8. <Step 1> • Open any terminal (e.g. Windows PowerShell) • Go to your work directory. • Clone the repository (With git, or download from github) • https://github.com/unvt/produce-cmv • Move to that repository, and install npm packages (npm install) Practice 2. Accessing PostGIS from your PC with node (Command) > git clone https://github.com/unvt/produce-cmv > cd produce-cmv > npm install You can download it from website, too.
10. • Please create config/default.hjson • Copy config/default-sample.hjson, then edit the necessary information. Practice 2 (cont.) I reduced the number of view for easy test. You can add osm_planet as well. Please add connection information for your PostGIS database. (OSM is not needed if you do not list them in relations.)
11. Practice 2 (cont.) • Please run test001.js • This is to show the number of records in each view.
12. • https://github.com/unvt/produce-cmv/blob/main/test001.js  Getting the response of sql as “res”  Nodejs modules  Defining sql  Getting view list  Creating connection info  Showing the result. FYI. Advanced content • pg is a npm module to access PostGIS server. • config is a npm module to provide config parameters from config/default.hjson (hjson is also needed.)
13. Practice 1 (cont.) • Please run test002.js • This is to show the first record in each view. • geom is written in geojson way.
14. FYI. Advanced content  First SQL to get the column list  Second SQL to get the column (LIMIT 1) • https://github.com/unvt/produce-cmv/blob/main/test002.js
15. Practice 2 (cont.) • Please run test003.js, then run test004.js • These two will take longer time. Please limit the number of views. (or check if your data is not too large) • Test03.js uses CUR to control the data flow. • Test04.js. We also use modify.js to edit the output flow.
16. You will find that test004.js has “tippecanoe” information. FYI. Advanced content And other unnecessary attributions are removed. This is the basic input to Tippecanoe to be converted into vector tile.
17. Practice 2 (cont.) • Please run test005.js • Then, you will get exported GeoJSON as text. • Test005.js would be faster than test004.js because writing to txt is faster than showing it in stdout.
18. Practice 2 (cont.) • You cannot run test006.js without Tippecanoe installed. • Test006.js should be run in the environment with Tippecanoe. • You will get output vector tile like you did in the practice 1. Test# Output stream Output test004.js stdout Console test005.js fs.write (fs is a npm package) text file test006.js tippecanoe.stdin vector tile
19. Summary • In the DEV server, we saw how we can create vector tile for small scale, i.e. for clearmap. • In our own PC, we tried accessing to PostgreSQL/PostGIS server with nodejs, and tried editing exported data sequence. • We cannot create vector tile in our own PC due to the lack of Tippecanoe access, but we can create vector tile in Linux, Docker or WSL environment, where we can use Tippecanoe. Output stream should be piped into Tippecanoe.stdin instead of stdout.
20. One more thing • Today’s practice is just for small scale with a simple code, meaning one single spatial ranges. • For vector tile conversion in larger scale, we will have large amount of data and we, therefore, need to use spatial extents to efficiently create vector tile. These scripts are in https://github.com/unvt/produce-gsc-6
21. References • https://qiita.com/T-ubu/items/3855054e394e3e7518fe • https://github.com/unvt/produce-cmv • Pictures from https://www.irasutoya.com/