2021-07-08_UNVT exercise day04

Material for a casual vector tile exercise (Day 04). Exercise of Vector tile hosting using nodejs.

Transcript

1. UNVT Exercise Day 4 8 July 2021 By Taro Ubukawa (Senior Geospatial Expert at UN Geospatial Information Section)
2. Whole schedule So far, about vector tile (VT), we have practiced the followings: • VT consumption in GIS tools like QGIS. • Simple VT tile styling. • VT hosting as static file. • VT data conversion. • Simple conversion • Advanced conversion with nodejs Today’s content: • Vector tile hosting • Delivering VT from mbtiles. You can launch your local server today. This means that you can consume your VT even without the access to the Internet.
3. Today’s contents - Data hosting (advanced) • Part 1: Establish a vector tile server in your local environment. • Part 2: Explanation how we can add Azure AD authentication to the server. Produce Host Style Optimize Import UNVT workflow of developing vector tile for base map Supported by UNVT, open source scripts Source DB Web APP Users Our focus today.
4. Do you know you can easily create a web server with nodejs?
5. Part 1: Establish a vector tile server in your local environment.
6. Part 1 (1): Check before you start. Today, the exercise is mainly for the windows users. • Can you use nodejs (ver 12.x) in your CLI? → If yes, you do not have to use Docker (unvt/nanban) → If no, let’s use Docker. Tips: You may remember, for data conversion, a windows user may need to use Docker (or WSL) to run the conversion tool “Tippecanoe.” But for hosting, nodejs is good enough. Tips: Let’s use nodejs ver. 12.x. Ver. 13 or later may not work well with certain package we use.
7. Part 1 (2): Prepare exercise data • Copy data from the MS Teams channel. Or download it at https://github.com/unvt/ezPrac (ex4-data) * mbtiles is not included in GitHub repository • Start command prompt (I used Windows Powershell), and move to the working directory.
8. Part 1 (3): The goal of this exercise • Using nodejs, we will establish a vector tile hosting server in local environment. • Port: 8836 (try http://localhost:8836/index.html when done) • Static hosting directory: htdocs • Log directory: log • Mbtiles Directory: mbtiles • Pbf file is delivered from mbtiles. • Confirm there is no pbf tile in the folders. We have only one mbitles file in the mbtiles directory. • The tile will served from http://localhost:8836/zxy/(mbtiles name)/{z}/{x}/{y}.pbf. • We will see our web map there. • http://localhost:8836/zxy/(mbtiles name)/{z}/{x}/{y}.pbf.
9. Part 1 (4): Start Docker (IF you do not use Docker, skip the first three lines.) • Then, start Docker (unvt/nanban), install tools > docker pull unvt/nanban > docker run -it --rm -v ${PWD}:/data -p 8836:8836 unvt/nanban > cd /data > npm install > npm install sqlite3 Here, I do not explain the preparation of the Docker environment. Please see https://docs.docker.com/docker-for-windows/install/ Tips: This time, you are specifying the port. Tips: Sqlite3 may be needed
10. Example:
11. Part 1 (5): Check if you are ready • Please confirm: • If you are at the exercise directory and can see all necessary files • Move to /data and check if you can see “package.json” • Try “node –version ” to see if you can use nodejs. (ver 12.x)
12. Part 1 (6): Confirm/edit config setting • Confirm/edit “default.hjson” in config directory. For this practice, you can use the current setting as is. • We do not have to care all components, but you can specify: • static hosting folder (htdocsPath), • Mbtiles directory (mbtilesDir), • log directory (logDirPath), • port for http, etc. • If you want to use https instead of http, please edit SSL/TLS key locations, store them in that place, and edit app.js.
13. Part 1 (7): Start the server • Run the server > node app.js * Ctrl + c to stop • Access the index.html to confirm if the server is working. http://localhost:8836/index.html • Try if you can download VT. http://localhost:8836/zxy/clearm ap-test/1/1/1.pbf * If there is a mbtiles file, pbf files are generated from it.
14. Part 1 (8): What we did? • Establish http server (nodejs/express) and host a static contents in htdocs. • We are using @mapbox/mbtiles package, and pbf files are generated from the source mbtiles to be delivered upon requests. Vector Tile Server (nodejs/express) Style VT in mbtiles VT in pbf library Font, etc. htdocs folder (static) Mbtiles folder Tips: If the data is small, you can simply load VT as pbf data (a static content).
15. Technically in detail • See app.js Packages used Static hosting PBF tile request is routed to the function (gettile, getMBTiles). Then, reply with pbf file where possible. Routing pbf request We use spdy for https
16. Can you see the small scale map? • “clearmap-test.mbtiles” is just for internal practice. I do not upload this file to GitHub. Now you can make VT, style, html, etc. by yourself. So, you can make your own web map APP.
17. (FYI) You can bring your web map to the offline environment. We can possibly use the Unite Street Map offline as well. → Demo
18. Part 1 (9): How can we add other vector tile and map APP? • Make your vector tiles (pbf / mbtile) and put it to the folders: • pbf format: static hosting at “htdocs” → http://localhost:8836/(foldername)/{z}/{x}/{y}.pbf • mbtiles format: Store it at “mbtiles” to be delivered as pbf upon requested → http://localhost:8836/zxy/(filename)/{z}/{x}/{y}.pbf • You can also add style, fonts, html files, js libraries at htdocs folder. • Please review previous exercise if necessary. • How to prepare style file for a web map • How to prepare html file for a web map
19. Part 1 (10): Tips for further development • For Stable operation. • Run nodejs server using PM2 so that we can run it as a deamon process. • If you use Linux server, using crontab to re-start the server regularly would be a good idea. • Increase the source • We may use more than one mbtiles file for certain source.(e.g. our prototype has 841 mbtiles for a single source covering the whole globe). We can use module approach. Please look at https://github.com/un-vector-tile-toolkit/onyx
20. Part 1: Conclusion • Using nodejs, we can establish a vector tile hosting server in local environment. • We can deliver pbf files, generated from mbtiles, from the hosting server. • We can now make our own web map app by: • Preparing vector tile, • Designing style, • Preparing html and other necessary files
21. Part 2: Adding Azure AD authentication function.
22. Why Azure AD authentication? Why? • It is a kind of standard authentication used in UN. How? • There is a good tutorial of deploying Azure AD authentication with nodejs/express server. • https://docs.microsoft.com/en- us/graph/tutorials/node?WT.mc_id=Portal- Microsoft_AAD_RegisteredApps&tutorial-step=3
23. Tutorial by Microsoft • Basic Idea: We use MSAL Node (Microsoft Authentication Library for Node.js) to deploy login function. You need to register your portal at Azure Portal to get your APP ID, etc.
24. Customize for UNVT server • Once authentication is done, adding a routing for VT delivery For Static hosting You can add vector tile delivery script here But, preparing this by yourself would be tough work… So, let’s do with an easier way.
25. Repositories we have • UNVT coesite: vector tile server (onyx with Azure AD authentication for actual use) https://github.com/un-vector-tile- toolkit/coesite • UNVT azure-onyx2: Simpler than coesite https://github.com/un-vector-tile- toolkit/azure-onyx2
26. Summary • Part 1: You learned how you can establish a vector tile server using nodejs. • Part 2: You learned how we can add Azure AD identication to the nodejs server.
27. References • Install Docker Desktop on Windows: https://docs.docker.com/docker-for-windows/install/ • UNVT onyx: https://github.com/un-vector-tile-toolkit/onyx • @mapbox/mbtiles: https://www.npmjs.com/package/@mapbox/mbtiles • Maputnik editor: https://maputnik.github.io/editor/ • Microsoft, Build Node.js Express apps with Microsoft Graph: https://docs.microsoft.com/en- us/graph/tutorials/node?WT.mc_id=Portal-Microsoft_AAD_RegisteredApps • Fujimura, H. (2021): UNVT Workshop: Introduction and Application https://speakerdeck.com/hfu/unvt-workshop-introduction-and-application
28. More practice? • https://github.com/unvt/ezPrac