DEPARTMENT OF OPERATIONAL SUPPORT UN VECTOR TILE TOOLKIT (UNVT) DEPLOYMENT IN UNGSC Technology and partnership - Wise use of open source tools for the operation support Taro Ubukawa (He/him) Senior Geospatial Expert, UN Geospatial Information Section At Unite Maps Conference 16 December 2021
Background – UN context (Partnership for Technology in PK, UN Geospatial Strategy, UN Open GIS) – Vector tile context • The Deployment Project – Overview – Vector tile cycles (Production, Styling, Hosting, Use) • Benefits – The added values of the project to the VT production in UN – The benefits from the projet to the world • Lessons leaned and Conclusion
2014 • Matching between challenges in Peacekeeping and solutions from the partners (member states, etc.) - Sometimes, we can find a solution from the partners that is not obtained from the market. I think partners can also learn from us. • UN Open GIS Initiative is one of the initiatives under this partnership. • Since March 2016 • An ongoing partnership Initiative for Technology in Peacekeeping. - Its uniqueness, I think, is that it put great emphasis on the use of open-source tool. • It aims to identify and develop an Open Source GIS bundle that meets the requirements of UN operations for both peace-building and peace-keeping. • It is released in 2021, aiming to further contribute to the goals described in the Data Strategy of SG. • It aims to set directions to support the implementation of the mandate of the UN Secretariat in the geospatial field. • It has 4 goals: - Provide universal fundamental global geospatial data, standards, methods, tools, and services - Enhance geospatial information benefits for mandate, operational and decision-making requirements - Increase capacity, expertise and integrated geospatial systems across the Organization - Raise awareness on geospatial information and harness international partnerships Background (1): UN Context Why Vector Tile Technology in UN Partnership for Technology in Peacekeeping Geospatial Strategy for the United Nations UN Open GIS Initiative https://operationalsupport.un.org/en/partnership-technology-peacekeeping#:~:text=The%20Partnership%20for%20Technology%20in,and%20technologies%20that%20have%20the https://www.un.org/geospatial/sites/www.un.org.geospatial/files/20210330_geospatial_strategy_for_the_united_nations-book.pdf , http://unopengis.org/unopengis/main/main.php Take-Away: We have a good strategy and framework to advance our technology with open-source.
Vector Tile context Rapid Development of Vector Map Technology in Web Mapping Vector Tile as an innovative way of geospatial information dissemination. https://speakerdeck.com/hfu/unvt-workshop-introduction-and-application Take-Away: We can enjoy the modern technology of web mapping to support UN operations • Machine readable • Data size is significantly smaller (but the designing of the tile is important!) • Flexible designing (one source can have many styles) • Advanced techniques, like voice reading, 3D expression, etc., are also available • Many geospatial organizations have started using this techniques to disseminate base maps Image from Fujimura (2021, UNVT workshop) Vector tile from GSI Japan Vector tile from Ordnance Survey, UK Image from Fujimura (2021, UNVT workshop)
Vector Tile Toolkit Deployment Project in UNGSC • By deploying UNVT, we can achieve efficient vector tile production and dissemination (in terms of cost, update frequency, tile design flexibility, various visualization, data consumption) from PostGIS data base in UNGSC. OSM source UN source Produce Host Style Optimize Import UNVT workflow of developing vector tile for base map Supported by UNVT, open source scripts Source DB (PostGIS) Users • Web APP • GeoPortal • Etc. Progress: Started in January 2020 • Production: Working with the data team, method was developed for the prod source. Daily/Weekly update of the mission area/world. • Styling: Thanks to the support by the visualization team, style was prepared, and being refined (can be seen in dev.) • Hosting: Nodejs/express server with Azure AD authentication was developed, but it still need some adjustment with UN GeoPortal. Way forward: • Further Work on hosting/consumption so that we can build servers in prod env. • Optimizing vector tile structure. Etc.
- Production • Semi-automatic production (as a scheduled task of Linux server) • Daily update of the Priority are (Blue) • Weekly update of the other - UN sourced data will be updated whenever necessary • If we convert whole globe at once: - 35 straight hours up to ZL15 - 50 straight hours up to ZL 16 • 841 mbtiles covering whole globe. - 130 GB in total up to ZL15 - 330 GB in total up tp ZL 16 • About 40 Vector tile layers from about 70 tables/views in the source PostGIS data base • Customized UNVT conversion tool (CLI) - Stored in UNVT’s repository (GitHub) - Tools used: – Nodejs (pg module) – Tippecanoe, Tile join (mapbox) - Extracted GeoJSON sequence is edited, and directly goes into the data conversion tool. What: VT of the whole world How: with Open Source Tools Produced Vector Tile Source (PostGIS) Vector Tiles (mbtiles) nodejs Tippica noe Repository: https://github.com/un-vector-tile-toolkit/produce-gsc-5 Area - ZL15 - ZL16 Priority 2h48m 3h16m 1 4h24m 5h53m 2 4h19m 5h31m 3 4h27m 5h12m 4 5h49m 7h27m 5 4h03m 6h25m 6 3h47m 6h14m 7 5h03m 8h11m Processing Time:
we design our vector tile in efficient manner? UNVT added value. Tips: Let’s think about the vector tile structure • Some proprietary tool does not have a flexibility of designing vector tile layer. It automatically converts the source into the vector tile. • Careful designing of the vector tile is important for your success. Vector Tiles do not necessarily have the same structures with its source. • Open-source tool provides a flexibility. We can select, edit, eliminate features from multiple source layers into a single vector tile layer - The number of the vector tile layers becomes less, then we need less layers for map styling. This results in quicker map drawings. Flexible structure of Vector tile layer • During the production, or designing the vector tile, we can check the size of vector tiles, then we can think about how we can reduce the size of the vector tile so that we can achieve the efficient dissemination. - I think this is the process that requires our expertise as GIS or geospatial experts. Manual work is also important Check VT size, and improve!! An example at the area 5-20-15 Source View VT layer Min Zoom Max Zoom Note unmap_bndl25_l bndl 0 2 National level unmap_bndl05_l 3 4 National level custom_unmap_bndl_l 6 15 minZ is based on admin class. unmap_bndl_l 5 15 un_unmik_bndl_l 7 15 un_unvmc_igac_bndl_l 6 15 An example: Different source views can be merged into one.
- Styling •JSON file based on Mapbox style specification (or MapLibre style specification) •Wise use of YAML files increases efficiency •We use UNVT/charites Style files • JSON format - Many lines (thousands) - In a single file - Hard to read/edit Efficient Styling with UNVT/Charites – Use of YAML files A story about the tool development Our tool Charites • Originally, we used HOCON parser to edit the style. These work was reported at UNVT workshop in May 2021 • Our partner, Geolonia, supported the shared idea, and contributed to develop a tool with YAML files. They contributed their tool at the UN Open GIS monthly meeting in October 2021. Let’s edit with • YAML format - Human readable - Structured file - Re-usable - Stored in the series of config files https://github.com/unvt/charites (Image from Geolonia)
the support and existing style by the visualization team, we developed the style for our open-source vector tile. – The VT structure is not exactly the same, we may need to further adjustment. Styling: Open source tool did not start from the scratch Used the common style Vector tile map in Geoportal (ArcGIS Online) Open Source Vector tile displayed in Geoportal (ArcGIS Online)
- Hosting • Simple and scalable server with the following functions: - Vector tile delivery - Web Map App (mapbox gl js v.1.x) - Vector Tile sever for Geoportal • Learning ArcGIS REST API, some json files are prepared so that we can consume our vector tiles from the UN Geoportal (ArcGIS online) • Nodejs MSAL package is used to add Azure AD authentication to the server. (Microsoft is providing nodejs module “msal” for the Azure AD authentication.) • We need a little more work to integrate this authentication with the authentication of the Geoportal. Hosting Server with nodejs Connection with Azue AD Adjustment for Geoportal Efforts to integrate open-source tools and proprietarily software Not only for ESRI, but also for other libraries! 3D view with mapbox gl js (ver. 1.x ) ESRI server Open source server Expected Library ArcGIS API for JavaScript Mapbox GL JS / MapLibre, etc TileSever Info JSON JSON is prepared for ESRI service VectorTile …/{z}/{y}/{x}.pbf (API returns from bundle file) …/{z}/{x}/{y}.pbf (Server returns from mbtiles file) Style A JSON in resources/styles A JSON at certain place Font Pbf files in resources/fonts PBF files at certain place Sprites (symbol) Image with JSON in resources/sprites Image with JSON in certain place Over zooming Using tilemap API Tile map will be prepared for ESRI service Authentication ESRI Geoportal Authentication Azure AD (MSAL)
Tile - Hosting (can be hosted by a small PC) • Vector Tile hosing with a small and single board computer We are learning some lessons/experience from the partners https://x.optgeo.org/a-1/#15.04/-69.00497/39.58404 Our partners, colleagues in GSI of Japan, are now hosting vector tile map around their Antarctic station with a Raspberry Pi as a server. This is because they can easily bring this web map to the area without the Internet connection. • Raspberry Pi - Around $100 - Small and single board PC Some News Did you know? Another example of application by our partner. • We can make box-cells with height from points clouds obtained from UAVs. They can be vector tiles (hosted raspberry pi). https://x.optgeo.org/kid- c/#16.41/35.155864/138.777703/73.2/60
Tiles can be consumed in various map libraries. • We also provides a web map to see the map. • Our tiles can be also consumed in ArcGIS based GeoPortal. • With a vector tile map, we can make a simple storytelling map. • We delivered a workshop on 22 October 2021. Vector Tile - Data Use With headlines As web map Storytelling map With Geoportal I will show a demo
the added value? And what is our challenge? Open Source tool provides some values to your important work of map delivery for operation supports. • Production – Better update frequency with semi-automatic process – Less cost for the software tools – Enabling a flexible design of vector tiles • Styling – Efficient styling with a tool jointly developed with the community • Hosting – Providing the data not only to the ESRI based geoportal, but for other libraries. (Expected use of vector tile beyond the ESRI geoportal.) • Integration with the proprietary software - Azure AD authentication - Geoportal (ESRI Geoportal) • Further refining the vector tile structure • Further refining the vector tile styles Challenges Benefits
the project: Growing techniques together with the Community With the people in the world (including non-UN colleagues) With UN colleagues • MS Teams channel • Workshops - UNVT workshop - UNVT exercises - UNVT storytelling workshop • We shared our experience through - Joining FOSS4G 2019, 2021 - UN Open GIS - Students use UNVT - (Possibly with Member States through UN-GGIM WG-Disasters, etc.) Even if we cannot share the data, we can share our techniques and experiences. Thank you for your interest and participation!
learned: • Open source tools may possibly provide various values to your work, such as cost-efficiency, innovative techniques, flexibilities, etc. • We sometimes need to be patient because we need to jointly explore the method with the community. • Open Source is not always free. We may need to allocate our resources including our time, own technical development, etc. • It is important to share any feedback to the community. Working with a good community is also important. Such community may provide an opportunity to catch up with the latest technology and to find the latest trend. • Use of the open source itself is not the purpose. Our aim is to add the values to our work via open source. We work should be focus on operation. Conclusion of this presentation: • I presented a specific example of the use of open source to support your work. • Through my presentation, I shared some example of advantage of wisely using the open source tools. • I would like to further encourage your active participation to our endeavors with open source, such as UN Open GIS Initiative, to enhance our geospatial information service. Lessons we learned and Conclusion This is the end of my presentation!! Take-Away: I am always happy to talk/consult with you about open source vector tile.
DEPARTMENT OF OPERATIONAL SUPPORT THANKS FOR WATCHING I FULLY SUPPORT THE IMPORTANCE OF “PARTNERSHIP FOR TECHNOLOGY” TROUGH THE OPEN SOURCE APPROACH ANY QUESTION? PLEASE FEEL FREE TO CONTACT ME AT [email protected]
ubukawa
maggiecrowley
akmur
reverentgeek
rasmusluckow
addyosmani
nonsquared
sachag
skipperchong
kastner
jponch
jakevdp
bermonpainter
