Building and Consuming Urban Digital Twins with Open-Source Tools

Slide 1

Slide 1 text

Building and Consuming Urban Digital Twins with Open-Source Tools Stefano Bovio Tobia Di Pisa Georgios Petrakis Digital Twin Toolbox 13th of June 2024

Slide 2

Slide 2 text

Introduction Digital Twin Toolbox 13th of June 2024

Slide 3

Slide 3 text

Introduction Consuming 3D data in WebGIS applications has increasingly become a requirement over the last years. 3D Tiles became one of the most common OGC standards for streaming and rendering 3D geospatial contents on the web such as: ● Photogrammetry like LiDAR-derived meshes ● 3D Buildings (.obj, .gltf, .glb ...) ● Point Clouds ● and more ... Digital Twin Toolbox 13th of June 2024

Slide 4

Slide 4 text

Introduction In response to ever-growing and more specific needs in this context, it is usually necessary to: ● Identify the best tools for viewing 3D data in 3D Tiles format ● Identify tools for converting datasets into 3D Tiles format in a correct and performing way but... what about doing that using Open Source tools? Digital Twin Toolbox 13th of June 2024 Use MapStore to visualize them first!

Slide 5

Slide 5 text

Introduction The Digital Twin Toolbox borns with the aim to support with the conversion processes to 3D Tiles by using Open Source tools: Digital Twin Toolbox 13th of June 2024 ● Defining pipelines for common data sources such as SHP and LAS files ● Providing necessary tools for visually inspecting and assessing datasets ● Allowing ways to manage classification, colorization and resampling ● Providing reliable tools to manage tiling options, CRS and georeferencing adjustments ● and many more…

Slide 6

Slide 6 text

Main objectives are: • Collect the best OS tools and libraries to process common data sources in the urban environment (SHP and LAS files for now) • Provide workflows to orchestrate a well-driven set of processing chains and methodologies to ○ Inspect and evaluate data ○ Prepare/process data ○ Convert input data in 3D Tiles ○ Preview data step by step • Provide an user friendly UI to facilitate the work All in a Dockerized environment! Introduction Digital Twin Toolbox 13th of June 2024

Slide 7

Slide 7 text

Use Case Digital Twin Toolbox 13th of June 2024

Slide 8

Slide 8 text

Use Case Objective: Reconstruction of a digital twin model in 3D Tiles format using sample data of the Washington D.C. city Digital Twin Toolbox 13th of June 2024

Slide 9

Slide 9 text

Use Case Steps overview: ● Download of data samples from the open data portal of Washington D.C. ● Data processing and 3D Tiles conversion using the open source digital- twin-toolbox solution ● Visualization of generated 3D Tiles inside a MapStore map viewer Data: ● Building volumes in Shapefile format (polygon geometries) ● Trees in Shapefile format (point geometries) ● Urban environment in LAS format in combination with orthophotos for colorization (point cloud structure) Link to sample data Digital Twin Toolbox 13th of June 2024

Slide 10

Slide 10 text

Use Case Digital Twin Toolbox is a collections of different tools/libraries and workflows inside a docker environment to generate 3D Tiles from common data sources such as Shapefiles and LAS files. This project is still a work in progress and this application is not production ready Digital Twin Toolbox 13th of June 2024

Slide 11

Slide 11 text

Use Case 3D Tiles format is designed for streaming and rendering massive 3D geospatial content such as Photogrammetry, 3D Buildings, BIM/CAD, Instanced Features, and Point Clouds Digital Twin Toolbox 13th of June 2024

Slide 12

Slide 12 text

Use Case MapStore is an highly modular Open Source WebGIS framework to create, manage and share maps, dashboard and geostories. The mapping engines currently supported are OpenLayers, LeafletJS and Cesium.js. Digital Twin Toolbox 13th of June 2024

Slide 13

Slide 13 text

Digital Twin Toolbox Setup Digital Twin Toolbox 13th of June 2024

Slide 14

Slide 14 text

Digital Twin Toolbox Setup Prerequisites: ● Docker - open platform for developing, shipping and running application ● Git - free and open source distributed version control system Setup steps: ● Clone the repository in your workspace directory git clone https://github.com/geosolutions-it/digital-twin-toolbox ● Navigate in the cloned digital-twin-toolbox repository cd ./digital-twin-toolbox ● Create an .env file by copying the provided .env.sample and renaming it cp .env.sample .env ● Start and build the application docker compose up Digital Twin Toolbox 13th of June 2024

Slide 15

Slide 15 text

Digital Twin Toolbox Setup Application by default runs at http://localhost:3000 Digital Twin Toolbox documentation: https://github.com/geosolutions-it/digital-twin-toolbox/wiki Digital Twin Toolbox 13th of June 2024

Slide 16

Slide 16 text

User Interface Digital Twin Toolbox 13th of June 2024

Slide 17

Slide 17 text

User Interface Controls: ● located on the top right corner of the screen ● change list of properties and action buttons based on the selected input file Digital Twin Toolbox 13th of June 2024

Slide 18

Slide 18 text

User Interface Data preview: ● covers all the screen background ● preview sample data in 3D ● live update of geometries for Shapefiles Digital Twin Toolbox 13th of June 2024

Slide 19

Slide 19 text

User Interface Process feedback: ● located on the top left corner of the screen ● shows logs for all the actions and processes initialized with the controls panel Digital Twin Toolbox 13th of June 2024

Slide 20

Slide 20 text

Workflows Digital Twin Toolbox 13th of June 2024

Slide 21

Slide 21 text

Workflows Digital Twin Toolbox provides following workflows: ● Conversion of shapefile data (polygons, lines, points) into 3D Tiles ● Processing point cloud data to fix/manage CRS, resample and colorization ● Conversion of point cloud data to 3D Tiles ● [Experimental] Conversion of lidar data to a 3D Mesh file and 3D Tiles Digital Twin Toolbox 13th of June 2024

Slide 22

Slide 22 text

Workflows The static/data/ folder contains all the input data (Shapefiles, LAS, …) The sample data needs to be unzipped and the content copied inside this folder: ● Shapefile as .zip folder with all sidecar files included ● Point cloud file as .las file ● Orthophoto for point cloud colorization as .tif files Digital Twin Toolbox 13th of June 2024

Slide 23

Slide 23 text

Workflows Conversion of shapefile polygons data into 3D Tiles Input data: Building_Footprints_join_Building_3D_EPSG_26985.zip Main tools dependencies: PostGIS, pg2b3dm Output: 3D Tiles Digital Twin Toolbox 13th of June 2024

Slide 24

Slide 24 text

Workflows Select Building_Footprints_join_Building_3D_EPSG_26985 (.zip) from input data section Data preview renders the building footprint surfaces Digital Twin Toolbox 13th of June 2024

Slide 25

Slide 25 text

Workflows Update the geometry by creating a volume based on properties: ● change the Upper limit height to create a volume using following expression: $minZ + (MAX_Z - MIN_Z) // $minZ is the average minimum z of the feature geometry (computed) // MAX_Z and MIN_Z are properties of the features Digital Twin Toolbox 13th of June 2024

Slide 26

Slide 26 text

Workflows Generate 3D Tiles using the default properties by clicking on the Create tileset button Tip: minimum and maximum geometric error could be adjusted to define the visibility range of the tileset in the map Digital Twin Toolbox 13th of June 2024

Slide 27

Slide 27 text

Workflows Preview of generated 3D Tiles in Cesium.js by clicking on the Tileset preview button in the feedback box Digital Twin Toolbox 13th of June 2024

Slide 28

Slide 28 text

Workflows Conversion of shapefile points data into 3D Tiles Input data: Urban_Forestry_Street_Tree_Benefits_EPSG_26985.zip Main tools dependencies: PostGIS, i3dm.export Output: 3D Tiles Digital Twin Toolbox 13th of June 2024

Slide 29

Slide 29 text

Workflows Select Urban_Forestry_Street_Tree_Benefits_EPSG_26985 (.zip) from input data section Data preview renders the tree points as cubes by default Digital Twin Toolbox 13th of June 2024

Slide 30

Slide 30 text

Workflows Update geometry replacing the default cube with a tree model: ● change the Model (.glb) field with the following string: “tree.glb” Notes: ● Additional 3D model in .glb format can be added in the static/glb folder ● Rotation and scale options supports expression or static values Digital Twin Toolbox 13th of June 2024

Slide 31

Slide 31 text

Workflows Generate 3D Tiles using the default properties by clicking on the Create tileset button Digital Twin Toolbox 13th of June 2024

Slide 32

Slide 32 text

Workflows Preview of generated 3D Tiles in Cesium.js by clicking on the Tileset preview button in the feedback box Digital Twin Toolbox 13th of June 2024

Slide 33

Slide 33 text

Workflows Processing point cloud data to fix/manage CRS, resample and colorization Input data: 1916.las, 1916.tif Main tools dependencies: PDAL Output: LAS (1916_processed.las) Digital Twin Toolbox 13th of June 2024

Slide 34

Slide 34 text

Workflows Select 1916 (.las) from input data section Data preview is empty by default Digital Twin Toolbox 13th of June 2024

Slide 35

Slide 35 text

Workflows Click on the Create a data sample for preview to generate the point cloud preview Note: this sample is limited to a maximum of 500000 points Digital Twin Toolbox 13th of June 2024

Slide 36

Slide 36 text

Workflows Retrieve metadata information by clicking on the Get projection button The projection authority code will be used in the point cloud processing Digital Twin Toolbox 13th of June 2024

Slide 37

Slide 37 text

Workflows Generate a new 1916_processed.las file with following changes: ● vertical datum shift with the EGM2008 geoid model From Crs = +init=EPSG:26985 +geoidgrids=./vdatum/egm08_25.gtx To Crs = EPSG:26985+4269 ● colorize the point cloud using the 1916.tif image Raster image = 1916.tif Digital Twin Toolbox 13th of June 2024

Slide 38

Slide 38 text

Workflows New file 1916_processed.las available in the data folder Digital Twin Toolbox 13th of June 2024

Slide 39

Slide 39 text

Workflows Conversion of point cloud data to 3D Tiles Input data: 1916_processed.las Main tools dependencies: py3dtiles Output: 3D Tiles Digital Twin Toolbox 13th of June 2024

Slide 40

Slide 40 text

Workflows Select 1916_processed (.las) from input data section then click the Create a data sample for preview button Now the preview shows the color of the raster image applied to the processed point cloud Digital Twin Toolbox 13th of June 2024

Slide 41

Slide 41 text

Workflows Retrieve metadata information by clicking on the Get projection button The projection authority code will be used in the tiling process Digital Twin Toolbox 13th of June 2024

Slide 42

Slide 42 text

Workflows Generate 3D Tiles by clicking on the Create tileset button using following properties: ● Add the projection authority number 26985 in the CRS number field ● Add a geometric error scale factor of 0.15 Digital Twin Toolbox 13th of June 2024

Slide 43

Slide 43 text

Workflows Preview of generated 3D Tiles in Cesium.js by clicking on the Tileset preview button in the feedback box Digital Twin Toolbox 13th of June 2024

Slide 44

Slide 44 text

Workflows [Experimental] Conversion of point cloud data to mesh (.ply format) Input data: 1916_processed.las Main tools dependencies: PDAL, Open3D Output: Mesh (1916_processed_mesh.ply) Digital Twin Toolbox 13th of June 2024

Slide 45

Slide 45 text

Workflows [Experimental] Select 1916_processed (.las) from input data section Nothing is rendered on the data preview for this type of process Digital Twin Toolbox 13th of June 2024

Slide 46

Slide 46 text

Workflows [Experimental] Click the Create mesh button with the default properties to generate a new .ply file named 1916_processed_mesh in the static/data/ folder Digital Twin Toolbox 13th of June 2024

Slide 47

Slide 47 text

Workflows [Experimental] Conversion of .ply mesh to 3D Tiles Input data: 1916_processed_mesh.ply, 1916.tif Main tools dependencies: Blender Python Output: 3D Tiles Digital Twin Toolbox 13th of June 2024

Slide 48

Slide 48 text

Workflows [Experimental] Select 1916_processed_mesh (.ply) from input data section Digital Twin Toolbox 13th of June 2024

Slide 49

Slide 49 text

Workflows [Experimental] Click on the Create a data sample for preview to generate the point cloud preview Note: this sample is a simplified version of the original mesh Digital Twin Toolbox 13th of June 2024

Slide 50

Slide 50 text

Workflows [Experimental] Generate 3D Tiles by clicking on the Create tileset button using: ● CRS property equal to EPSG:26985 ● Orthophoto image equal to 1916.tif Note: this process can take up to ~ 30 minutes Digital Twin Toolbox 13th of June 2024

Slide 51

Slide 51 text

Workflows [Experimental] Preview of generated 3D Tiles in Cesium.js by clicking on the Tileset preview button in the feedback box Digital Twin Toolbox 13th of June 2024

Slide 52

Slide 52 text

Tilesets Preview Digital Twin Toolbox 13th of June 2024

Slide 53

Slide 53 text

Tilesets Preview The static/tilesets/ folder contains all generated 3D Tiles outputs and they can be previewed in a Cesium.js client using the url: http://localhost:3000/preview/?{tileset-name} Digital Twin Toolbox 13th of June 2024

Slide 54

Slide 54 text

Tilesets Preview Alternatively tilesets can be previewed in an embedded MapStore map viewer application by selecting tilesets on the controls panel and then clicking on “Show selected tilesets in MapStore” button Tip: refresh the page to visualize the latest generated tilesets Digital Twin Toolbox 13th of June 2024

Slide 55

Slide 55 text

Tilesets Preview Here the final map with all the generated 3D Tiles layer for the Washington D.C. urban sample rendered in MapStore with the help of the Cesium.js map engine MapStore offers different tools to interact with 3D Tiles Digital Twin Toolbox 13th of June 2024

Slide 56

Slide 56 text

Tilesets Preview [Example] Point cloud shading properties to improve points visibility Digital Twin Toolbox 13th of June 2024

Slide 57

Slide 57 text

Tilesets Preview [Example] Style editor with possibility to filter the 3D Tiles feature by property Identify panel with information about selected feature Digital Twin Toolbox 13th of June 2024

Slide 58

Slide 58 text

Tilesets Preview [Example] Style editor with possibility to filter the 3D Tiles feature by property Identify panel with information about selected feature Digital Twin Toolbox 13th of June 2024

Slide 59

Slide 59 text

Tilesets Preview [Example] Different type of measurements for the 3D environment: polyline distance, area, point coordinates, height from terrain, angle and slope Digital Twin Toolbox 13th of June 2024

Slide 60

Slide 60 text

Tilesets Preview Here we presented few examples of the MapStore capabilities, Below links to learn more about the project: ● Documentation with user and developer guides https://docs.mapstore.geosolutionsgroup.com/en/v2024.01. 01/ ● Repository https://github.com/geosolutions-it/MapStore2 ● Latest release https://github.com/geosolutions- it/MapStore2/releases/tag/v2024.01.01 Digital Twin Toolbox 13th of June 2024

Slide 61

Slide 61 text

Ongoing Work: Point cloud Classification Digital Twin Toolbox 13th of June 2024

Slide 62

Slide 62 text

What is point cloud classification ? Digital Twin Toolbox 13th of June 2024 Point clouds images source: Atik et al. 2024 Classified point clouds

Slide 63

Slide 63 text

Why point cloud classification is important? Digital Twin Toolbox 13th of June 2024 Point cloud classification is a crucial part of 3D urban reconstruction since it provides: ● Improved data interpretation ● Acceleration of the data analysis ● Information about the land use / land cover of the scene ● Potential tool for further point cloud processing ● Useful for many other applications (e.g Self-driving cars, inspection, autonomous navigation etc)

Slide 64

Slide 64 text

Point cloud classification using AI Digital Twin Toolbox 13th of June 2024 ● Machine learning algorithms: SVM, Random Forest, Gradient Boosted Trees etc ● Deep learning architectures: PointNet, PointNet++, PointCNN, etc Random Forest architecture Image source: https://www.javatpoint.com/machin e-learning-random-forest-algorithm

Slide 65

Slide 65 text

Experimentation & Results Digital Twin Toolbox 13th of June 2024 Experimentation pipeline, using AI-based algorithms for point cloud classification Dataset preparation heterogeneity Classes selection Parameterization with unknown datasets Training Model extraction Model testing Comparison with other models

Slide 66

Slide 66 text

Results of a pre-trained model Digital Twin Toolbox 13th of June 2024 Results using a pre-trained model, trained with the classes: ground, buildings, vegetation and vehicles Point cloud by ODMSemantic3D Prediction of the pre-trained model with ground in gray, buildings in orange, vegetation in green and vehicles in yellow

Slide 67

Slide 67 text

Results of a pre-trained model Digital Twin Toolbox 13th of June 2024 Prediction of the pre-trained model using the Washington DC open LiDAR

Slide 68

Slide 68 text

Training & fine-tuning Digital Twin Toolbox 13th of June 2024 ● Goal: Focused on urban areas ● Classes: Vegetation, buildings, ground, water ● Parameters tuning Training dataset by Washington DC open LiDAR

Slide 69

Slide 69 text

Training & fine-tuning Digital Twin Toolbox 13th of June 2024 Results using samples from densely urban scenes of Washington DC open LiDAR

Slide 70

Slide 70 text

Training & fine-tuning Digital Twin Toolbox 13th of June 2024 Results using samples from sparsely urban scenes of Washington DC open LiDAR

Slide 71

Slide 71 text

Training & fine-tuning Digital Twin Toolbox 13th of June 2024 Results using samples by GeoPortal of Genova

Slide 72

Slide 72 text

Classification results on 3D Tiles Digital Twin Toolbox 13th of June 2024 Results by Washington DC open LiDAR dataset, displayed on the 3D Tiles of the Digital-Twin-Toolbox

Slide 73

Slide 73 text

Future work Digital Twin Toolbox 13th of June 2024

Slide 74

Slide 74 text

1. Further improvement of classification capabilities of point cloud data (including UI support) 1. More advanced and complete support for photogrammetry processes 1. Automation of the processing chains 1. Support to include LODs and further improve the Tiling System We have in plan to work on a bunch of significant functionalities to enrich the toolbox capabilities: That’s all for a first release of the Digital Twin Toolbox this year! Future work Digital Twin Toolbox 13th of June 2024

Slide 75

Slide 75 text

Check it out on Github: https://github.com/geosolutions- it/digital-twin-toolbox Pre-Release at: https://github.com/geosolutions-it/digital-twin-toolbox/releases/tag/v1.0.0-rc Online Documentation: https://github.com/geosolutions- it/digital-twin-toolbox/wiki Tutorials are also available in the WIKI: https://github.com/geosolutions-it/digital-twin-toolbox/wiki/Tutorials Online resources Digital Twin Toolbox 13th of June 2024

Slide 76

Slide 76 text

That’s all folks! Questions? [email protected] Digital Twin Toolbox 13th of June 2024