-Tokyo Digital Twin Project 2021- Demonstration 03 Report

Transcript

1. Tokyo Digital Twin Project Demonstration 03 People Participation in Digital Twin Updates Report
2. １． Background and Overview ２．Area ３．Aboveground: Base Point Cloud ４．Aboveground: Smartphone LiDAR Point Cloud ５． Underground smartphone LiDAR point cloud acquisition superimposition ６．Ueno park participatory demonstration ７．Reflection on the Tokyo Met. Digital Twin 3D Viewer ８．AR Display Demo ９．Results and Issues 10. Future Directions 2 Contents
3. 3 １．Background and Overview
4. 4 Background The feasibility of using smartphone LiDAR as a tool for updating 3D city data needs to be verified. 3D map update using smartphone LiDAR ◼ 3D maps that have been developed need to be updated in the future as the city is updated. ◼ LiDAR Sensor installed on Smartphones enable more people to perform 3D scans. ◼ Extensive 3D city models and point cloud data (3D maps) are available for various municipalities. Tokyo residents participate in 3D map update using their smartphones, enabling immediate and low-cost updates of 3D map according to their needs.
5. Goal ◼ Forming an ecosystem in which the general public, including Tokyo residents, participate in updating the system. ※ Features data added to the 3D city model is expected to be updated at a faster cycle than buildings data. Technical verification was conducted for the establishment of a future point cloud data update ecosystem. ◼ The superimposition of point cloud data on a 3D city model displayed elements of the city that cannot be represented by the city model on a map. ◼ A mechanism to acquire and update point cloud data with low cost, immediacy, and applicability (utilization of popular media) based on point cloud acquired by smartphone LiDAR was considered. Overview Technical verification and use case studies to realize effective 3D map update with the participation of Tokyo residents. 5 Goal and Overview
6. The construction of an ecosystem and autonomous update of 3D map with Tokyo residents were advocated. Long-term goal (2040) : The realization of Smart Tokyo by utilizing the ecosystem ◼ Trial of point cloud acquisition and update system ◼ Arrangement of point cloud acquisition methodology for participation of Tokyo residents ◼ Arrangement of technical requirements for the formation of an ecosystem ◼ Consideration of a use case study ◼ Realization of autonomous and needs-based update of 3D map with the participation of Tokyo residents Examples of specific services： ・Companies update their advertisements and other information on the Digital Twin ・Part of infrastructure in need of repair and barrier-free difficulties are visually collected ・Display barrier-free difficulties on the map Mid-term goal (2030): The formation of a complete Digital Twin ecosystem Short-term goal: the demonstration of technology and the embodiment of ecosystem Scope of this demonstration ◼ Formation of point cloud acquisition and update system ◼ Use case (feedback to physical space) ◼ Consideration of a system which promote update, such as the improvement of the value of the point cloud acquisition experience, providing incentives 6 Goal and Overview
7. The verification and arrangement details for each phase of the demonstration were clarified. Demonstration scope Items to be verified and organized Trial of point cloud acquisition and update system 【Aboveground: base point cloud】 ・Points to consider in the measurement process (where/how the reference/targeting points were established in an environment prone to GNSS turbulence) ・Data editing method (how the data acquired in an environment prone to GNSS turbulence was error-corrected, what considerations were made for objects difficult-to-acquire such as glass, etc.) 【Ground and underground: superimposed point cloud and method】 ・Features that can be acquired and superimposed using the iPhone12 were confirmed. (verification was done by area characteristics, such as skyscrapers, underground, etc.) ・Arrangement of the superimposition method between the base point cloud and the superimposed point cloud. ・Arrangement of requirements for applications to be used 【Web: publication method】 ・Arrangement of the methods for conversion, layer organization, display, etc. of acquired data 【Whole】 ・Consideration of current technical hurdles and future vision for each item 7 Goal and Overview
8. The verification and arrangement details for each phase of the demonstration were clarified. Demonstration Scope Items to be verified and organized Arrangement of point cloud acquisition methodologies for Tokyo residents' participation ・Arrangement of manuals, precautions for point cloud acquisition by Tokyo residents (who have no knowledge of point cloud acquisition) using iPhones. Arrangement of technical requirements for ecosystem formation ・Consideration of a mechanism to promote update, such as improving the value of the point cloud acquisition experience and providing incentives Verification of the use case study ・Arrangement of use cases possible with current technology and use cases expected in the future 8 Goal and Overview
9. Based on the goals of the demonstration, the target objects and use cases were summarized. Area Subject features Use case Outdoor ground area • Trees • Steps, stairs • Advertisements, signboards, vending machines • Verification of immediate and low-cost features' acquisition, display, and addition on urban models. • Time change displayed by AR (from "without" state to "with" state) Indoor underground building area • Advertisements • Signboards, braille blocks • Accessible toilet • Reflection of movable features and textures installed in the station on the 3D model. • Acquisition of barrier-free status of the station and consideration of it on the 3D model • Prior confirmation of 3D model of the Accessible toilet facilities details 9 Goal and Overview
10. 10 ２．Area
11. Nishi-Shinjuku was selected as the target area based on the field characteristics and various related measures. By bringing together diverse stakeholders and Tokyo residents, the creation of Digital Twin promotion community is expected. ◼ Nishi-Shinjuku is a town visited by a diverse range of stakeholders, including real estate workers and professionals and is expected to become a nucleus of a digital twin promotion community. As a leading Smart Tokyo implementation area, collaboration with other initiatives is expected. ◼ Nishi-Shinjuku is an area where "Smart Tokyo" will be implemented in advance and is expected to collaborate with the "Nishi-Shinjuku Smart City Project" including the Tokyo Data Highway, as well as with various other initiatives around the Tocho Building. In addition to the existence of areas with various features, the result of last year's verification experiment of point cloud data from Tochomae Station on the Oedo Line can be utilized. ◼ Various geographic areas, such as skyscrapers, underground makes this area suitable for GPS performance verification. ◼ Various features such as signboards, plantings, steps, etc. can be acquired. ◼ Point cloud data of Tochomae Station on the Oedo Line, the result of last year's verification experiment, can be utilized. Demonstration area Reasons for selection of target area 11 Nishi-Shinjuku Area Area
12. Ground Underground Ground ©GeoTechnologies Inc. 許諾番号：PL1702 The locations of point cloud acquisition and the target objects to be acquired were selected. ◼ Point cloud acquisition and update with smartphones is characterized by low cost, immediacy, and applicability (use of diffused media) ◼ Features (e.g., planted trees, signboards) that can show change over time by point cloud superimposition, display on Terria or AR, features (e.g., barrier-free related objects) requiring detailed confirmation and areas where the base point cloud is missing are acquired as features in which the above 3 characteristics effectively function. ◼ The area shown in the figure below was selected as the demonstration area as it contains many of the above features. 【Ground】 Around Nishi-Shinjuku Mitsui Bldg. (stairs, curbs, plantings) 【Ground】 Around Shinjuku Station west exit electric town （vending machines, signage, curbs, plantings, road surfaces） 【Underground】 Oedo Line Tochomae Station (advertisements, braille blocks, and Accessible toilet) 12 Selection of demonstration area
13. 13 ３．Aboveground: Base Point Cloud
14. The data format of the base point cloud was determined. ◼ Based on the superimposition with the point cloud data acquired by LiDAR for cell phones and the display on the 3D viewer (Terria), the data format of the base point cloud was discussed and decided as follows: ◼ ①Format: LAS format ◼ ②Geodetic system：JGD2011 ◼ ③Coordinate system: Plane rectangular coordinates ◼ ④Height: Elevation ◼ ⑤Unit: Meter ◼ ⑥Geoid Model: Japanese Geoid 2011 ◼ ⑦Reference ellipsoid：GRS80 14 Aboveground: Base Point Cloud
15. The operation plan for base point cloud acquisition was implemented. ◼ Review and implement an operation plan base point cloud acquisition out of the demonstration. GLS-2000 middle and long type （Right figure：laser scanner、 left diagram：target plate） ◼ size：23 × 29 × 41cm （Main unit only, excluding legs） ◼ weight：10kg Operation content and equipment Worker Operation date and time Reference point selection and observation equipment: Total station 6 persons: 2 acquirers, 1 assistant, 1 guard, 2 overall supervisors 8/31 and 9/1 (Tue & Wed) Daytime ※It was judged feasible to conduct the work during the daytime due to the short time of the reference point measurement. Installation and observation of pass points, and point cloud acquisition equipment：GLS- 2000 8 persons: 4 acquirers, 1 assistant, 1 guard, 2 overall supervisors 9/6-8 (Mon-Wed) at night ※Conducted at night to maintain the accuracy and prevent the increase in the occlusion processing of the point cloud data as there are fewer pedestrians and vehicles at night. Base Point Cloud Acquisition Schedule ８/31（Tue） ・ 9/1（Wed） daytime Reference point survey 9/6（Mon） ・ 7（Tue） at night Establishment and observation of pass points, point cloud surveying ～About 6.5 weeks Cleaning operations, accuracy verification 15 Aboveground: Base Point Cloud
16. Operation① Measurement of reference points Reference point selection and measurement Selected reference points M a p d e l i v e r e d b y Z e n r i n I n c . ◼ Measurement work was performed in the ground area. ◼ The operation was conducted with attention to safety measures and measures against COVID-19 infection. 16 Aboveground: Base Point Cloud Click to add text
17. Operation② Acquisition of point cloud ◼ Measurement work was performed in the ground area. ◼ The operation was conducted with attention to safety measures and measures against COVID-19 infection. Point cloud acquisition implementation scenery Acquisition Result 17 Aboveground: Base Point Cloud
18. The data accuracy of reference point measurements was compared and verified among the measurement methods. ◼ In light of the fact that point cloud acquisition was conducted in an environment where GNSS is easily disturbed, the data and its accuracy were carefully examined as follows: ◼ In implementing the reference point measurement, "measurement by the VRS method of the network RTK method (VRS-RTK measurement) " was implemented as a comparison to the "measurement method using a total station to measure a new point from the city district reference point (MLIT) and assign coordinates to the new point", and the results confirmed that the VRS-RTK measurement is not suitable as an implementation method. ◼ Then, to check the accuracy of the coordinates (X and Y coordinates) assigned to the point cloud data by the measurement method using a total station, the values of the coordinates (plane rectangular coordinates) were input into the Geospatial Information Authority of Japan's "Conversion to Latitude and Longitude" software, and it was confirmed that no differences occurred in the displayed locations (= no accuracy problems). - 【Measurement results using the VRS method of the networked RTK method】 - Ten points were measured around the Nishi-Shinjuku Mitsui building, and 13 points were measured around the Shinjuku Station west exit electric town. In consideration of multi-paths and other factors, each measurement took at least 3 minutes to complete (the RTK method is inherently extremely short, 10 seconds for completion). In addition, two measurements were taken at each location for inspection. - As a result, fix solution was obtained at one point in each measurement area, and at the other points, only the coordinates of the float solution were obtained. In addition, the "3D CQ" value of the coordinates of the float solution (a smaller value indicates a smaller error in the coordinates) are large, indicating that the accuracy as coordinates is low. Based on the above results, we confirmed that VRS-RTK measurement is not suitable for reference point measurement in an area with a high concentration of skyscrapers, as in this case. 18 Aboveground: Base Point Cloud
19. The accuracy of the coordinates assigned to the point cloud data was confirmed to be acceptable. ◼ In light of the fact that the survey was conducted in an environment where GNSS is easily disturbed, the coordinates assigned to the point cloud data were examined. ◼ The coordinates of 4 points in measurement area ① and 5 points in measurement area ② were extracted from the point cloud data. Using the "Conversion to Latitude and Longitude" software, the position indicated by each coordinate on a map were checked if any differences in position occurred. ◼ As a result, there was no significant difference in the position of each coordinate, and it was determined that the coordinates assigned to the point cloud data were accurately measured. 19 Point cloud coordinates examined (top) / Details (bottom) Check the location of coordinates on map Aboveground: Base Point Cloud
20. 20 4. Aboveground: Smartphone LiDAR Point Cloud 4.1 Verification of Smartphone LiDAR Performance and Data Specifications
21. Location data based on available information Azimuthal information Base point cloud cut out based on rough location information Point cloud acquired Azimuth-corrected acquired point cloud Rough location information Overview of data flow for point cloud automatic superimposition Automatic superimposition technology and data flow were developed and verified. ◼ Symmetry Dimensions Inc. developed an automatic point cloud superimposition system based on its existing technology. ◼ Automatic superimposition are conducted to the base point cloud using the acquired point cloud geometry, smartphone GPS coordinates, and compass (azimuth information). 21 LiDAR Performance and Data Specifications O r i g i n a l i m a g e c r e a t e d b y S y m m e t r y D i m e n s i o n s I n c .
22. <Appendix> The characteristics of the automatic superimposition system used in the demonstration: ◼ In automatic superimposition of smartphone LiDAR point cloud to the base point cloud, it is desirable to acquire not only the features points to be acquired, but also the features points of the surrounding walls, pillars, and other shapes of the location. ◼ GPS and azimuthal information should also be obtained for use in superimposition. 22 Example of automatic superimposition not working (few feature points) Example of automatic superimposition on an incorrect angle depending on the compass value Example of point cloud with easy automatic superimposition (many feature points) LiDAR Performance and Data Specifications
23. The quality of the point cloud acquired by the smartphone and the output quality of the application were verified. ◼ Attempted to acquire point cloud using iPhone12 pro and several existing apps. We confirmed that the point density required for automatic superimposition and 3D viewer display can be ensured. ◼ The point cloud was acquired and compared in two different qualities on the application, and it was confirmed that the difference in accuracy appeared in "undulations of planes" and "edges of columns". Signs, object distance 0.5 to 1 m, output quality High Car, object distance 1m (upper left) / 0.3m (lower right), output quality High 23 LiDAR Performance and Data Specifications
24. The flow and format from data acquisition to display on the web were organized. ◼ The flow of acquiring data and displaying it in the 3D viewer on the website was organized. ◼ With the data format organized, the format of the point cloud for superimposition was decided to follow the base point cloud. ◼ Each point cloud was converted to 3D Tiles format using FME and displayed on Terria. ZENRIN Corporation, Type S Ltd. Superimposition of acquired point cloud ◼ Automatic superimposition of point clouds for superimposition ◼ Move mesh according to point cloud position Symmetry Dimensions Inc. Display on Web site ◼ Convert point cloud to 3DTiles ◼ Display by Terria (Cesium) Pacific Spatial Solutions Inc. Base point cloud Point cloud and Mesh for superimposition （At the time of demonstration obtained by Symmetry Dimensions Inc. ） Adjustment of base point cloud delivery format Superimposed point cloud（las）・Mesh(obj) 24 Base point cloud data acquisition Format LAS format Geodetic system JGD2011 Coordinate system Plane rectangular coordinates Height Elevation Unit Meter Geoid model Japanese Geoid 2011 Conformal ellipsoid GRS80 Acquisition of point cloud data for superposition ◼ Acquired by Smartphone LiDAR ◼ The format of the superposed point cloud follows that of the base point cloud LiDAR Performance and Data Specifications Tokyo residents and other general public
25. 25 4. Aboveground: Smartphone LiDAR Point Cloud 4.2 LiDAR Point Cloud Acquisition and Superimposition
26. The details of the target objects in the area around the Nishi-Shinjuku Mitsui Bldg. were organized. ※Image shows base point cloud data before noise elimination Stairs Curbs Enlarged map of the area where the shooting was conducted Area base point cloud around Nishi-Shinjuku Mitsui bldg. ©GeoTechnologies Inc. license number：PL1702 26 Aboveground: LiDAR Acquisition/Superimposition ◼ In the area around the Nishi-Shinjuku Mitsui building, stairs, steps, and curbs were photographed from the perspective of "complementing the base point cloud" based on the plan.
27. The details of the target objects in the area around the Nishi-Shinjuku Mitsui Bldg. were organized. Objects to be photographed Use case details Stairs and steps ◼ Part of the staircase was missing due to the installation of equipment, and completion by point cloud superimposition could be seen. Curbs and plantings ◼ The aptitude for the curbs, plantings, and surrounding ground near the equipment to be supplemented by the superimposed point cloud was assumed because the base point cloud was not well acquired. ①Stairs・steps ②Curbs 27 ※Image shows base point cloud data before noise elimination Aboveground: LiDAR Acquisition/Superimposition ◼ In the area surrounding the Nishi-Shinjuku Mitsui building, stairs, steps, and curbs were photographed from the perspective of "complementing the base point cloud" based on the plan.
28. ※Image shows base point cloud data before noise elimination 28 Sidewalk surface Curbs and plantings ©GeoTechnologies Inc. license number：PL1702 Shinjuku station west exit electric town area base point group The target objects in the electric town area at the west exit of Shinjuku station area were as follows: ◼ Regarding the shooting in the electric town area at the west exit of Shinjuku station, from the viewpoint of "complementing the base point cloud," sidewalk surfaces, plants, steps, etc. that were missing due to running parked vehicles, pedestrians, acquisition equipment, etc. were shot. Aboveground: LiDAR Acquisition/Superimposition
29. Sidewalk surface Curbs and plantings Features to be photographed Use Case Details Sidewalk surface ◼ Part of the stairs were missing due to pedestrians and parked vehicles and could be complemented by point cloud superimposition. Curbs and plantings ◼ The aptitude for the curbs, plantings, and surrounding ground near the equipment to be supplemented by the superimposed point cloud was assumed because the base point cloud was not well acquired. ※ Image shows base point cloud data before noise elimination 29 The target objects in the electric town area at the west exit of Shinjuku station were as follows: ◼ Regarding the shooting in the Electric Town area at the west exit of Shinjuku Station, from the viewpoint of "complementing the base point cloud," sidewalk surfaces, plants, steps, etc. that were missing due to running parked vehicles, pedestrians, acquisition equipment, etc. were shot. Aboveground: LiDAR Acquisition/Superimposition
30. The details of the demonstration area in the electric town area at the west exit of Shinjuku station were organized. Vending machines Store signboards ◼ In Shinjuku station west exit electric town area②, vending machines and store signboards were photographed from the perspectives of "updating and adding base point cloud". ※ Image shows base point cloud data before noise elimination 30 ©GeoTechnologies Inc. license number：PL1702 Aboveground: LiDAR Acquisition/Superimposition
31. Features to be photographed Use case details Vending machines ◼ Product lineup changed from time to time and could be updated by superimposed point clouds. Store signboards ◼ Signboards of restaurants, etc., which did not exist when the base point cloud was taken at night, could be added by the superimposed point cloud. Vending machines Store Signboards 31 ※ Image shows base point cloud data before noise elimination The details of the demonstration area in the electric town area at the west exit of Shinjuku station were organized. ◼ In Shinjuku station west exit electric town area ②, vending machines and store signboards were photographed from the perspective of "updating and adding base point clouds". Aboveground: LiDAR Acquisition/Superimposition
32. Point cloud of target objects was acquired in the aboveground area. ◼ As with video recording, point cloud could be acquired by holding up a smartphone. ◼ During the acquisition process, the accompanying person displayed work in progress sign to the people around for privacy and safety concerns. Shooting of the stairs Shooting of the vending machines 32 Aboveground: LiDAR Acquisition/Superimposition
33. Point clouds with successful automatic superimpositions related to use cases were made public. ◼ The landmarks present in the public data were as follows: ◼ Point clouds that succeeded in automatic superimposition and that showed use cases were released. Stairs' area around Nishi-Shinjuku Mitsui bldg. Curbs and plantings' area around Nishi-Shinjuku Mitsui bldg. ◼ Stairs ◼ Curbs and plantings 33 Aboveground: LiDAR Acquisition/Superimposition
34. ◼ Store signboards ◼ Sidewalk surface ◼ Vending machines Shinjuku Station west exit electric town area vending machines Shinjuku station west exit electric town area main street 34 ◼ The landmarks present in the public data were as follows: ◼ Point clouds that succeeded in automatic superimposition and that showed use cases were released. Point clouds with successful automatic superimpositions related to use cases were made public. Aboveground: LiDAR Acquisition/Superimposition
35. 35 5. Underground: Smartphone LiDAR Point Cloud Acquisition and Superimposition
36. The demonstration area and the target objects in Tochomae station were selected. Source） https://www.kotsu.metro.tokyo.jp/subway/stations/tochomae.html#barrierfree Created from screenshot (retrieved on 2021/09/02) ②Accessible toilet ③Braille block from ticket gate to front of restroom ①Around objects inside ticket gates ④Wall ads and posters ：Shooting range ◼ The area enclosed by the green line in the figure below was taken based on the land property to be acquired. ⑤Exterior wall surfaces of ticket gates (including ticket vending machines and advertisements), stairs, and ramps 36 Underground: LiDAR Acquisition/Superimposition
37. Point cloud of target objects was acquired in the underground area. ◼ Considering privacy and safety concerns, the work was conducted late at night when there were no passengers using the station. Shooting of braille blocks Shooting of multifunctional restroom and its surrounding area 37 Underground: LiDAR Acquisition/Superimposition
38. Point cloud and mesh for the target objects in the Tochomae station demonstration were acquired. ◼ Photography work was conducted in the Tochomae station area, focusing on the features below. ◼ Mirrored portions of objects were difficult to acquire. Object of the shootings Point cloud acquisition status Remarks Objects in ticket gates × • This was organized as a difficult condition for point cloud acquisition because many of the mirrored surfaces could not be acquired. Toilet （Includes Accessible toilet） 〇 • Mirror surface area was chipped. Braille blocks from ticket gates to restrooms 〇 • These were obtained without problems. Part of braille blocks outside ticket gates 〇 Corporate advertisements, posters, etc. 〇 Wall outside ticket gate 〇 Stairs and ramps 〇 38 Underground: LiDAR Acquisition/Superimposition
39. Point clouds with successful automatic superimpositions related to use cases were released as underground demonstration results. ◼ Braille blocks from ticket gates to restrooms ◼ Point cloud that succeeded in automatic superimposition and that showed use cases were released. However, only for "Accessible toilet", manual superimposition was performed because the base point cloud did not exist. ◼ The features present in the public data were as follows: B2F Braille blocks in ticket gates B2F Accessible toilet （Men side） 39 ◼ Facilities in the Accessible toilet B2F Accessible toilet （Women side） ◼ Facilities in the Accessible toilet Underground: LiDAR Acquisition/Superimposition
40. ◼ Point clouds that succeeded in automatic superposition and that showed use cases were released. ◼ The features present in the public data were as follows: ◼ Stairs B2F Stair area B2F Area in front of ticket machine B2F Slope Area B2F Braille block outside ticket gates 40 ◼ Ticket machines ◼ Posters ◼ Wall map ◼ Braille blocks outside ticket gates ◼ Slope ◼ Wall outside ticket gate Point clouds with successful automatic superimpositions related to use cases were released as underground demonstration results. Underground: LiDAR Acquisition/Superimposition
41. 41 6. Ueno Park Demonstration
42. The goals and overview of the participatory demonstration were as follows: ◼ A participatory demonstration in Ueno Park with the Tokyo University of the Arts was conducted. ◼ The objectives, concept and detailed contents were as follows: Items Contents Objective ① Create a space for diverse people to be aware of and participate in the digital twin. ② Practice point cloud acquisition using smartphone LiDAR and brush up the point cloud acquisition know-how organized in Demonstration 03. Contents of implementation ◼ Photograph and archive point clouds of "features that change over time". ...Objective ① ◼ Provide those who are not familiar with point cloud acquisition with an experience of point cloud superposition and update. ...Objective ② ◼ Plan to spread awareness of the Digital Twin Project by disseminating the initiatives on SNS by the organizers and participants. ...Objective ① Details of implementation ① Participants will use a smartphone LiDAR to take pictures of advertising billboards, features, etc. in Ueno park and acquire point clouds. ② The acquired point cloud is superimposed on the base point cloud and digitally stored on the 3D viewer. 42 Ueno Park Demonstration
43. The demonstration area and target objects for the participatory demonstration were as follows: Features to be photographed Significance of shooting at the Ueno No Mori Demonstration ①Signboards and others update ◼ Photograph and update frequently updated items such as advertisements for limited time exhibits. ②Display and confirmation of equipment, etc. on a 3D viewer ◼ Acquire and display features that are useful to see on a 3D map. ③Base point cloud completion ◼ Smartphone LiDAR is used to acquire point clouds to supplement point cloud that are difficult to acquire with point cloud acquisition equipment with a high viewpoint (taller than a person's height). ◼ After consultation with the Tokyo University of the Arts and related departments, the point cloud around the grand fountain and the point cloud in front of the police box were decided as the base point clouds to be displayed in the 3D viewer. ◼ Based on the base point cloud coverage, significant features to be photographed during the demonstration were selected. 【Reference: Base point cloud status acquired at the Tokyo University of the Arts】 ◼ It was highly comprehensive and includes detailed geographic features in its representation. ◼ It was expected to be supplemented and updated by the Tokyo University of the Arts and other institutions in the future. 43 Around the Grand Fountain Front of a police box Ueno Park base point cloud displayed in the viewer Ueno Park Demonstration
44. Target Object① Information update ◼ In areas with signboards, such as the area around the grand fountain plaza, the latest signboards content will be acquired by smartphone LiDAR scanning, and efforts will be made to replace and improve the readability of the signage. Property acquired Signboard Scope Around 50 ㎡(spot acquisition) Time Less than 30 minutes Guide signboard, ground maps, etc. 44 Point cloud alone was insufficient for signboard legibility Ueno Park Demonstration
45. Target Object② Acquisition by participants ◼ Each participant reviewed and selected 1 type of item in the park "that would be useful/enjoyable to see detailed information in a 3D viewer" and "that is missing in the base point cloud data" and obtained a point cloud/mesh. ◼ Participants had checked 3D viewers, etc. in advance and considered what to acquire. ◼ Acquired landmarks were superimposed on a 3D viewer so that the detailed status of the landmarks could be checked in virtual space. Example: Areas with missing points in the base point cloud ※Of these, areas that are difficult to supplement in future work are shown Example of features: Toilet area /Accessible toilet (large making it extremely difficult) Example of features: Trash cans Property acquired Ground, plants, etc Scope Approx. 20 ㎡ Time A little over 30 minutes 45 Ueno Park Demonstration
46. Apps used by participants and data collection method were as follows: 1. Participants must create a Dropbox account in advance and provide the secretariat with their email address. 2. The secretariat will invite the collected email addresses to the owner's Dropbox. 3. On the day of the event, participants will store the acquired point cloud from the application to Dropbox. Data collection method: Upload to Dropbox provided by the secretariat Apps ◼ Scaniverse (Can export to various data formats for free) ※If in possession, other applications such as Polycam could be used. ◼ Format of data to be submitted: obj (mesh) / ply/las (point cloud) 46 ◼ The above data collection method was selected as a method that allows the acquired data to be shared on the spot. ◼ The following is a discussion of the apps that participants were asked to use and how data were collected from the apps. Ueno Park Demonstration
47. The participatory demonstration created a place for Digital Twin practice involving many entities. ◼ The demonstration was conducted, and the following results were obtained. ◼ Implementation of projects in collaboration with the Tokyo University of the Arts ◼ Create opportunities for the general public, including Tokyo residents, to become aware of and participate in the Digital Twin ◼ Practice of point cloud acquisition by inexperienced persons （Brush up know-how based on guidance by experts) Participatory Demonstration Scenery 47 Ueno Park Demonstration
48. The point cloud and mesh acquired in the participatory demonstration were superimposed. (Signboard) ◼ The signboard point cloud and mesh superimposed and displayed in the 3D viewer were shown below. The results were also available on Sketchfab. Prof. Furuhashi, Aoyama Gakuin University Mr. Houshitou, student at Tokyo University of the Arts Mr. Ito, HoloLab Inc. Mr. Iwama 48 Ueno Park Demonstration
49. The point cloud and mesh acquired in the participatory demonstration were superimposed. (Free acquisition) ◼ Free acquisition point cloud and mesh superimposed and displayed in the 3D viewer were shown below. The results were also available on Sketchfab. Mr. Iwama, Steps behind toilet Prof. Akita, Tokyo University of the Arts Behind the drinking fountain Mr. Houshitou, student at Tokyo University of the Arts Disaster Water Supply Stations Digital Services Bureau Representative Steps around fountain Mr. Ito, HoloLab Inc. Coiin-operated locker Prof. Furuhashi, Aoyama Gakuin University Exterior view of restroom 49 Ueno Park Demonstration
50. 50 7. Reflection on the Tokyo Met. Digital Twin 3D Viewer
51. Ground base point cloud was displayed on the 3D viewer. ◼ The ground-based point cloud was tried to be displayed on the 3D viewer with several point sizes. It was decided to release the point clouds with "Point Size: 2", which had a good balance between visibility and rendering performance. Note: The smaller the point size, the better the rendering performance on the 3D viewer (Terria). A1 Point size: 1 (Default size) A2 Point size: 2 A1 Point size: 4 A2 Point size: 1 51 Digital Twin 3D Viewer
52. Point cloud and mesh acquired with smartphones were displayed on 3D viewer. ◼ Above- and below-ground smartphone LiDAR point cloud and mesh were displayed on a 3D viewer. ◼ In confirming detailed textures, mesh data, which could be generated simultaneously, was found to be more suitable than point cloud data. ◼ However, the mesh display needed to be adjusted for each application in terms of data conversion method, light-receiving settings, etc. Point cloud display on Terria Mesh display on Terria （Signs on the stairs to the home floor can be seen more clearly.） 52 Digital Twin 3D Viewer
53. Accessible toilet's interior mesh data was released on Sketchfab. ◼ Because it was difficult to move to an appropriate viewpoint and adjust the brightness of the mesh data of Accessible toilet on a 3D viewer, a detailed display with the external service "Sketchfab" and embedding in an information dissemination site were implemented. ◼ The details of the facilities of the " Accessible toilet" could be easily confirmed on the 3D screen prior to the visit. 53 Digital Twin 3D Viewer
54. 54 8. AR Display Demo
55. The demonstration of AR display of point cloud data acquired by smartphone. Source: AR display image created by Symmetry Dimensions Inc. ◼ The AR display function demo of point cloud data (superposed data acquired using the iPhone) was provided using existing technology from Symmetry Dimensions Inc. ◼ The system allowed point cloud data to be viewed on an application along with smartphone camera images. On-site, etc. Check on the app Image of AR display demo 55 AR Display Demo point cloud data
56. The point cloud of stair was displayed in overlay through AR. ◼ AR display of point cloud of stairs in front of Nishi-Shinjuku Mitsui Bldg. was implemented. ◼ As a future use case, it would be possible to use AR to confirm the locations reported by the Tokyo residents using 3D point clouds for information such as "barrier-free concerns" or "damage to stairs, etc.". Result of AR display demo 56 AR Display Demo
57. 57 9. Results and Issues
58. Technology verification was successful and use case was organized this year. ◼ New technology that automatically aligns point clouds acquired with smartphones to the corresponding positions in the base large-scale point cloud was realized. ◼ Technology to lower the hurdle for Tokyo residents' participation by automating 3D map update with point cloud and mesh was verified. Automatic update of base point cloud by point cloud acquired by smartphone ◼ Acquisition of data from smartphones, their formats and their aggregation methods were considered for 3D map update with point cloud. Arrangement of technical requirements for utilizing point clouds acquired by smart phones ◼ Features and scenes where 3D map update with Tokyo residents' participation using point cloud and mesh were effective were examined. ◼ The feasibility and usefulness of "update of textures such as vending machines and advertisements," "addition of time-limited features such as signboards," and "display of barrier-free information" were confirmed. Validation of use case 58 Results and Issues
59. The technical aspects of smartphone LiDAR remained issues. ◼ The existence of areas that cannot be acquired (e.g., mirror surfaces) and where errors occur due to smartphone LiDAR performance (e.g., large objects over several tens of meters), make it difficult to update 3D maps using smartphones at this time. ◼ It is necessary to examine use case and future scope by conducting a close examination of areas that cannot be acquired and where errors occur due to smartphone LiDAR performance. Existence of areas that cannot be acquired and where errors occur due to smartphone LiDAR performance ◼ There are conditions that hinder automatic superimposition, such as underground or buildings where positional accuracy is not accurate as smartphone LiDAR requires azimuth and GPS data. ◼ The presence of noise such as vehicles and pedestrians, repetitive structures such as the columns at the Tochomae Station, and large differences between the reference point cloud data and the smartphone scan point cloud also hinder automatic superimposition. ◼ Effects of other conditions should be clarified in the future. ◼ To cope with conditions currently known to be unfavorable for automatic superimposition, it is possible to respond with methodologies (periodic update of reference point cloud data and point cloud acquisition during periods of low noise) and equipment (use of equipment and technology to measure the position and orientation of smartphone, such as beacons and markers). Existence of conditions and environments unfavorable to automatic superimposition 59 Results and Issues
60. Administrative arrangement and study on the mechanism of participation are needed for Tokyo residents' participation. ◼ Challenges, such as confirming the accuracy of the data and unifying specifications exist for using point cloud data collected from Tokyo residents for Tokyo 3D map update. ◼ In the future, arrangement within administration would be needed based on the above issues. Issues on the usage of point cloud gathered from Tokyo residents for administrative map update ◼ It is necessary to consider how to realize an ecosystem whereby the Tokyo residents would voluntarily acquire urban point cloud and upload them to the 3D map update. ◼ In considering the development of smartphone applications and SDK in the future, it would be necessary to consider mechanisms and incentives for participation in 3D map update, such as incorporating gamification-like mechanisms in scanner applications or providing incentives such as points. Building an ecosystem where Tokyo residents can participate in updating 3D maps ◼ The demonstration was conducted under a privacy and safety policy that was appropriate for the area. ◼ Continuous consolidation of considerations, guidelines, etc. will be necessary when TMG in general acquires data in the future. Consideration for privacy and security during data acquisition 60 Results and Issues
61. 61 10. Future Directions
62. Systemization and community formation are necessary to realize 3D map update. ◼ In this year, verification of automatic superimposition technology and use case that contribute to 3D map update using smartphone LiDAR were studied. In addition, a framework for diverse people to participate in map update was tested through a participatory demonstration. ◼ Future preparations for the implementation of 3D map update with Tokyo residents' participation may include system studies, legal issues consolidation, and implementation of events aimed at improving Tokyo residents' participation and understanding, as well as core community formation. 62 Future Directions FY2021 technical demonstration After FY2022 3D map update community formation Future Goals 3D map update implementation ◼ Point cloud automatic superimposition technology that is effective for 3D map update with Tokyo residents' participation was demonstrated. ◼ Consideration of use case ◼ Implementation of an event for Tokyo residents to participate in 3D map update ◼ Formation of a core community in collaboration with communities interested in technology ◼ Consideration of a framework (including systems, applications, etc.) in which Tokyo residents can participate on a regular basis ◼ Formation of a map update community centered on the core community ◼ Completion of internal administrative arrangement, including legal perspectives