Taming Billions of LIDAR Points with GPU Database OmniSci

Presented at 2018 Data Science Conference on October 9, 2018 under the topic of Urban-Analytics and Social-Media Applications.

Transcript

1. Taming Billions of LIDAR Points with GPU Database OmniSci 2018 Data Science Conference Michael Flaxman & Veda Shankar, OmniSci | October 9th, 2018
2. Veda Shankar Sr Developer Advocate , OmniSci Community veda.shankar@omnisci.com Michael Flaxman Founder, Geodesign Technologies mflaxman@geodesigntech.com slides: https://speakerdeck.com/omnisci
3. © OmniSci 2018 Data Grows Faster Than CPU Processing Data Growth 40% per year CPU Processing Power 20% per year
4. © OmniSci 2018 4 Core Density Makes a Huge Difference
5. © OmniSci 2018 5 Advanced Memory Management
6. The Fastest Software Designed for the Fastest Hardware HARNESS GPUs
7. © OmniSci 2018 7 OmniSci Innovations Powering Extreme Analytics 3-Tier Memory Caching Query Compilation In-Situ Rendering
8. © OmniSci 2018 8
9. DEMO : OmniSci Immerse
10. © OmniSci 2018 10
11. © OmniSci 2018 11 Why Use a GPU Database? • Challenges with LIDAR ◦ LIDAR data is typically HUGE ◦ LIDAR is not natively well-structured ◦ Conventional desktop tools require massive downloads & processing • Solution with OmniSci ◦ Interact with big data without moving it ◦ Can handle millions to billions of points natively ◦ Can cross-filter to conduct quality assurance ◦ Persist and store “only the good stuff” for efficient access, eliminate piles of tiles file management ◦ Run large queries faster than postGIS
12. © OmniSci 2018 12 OmniSci Geospatial Features • Geospatial objects ◦ POINT, LINESTRING, POLYGON, MULTIPOLYGON • Geospatial File Formats ◦ GeoJSON, ESRI Shapefile, KML and CSV/TSV with WKT • Geospatial Functions ◦ Geometry Constructors ◦ Geometry Editors ◦ Geometry Accessors ◦ Spatial Relationships and Measurements ▪ ST_Distance, ST_Contains, ST_Within, ST_Area, ST_Perimeter, ST_Length
13. © OmniSci 2018 13 Ingesting LIDAR Data • Coordinate Systems & LIDAR ◦ LIDAR data is typically available in local coordinate systems ◦ OmniSci ver4 supports Web Mercator & Geographic Coordinates (WGS84 lat/lon) ◦ Back-projection into Geo-coordinates usually required • LIDAR to OmniSci Readable Format ◦ PDAL is the best tool to convert LIDAR files to formats readable by OmniSci ◦ PDAL is an open source library installable locally using Conda ▪ conda install -c mathieu pdal ◦ PDAL Docker image available for easy execution ▪ !sudo docker run -v {laz_dir}:/data:z pdal/pdal:1.7 pdal translate -i /data/{laz_file} -o /data/{csv_file} -f filters.reprojection --filters.reprojection.out_srs="EPSG:4326"
14. © OmniSci 2018 14 Using PDAL command line to convert LIDAR files import glob, os filespec = “laz/*.laz" fieldspec = 'X:7,Y:7,' # keep all 7 decimal places fieldspec += 'Z:2,' # keep 2 places of precision fieldspec += ‘Intensity:0,ReturnNumber:0,’ fieldspec += ‘NumberOfReturns:0,Classification:0,' fieldspec += 'HeightAboveGround:2' for counter, file in enumerate(glob.glob(filespec)): outfile = file.replace('.laz','.csv') !pdal translate -i {file} -o {outfile} \ -f filters.smrf -f filters.hag \ -f filters.mortonorder \ --writers.text.keep_unspecified="false" \ --writers.text.order={fieldspec}
15. © OmniSci 2018 15 Python Jupyter Notebook Sample Code 3-Tier Memory Caching Query Compilation In-Situ Rendering • https://github.com/omnisci/community-geo-demos
16. © OmniSci 2018 16 Case Study : Lake Tahoe • Tahoe is a beautiful resort that looks like a national park ◦ But contains 50,000 buildings ◦ It is also highly prone to fire • Current Fire Risk Maps are Landscape Scale • Based on National 30m Landsat Classifications • Ignores individual structures in the woods • Tahoe’s fire risk is all about houses in the woods
17. © OmniSci 2018 17 Project Goals • Ultimately, to build better fire models For Wildland Urban Intermix • Develop an open database Including urban forest Structural characteristics • Develop FOSS workflows for LIDAR -> OmniSci • Characterize Low, Medium & High Vertical vegetation density
18. DEMO
19. © OmniSci 2018 Four Ways to Get Started GitHub repo OPEN SOURCE Website download COMMUNITY OmniSci as a service OMNISCI CLOUD Contact sales ENTERPRISE 19 Ask questions and share your experiences @ https://community.omnisci.com
20. THANK YOU