Visualizing large geospatial data - hands-on MapD

This evening will be about creating geographic maps. About what tools are out there for which purposes and what the challenges are.

One of them for sure is how to deal with extremely large data sets. And we are excited to have Aaron Williams from MapD with us for this one. He will do a short workshop where we can use their web-based data visualization interface, which uses the GPU to accelerate speed and rendering capabilities.

About MapD
MapD is designed to leverage the massively parallel processing of GPUs alongside traditional CPU compute, for extraordinary increases in performance at scale. (More here: http://www.mapd.com)

Transcript

1. © MapD 2018 Vancouver Data Visualization Meetup | August 27, 2018
2. © MapD 2018 Aaron Williams VP of Global Community @_arw_ aaron@mapd.com /in/aaronwilliams/ /williamsaaron slides: https://speakerdeck.com/mapd
3. © MapD 2018 © MapD 2018 Agenda • Intro to GPU Analytics and MapD • Tutorial A: Importing Data into MapD Cloud • Tutorial B: Rendering with Vega 3
4. © MapD 2018 Do This Now If you want to participate in the tutorials, sign up for a free trial account on MapD Cloud http://mapd.cloud 4
5. Core Density Makes a Huge Difference 5 GPU Processing CPU Processing 40,000 Cores 20 Cores *fictitious example Latency Throughput CPU 1 ns per task (1 task/ns) x (20 cores) = 20 tasks/ns GPU 10 ns per task (0.1 task per ns) x (40,000 cores) = 4,000 task per ns Latency: Time to do a task. | Throughput: Number of tasks per unit time.
6. © MapD 2018 MapD is the analytics platform created for GPUs
7. © MapD 2018 © MapD 2018 7 DEMO TIME!
8. © MapD 2018 Advanced memory management Three-tier caching to GPU RAM for speed and to SSDs for persistent storage 8 SSD or NVRAM STORAGE (L3) 250GB to 20TB 1-2 GB/sec CPU RAM (L2) 32GB to 3TB 70-120 GB/sec GPU RAM (L1) 24GB to 256GB 1000-6000 GB/sec Hot Data Speedup = 1500x to 5000x Over Cold Data Warm Data Speedup = 35x to 120x Over Cold Data Cold Data COMPUTE LAYER STORAGE LAYER Data Lake/Data Warehouse/System Of Record
9. © MapD 2018 MapD Core: Query Compilation with LLVM
10. © MapD 2018 MapD Immerse: Hybrid Rendering
11. © MapD 2018 Tutorial A: Importing Data into MapD Cloud 11
12. © MapD 2018 Last Chance ... If you want to participate in the tutorials, sign up for a free trial account on MapD Cloud http://mapd.cloud 12
13. © MapD 2018 Step 1: MapD Immerse Basics 1. View your data in the Data Manager 2. Import data a. Local CSV b. S3 Bucket 3. View your dashboards 4. Create a new dashboard a. SAVE! 13
14. © MapD 2018 Geospatial Objects Type Description POINT A point described by two coordinates. LINESTRING A sequence of 2 or more points and the lines that connect them. POLYGON A set of one or more rings (closed line strings), with the first representing the shape (external ring) and the rest representing holes in that shape (internal rings) MULTIPOLYGON A set of one or more polygons.
15. © MapD 2018 Step 2: Loading MapD Shapefiles into Immerse 1. Polygons SF City and County Subdivision Parcels MULTIPOLYGONS in GeoJSON https://s3.amazonaws.com/mapd-data/geodata/citylots.json 2. Points SF City-owned Critical Facilities POINTS in ESRI Shapefile https://s3.amazonaws.com/mapd-data/geodata/sffacs_current.zip 15
16. © MapD 2018 Tutorial B: Rendering with Vega 16
17. © MapD 2018 Load Using MapDQL • Clone the repo • git clone https://github.com/omveda/mapd-vega-mapboxgl-geo • Run the example (from the repo folder) • yarn install • npm start • Go to http://localhost:8080 • Change vegaspec.js to see points that are a specific distance from the utility lines • “name”: "sf_facilities", “sql”: “select conv_4326_900913_x(ST_X(A.mapd_geo)) as x, conv_4326_900913_y(ST_Y(A.mapd_geo)) as y, A.rowid from sf_facilities A, utility_lines B WHERE ST_Distance(ST_Transform(A.mapd_geo), ST_Transform(B.orig_geom)) < 20” 17
18. © MapD 2018 MapD VEGA MapD Vega is based on the open-source Vega Specification, it has been adapted to drive the rendering engine directly on the result set of a SQL query without ever requiring the data to leave the GPU. The MapD Connector API makes it easy to send the Vega JSON to the backend, which renders the visualization and returns a base64-encoded PNG image to the client. A Vega specification includes: • a data property that specifies and filters data source(s). • a marks property that defines the basic visualization graphic of a data item. • a scales property that defines geometry or applies additional attributes to the data item visualization. • viewing area dimensions.
19. © MapD 2018 Geospatial Functions Spatial Relationship and Measurement Functions Description ST_Distance Returns shortest planar distance between geometries. Returns shortest geodesic distance between geographies (in meters, limited support). ST_Contains Returns true if first geometry contains the second one.
20. © MapD 2018 Geospatial Functions Geometry Constructor Functions Description ST_GeomFromText Return a specified geometry from Well-Known Text representation (WKT) ST_GeogFromText Return a specified geography from Well-Known Text representation (WKT)
21. © MapD 2018 Geospatial Functions Geometry Editor Functions Description ST_Transform Return a new geometry with its coordinates transformed to a different spatial reference. The only supported transform in this release is WGS84 to Web Mercator, e.g. ST_Transform(ST_GeogFromText('POINT(-71.064544 42.28787)',4326),900913) ST_SetSRID Set the SRID on a geometry to a particular integer value.
22. © MapD 2018 Geospatial Functions Geometry Accessor Functions Description ST_XMin Returns X minima of a geometry. ST_XMax Returns X maxima of a geometry. ST_YMin Returns Y minima of a geometry. ST_YMax Returns Y maxima of a geometry. ST_StartPoint Returns the first point of a LINESTRING as a POINT. ST_EndPoint Returns the last point of a LINESTRING as a POINT. ST_PointN Return the Nth point of a LINESTRING as a POINT. ST_SRID Returns the spatial reference identifier for the underlying object
23. © MapD 2018 © MapD 2018 • mapd.com/demos Play with our demos • mapd.cloud Get a MapD instance in less than 60 seconds • mapd.com/platform/download-community/ Download the Community Edition • community.mapd.com Ask questions and share your experiences 23 Next Steps
24. © MapD 2018 Aaron Williams VP of Global Community @_arw_ aaron@mapd.com /in/aaronwilliams/ /williamsaaron slides: https://speakerdeck.com/mapd Thank you! Questions?