Enhancing GIS with Cloud Technology and Serverless Computing

Cloud and serverless technologies have been growing rapidly in the computing and GIS space over the past several
years. These advancements are allowing individuals and organizations to quickly and cost effectively spin up and down
computing resources on an as needed, scalable basis. Gone are the days of needing to obtain, provision, and maintain
physical server resources for every IT need. Cloud computing resources can instead be created or decommissioned in a
matter of minutes and GIS SaaS providers such as ArcGIS Online (AGOL) allow for rapid deployment of GIS solutions with
or without the need for enterprise GIS server technology. A hybrid GIS solution can be achieved whereby AGOL web
maps and hosted layers allow for a rapid solution to business problems while AWS cloud resources can be leveraged to
fill the missing pieces of web hosting and on demand compute power. This talk will cover use cases, and their associated
cost effectiveness, for leveraging Amazon Web Service (AWS) cloud technologies to enhance the capabilities of the Esri
GIS stack at three distinct levels of integration: 1) Using AWS Lambda functions as a computation engine for data ETL in
AGOL hosted feature layers, 2) Amazon S3 buckets and CloudFront for hosting web applications customized using the
Esri Web App Builder SDK, and 3) Full custom GIS application development using AWS EC2, API Gateway, and Relational
Database Service. By combining the powerful spatial capabilities of ArcGIS Online with the ease, availability, and
scalability of cloud-based hosting solutions you can build powerful and compelling spatial applications that are
affordable and easily maintainable.

Transcript

1. Enhancing GIS with Cloud Technology and Serverless Computing Yogesh Dhanapal Chaz Mateer
2. Introductions • Yogesh Dhanapal ◦ Senior GIS Application Developer – E- Government Solutions ◦ yogesh.dhanapal@timmons.com • Chaz Mateer, GISP ◦ GIS Application Developer – Geospatial Enterprise Solutions ◦ chaz.mateer@timmons.com • Timmons Group ◦ Civil Engineering and Geospatial Technology ◦ http://www.timmons.com/gis
3. 1. Cloud Tech Overview 2. Hosting a static web app ◦ Use Case: Custom web app builder or ArcGIS JS API on AWS S3 3. Serverless computing ◦ Use Case: Post processing AGOL hosted content 4. Full stack development ◦ Use Case: Fully custom web app development Agenda
4. Cloud Tech Overview • “Delivery of hosted services over the internet…” • Features ◦ Self-service provisioning ◦ Elasticity ◦ Pay per use ◦ Workload resilient ◦ Migration flexibility
5. Deployment Models
6. Common Vendors
7. Market Share
8. • S3 • RDS • Cloud Front • Lambda • API Gateway • EC2 • Rte53 • Lots of buzz words! Essentially these encapsulate your common IT workflows and functions. Amazon Web Services
9. Cloud Tech and GIS • AGOL – SaaS • Leverage AWS to support existing GIS tech • RDS for scalable geodatabase storage • EC2 for hosting ArcGIS Server/Enterprise • S3 for data storage • Lambda for scheduled tasks and data ETL jobs
10. Cost, scalability, ease of access and control Why Cloud?
11. Allows us to not think of infrastructure as a pet… Infrastructure as code.
12. Use Cases Cloud and GIS Hosting GIS web app Hosting Tiled Map Services Geospatial analysis as api Authoring Scheduled Task Hosting traditional GIS Web Applications
13. Hosting GIS web app Use Case Goal • Serve a GIS web application • Could be developed from scratch using ArcGIS Javascript Api, mapbox, openlayers or leaflet • ArcGIS Web Appbuilder application with custom widgets
14. Technology Used • Amazon S3 – storage -> website content • Cloudfront – content delivery network to speed up web (Optional) ◦ Required for HTTPS • Route 53 - DNS • AWS Certificate Manager - SSL certificate (Optional) ◦ Required for HTTPS
15. Use Case: Guardrail Windshield Assessment
16. • Integrate very well with ArcGIS Online and Portal for ArcGIS Identity Security
17. • AWS ResourceCostAmazon S3 Service (US-East) - $0.13 per month • Amazon Route 53 Service - $0.9 per month • Amazon CloudFront Service - $1.94 per month • Amazon CloudWatch Service (US-East) - $0 per month • AWS Data Transfer Out - $0 per month • AWS Support (Basic) - $0 per month • AWS SSL Certificate – $0 per month Note: annual domain registration and renewal fee of $ 12 Total Operating cost $3.02 per month
18. Key Takeaways • Ideal for hosting public facing static GIS web applications • Low cost • High performance • Scalable • Integrates very well with AGOL and Portal for ArcGIS security
19. DevOps Pipeline
20. Hosting Tiled Map Services Use Case Goal • Serve large GIS dataset as Map service like crash points, LRS route network • Fast • Scalable • Map should be rendered in all zoom levels • Solution: Vector Tile Maps
21. Technology Used • Amazon S3 – storage -> store vector tiles • Tippecanoe - CLI tool to generate vector tiles from large geojson file • GDAL - geospatial tool to generate geojson from other spatial formats like shapefile (Optional) • AWS Code build - Runs container on demand ◦ Executes Tippecanoe tool to generate tiles and push to S3 tippecanoe --output-to-directory=tiles/aawdt geojson/aawdt.json --no-tile-size-limit (simplified) aws s3 sync `pwd`/tiles/aawdt s3://vector-tiles-map/tiles/aawdt --delete --content- type "application/x-protobuf"
22. Consuming Vector Tile map service • Tiled service endpoint of Annual Average Weekday Traffic data https://s3.amazonaws.com/vector-tiles- map/tiles/aawdt/{z}/{x}/{y}.pbf • Using vector tiles in Mapbox GL js
23. Key Takeaways • Ideal for large GIS data - static • Zero licensing cost • High performance • Scalable • Integrates very well with Mapbox gl js, ArcGIS Javascript Api and Open Layers • Works well in modern browsers because of webgl support • Same approach can be extended raster data by using rasterio (python library for working with raster) instead of Tippecanoe • ArcGIS Pro can used to perform similar approach but automation is not straight forward
24. Geospatial Analysis as API Use Case Goal • Develop a API which performs geospatial analysis • Fast • Scalable
25. Technology Used • Amazon S3 – storage -> store input data (optional) • Libraries ◦ Geospatial - numpy, rasterio, shapely, mercantile, supermercado, fiona ◦ Image Analysis & Machine Learning - numpy, scipy, scikit-learn, rasterio, OpenCV • API Gateway - used to develop API at scale • Lambda - runs the geospatial analysis and return results
26. Use Case: API to perform reverse geocoding at Scale Serverless API solution • Develop solution using api gateway, lambda • Use US Census bureau state boundaries and use point in polygon to determine the which state the proven point is • Pick nearby boundary in case point in not in any of the boundary (in order to handle situation - some of the bridges are not covered in boundary layer) • Process one million records in 30 seconds • Tax service company - calculates tax for trucks travel • They have lat, lon for truck locations and they want to determine which state that truck was travelling which is need to calculate tax they need to pay for each state • They have millions of GPS locations • Existing solution - using google reverse geocode api ◦ Takes hours to execute, more usage cost
27. $20/month ~3 million requests/month (old solution cost $5,000/month)
28. Key Takeaways • Ideal for simple and complex geospatial operations that can be parallelized • Zero licensing cost • High performance • Scalable • Same approach can be extended to raster or machine learning operations by choosing appropriate libraries.
29. Authoring Scheduled Task Use Case Goal • Develop scheduled task that tiggers at regular intervals like every 5 mins • Low maintenance • Cost effective
30. Technology Used • Cloudwatch - aws service that triggers lambda at specified intervals • Lambda - Executes a logic
31. Use Case: Enriching of AGOL Hosted Service Data • Rest Areas Assessment system using Collector for ArcGIS • generates rich reports and add is as attachment to AGOL hosted feature • Scoring process On Field Collected Data • Approval workflow on Field collected data ◦ Field inspector completes the initial inspection using collector for ArcGIS ◦ Field supervisor reviews and Submits to Stakeholder ◦ Stakeholder reviews and Accepts or Rejects the inspection report
32. $10/month
33. Key Takeaways • Ideal for performing ETL jobs, reports generation and data processing • Low cost • Low maintenance
34. Hosting traditional GIS Web Applications Use Case • Deploy enterprise GIS application solution on cloud infrastructure
35. Hosting traditional GIS Web Applications • AWS EC2 – Web application servers and ArcGIS Enterprise • AWS RDS – Enterprise geodatabase • AWS Simple Email Service – Sending Emails • AWS Lambda – containerized compute power
36. • WebApp – Angular Application Hosted in S3 • WebAPI – Elastic beanstalk running ASP.NET Core • GIS Server – EC2 hosting ArcGIS Enterprise • Geodatabase – PostgreSQL RDS • Automation – Jenkins Application Components
37. Pathways for Planning
38. Benefits (costs are similar to on-prem) Scalability and availability
39. Questions? yogesh.dhanapal@timmons.com chaz.mateer@timmons.com