Design Patterns of GeoData Driven Apps

Modern geospatial applications are becoming more and more complex. Whether it be an application for self driving cars or prediction engine for energy forecasting the number of moving pieces are increasing. Applications are dealing with real-time data, often with errors and/or missing data. In addition to this skills specialization means teams working on these applications have to collaborate and communicate across different aspects of the software in a meaningful way. This can range from machine learning and data science workflows to delivering meaningful insights to end users. In this talk we will describe design patterns of geo data pipelines in the age of big data & machine learning. We will provide tips on the open source options available to develop such architectures. Where possible we will identify gaps in the market.

Transcript

1. Final presentation wherecamp berlin 2019 1
2. Design Patterns of GeoData Driven Apps Shoaib Burq https://geografia.com.au http://deeplearningbox.com WhereCamp 2019 (Berlin)
3. Complexity in Data Driven Apps 3 greater number of data sources more stakeholders more applications bigger datasets
7. 7 Client Name
8. v1 (2017) - Traditional Geo Applications 8
9. v1: Challenges 9 No Quality Checks Poor Schema Enforcement Time Consuming Not Scalable for Growth
11. 11 v2: add workflow system
12. 12 v2: add workflow system Add S3 to Store Raw Data Trigger Workflow (AWS lambda)
13. 13 v2: add workflow system Add S3 to Store Raw Data Trigger Workflow (AWS lambda) https://docs.prefect.io/core
14. 14 Forecasting with Machine / Deep Learning
15. v3: Doing Data Science 15
16. v3: Doing Data Science 16 Workstation with GPUs to train ML & DL Models ————————— deeplearningbox.com —————————
17. v3: Doing Data Science 17 Workstation with GPUs to build Deep Learning Models Competing for Resources with applicatioon Manual Deployment of ML Models
18. v4: ML Model Tracking + Deployments 18
19. v4: PostgreSQL read only replicas 19
20. v4: MLFlow + RDMBS scaling 20
21. v4: MLFlow + RDMBS scaling 21 Fix: Competing for Resources Fix: ML Experiment Track & Deploy
22. Future expected demands from customers & team 22
23. Future Interactive Big Data Queries via Hadoop + Presto 23
24. Future Real-Time events integration (e.g. social media feed) 24
25. Thank you! Questions? Shoaib Burq https://geografia.com.au | http:// deeplearningbox.com 25