Nuno Oliveira Simone Giannecchini Andrea Aime GeoSolutions Processing and publishing Maritime AIS data with GeoServer and Databricks in Azure 

GeoSolutions Enterprise Support Services Deployment Subscription Professional Training Customized Solutions GeoNode • Offices in Italy & US, Global Clients/Team • 30+ collaborators, 25+ Engineers • Our products • Our Offer 

Affiliations We strongly support Open Source, it is in our core We participate in OGC testbeds and get funded to advance new open standards We support standards critical to GEOINT 

Big data and maritime security 

Using maritime data as our use case • Maritime Data is produced by a variety of sources: • Ships positions AIS, LRIT • Maritime assets ports, navigational aid systems, … 

The use case in numbers • In 24 hours: • We receive up to 50 millions positions reports • We handle up to 500K different ships • Peaks of activity during daylight: • Up to 2500 messages per second! • Azure data lake with 7 years of data: ~125 billion positions! 

Big data? • We can start with the usual three V’s: • Velocity • Volume • Variety • A practical definition from Wikipedia: • Big data refers to data sets that are too large or complex to be dealt with by traditional data-processing application software. • We already covered velocity and volume, what about variety? 

Maritime data overview • Provide a foundation for informed decision-making applications: • Maritime traffic monitoring • Search and rescue operations • Environmental marine disasters monitoring • … • Several datasets need to be combined: • Fisheries data • Ships registries information • …. Interoperability! 

• Authorization rights need to be respected: • Different authorization rights will result: Authorization rights … In different views of maritime assets! t1 Ships Sensors SAT-AIS T-AIS User 1 can see all vessels positions. User 2 can only see SAT-AIS vessels positions. t1 t0 t0 t1 t1 t1 t0 

Visualize in real time ships positions 

Use case overview • Displays the latest position for each known vessel in the last 24 hours. • System designed to handle up to 5K positions per second 432 millions positions per day! • Positions are enriched with several datasets, e.g. fisheries. 

Real time ships positions • Real time maritime picture displayed using a style that color each vessel according to its type: 

Real time ships positions • Real time maritime picture displaying only fishing vessels colored based on their fishing gear: 

Real time navigation to aid systems • Real time aircraft search and rescue operation: 

Real time navigation to aid systems • Real time aids to navigation systems positions: 

Real time ships positions • Real time maritime picture displayed using a polar projection (EPSG:5041): 

Advanced styling • Real time maritime picture displaying only cargo vessels, some of them highlighted: 

Reduced deployment diagram in Azure Kafka Cluster (Critical: RAM and DISK) SaaS - Managed Kafka Cluster PostgreSQL postgresql (32 vCPUs, 160 GB RAM, 8 TB, 20000 IOPS) SaaS - Gen 5, 32 vCore reads \ writes from topics kafka.head.x (2 VCPUs, 16 GB RAM, 135 GB DISK) kafka.worker.x (4 VCPUs, 4 GB RAM, 200 GB DISK) kafka.zookeeper.x (2 VCPUs, 14 GB RAM, 135 GB DISK) kafka.storage (1TB SSD) Kubernetes Cluster aks.x SaaS - DS2 v2 x3 x2 x3 x3 Ingestion Cluster (Critical: CPU and RAM): ingestion.1 (4 CPU, 8 GB RAM, 64 SSD + 64 PREMIUM SSD) IaaS - F4s v2 ingestion.2 (4 CPU, 8 GB RAM, 64 SSD + 64 PREMIUM SSD) GeoServer Cluster (Critical: CPU) geoserver.1 (4 CPU, 8 GB RAM, 64 SSD + 32 PREMIUM SSD) PaaS - F4s v2 geoserver.2 (4 CPU, 8 GB RAM, 64 SSD + 32 PREMIUM SSD) reads tables writes to tables manages x2 

Historical vessels tracks visualization 

Use case overview • Retrieve ship(s) historical positions from an Azure Data Lake: • Make them available through GeoServer OGC WFS, WMS and WPS services. • We can afford an initial preparation time, but then we need to be fast! ~125 billion positions! 

Ship positions correlation Annual voyage route of a bulk carrier from Asia to the Americas, Africa, and Europe 

Multiple ship tracks correlation Routes of 15 fishing vessels tracked over a 6-month period 

Ships in area identification 24-hour vessel traffic routes through the Strait of Gibraltar 

Read \ Write Architecture overview Azure Delta Lake Apache Spark SQL End-Point (Photon) Databricks Databricks Connector Read \ Write Coordination Read PG cache Cache 

The End Questions? [email protected] [email protected] [email protected]