From Raw to Ready: Rapid Sentinel-2 Index Workflows for Time-Sensitive Use Cases

This project began after a major data loss incident where more than 75,000 Sentinel-2 scenes and vegetation index products were lost just four days before launch. The team rebuilt the entire processing workflow under extreme time pressure using a cloud-native, open-source architecture with containers, scalable workers, and Kubernetes.

The new pipeline covers Sentinel-2 ingestion, cloud and cirrus masking, NDVI/LAI computation, COG conversion, STAC registration, and data delivery via OGC API-EDR.
All datasets were successfully regenerated within four days, demonstrating a reproducible, scalable workflow suitable for real operational applications in agriculture and large-area environmental monitoring.

Transcript

1. From Raw to Ready: Rapid Sentinel-2 Index Workflows for Time-Sensitive Use Cases Vallaris Team FOSS4G AUCKLAND 2025 17 Nov - 23 Nov 2025
2. Vallaris Maps We are Modern Geospatial Solution Ref: https://unsplash.com/photos/ucYWe5mzTMU Vallaris Maps is a geomatics platform that enables easy map creation with high flexibility in usage and delivers data services according to standards certified by The Open Geospatial Consortium (OGC®) – the first in Asia. It offers diverse options for map utilization on various devices including workstation, smartphones and tablets. It serves multiple industries and businesses.
3. Natpakal Maneerat Geo-informatics Engineer Siriwat Suttipanyo GIS Frontend Developer About Speaker
4. The Incident 4 Days Before Launch An issue occurred with the storage used for storing processed results. Storage failure The processed imagery generated from Sentinel-2 satellite data was lost. ~25,000 Sentinel-2 scenes lost Previously processed NDVI and LAI data have disappeared, including approximately 50,000 assets. NDVI & LAI indexes gone Only four days remain to complete the required work. Only 4 days remaining All data must be reprocessed within the deadline, as no backup was available. No backup 4
5. Using Sentinel-2 tiling grid for the Thailand area. Reprocess all Sentinel-2 scenes (2022–2025) Index all processed outputs into a STAC Catalog for organized Register results into STAC Process based on the Sentinel-2 tiling grid Recompute NDVI & LAI Convert results to Cloud-Optimized GeoTIFF Convert to COG Mission Required Sentinel-2 imagery from 2022–2025 is reprocessed and organized using Thailand’s tiling grid, with NDVI and LAI recalculated. Clouds are removed, outputs converted to COG, and all results indexed in a STAC Catalog for efficient access and visualization. Remove clouds and cirrus clouds from imagery Apply Cloud Map 5
6. Sentinel-2 tiling grid 6
7. Cloud-Native Principles Cloud-Native principles provided the ideas and capabilities that enabled the processing of large numbers of satellite imagery scenes to meet the goals. Principles A workflow designed to run efficiently on the cloud, with easy scalability and management. Cloud-native workflow Each algorithm is packaged in its own container, ensuring consistent execution, easy version control, and portable environments. Containers for every elements Processing workers do not store internal state, allowing them to be restarted or scaled out instantly without affecting operations. Stateless processing workers Using open standards such as STAC, COG, and EDR makes it easier to search, access, and share geospatial data with interoperability. STAC / COG / EDR (open standards) The processing pipeline can be repeated with identical results, reducing errors and enabling full traceability. Fully reproducible pipeline The system automatically adds more processing workers based on workload, enabling efficient handling of large-scale tasks. Horizontal auto-scaling 7
8. 8 Process workflow
9. 9 Process Monitoring
10. 10 Scalable Processing Workers Workers that can be easily scaled to handle varying Each worker runs in a container, encapsulating specific algorithms like Python NDVI or SNAP LAI for consistent execution. 1 The system automatically adjusts the number of workers according to workload demand. 2 Container-based workers Auto-scale up/down based on job volume (Kubernetes) 10
11. Tasks are managed via a queue, enabling orderly and parallel processing. 3 Workers do not store internal state, so they can be restarted or replaced without disrupting processing. 4 Designed to fully leverage cloud infrastructure for efficiency, flexibility, and scalability. 5 Queue-driven execution (RabbitMQ) Stateless design Optimized for cloud- native environments 11
12. Quick recovery from failures without manual infrastructure configuration, reducing downtime. 6 Faster recovery, no manual infra setup 12
13. NDVI & LAI Processing **It is a part of the processing code. 13
14. COG Output Transform all outputs into COG (Cloud- Optimized GeoTIFF) format for efficient cloud storage and access. Convert every output → Cloud Optimized GeoTIFF Enable quick search and retrieval of datasets via the STAC Catalog. Fast STAC lookup Structure data for efficient per-pixel access, making it suitable for Earth Data Record (EDR) analysis. Optimized for EDR pixel access Original Resolution Image Zoom factor 14
15. STAC Registration Arrange data hierarchically by quarter, year, and month for easy navigation. Organized by Quarter → Year → Month Maintain a single, consolidated catalog of all assets for consistency and discoverability. Unified asset catalog Fully compatible with any STAC-compliant client, enabling flexible access and integration. Works with any STAC client 15
16. EDR for Applications Provide access to data through the OGC API for Environmental Data Retrieval (EDR) standards. OGC API — Environmental Data Retrieval Allow retrieval of exact pixel values for precise analysis. Pixel-level value extraction Support real-time visualization in web or GIS frontend applications. Works with any STAC Instant rendering in frontend apps 16
17. Real world Use case Monitor plant health using NDVI data processed using EDR query NDVI Monitoring Use the LAI to recommend fertilizer application for rice quality Fertilizer Recommendation
18. 18 Why It Worked Recovery succeeded due to an architecture built on open-standards, containerization, cloud-native principles, modularity, and reproducibility. This incident proved that FOSS4G principles aren't just theoretical — they provided the foundational strength needed to rebuild 25,000 scenes in just four days. Open-Standards Based Adherence to open standards (STAC, COG, EDR) facilitated seamless integration and data accessibility, enabling rapid reconstruction. Modular A modular design allowed independent development and rapid iteration of processing steps, accelerating the recovery process significantly. Containerized Every algorithm packaged in containers ensured consistent environments and easy deployment across diverse compute resources, critical for fast scaling. Reproducible Fully reproducible pipelines guaranteed data integrity and allowed for quick re-execution of lost processes with confidence, saving valuable time. Cloud-Native Leveraging cloud-native architecture enabled dynamic scaling and efficient resource utilization, allowing us to meet demands even with limited compute capacity. 18
19. If you have any question, Feel free to contact us any time on (phone cell) or contact us by (email). We will get back to you as soon as we can I-bitz company limited 56/3 Soi Bunyu, Dindang rd. Phayathai Bangkok, Thailand 10400 Office Hours Monday-Friday 09.00-17.00 Get In Touch (+66) 2278 7913 More Information www.i-bitz.co.th sales@i-bitz.co.th Contact Us