Advancing Earth Science with Elasticsearch at Terradue

Terradue develops and operates large cloud solutions for processing Earth observation satellite data, and uses Elasticsearch to empower users to expound it. Learn how they use the Elasticsearch .NET client to tackle challenges such as geohazards for rapid response monitoring of volcanoes or earthquakes...

Emmanuel Mathot l Technical Leader l Terradue

Transcript

1. Wednesday March 08 @emmanuelmathot Advancing Earth Science with Elasticsearch at Terradue Emmanuel Mathot, Tech Lead
3. ©ESA/PPO.LABS/NORUT/NGU
5. Central Italy Earthquake August 24th
9. B First observation Second observation Post-event image : 2016 Aug 24th Pre-event image : within 6 months before with min 80% coverage Baseline < 120m Burst Sync < 5 ms
11. "_source": { "platformShortName": "S1A", ... "modified": "2016-08-29T12:05:55.72414", "published": "2016-08-29T12:05:55.722029Z", "startDate": "2016-08-26T07:19:43.426903+02:00", "endDate": "2016-08-26T07:20:11.341358+02:00", "geometry": { "coordinates": [ [ [ 12.880328, 40.986172 ], [ 9.856127, 41.385281 ], [ 10.212059, 43.060333 ], [ 13.319739, 42.661736 ], [ 12.880328, 40.986172 ] ] ], "type": "Polygon" }, "identifier": "S1A_IW_SLC__1SDV_20160826T051943_20160826T052011_012767_0141C5_3599", ... }
13. POST sentinel1/eopfeature/_search { "query": { "and": { "filters": [{ "geo_shape" : { "geometry" : { "relation" : "intersects", "shape" : { "coordinates": [[ [ 12.880328, 40.986172 ], [ 9.856127, 41.385281 ], ... ]], "type" : "Polygon" }}}, "range": { "start": { "gte": "2016-02-24", "lte": "2016-08-23"}}, }, "term": { "track": "128" } ]} }}
14. ü Post-event image : 2016 Aug 24th ü Pre-event image : within 6 months before with min 80% coverage q Baseline < 120m q Burst Sync < 5 ms
16. { "platform": "S1A", "type": "SLC", "time": "2017-02-06T08:31:48.027691Z", "frame": "EarthFixed", "absoluteOrbit": 15160, "position": [ -5613366.283966, 4302302.834969, -256349.609754 ], "velocity": [ 756.880948, 1415.929115, 7425.469561 ] }
17. OSV 1 (tn) OSV 2 (tn+10s) OSV 3 (tn+20s) OSV 4 (tn+30s) OSV 5 (tn+40s) tn + 24.341s
18. POST sentinel1/eoporbit/_search { "aggs": { "OSV": { "scripted_metric": { "init_script": { "_comment" : "Init array of vectors and the timestamp", "file": "init_osv.py" }, "map_script" : { "_comment" : "Load orbit state vectors", "file": "load_osv.py" }, "reduce_script" : { "_comment" : "Robust Cublic Spline interpolation", "file": "cubic_spline_interpolation.py" } } } ~ 50 orbit state vectors
19. First observation Second observation
20. { "platform": "S1A", "azimuthTime": "2017-02-06T08:30:45.951147Z", "slantRangeTime": 0.005821934184880311, "line": 15174, "pixel": 12660, "latitude": -4.903591527542559, "longitude": 147.6990386159871, "height": -0.000001518987119197845, "incidenceAngle": { "value": 38.98923179289942, "uom": "deg" }, "elevationAngle": { "value": 34.53983660078405, "uom": "deg" }, "modified": "2017-02-06T15:11:51.464773Z" }
21. POST sentinel1/eoppoint/_search { "aggs": { "slantRange": { "avg": { "field": "slantRangeTime" } } }, "query": { "fuzzy": { "azimuthTime": { "value" : "2016-08-24T10:05:07.276Z", "fuzziness" : "700ms" } } } }
22. B First observation Second observation S
23. ~ 1,000,000 scenes ~ 13,000,000 orbit state vectors ~ 270,000,000 ground points # documents
34. { "platform": "S1A", "dataset_id": "S1A_IW_GRDH_1SDV_20160729T182822_20160729T182847_012367_013456_E8BF", "num_lines": 25184, "num_pixels": 16804, "stat_fn": "norm", "stat_fn_paramaters": { “mean": -13.9180515377, “stddev": 2.52405823515 }, "dB_spread": { "pctl5th": -18.6049933434, "pctl95th": -10.1168370247, "diff": 8,4881563187 }, "modified": "2017-02-06T15:11:51.464773Z" }
35. POST sentinel1/backscatter_profile/_search { "aggs": { "anomaly": { "scripted_metric": { "init_script": { "_comment" : "Init functions & anomaly indices", "file": "init_anomaly_idx.py" }, "map_script" : { "_comment" : "Load PDF and CDF from stat fn", "file": "load_stat.py" }, "reduce_script" : { "_comment" : “Compute anomaly indices", "file": "compute_anomaly_idx.py" } }, }
36. i1 0.25463341 i2 0.18215659 i3 0.31308018 i1 0.00552214 i2 0. i3 0.00552214 i1 0.01139076 i2 0.05227872 i3 0.05350527 i1 0.03202082 i2 0.16190536 i3 0.16504144 i1 0. i2 0.03447718 i3 0.03447718 i1 0.15261477 i2 0. i3 0.15261477 i1 0.64931845 i2 0.41153427 i3 0.76874892 i1 1. i2 1. i3 1.41421356
38. Elasticsearch Elasticsearch.net + NEST OpenSearch Formats Geo & Time EOP Correlation Geo JSON Geo RSS KML Terradue Cloud Platform Catalogue
39. Elasticsearch Nodes Nodes Nodes HA + Load balancer 5 Intel® Xeon® E3-1275 v5 Quad-Core Skylake RAM 64GB DDR4 ECC 2 x 512 GB SSD (RAID 1)
40. 0 2 4 6 8 10 12 2014 2015 2016 2017 2018 2019 2020 Millions Sentinel-1 Sentinel-2 Sentinel-3 Sentinel-4 Sentinel-5
44. Questions ?