Advancing Earth Science with Elasticsearch at Terradue

Slide 1

Slide 1 text

Wednesday March 08 @emmanuelmathot Advancing Earth Science with Elasticsearch at Terradue Emmanuel Mathot, Tech Lead

Slide 2

Slide 2 text

No content

Slide 3

Slide 3 text

©ESA/PPO.LABS/NORUT/NGU

Slide 4

Slide 4 text

No content

Slide 5

Slide 5 text

Central Italy Earthquake August 24th

Slide 6

Slide 6 text

No content

Slide 7

Slide 7 text

No content

Slide 8

Slide 8 text

No content

Slide 9

Slide 9 text

B First observation Second observation Post-event image : 2016 Aug 24th Pre-event image : within 6 months before with min 80% coverage Baseline < 120m Burst Sync < 5 ms

Slide 10

Slide 10 text

No content

Slide 11

Slide 11 text

"_source": { "platformShortName": "S1A", ... "modified": "2016-08-29T12:05:55.72414", "published": "2016-08-29T12:05:55.722029Z", "startDate": "2016-08-26T07:19:43.426903+02:00", "endDate": "2016-08-26T07:20:11.341358+02:00", "geometry": { "coordinates": [ [ [ 12.880328, 40.986172 ], [ 9.856127, 41.385281 ], [ 10.212059, 43.060333 ], [ 13.319739, 42.661736 ], [ 12.880328, 40.986172 ] ] ], "type": "Polygon" }, "identifier": "S1A_IW_SLC__1SDV_20160826T051943_20160826T052011_012767_0141C5_3599", ... }

Slide 12

Slide 12 text

No content

Slide 13

Slide 13 text

POST sentinel1/eopfeature/_search { "query": { "and": { "filters": [{ "geo_shape" : { "geometry" : { "relation" : "intersects", "shape" : { "coordinates": [[ [ 12.880328, 40.986172 ], [ 9.856127, 41.385281 ], ... ]], "type" : "Polygon" }}}, "range": { "start": { "gte": "2016-02-24", "lte": "2016-08-23"}}, }, "term": { "track": "128" } ]} }}

Slide 14

Slide 14 text

ü Post-event image : 2016 Aug 24th ü Pre-event image : within 6 months before with min 80% coverage q Baseline < 120m q Burst Sync < 5 ms

Slide 15

Slide 15 text

No content

Slide 16

Slide 16 text

{ "platform": "S1A", "type": "SLC", "time": "2017-02-06T08:31:48.027691Z", "frame": "EarthFixed", "absoluteOrbit": 15160, "position": [ -5613366.283966, 4302302.834969, -256349.609754 ], "velocity": [ 756.880948, 1415.929115, 7425.469561 ] }

Slide 17

Slide 17 text

OSV 1 (tn) OSV 2 (tn+10s) OSV 3 (tn+20s) OSV 4 (tn+30s) OSV 5 (tn+40s) tn + 24.341s

Slide 18

Slide 18 text

POST sentinel1/eoporbit/_search { "aggs": { "OSV": { "scripted_metric": { "init_script": { "_comment" : "Init array of vectors and the timestamp", "file": "init_osv.py" }, "map_script" : { "_comment" : "Load orbit state vectors", "file": "load_osv.py" }, "reduce_script" : { "_comment" : "Robust Cublic Spline interpolation", "file": "cubic_spline_interpolation.py" } } } ~ 50 orbit state vectors

Slide 19

Slide 19 text

First observation Second observation

Slide 20

Slide 20 text

{ "platform": "S1A", "azimuthTime": "2017-02-06T08:30:45.951147Z", "slantRangeTime": 0.005821934184880311, "line": 15174, "pixel": 12660, "latitude": -4.903591527542559, "longitude": 147.6990386159871, "height": -0.000001518987119197845, "incidenceAngle": { "value": 38.98923179289942, "uom": "deg" }, "elevationAngle": { "value": 34.53983660078405, "uom": "deg" }, "modified": "2017-02-06T15:11:51.464773Z" }

Slide 21

Slide 21 text

POST sentinel1/eoppoint/_search { "aggs": { "slantRange": { "avg": { "field": "slantRangeTime" } } }, "query": { "fuzzy": { "azimuthTime": { "value" : "2016-08-24T10:05:07.276Z", "fuzziness" : "700ms" } } } }

Slide 22

Slide 22 text

B First observation Second observation S

Slide 23

Slide 23 text

~ 1,000,000 scenes ~ 13,000,000 orbit state vectors ~ 270,000,000 ground points # documents

Slide 24

Slide 24 text

No content

Slide 25

Slide 25 text

No content

Slide 26

Slide 26 text

No content

Slide 27

Slide 27 text

No content

Slide 28

Slide 28 text

No content

Slide 29

Slide 29 text

No content

Slide 30

Slide 30 text

No content

Slide 31

Slide 31 text

No content

Slide 32

Slide 32 text

No content

Slide 33

Slide 33 text

No content

Slide 34

Slide 34 text

{ "platform": "S1A", "dataset_id": "S1A_IW_GRDH_1SDV_20160729T182822_20160729T182847_012367_013456_E8BF", "num_lines": 25184, "num_pixels": 16804, "stat_fn": "norm", "stat_fn_paramaters": { “mean": -13.9180515377, “stddev": 2.52405823515 }, "dB_spread": { "pctl5th": -18.6049933434, "pctl95th": -10.1168370247, "diff": 8,4881563187 }, "modified": "2017-02-06T15:11:51.464773Z" }

Slide 35

Slide 35 text

POST sentinel1/backscatter_profile/_search { "aggs": { "anomaly": { "scripted_metric": { "init_script": { "_comment" : "Init functions & anomaly indices", "file": "init_anomaly_idx.py" }, "map_script" : { "_comment" : "Load PDF and CDF from stat fn", "file": "load_stat.py" }, "reduce_script" : { "_comment" : “Compute anomaly indices", "file": "compute_anomaly_idx.py" } }, }

Slide 36

Slide 36 text

i1 0.25463341 i2 0.18215659 i3 0.31308018 i1 0.00552214 i2 0. i3 0.00552214 i1 0.01139076 i2 0.05227872 i3 0.05350527 i1 0.03202082 i2 0.16190536 i3 0.16504144 i1 0. i2 0.03447718 i3 0.03447718 i1 0.15261477 i2 0. i3 0.15261477 i1 0.64931845 i2 0.41153427 i3 0.76874892 i1 1. i2 1. i3 1.41421356

Slide 37

Slide 37 text

No content

Slide 38

Slide 38 text

Elasticsearch Elasticsearch.net + NEST OpenSearch Formats Geo & Time EOP Correlation Geo JSON Geo RSS KML Terradue Cloud Platform Catalogue

Slide 39

Slide 39 text

Elasticsearch Nodes Nodes Nodes HA + Load balancer 5 Intel® Xeon® E3-1275 v5 Quad-Core Skylake RAM 64GB DDR4 ECC 2 x 512 GB SSD (RAID 1)

Slide 40

Slide 40 text

0 2 4 6 8 10 12 2014 2015 2016 2017 2018 2019 2020 Millions Sentinel-1 Sentinel-2 Sentinel-3 Sentinel-4 Sentinel-5