Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Advancing Earth Science with Elasticsearch at Terradue

Dd9d954997353b37b4c2684f478192d3?s=47 Elastic Co
March 08, 2017

Advancing Earth Science with Elasticsearch at Terradue

Terradue develops and operates large cloud solutions for processing Earth observation satellite data, and uses Elasticsearch to empower users to expound it. Learn how they use the Elasticsearch .NET client to tackle challenges such as geohazards for rapid response monitoring of volcanoes or earthquakes...

Emmanuel Mathot l Technical Leader l Terradue

Dd9d954997353b37b4c2684f478192d3?s=128

Elastic Co

March 08, 2017
Tweet

Transcript

  1. Wednesday March 08 @emmanuelmathot Advancing Earth Science with Elasticsearch at

    Terradue Emmanuel Mathot, Tech Lead
  2. None
  3. ©ESA/PPO.LABS/NORUT/NGU

  4. None
  5. Central Italy Earthquake August 24th

  6. None
  7. None
  8. None
  9. B First observation Second observation Post-event image : 2016 Aug

    24th Pre-event image : within 6 months before with min 80% coverage Baseline < 120m Burst Sync < 5 ms
  10. None
  11. "_source": { "platformShortName": "S1A", ... "modified": "2016-08-29T12:05:55.72414", "published": "2016-08-29T12:05:55.722029Z", "startDate":

    "2016-08-26T07:19:43.426903+02:00", "endDate": "2016-08-26T07:20:11.341358+02:00", "geometry": { "coordinates": [ [ [ 12.880328, 40.986172 ], [ 9.856127, 41.385281 ], [ 10.212059, 43.060333 ], [ 13.319739, 42.661736 ], [ 12.880328, 40.986172 ] ] ], "type": "Polygon" }, "identifier": "S1A_IW_SLC__1SDV_20160826T051943_20160826T052011_012767_0141C5_3599", ... }
  12. None
  13. POST sentinel1/eopfeature/_search { "query": { "and": { "filters": [{ "geo_shape"

    : { "geometry" : { "relation" : "intersects", "shape" : { "coordinates": [[ [ 12.880328, 40.986172 ], [ 9.856127, 41.385281 ], ... ]], "type" : "Polygon" }}}, "range": { "start": { "gte": "2016-02-24", "lte": "2016-08-23"}}, }, "term": { "track": "128" } ]} }}
  14. ü Post-event image : 2016 Aug 24th ü Pre-event image

    : within 6 months before with min 80% coverage q Baseline < 120m q Burst Sync < 5 ms
  15. None
  16. { "platform": "S1A", "type": "SLC", "time": "2017-02-06T08:31:48.027691Z", "frame": "EarthFixed", "absoluteOrbit":

    15160, "position": [ -5613366.283966, 4302302.834969, -256349.609754 ], "velocity": [ 756.880948, 1415.929115, 7425.469561 ] }
  17. OSV 1 (tn) OSV 2 (tn+10s) OSV 3 (tn+20s) OSV

    4 (tn+30s) OSV 5 (tn+40s) tn + 24.341s
  18. POST sentinel1/eoporbit/_search { "aggs": { "OSV": { "scripted_metric": { "init_script":

    { "_comment" : "Init array of vectors and the timestamp", "file": "init_osv.py" }, "map_script" : { "_comment" : "Load orbit state vectors", "file": "load_osv.py" }, "reduce_script" : { "_comment" : "Robust Cublic Spline interpolation", "file": "cubic_spline_interpolation.py" } } } ~ 50 orbit state vectors
  19. First observation Second observation

  20. { "platform": "S1A", "azimuthTime": "2017-02-06T08:30:45.951147Z", "slantRangeTime": 0.005821934184880311, "line": 15174, "pixel":

    12660, "latitude": -4.903591527542559, "longitude": 147.6990386159871, "height": -0.000001518987119197845, "incidenceAngle": { "value": 38.98923179289942, "uom": "deg" }, "elevationAngle": { "value": 34.53983660078405, "uom": "deg" }, "modified": "2017-02-06T15:11:51.464773Z" }
  21. POST sentinel1/eoppoint/_search { "aggs": { "slantRange": { "avg": { "field":

    "slantRangeTime" } } }, "query": { "fuzzy": { "azimuthTime": { "value" : "2016-08-24T10:05:07.276Z", "fuzziness" : "700ms" } } } }
  22. B First observation Second observation S

  23. ~ 1,000,000 scenes ~ 13,000,000 orbit state vectors ~ 270,000,000

    ground points # documents
  24. None
  25. None
  26. None
  27. None
  28. None
  29. None
  30. None
  31. None
  32. None
  33. None
  34. { "platform": "S1A", "dataset_id": "S1A_IW_GRDH_1SDV_20160729T182822_20160729T182847_012367_013456_E8BF", "num_lines": 25184, "num_pixels": 16804, "stat_fn":

    "norm", "stat_fn_paramaters": { “mean": -13.9180515377, “stddev": 2.52405823515 }, "dB_spread": { "pctl5th": -18.6049933434, "pctl95th": -10.1168370247, "diff": 8,4881563187 }, "modified": "2017-02-06T15:11:51.464773Z" }
  35. POST sentinel1/backscatter_profile/_search { "aggs": { "anomaly": { "scripted_metric": { "init_script":

    { "_comment" : "Init functions & anomaly indices", "file": "init_anomaly_idx.py" }, "map_script" : { "_comment" : "Load PDF and CDF from stat fn", "file": "load_stat.py" }, "reduce_script" : { "_comment" : “Compute anomaly indices", "file": "compute_anomaly_idx.py" } }, }
  36. i1 0.25463341 i2 0.18215659 i3 0.31308018 i1 0.00552214 i2 0.

    i3 0.00552214 i1 0.01139076 i2 0.05227872 i3 0.05350527 i1 0.03202082 i2 0.16190536 i3 0.16504144 i1 0. i2 0.03447718 i3 0.03447718 i1 0.15261477 i2 0. i3 0.15261477 i1 0.64931845 i2 0.41153427 i3 0.76874892 i1 1. i2 1. i3 1.41421356
  37. None
  38. Elasticsearch Elasticsearch.net + NEST OpenSearch Formats Geo & Time EOP

    Correlation Geo JSON Geo RSS KML Terradue Cloud Platform Catalogue
  39. Elasticsearch Nodes Nodes Nodes HA + Load balancer 5 Intel®

    Xeon® E3-1275 v5 Quad-Core Skylake RAM 64GB DDR4 ECC 2 x 512 GB SSD (RAID 1)
  40. 0 2 4 6 8 10 12 2014 2015 2016

    2017 2018 2019 2020 Millions Sentinel-1 Sentinel-2 Sentinel-3 Sentinel-4 Sentinel-5
  41. None
  42. None
  43. None
  44. Questions ?