The Euclid QLA Processing Framework

A description of the Euclid mission, the Operations Centre, and the Quick Look Processing Framework

Transcript

1. The Euclid QLA Processing Framework J. C. González Euclid SOC Team @ ESAC November 11th, 2016
2. The QLA Processing Framework J.C.González – Euclid SOC Team @ ESAC Nov.2016 Slide 2 of 25 Outline  The Euclid Mission  Euclid Operations and Architecture  Quick-Look Analysis System  QLA Processing Framework
3. The QLA Processing Framework J.C.González – Euclid SOC Team @ ESAC Nov.2016 Slide 3 of 25 The Euclid Mission – Overview (I) Goal and Scientific Objectives Understand the nature of Dark Energy and Dark Matter  Measure the DE equation of state parameters wp and wa to a precision of 2% and 10%, respectively, using both expansion history and structure growth, in order to be able to discriminate between a true cosmological constant and other models of Dark Energy  Measure g, the exponent of the growth factor, with precision enough to distinguish General Relativity and a wide range of modified-gravity theories  Test the Cold Dark Matter paradigm for hierarchical structure formation, and measure the sum of the neutrino masses with high precision  Constrain ns , the spectral index of primordial power spectrum, to percent accuracy when combined with Planck, and to probe inflation models by measuring the non-Gaussianity of initial conditions parametrized by fNL to a 1s precision of ~2.
4. The QLA Processing Framework J.C.González – Euclid SOC Team @ ESAC Nov.2016 Slide 4 of 25 The Euclid Mission – Overview (II) Primary Cosmological Probes Weak Gravitational Lensing Gravitational potential of “foreground” structures perturbs the paths of photons emitted by distant galaxies. The amplitude of the distortion gives a direct measure of the gravitational tidal field, which is used to “map” the distribution of (dark and luminous) matter directly. The WL signal can be extracted by measuring the correlations in the galaxy shapes. Photometric redshift for 1.5 billion sources with shapes in 10 slices  weak lensing tomography Galaxy Clustering Baryon Acoustic Oscillations Measure redshift and position of galaxies Obtain the power spectrum for a given redshift bin Determine scale length from the “wiggles” – the acoustic peaks - in the power spectrum Redshift Space Distortions Measure redshift and position of galaxies Determine the growth rate f(z) for a given redshift bin: f(z)=[Ωm(z)]γ Dark Matter
5. The QLA Processing Framework J.C.González – Euclid SOC Team @ ESAC Nov.2016 Slide 5 of 25 Launcher: Soyuz ST-2.1 B from Kourou Orbit: Large Sun-Earth Lagrange point 2 (SEL2), free insertion orbit Pointing: (99.7% CL) 75 mas rel.error/700 s, 7.5 arcsec abs.error Obs.mode: Step and stare, 4 dither frames per field, VIS and NISP common FoV = 0.54 deg2 Lifetime: 7 years Operations: 4 hours per day contact, more than one ground station to cope with seasonal visibility variations. Comms: Maximum science data rate of 850 Gbit/day downlink in K band (26GHz), steerable HGA Wide Survey: Required: 15,000 deg2 Step and stare with 4 dither pointing per field. Deep Survey: Total Area: 40 deg2 2 magnitudes deeper than wide survey in at least 2 patches of ≥10 deg2 Deep fields must be in clean regions and observable throughout the year, hence near the Ecliptic poles The Euclid Mission – Overview (III) Spacecraft Telescope: 1.2 m Korsch, 3 mirror anastigmat, f=24.5 m VIS: 0.787×0.709 deg2 36 arrays 4k×4k CCD Visual Imaging: 550– 900 nm 24.5mag 10σ ext.src. 0.1 arcsec NISP: 0.763×0.722 deg2 16 arrays 2k×2k NIR sensitive HgCdTe detectors NIR Imaging Photometry: Y (920-1146nm), J (1146-1372nm), H (1372-2000nm) 24mag 5σ point source 0.3 arcsec NIR Spectroscopy: 1250-1850 nm (0.8<z<1.8 from Ha) 2×10-16 (4.5σ) for 0.5 arcsec diameter R≥380 Payload Surveys
6. The QLA Processing Framework J.C.González – Euclid SOC Team @ ESAC Nov.2016 Slide 6 of 25 Euclid Operations Main Entities (I) Mission Operations Centre External Data Providers Science Operations Centre Science Data Centres Euclid Archive IOT’s Coordinator IOT VIS IOT NISP Science Ground Segment
7. The QLA Processing Framework J.C.González – Euclid SOC Team @ ESAC Nov.2016 Slide 7 of 25 Euclid Operations Main Entities (II) The Euclid Mission Operations Centre at ESOC is the responsible for the consolidation and uplink of all the telecommanding to Euclid, including those related to instrument operations. It is also the responsible to obtain the telemetry generated on board, and make it available to the SOC. Being the responsible to uplink telecommands, it is the final responsible to guarantee the instrument safety and health. The MOC shall also act as configuration control holder of the instruments operational DBs (MIB), being the entity in charge of its formal dissemination to SOC and operational deployment. The Science Operations Centre at ESAC is the responsible of the Survey execution. It generates the required commanding requests to ensure its fulfillment as well as the routine calibration plan. It also acts as formal interface from the IOT’s to the MOC during the routine mission. The SOC is the entity responsible to gather the data provided by the MOC and make it available to the rest of the Science Ground Segment in appropriate format, generating the Level 1. The Science Data Centres are entrusted to run pipelines to process the Science Telemetry of the instruments. Finally, the Instrument Operations Teams are the entities responsible to define the Instrument Operations, mainly the instrument operational procedures, the instrument DBs and the criteria for instrument safety and health, as well as for instrument monitoring. The two IOT’s will have a single IOT coordinator that will act as single point of contact towards SOC (and therefore MOC). MOC SOC SDCs IOTs
8. The QLA Processing Framework J.C.González – Euclid SOC Team @ ESAC Nov.2016 Slide 8 of 25 Science Operations Centre SOC Components QLA Quick Look Analysis HMS Health Monitoring System ESS Euclid Survey System L1P Level 1 Processing EAS@SOC Euclid Archive System at SOC SOC Storage Mission Operations Centre Scientific data S/C HK data Level 1 VIS & NISP products QLA Parameters QLA Diagnostics Reports SCS SOC Command System SIS SOC Interface System Commanding requests
9. The QLA Processing Framework J.C.González – Euclid SOC Team @ ESAC Nov.2016 Slide 9 of 25 The Quick-Look Analysis Subsystem Responsibilities  Automatically check S/C data, produce and ingest reports in the Euclid Archive System within 48 hours from data retrieval to confirm that the data are suitable for scientific analysis and eventually quickly react to issues. The IOT (instrument operation team) and EC (Euclid consortium) will investigate the data in further details  Facilitate data inspection by the user (mainly instrument scientists) in case further data inspection is needed Capable of accessing detectors data, showing it and/or exporting it to external programs, etc.
10. The QLA Processing Framework J.C.González – Euclid SOC Team @ ESAC Nov.2016 Slide 10 of 25 QLA QLA Decomposition (I) Analysis and Processing SDA PDA ADA PRO CDA VIM QAF REP PAR CAM SIF DDS TM CDFP TM Aux Data Configuration Logs Visualization Quality Flags Quality Reports Parameter File Survey EAS Data LE1 HMS Alerts HMS Limit File HMS Reports Instr. Status Data QLA Processing Framework QLA Diagnostic Tools
11. The QLA Processing Framework J.C.González – Euclid SOC Team @ ESAC Nov.2016 Slide 11 of 25 QLA Decomposition (II)  QLA Processing Framework (QPF): provides the processing framework to execute system functionalities, implements the main HMI, the data access and persistence mechanisms, the logging and the management of the tasking, and the gathering and collection of reports. The QPF also runs the LE1 Processor to generate LE1 data products.  QLA Diagnostic Tools (QDT): they implement different algorithms and functions to perform over the data. They encompass data extraction (HKTM and parametric data), data processing and reporting:  Common (including HKTM extraction and processing)  VIS functions  NISP functions  AOCS functions On top of this, QLA operation will rely on and integrate external tools for more advanced and specific data analysis (e.g. Sextractor, Scamp) and visualisation (e.g. DS9).
12. The QLA Processing Framework J.C.González – Euclid SOC Team @ ESAC Nov.2016 Slide 12 of 25 The QLA Processing Framework - QPF Main Architecture Drivers Independent execution: Each of the components works independently of the others. Message-based communication: The activities of the different components are synchronized by means of message-based communication. Components process the messages they receive, and perform their tasks accordingly Service oriented: Each of the components behaves like a service provider element, acting upon request (after a message from another component). Encapsulated task execution: The Orchestration function provides to the Task Manager function the information about the processing task that must be executed on a data set. The Task Manager function is devoted to the launching of the different tasks, that will be encapsulated and isolated from the main host system. File based input and output is the only interface between the system and the different Processing Elements. Database storage: The configuration, products metadata, task execution information, exchanged messages, alarms, etc. are stored in a dedicated DB, to facilitate the system operation as well as the access form external applications to the system information. System persistence: Strategy to allow it to be persistent upon failures. QPF is deeply Euclid-agnostic Orchestration rules & processing tasks to be executed are highly configurable, and they can be embedded in their specific environments Easily exportable to other environments
13. The QLA Processing Framework J.C.González – Euclid SOC Team @ ESAC Nov.2016 Slide 13 of 25 QPF Responsibilities  Receive and place in local storage science / HKTM data and events  Perform orchestration of incoming data for LE1 products generation  Perform orchestration of generated LE1 for QLA Diagnostics execution  Diagnostics on VIS, NISP-P, NISP-S science data (i.e. including calibration exposures)  Analysis of HK parameters from instruments and S/C  Analysis of AOCS data for pointing performance assessment  Perform processing tasks management across the specified QPF Core nodes in the configured network  Management of system & diagnostics alerts  Maintenance of Local Archive (circular buffer with 1-2 weeks of data)  Ingestion of generated products and reports, and registration of their corresponding metadata in Euclid Archive System at SOC  Update of HMS and/or ESS with QLA parameters.  Perform user-initiated on-demand reprocessing of products in local archive (through QPF HMI)
14. The QLA Processing Framework J.C.González – Euclid SOC Team @ ESAC Nov.2016 Slide 14 of 25 QPF Architecture Processing Node(s) Processing Node(s) Main Host Processing Node(s) Docker Processing Element 1 Docker Processing Element 2 Docker Processing Element M … … Event Manager EvtMng HMI Data Manager DataMng Task Manager TaskMng START / STOP START / STOP START / STOP INDATA TSK_RES TSK_PROC TSK_RES TSK_PROC START / STOP DB Central (External) Archive TSK_RES Local Storage Task Orchestrator TaskOrc TSK_PROC INDATA TSK_PROC TSK_RES CMD CMD Task Agent 1 TaskAg Task Agent N TaskAg Log Manager LogMng Shared area QPF Core Main QPF HMI QPF Processing Core(s)
15. The QLA Processing Framework J.C.González – Euclid SOC Team @ ESAC Nov.2016 Slide 15 of 25 Some QPF Sequence Diagrams (I) TaskOrc TaskMng TaskAg i MSG_TASK_PROC PROCESSING RULE FIRING MSG_TASK_PROC check rules rule fired Docker script agent selection task preaparation Docker container Docker script launch Container execution loop: monitoring request container status info container status info container id MSG_TASK_RES
16. The QLA Processing Framework J.C.González – Euclid SOC Team @ ESAC Nov.2016 Slide 16 of 25 Some QPF Sequence Diagrams (II) TaskOrc TaskMng TaskAg i PROCESSING TASK COMPLETION Docker container loop: monitoring request container status info container status info MSG_TASK_RES DataMng EvtMng MSG_TASK_RES Task Completion MSG_TASK_RES MSG_TASK_RES
17. The QLA Processing Framework J.C.González – Euclid SOC Team @ ESAC Nov.2016 Slide 17 of 25 QPF Technologies  Coding: C++  Msgs/Config/Reports: JSON  Internal DB: PostgreSQL (v-9.2+ x64)  Messaging: 0MQ Library (v-4.1.1) + C++ binding  Processing: Docker  QPF HMI: Qt 5.4+
18. The QLA Processing Framework J.C.González – Euclid SOC Team @ ESAC Nov.2016 Slide 18 of 25 QPF HMI – Some screenshots (I) Flow of messages between the different QPF components
19. The QLA Processing Framework J.C.González – Euclid SOC Team @ ESAC Nov.2016 Slide 19 of 25 QPF HMI – Some screenshots (II) List of processing tasks being executed
20. The QLA Processing Framework J.C.González – Euclid SOC Team @ ESAC Nov.2016 Slide 20 of 25 QPF HMI – Some screenshots (III) Task Processing Agents status and task statistics
21. The QLA Processing Framework J.C.González – Euclid SOC Team @ ESAC Nov.2016 Slide 21 of 25 QPF HMI – Some screenshots (IV) Local Archive (products, logs and QLA reports). DS9 invoked from QPF HMI for one image.
22. The QLA Processing Framework J.C.González – Euclid SOC Team @ ESAC Nov.2016 Slide 22 of 25 QPF HMI – Some screenshots (V) Exploring the content of one generated QLA JSON report. A warning has been raised.
23. The QLA Processing Framework J.C.González – Euclid SOC Team @ ESAC Nov.2016 Slide 23 of 25 QPF HMI – Some screenshots (VI) Configuration Tool is invoked from Tools menu.
24. The QLA Processing Framework J.C.González – Euclid SOC Team @ ESAC Nov.2016 Slide 24 of 25 Next steps  Full control of processing tasks  Reprocessing  Filtering & Analysing QLA JSON Reports  Define a periodic report from a set of QLA JSON reports  Additional processing tasks to be included in the system  Study of parallelisation strategies  Light, web-based front end for monitoring purposes
25. Thanks