The Euclid Quick Look Analysis System

Transcript

1. The Euclid SOC Quick-Look Analysis System J. C. González Euclid SOC Team @ ESAC May, 2017
2. Euclid QLA System & Processing Framework J.C.González – Euclid SOC Team @ ESAC May 2017 Slide 2 of 13 Euclid Operations Main Entities (I) Mission Operations Centre External Data Providers Science Operations Centre Science Data Centres Euclid Archive IOT’s Coordinator IOT VIS IOT NISP Science Ground Segment
3. Euclid QLA System & Processing Framework J.C.González – Euclid SOC Team @ ESAC May 2017 Slide 3 of 13 Science Operations Centre SOC Components QLA Quick Look Analysis HMS Health Monitoring System ESS Euclid Survey System L1P Level 1 Processing EAS@SOC Euclid Archive System at SOC SOC Storage Mission Operations Centre Scientific data S/C HK data Level 1 VIS & NISP products QLA Parameters QLA Diagnostics Reports SCS SOC Command System SIS SOC Interface System Commanding requests
4. Euclid QLA System & Processing Framework J.C.González – Euclid SOC Team @ ESAC May 2017 Slide 4 of 13 The Quick-Look Analysis Subsystem Responsibilities q  Automatically check S/C data, produce and ingest reports in the Euclid Archive System within 48 hours from data retrieval to confirm that the data are suitable for scientific analysis and eventually quickly react to issues. The IOT (instrument operation team) and EC (Euclid consortium) will investigate the data in further details q  Facilitate data inspection by the user (mainly instrument scientists) in case further data inspection is needed Capable of accessing detectors data, showing it and/or exporting it to external programs, etc. q  Generate LE1 products from S/C data (not QLA but SOC resp) LE1 processor is NOT developed by the SOC, but the SOC is responsible for its execution in order to generate the LE1 products, and archival
5. Euclid QLA System & Processing Framework J.C.González – Euclid SOC Team @ ESAC May 2017 Slide 5 of 13 QLA QLA Decomposition (I) Analysis and Processing SDA PDA ADA PRO CDA VIM QAF REP PAR CAM SIF DDS TM CDFP TM Aux Data Configuration Logs Visualization Quality Flags Quality Reports Parameter File Survey EAS Data LE1 HMS Alerts HMS Limit File HMS Reports Instr. Status Data QLA Processing Framework QLA Diagnostic Tools
6. Euclid QLA System & Processing Framework J.C.González – Euclid SOC Team @ ESAC May 2017 Slide 6 of 13 QLA Decomposition (II) q  QLA Processing Framework (QPF): provides the processing framework to execute system functionalities, implements the main HMI, the data access and persistence mechanisms, the logging and the management of the tasking, and the gathering and collection of reports. The QPF also runs the LE1 Processor to generate LE1 data products. q  QLA Diagnostic Tools (QDT): they implement different algorithms and functions to perform over the data. They encompass data extraction (HKTM and parametric data), data processing and reporting: §  Common (including HKTM extraction and processing) §  VIS functions §  NISP functions §  AOCS functions On top of this, QLA operation will rely on and integrate external tools for more advanced and specific data analysis (e.g. Sextractor, Scamp) and visualisation (e.g. DS9).
7. Euclid QLA System & Processing Framework J.C.González – Euclid SOC Team @ ESAC May 2017 Slide 7 of 13 QPF Architecture Processing Node(s) Processing Node(s) Main Host Processing Node(s) Docker Processing Element 1 Docker Processing Element 2 Docker Processing Element M … … Event Manager EvtMng HMI Data Manager DataMng Task Manager TaskMng START / STOP START / STOP START / STOP INDATA TSK_RES TSK_PROC TSK_RES TSK_PROC START / STOP DB Central (External) Archive TSK_RES Local Storage Task Orchestrator TaskOrc TSK_PROC INDATA TSK_PROC TSK_RES CMD CMD Task Agent 1 TaskAg Task Agent N TaskAg Log Manager LogMng Shared area QPF Core Main QPF HMI QPF Processing Core(s)
8. Euclid QLA System & Processing Framework J.C.González – Euclid SOC Team @ ESAC May 2017 Slide 8 of 13 Initial usage of Docker in QLA Master Host QPF Master Core Docker service Container Container Container … Processing Host QPF Processing Core Docker service Container Container Container … Processing Host QPF Processing Core Docker service Container Container Container … ...
9. Euclid QLA System & Processing Framework J.C.González – Euclid SOC Team @ ESAC May 2017 Slide 9 of 13 Characteristics of processors execution in QPF q  Execution of processors is triggered by very simple rules q  Docker container based processors are launched via Bash scripts from main application: very flexible §  QDT processing is quite fast, so QDT Docker containers are short lived (~mins) q  Data circulation across system is based in scp (secure copy) q  I/O data provision is done using Docker volumes (mapped paths) q  Local Archive is file/directory based, with metadata stored in local database
10. Euclid QLA System & Processing Framework J.C.González – Euclid SOC Team @ ESAC May 2017 Slide 10 of 13 Future ways to enhance processing & monitoring q  Processing: §  Docker cointainer networks §  Docker Swarm (scalable processing service) §  Explore alternatives (Kubernetes, …) q  Archiving: §  Use more standard system for data archiving q  Data circulation: §  Use standard data provision system (foreseen HTTP server) q  Standard System Monitoring Tools q  Web based Status Information q  Data provision and reporting subscription services q  Standard Business Rules Tool (not needed so far) q  VOSpace interaction? ✓ ✓ ✓ Marked elements are already 80% implemented for V2.0
11. Euclid QLA System & Processing Framework J.C.González – Euclid SOC Team @ ESAC May 2017 Slide 11 of 13 External Orchestrator Monitoring System Docker Containers & Services in QPF Processing Host QPF Processing Core Docker service Container Container Container … Processing Host QPF Processing Core Docker service Container Container Container … Swarm Processing Node Docker service Docker Swarm Node Layer Service Service Service … Service Swarm Processing Node Docker service Docker Swarm Node Layer Service Service Service … Service Local Archive EAS HMI Monitoring & Alerts Analysis Data Server QPF Master Core Master Host Docker service Container Container Container … Docker Swarm Manager Data Server
12. Euclid QLA System & Processing Framework J.C.González – Euclid SOC Team @ ESAC May 2017 Slide 12 of 13 Containers vs Services images in QPF Containers Image Support Libraries Image OS Layer Support Libraries Image OS Layer Services Image I/O Products & Logs Processing Handler QLA Diagnostic Tools FS Mapping I/O Products & Logs Processing Handler Processing Software FS Mapping Volume Mapping Layer Volume Mapping Layer
13. Thanks