The Square Kilometre Array: Overview and Engineering Update

The Square Kilometre Array is currently undergoing the Preliminary Design Reviews for its composing elements, and is thus at a critical point on its way to becoming ready for construction starting in 2018. In this talk we will provide an overview of the SKA, its composing elements, and their status, with emphasis on the Telescope Manager and the Science Data Processor, respectively the Monitoring & Control system, and Pipeline. We will see how do they compare with their ALMA equivalents, and how is the SKA similar/different from ALMA, and what are the main issues to be solved regarding the SKA computing infrastructure.

Talk given on Dec 10, 2014, at the Centre for Mathematical Modelling (CMM) of the University of Chile.

Transcript

1. The Square Kilometre Array: Overview and Engineering Update CMM, Dec 10 2014 Juande Santander-Vela
2. • Who am I? • SKA Science • Telescopes, Sites, Technologies • SKA Organisation • Engineering Progress • SKA as an Exaflop Tomograph • Distributed SKA Control • What’s Next? Outline CMM U. Chile, December 10, 2014
3. Who am I? CMM U. Chile, December 10, 2014
4. CMM U. Chile, December 10, 2014 • Systems Engineer for the SDP/TM elements • Former VIA-SKA Project Manager • VLT Archive Applied Scientist, later ALMA Science Archive Front-/Back-end Developer at ESO • Ph.D. in Radio Astronomical data archives and Virtual Observatory • Software Analyst/Manager Who am I?
5. Great observatories of the present CMM U. Chile, December 10, 2014 ALMA SCO HQ, Dec 11 2014
6. …and future CMM U. Chile, December 10, 2014 ALMA SCO HQ, Dec 11 2014
7. …and future CMM U. Chile, December 10, 2014 ALMA SCO HQ, Dec 11 2014
8. …and (near) future CMM U. Chile, December 10, 2014 ALMA SCO HQ, Dec 11 2014
9. SKA Science CMM U. Chile, December 10, 2014
10. Fundamental Forces & Particles • Gravity • Radio pulsar tests of General Relativity • Gravitational Waves • Dark Energy • Magnetism • Origin and evolution of Cosmic Magnetism
 Origins • Galaxies and the Universe • Cosmic Dawn • First Galaxies • Galaxy Assembly & Evolution • Stars, Planets & Life • Protoplanetary Disks • Biomolecules • SETI SKA Science Goals CMM U. Chile, December 10, 2014 XX Century: We discovered our place in the Universe XXI Century: We understand the Universe we inhabit
11. • Initial SKA Science Book (2004): 5 KSPs, 43 additional ones • Updated SKA Science Book (2014): 13 Phase 1 Headline Projects, 130 Chapters • Projects reduced, parameterised from ~43 projects, through science prioritisation exercise with SKAO and SWGs SKA Science Goals CMM U. Chile, December 10, 2014
12. • Astrobiology (“The Cradle of Life”) • Project Scientist: Tyler Bourke • WG Chair: Melvin Hoare • Galaxy Evolution – Continuum • Project Scientist: Jeff Wagg • WG Chairs: Nick Seymour & Isabella Prandoni • Cosmic Magnetism • Project Scientist: Jimi Green • WG Chairs: Melanie Johnston- Hollitt & Federica Govoni • Cosmology • Project Scientist: Jeff Wagg • WG Chair: Roy Maartens • Epoch of Reionisation & Cosmic Dawn • Project Scientist: Jeff Wagg • WG Chair: Leon Koopmans • Galaxy Evolution – HI • Project Scientist: Jimi Green • WG Chairs: Lister Staveley-Smith & Tom Osterloo • Pulsars (“Strong field tests of gravity”) • Project Scientist: Jimi Green • WG Chairs: Ben Stappers & Michael Kramer • Transients • Project Scientist: Tyler Bourke • WG Chairs: Rob Fender & J.-P. MacQuart SKA Science Goals: WGs CMM U. Chile, December 10, 2014
13. SKA Science Goals CMM U. Chile, December 10, 2014 Year in the life of SKA1: Mock-up 3-years scheduling to constrain operations, computation
14. Telescopes, Sites, Technologies CMM U. Chile, December 10, 2014
15. SKA Telescopes CMM U. Chile, December 10, 2014 SKA1-Low
16. SKA Telescopes CMM U. Chile, December 10, 2014 SKA1-Mid
17. SKA Telescopes CMM U. Chile, December 10, 2014 SKA1-Survey
18. CMM U. Chile, December 10, 2014
19. CMM U. Chile, December 10, 2014
20. SKA Organisation CMM U. Chile, December 10, 2014
21. • Australia (DoI) • Canada (NRCZHIA) • China (MOST) • Germany (BMBF) • India (NCRA) • Italy (INAF) • Netherlands (NWO) • New Zealand (MED) • South Africa (DST) • Sweden (Chalmers) • UK (STFC) SKA Organisation CMM U. Chile, December 10, 2014 Currently UK Company Limited by Guarantee → Treaty Organisation Other partners in negotiation, expected
22. SKA Governance: Pre-Construction CMM U. Chile, December 10, 2014 * * * * * * * * * * * * * * * * * * SKA Governance Members$ Board$of$Directors$ $ SKA$Office$ ~50$staff$ $ Director;General$ 11$Design$ConsorXa$ ~500$ScienXsts$&$Engineers$ Strategy$&$$ Business$Dev$ Cmte$ Finance$ Cmte$ ExecuXve$Cmte$ Science$&$Eng$ Advisory$Cmte$ €23.4M$ (cash)$ ~$€125M$ (in;kind)$
23. SKA International Design Consortia CMM U. Chile, December 10, 2014 Project Management and System Engineering Team based at JBO (UK) ~500 scientists & engineers in institutes & industry in 11 Member countries
24. Engineering Progress CMM U. Chile, December 10, 2014
25. 64 x 13.5m offset Gregorian antennas 8km maximum baseline length First receivers: 0.9 – 1.67 GHz (L-band) 0.58 – 1 GHz (UHF) 770 MHz bandwidth Early operations 2016/7 SKA1 Precursors: MeerKAT CMM U. Chile, December 10, 2014 First light with L-band receiver and digitiser erKAT
26. SKA1 Precursors: MeerKAT CMM U. Chile, December 10, 2014 Lot’s more of infrastructure already in place: dish assembly shed; bunkered and RFI shielded data processor room and power system
27. SKA1 Precursors: MeerKAT CMM U. Chile, December 10, 2014 Lot’s more of infrastructure already in place: dish assembly shed; bunkered and RFI shielded data processor room and power system
28. SKA1 Precursors: ASKAP CMM U. Chile, December 10, 2014 36 x 12m antennas 3-axis movement 30m – 6km baselines Novel PAF receiver 700MHz – 1800MHz Wide field of view Fast survey speed
29. BETA • 6 Mk I PAFs and digital systems • Commissioning, learning about beamforming 
 Lesson: deploy early, test, ensure adequate gap to production commitment ASKAP • First Mk II PAF installed • Results expected in next few weeks • Production of Mk II PAFs install through to Feb 2016 SKA1 Precursors: ASKAP CMM U. Chile, December 10, 2014
30. SKA1 Precursors: MWA CMM U. Chile, December 10, 2014 Murchison Widefield Array, Operational 17 refereed papers published, more coming
31. 3 dish prototypes in testing CMM U. Chile, December 10, 2014
32. 3 dish prototypes in testing CMM U. Chile, December 10, 2014
33. 3 dish prototypes in testing CMM U. Chile, December 10, 2014
34. SKA as an Exaflop Tomograph CMM U. Chile, December 10, 2014
35. • Dishes, feeds, receivers (N=250 → 2500) • Low and mid aperture arrays (n=250k → 1000k) • Signal transport (~1 Pb/s → 10 Pb/s) • Signal processing (exa-MACs) • Software engineering & algorithm development • High performance computing (exa-flop capability) • Data storage (exa-byte capacity) • (Distributed) power requirements (10 → 50MW) Engineering Challenges CMM U. Chile, December 10, 2014
36. SKA High-Level Architecture CMM U. Chile, December 10, 2014 SKA-TEL-SKO-DD-001 Revision : 1 Figure 1 A schematic diagram of the SKA Observatory, showing the geographical locations of site entities (telescopes), the entities at regional centres (Host Country Headquarters and Science Data Processing), and Regional Science & Engineering Centre(s) Regional Science & Engineering Centre(s) SKA Observatory Global Headquarters Remote stations on spiral arms Science Computing Remote Station Remote Station Remote Station Host Country Headquarters Central Signal Processing SKA1-low SKA1-survey Core Arrays Australia Remote stations on spiral arms Science Computing Remote Station Remote Station Remote Station Host Country Headquarters Central Signal Processing Mid-Freq Aperture Array (SKA2) SKA1-mid Core Arrays South Africa
37. CMM U. Chile, December 10, 2014 SDP F2F Meeting, April 7th 2014, Cambridge SKA Elements & Interfaces INFRA DSH LFAA CSP SDP TM Command control flow Data flow SaDT Timing Conceptual figure: it should be split by telescope
38. CMM U. Chile, December 10, 2014 SDP F2F Meeting, April 7th 2014, Cambridge SKA Elements & Interfaces INFRA DSH/LFAA CSP SDP TM Command control flow Data flow SaDT Timing Conceptual figure: it should be split by telescope
39. • Software Defined Networking • Allows easy partitioning, sub-arraying, quality of service, commensality… • Software Defined Beamforming (-Low, - Survey) • Software Defined Computing • Data-Flow Driven Science Data Processing SKA is a Software-Defined Telescope CMM U. Chile, December 10, 2014 Lots of computing challenges
40. CMM U. Chile, December 10, 2014 Science Data Processor Local M&C Science Data Processor Telescope Manager Correlator / Beamformer Data Routing Ingest Visibility processing Multiple Reads Time Series Search Multiple Reads Data$Buffer Data Routing Time Series Processing Image Plane Processing Data Prodcuts Sky Models, Calibration Parameters ... Meta Data Master Controller Master Controller Local M&C Database Tiered Data Delivery
41. CMM U. Chile, December 10, 2014 Performance Modelling •  Imaging and calibration determines system sizing •  Data is Buffered after Ingest •  Detailed analysis building on Cornwell: •  Imaging o  AW-projection o  Plus w-snapshots o  Plus faceting (working memory size) •  Continuum imaging always required for calibration o  Field of view to second zero of beam •  Maximal case includes “science discovery” mode of forming high spectral-resolution image cube over primary beam •  Commensal fast-imaging mode for slow transients Processing maximal (Pflop) Ingest (GB/s) Use Case examples (GB/s) LFAA! 100! 7300! 245! Survey! 135! 4600! 995! Mid! 360! 3300! 255! Detailed analysis is much more complicated
42. CMM U. Chile, December 10, 2014
43. CMM U. Chile, December 10, 2014
44. SDP Parallelism: Frequency CMM U. Chile, December 10, 2014 x Time: accumulate visibilities into separate grids and combine later x Beams Simplified examples are illustrated in the following figures. Figure 5: Illustrating data parallelism based on frequency selection only (c.f. LOFAR)
45. SDP Parallelism: uv-space partition CMM U. Chile, December 10, 2014 Figure 6: Managing memory bandwidth by distributing regions of target uv-space over nodes followed by a gather and FFT to image space.
46. SDP Parallelism: spatial partition CMM U. Chile, December 10, 2014 Figure 6: Managing memory bandwidth by distributing regions of target uv-space over nodes followed by a gather and FFT to image space. Figure 7: Managing IO bandwidth from local storage by distributing source data over multiple nodes. Target gridded visibility plane is duplicated then an accumulation step is performed over the data island over which the source data were distributed 3.5 Data flow programming As discussed in Section 4 a data-flow approach is adopted for the top-level SDP architecture. In this section we provide some background discussion to the established concept of data-flow
47. >1 EXAFLOP/S Correlation Data Product Generation Temporary Storage Long Term Storage High Availability Storage / DB On-Demand Processing CAN’T STORE IT! 1 DAY STREAM = 150 DAYS GLOBAL INTERNET TRAFFIC PROCESSING NEEDS 109 TOP RANGE PCS Antenna & Fronted-End Systems 30-300 PETAFLOP/S 30 PETAFLOP/S SKA Data Flow and Processing CMM U. Chile, December 10, 2014
48. Distributed SKA Control CMM U. Chile, December 10, 2014
49. • Need to find COTS-based solution • Move away from expensive hard real-time solutions • Switched network inside products • Orchestration instead of deterministic control • Certain amount of product independence SKA Control Problem CMM U. Chile, December 10, 2014
50. CMM U. Chile, December 10, 2014 Figure 1. ALMA Software Subsystems interaction diagram (source: ALMA Software documentation)
51. CMM U. Chile, December 10, 2014 TM.TELMGT TM.OBSMGT DSH.LMC CSP.LMC SDP.INGEST SDP.PIP.* SDP.ARCH SDP.LMC Commands Data Monitoring/Status CSP.CORR DSH.RECEIVER SKA Observation interaction
52. CMM U. Chile, December 10, 2014 Ethernet switching Middleware TBD (EPICS/TANGO/OPC UA) DSH.LMC SADT.DDBH SADT.TM CSP.LMC SADT.SAT
53. CMM U. Chile, December 10, 2014 TM.TELMGT TM.OBSMGT TM.TELMOD SDP.PIP.
54. What’s Next? CMM U. Chile, December 10, 2014
55. CMM U. Chile, December 10, 2014 2016 2015 2017 2018 2019 2020 2021 2022 2023 2014 High/level/design Rebaselining/submissions/(RBS) Preliminary/Design/Review/(elements) Detailed/design Prototype/systems/deployed Critical/Design/Review SKA1/construction/approved Procurement SKA1/early/science SKA1/construction SKA2/detailed/design Design/consortia/start SKA2/procurement SKA2/construction/starts SKA2/concept/development Board/RBS/approval 3K5/Mar/15 15K22/Sep/14 1/Nov/13 PreKconstruction/Stage/1 PreKconstruction/Stage/2 Production*Readiness*Review*(4*dish*array) Construction*Readiness*Review* (1*dish*line9up) New/SKA/Organisation/in/place Key/Doc/Set/&/Prospectus Formal/negotiations Ratification/of/Agreements Organisation/has/minimum/no./Members/with/Agreements/ratified Agreements/in/Key/Doc/Set/signed Andrea/Casson,/SKAO/Project/Controller,/Sept/2014 System/PDR Integration/testing/on/site Advanced/Instrumentation/Prog. KEY:/Blue/=/SKA1/science/&/engineering;/orange/=/policy;/green/=/SKA2;/milestones*in*italics*=/proposals/under/discussion The SKA1 timeline
56. CMM U. Chile, December 10, 2014 Spain SKA Day, October 23rd, Granada The SKA1 timeline Issue%Baseline% Design Science% Assessment% Workshops Proposal% Feedback Level%1% Requirements% Feedback Science% Conference Engineering% Meeting Engineering%Change%Proposals Designs%&% Costs SKAO%Review Technical% Advice (working% groups) SEAC Board Consortium%Design%Activities Level%0% Requirements Brief% SEAC 2013 2015 Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar 2014 PDR Science Review%Panel Science Review% Panel !! !! !! !! !! !! !!(43) !! !! !! !! Issue%Baseline% Design Science% Assessment% Workshops Proposal% Feedback Level%1% Requirements% Feedback Science% Conference Engineering% Meeting Engineering%Change%Proposals Designs%&% Costs SKAO%Review Technical% Advice (working% groups) SEAC Board Consortium%Design%Activities Level%0% Requirements Brief% SEAC 2013 2015 Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar 2014 PDR Science Review%Panel Science Review% Panel
57. CMM U. Chile, December 10, 2014 The SKA1 timeline Issue%Baseline% Design Science% Assessment% Workshops Proposal% Feedback Level%1% Requirements% Feedback Science% Conference Engineering% Meeting Engineering%Change%Proposals Designs%&% Costs SKAO%Review Technical% Advice (working% groups) SEAC Board Consortium%Design%Activities Level%0% Requirements Brief% SEAC 2013 2015 Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar 2014 PDR Science Review%Panel Science Review% Panel Level%1% Requirements% Feedback Engineering% Meeting Engineering%Change%Proposals Designs%&% Costs SKAO%Review Technical% Advice (working% groups) SEAC Boar Consortium%Design%Activities Level%0% Requirements Brief% SEAC Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar PDR
58. CMM U. Chile, December 10, 2014 • Formed by External Reviewers, AG representative, SE representative, PM, Observers • Chaired by a selected External Reviewer • SKAO Project Manager acts as Secretary • SKAO personnel contributes to OAR • Consortia respond to Consolidated OAR before PDR review • Face-to-face for outstanding actions, 1-2 days PDR panels and process
59. CMM U. Chile, December 10, 2014 • Element overall design report. • Specific design reports for all second tier subsystems (the Element being the first tier), including manufacturing/procurement approach, transport, installation, verification, support… • Finite element models (if needed). • Software architecture models and use cases. • Parametric models. • Analysis reports: Finite element modelling (including all mechanical and thermal load cases), thermal analysis, fluid dynamics, RFI/EMC analysis, safety analysis, hazard analysis… • Cost analysis (development, construction, verification, commissioning & steady state ops). • Preliminary construction plans. • Configuration items list. • Compliance matrix. • Preliminary maintenance plan. • Preliminary verification plan. • Schedule estimate for construction. • High level risk register. • Draft Statement of work for the construction of the Element. • Draft technical specifications for the construction of the Element. PDR documentation
60. CMM U. Chile, December 10, 2014 • Interface Control Documents (ICDs) • Owned by the SKAO, developed between the Consortia • Technology Readiness Level Assessment • Controlled through Risk Register right now PDR documentation (in parallel)
61. Questions? Juande Santander-Vela j.santander-vela@skatelescope.org