Agile Systems Engineering & Agile at SKA Scale

Systems Engineering (SE) is the set of processes and documentation required for successfully realising large-scale engineering projects, but the classical approach is not a good fit for software-intensive projects, especially when the needs of the different stakeholders are not fully known from the beginning, and requirement priorities might change. The SKA is the ultimate software-enabled telescope, with enormous amounts of computing hardware and software required to perform its data reduction. We give an overview of the system and software engineering processes in the SKA1 development, and the tension between classical and agile SE.

Transcript

1. Agile Software Engineering
   & System Engineering at
   SKA scale
   Juande Santander-Vela
   SKA SW Systems Engineer
   SQUARE KILOMETRE ARRAY
   Exploring the Universe with the worlds’ largest radio telescope
   Astronomical Data Analysis
   Software & Systems
   (on behalf of the SE team)
   

   View Slide

2. SKA Key Science Drivers:
   The history of the Universe
   Cosmic Dawn
   
   (First Stars and Galaxies)
   Galaxy Evolution
   
   (Normal Galaxies z~2-3)
   Cosmology
   
   (Dark Energy, Large Scale Structure)
   Cosmic Magnetism
   
   (Origin, Evolution)
   Cradle of Life
   
   (Planets, Molecules, SETI)
   Testing General Relativity
   
   (Strong Regime, Gravitational Waves)
   Exploration of the Unknown
   Broadest range of
   science of any facility,
   worldwide
   

   View Slide

3. SKA Observatory Vision
   Exploring the Universe with the worlds’ largest radio telescope
   3 sites
   2 telescopes
   1 observatory
   Design Phase:
   ~ €200M; 600 scientists+engineers,
   
   80% complete
   SKA Phase 1 (SKA1)
   Construction: 2019 – 2025
   
   Construction cost cap: €674.1M
   (2016 inflation-adjusted)
   
   Operations cost: (estimate) €89M/yr
   
   MeerKAT integrated
   Observatory Development Programme
   (€20M/year planned)
   
   SKA Regional centres out of scope of
   centrally-funded SKAO.
   SKA Phase 2: start mid-2020s
   ~2000 dishes across 3500km of
   Southern Africa
   
   Major expansion of SKA1-Low
   across Western Australia
   >50 years lifetime!
   Drives need for reliability,
   and adaptability
   

   View Slide

4. SKA Observatory Vision
   Exploring the Universe with the worlds’ largest radio telescope
   3 sites
   2 telescopes
   1 observatory
   Design Phase:
   ~ €200M; 600 scientists+engineers,
   
   80% complete
   SKA Phase 1 (SKA1)
   Construction: 2019 – 2025
   
   Construction cost cap: €674.1M
   (2016 inflation-adjusted)
   
   Operations cost: (estimate) €89M/yr
   
   MeerKAT integrated
   Observatory Development Programme
   (€20M/year planned)
   
   SKA Regional centres out of scope of
   centrally-funded SKAO.
   SKA Phase 2: start mid-2020s
   ~2000 dishes across 3500km of
   Southern Africa
   
   Major expansion of SKA1-Low
   across Western Australia
   >50 years lifetime!
   Drives need for reliability,
   and adaptability
   

   View Slide

5. SKA Organisation
   Exploring the Universe with the worlds’ largest radio telescope
   ! Australia (DoI&S)
   
   " Canada (NRC-HIA)
   
   # China (MOST)
   
   $ India (DAE)
   
   % Italy (INAF)
   
   & Netherlands (NWO)
   
   ' New Zealand (MED)
   
   ( South Africa (DST)
   
   ) Sweden (Chalmers)
   
   * UK (BEIS/STFC)
   In discussion with:
   
   + Germany
   
   , France
   
   - Portugal
   
   . Spain
   
   / Switzerland
   
   0 Japan
   
   1 South Korea
   
   In the process of
   becoming an Inter-
   Governmental
   Organisation
   

   View Slide

6. SKA1 Design Consortia
   Exploring the Universe with the worlds’ largest radio telescope
   

   View Slide

7. SKA1 Design Consortia
   Exploring the Universe with the worlds’ largest radio telescope
   Software
   Heavy
   Elements
   

   View Slide

8. Have
   ~6-10 × FTE
   wrt SKAO
   SKA1 Design Consortia
   Exploring the Universe with the worlds’ largest radio telescope
   

   View Slide

9. 1000 km
   Map data ©2017 Terms of Use
   Filter Consortia
   Assembly, Integration & Verification
   Central Signal Processor
   Dish
   Infrastructure Australia
   Infrastructure South Africa
   Low-Frequency Aperture Array
   Mid-Frequency Aperture Array
   Science Data Processor
   Signal and Data Transport
   Telescope Manager
   Wideband Single Pixel Feeds
   Check/uncheck all
   GMT-7
   GMT+12
   Exploring the Universe with the worlds’ largest radio telescope
   19 hour timezone span
   

   View Slide

10. SKA1-Low locations
    1600 km
    N
    ➤
    ➤
    N
    Data SIO, NOAA, U.S. Navy, NGA, GEBCO
    Data SIO, NOAA, U.S. Navy, NGA, GEBCO
    Image Landsat / Copernicus
    Image Landsat / Copernicus
    SKA1-Mid locations
    500 km
    N
    ➤
    ➤
    N
    Image Landsat / Copernicus
    Image Landsat / Copernicus
    Data SIO, NOAA, U.S. Navy, NGA, GEBCO
    Data SIO, NOAA, U.S. Navy, NGA, GEBCO
    SKA1 Sites
    Exploring the Universe with the worlds’ largest radio telescope
    SKA1-LOW: 50 – 350 MHz
    Phase 1: ~130,000 antennas
    
    across 65km
    SKA1-Mid: 350 MHz – 24 GHz
    Phase 1: 200 15-m dishes
    
    across 150 km
    

    View Slide

11. SKA1 Telescopes
    Exploring the Universe with the worlds’ largest radio telescope
    SKA1-LOW: 50 – 350 MHz
    Phase 1: ~130,000 antennas
    
    across 65km
    SKA1-Mid: 350 MHz – 24 GHz
    Phase 1: 200 15-m dishes
    
    across 150 km
    

    View Slide

12. SKA1 Notional Data Flow
    Exploring the Universe with the worlds’ largest radio telescope
    2 Pb/s 7.2 Tb/s
    7.2 Tb/s
    8.8 Tb/s
    8.8 Tb/s
    ~50 PFLOPS
    ~5 Tb/s
    ~250 PFLOPS
    Long Term
    Storage
    300 PB/year
    LFAA Data Network Digital Data Backhaul
    Digital Data Backhaul
    CSP-SDP Network
    

    View Slide

13. SKA1 Notional Data Flow
    Exploring the Universe with the worlds’ largest radio telescope
    2 Pb/s 7.2 Tb/s
    7.2 Tb/s
    8.8 Tb/s
    8.8 Tb/s
    ~50 PFLOPS
    ~5 Tb/s
    ~250 PFLOPS
    Long Term
    Storage
    300 PB/year
    LFAA Data Network Digital Data Backhaul
    Digital Data Backhaul
    CSP-SDP Network
    

    View Slide

14. SKA1 Notional Data Flow
    Exploring the Universe with the worlds’ largest radio telescope
    2 Pb/s 7.2 Tb/s
    7.2 Tb/s
    8.8 Tb/s
    8.8 Tb/s
    ~50 PFLOPS
    ~5 Tb/s
    ~250 PFLOPS
    Long Term
    Storage
    300 PB/year
    LFAA Data Network Digital Data Backhaul
    Digital Data Backhaul
    CSP-SDP Network
    

    View Slide

15. SKA1 Notional Data Flow
    Exploring the Universe with the worlds’ largest radio telescope
    2 Pb/s 7.2 Tb/s
    7.2 Tb/s
    8.8 Tb/s
    8.8 Tb/s
    ~50 PFLOPS
    ~5 Tb/s
    ~250 PFLOPS
    Long Term
    Storage
    300 PB/year
    LFAA Data Network Digital Data Backhaul
    Digital Data Backhaul
    CSP-SDP Network
    Similar to LOFAR
    in flexibility.
    See O4.2 by
    Adriaan Renting
    

    View Slide

16. How to develop so
    much software?
    Exploring the Universe with the worlds’ largest radio telescope
    

    View Slide

17. Agile
    Exploring the Universe with the worlds’ largest radio telescope
    

    View Slide

18. Agile, Scaled Agile
    Exploring the Universe with the worlds’ largest radio telescope
    (Framework)
    

    View Slide

19. SAFe for Lean Enterprises Large Solution SAFe
    ®
    Exploring the Universe with the worlds’ largest radio telescope
    Agile Teams, 5-7
    people (including
    Scrum Master),
    ±2 people
    
    Agile Release
    Trains (ARTs)
    in Definition
    https://scaledagileframework.com/
    

    View Slide

20. Agile Manifesto
    • Individuals and interactions over processes and tools
    
    • Working software over comprehensive documentation
    
    • Customer collaboration over contract negotiation
    
    • Responding to change over following a plan
    Exploring the Universe with the worlds’ largest radio telescope
    http://agilemanifesto.org
    

    View Slide

21. The emphasis is on Quality
    Exploring the Universe with the worlds’ largest radio telescope
    Nick Rees talk I10.1
    (Thursday 26th, 9:00 AM)
    

    View Slide

22. What about
    Systems Engineering?
    Exploring the Universe with the worlds’ largest radio telescope
    

    View Slide

23. INCOSE, 2004
    “Systems engineering [SE] is an interdisciplinary approach and
    means to enable the realisation of successful systems […] [SE]
    integrates all the disciplines and specialty groups into a team effort
    forming a structured development process […] from concept to
    production to operation. [SE] considers both the business and the
    technical needs of all customers with the goal of providing a quality
    product that meets the user needs.”
    Exploring the Universe with the worlds’ largest radio telescope
    

    View Slide

24. INCOSE, 2004
    “Systems engineering [SE] is an interdisciplinary approach and
    means to enable the realisation of successful systems […] [SE]
    integrates all the disciplines and specialty groups into a team effort
    forming a structured development process […] from concept to
    production to operation. [SE] considers both the business and the
    technical needs of all customers with the goal of providing a quality
    product that meets the user needs.”
    Exploring the Universe with the worlds’ largest radio telescope
    

    View Slide

25. INCOSE, 2004
    “Systems engineering [SE] is an interdisciplinary approach and
    means to enable the realisation of successful systems […] [SE]
    integrates all the disciplines and specialty groups into a team effort
    forming a structured development process […] from concept to
    production to operation. [SE] considers both the business and the
    technical needs of all customers with the goal of providing a quality
    product that meets the user needs.”
    Exploring the Universe with the worlds’ largest radio telescope
    

    View Slide

26. INCOSE, 2004
    “Systems engineering [SE] is an interdisciplinary approach and
    means to enable the realisation of successful systems […] [SE]
    integrates all the disciplines and specialty groups into a team effort
    forming a structured development process […] from concept to
    production to operation. [SE] considers both the business and the
    technical needs of all customers with the goal of providing a quality
    product that meets the user needs.”
    Exploring the Universe with the worlds’ largest radio telescope
    

    View Slide

27. INCOSE, 2004
    “Systems engineering [SE] is an interdisciplinary approach and
    means to enable the realisation of successful systems […] [SE]
    integrates all the disciplines and specialty groups into a team effort
    forming a structured development process […] from concept to
    production to operation. [SE] considers both the business and the
    technical needs of all customers with the goal of providing a quality
    product that meets the user needs.”
    Exploring the Universe with the worlds’ largest radio telescope
    

    View Slide

28. David Long, CEO Vitech Corp
    “The role of Systems Engineers is to encode
    and translate engineering to all stakeholders.”
    Exploring the Universe with the worlds’ largest radio telescope
    

    View Slide

29. Nick Rees, SKAO Head of Computing
    “Systems Engineers capture and remind us
    of all the bullshit stakeholders say about
    the system.”
    Exploring the Universe with the worlds’ largest radio telescope
    

    View Slide

30. Exploring the Universe with the worlds’ largest radio telescope
    

    View Slide

31. Exploring the Universe with the worlds’ largest radio telescope
    

    View Slide

32. Exploring the Universe with the worlds’ largest radio telescope
    

    View Slide

33. Juande Santander-Vela, SKAO Software SE
    “Systems Engineers have to provide useful
    engineering artefacts to stakeholders, notice
    still missing definition, and supply it ASAP”
    Exploring the Universe with the worlds’ largest radio telescope
    Freshly coined for
    #ADASS2017CL!
    

    View Slide

34. Juande Santander-Vela, SKAO Software SE
    “Systems Engineers have to provide useful
    engineering artefacts to stakeholders, notice
    still missing definition, and supply it ASAP”
    Exploring the Universe with the worlds’ largest radio telescope
    Or assess and
    record the risk of
    not doing it
    Freshly coined for
    #ADASS2017CL!
    

    View Slide

35. INCOSE, 2004
    “Systems engineering [SE] is an interdisciplinary approach and
    means to enable the realisation of successful systems […] [SE]
    integrates all the disciplines and specialty groups into a team effort
    forming a structured development process […] from concept to
    production to operation. [SE] considers both the business and the
    technical needs of all customers with the goal of providing a quality
    product that meets the user needs.”
    Exploring the Universe with the worlds’ largest radio telescope
    With an assessment of
    what are the risks against
    quality and user needs
    

    View Slide

36. SE must be Agile, or it
    provides diminished value
    Exploring the Universe with the worlds’ largest radio telescope
    

    View Slide

37. Agile Manifesto
    • Individuals and interactions over processes and tools
    
    • Working software over comprehensive documentation
    
    • Customer collaboration over contract negotiation
    
    • Responding to change over following a plan
    Exploring the Universe with the worlds’ largest radio telescope
    http://agilemanifesto.org
    

    View Slide

38. Agile SE
    • Keep communication with all stakeholders involved in the project, with
    speedy and clear dissemination of agreements
    
    • Continuous integration of the engineering artefacts… and code,
    preferably self-documented
    
    • Get devs (and SEs) that are invested, not just
    contracted → Gain‑share frameworks
    
    • Have systems & processes that accommodate system (and code)
    change and prove requirements are satisfied → don’t do document-
    centric requirements management
    Exploring the Universe with the worlds’ largest radio telescope
    

    View Slide

39. Agile SE
    • Keep communication with all stakeholders involved in the project, with
    speedy and clear dissemination of agreements
    
    • Continuous integration of the engineering artefacts… and code,
    preferably self-documented
    
    • Get devs (and SEs) that are invested, not just
    contracted → Gain‑share frameworks
    
    • Have systems & processes that accommodate system (and code)
    change and prove requirements are satisfied → don’t do document-
    centric requirements management
    Exploring the Universe with the worlds’ largest radio telescope
    At all phases! Including design,
    construction, commissioning, and
    operations…
    Don’t disband SE!
    

    View Slide

40. Agile SE
    • Keep communication with all stakeholders involved in the project, with
    speedy and clear dissemination of agreements
    
    • Continuous integration of the engineering artefacts… and code,
    preferably self-documented
    
    • Get devs (and SEs) that are invested, not just
    contracted → Gain‑share frameworks
    
    • Have systems & processes that accommodate system (and code)
    change and prove requirements are satisfied → don’t do document-
    centric requirements management
    Exploring the Universe with the worlds’ largest radio telescope
    At all phases! Including design,
    construction, commissioning, and
    operations…
    Don’t disband SE!
    

    View Slide

41. Agile SE
    • Keep communication with all stakeholders involved in the project, with
    speedy and clear dissemination of agreements
    
    • Continuous integration of the engineering artefacts… and code,
    preferably self-documented
    
    • Get devs (and SEs) that are invested, not just
    contracted → Gain‑share frameworks
    
    • Have systems & processes that accommodate system (and code)
    change and prove requirements are satisfied → don’t do document-
    centric requirements management
    Exploring the Universe with the worlds’ largest radio telescope
    At all phases! Including design,
    construction, commissioning, and
    operations…
    Don’t disband SE!
    

    View Slide

42. Agile SE
    • Keep communication with all stakeholders involved in the project, with
    speedy and clear dissemination of agreements
    
    • Continuous integration of the engineering artefacts… and code,
    preferably self-documented
    
    • Get devs (and SEs) that are invested, not just
    contracted → Gain‑share frameworks
    
    • Have systems & processes that accommodate system (and code)
    change and prove requirements are satisfied → don’t do document-
    centric requirements management
    Exploring the Universe with the worlds’ largest radio telescope
    Partners, not
    contractors/suppliers
    mentioned by
    JC Guzman on I2.1
    At all phases! Including design,
    construction, commissioning, and
    operations…
    Don’t disband SE!
    

    View Slide

43. Agile SE
    • Keep communication with all stakeholders involved in the project, with
    speedy and clear dissemination of agreements
    
    • Continuous integration of the engineering artefacts… and code,
    preferably self-documented
    
    • Get devs (and SEs) that are invested, not just
    contracted → Gain‑share frameworks
    
    • Have systems & processes that accommodate system (and code)
    change and prove requirements are satisfied → don’t do document-
    centric requirements management
    Exploring the Universe with the worlds’ largest radio telescope
    Partners, not
    contractors/suppliers
    mentioned by
    JC Guzman on I2.1
    At all phases! Including design,
    construction, commissioning, and
    operations…
    Don’t disband SE!
    

    View Slide

44. Agile-ish Practices at SKAO
    • SE planning
    
    • We have a full plan, from which the backlog is constructed
    
    • Progress reviewed every 2 weeks, with forward look to another 2 weeks
    
    • Systems Modelling
    
    • Integrated System Model for SKA1_Common and SKA1_Low in Cameo SysML
    
    • Second model for SKA1_Mid in Vitech CORE
    
    • Used for ICDs, discussions
    
    • SKA1_Mid and SKA1_Low Block Diagrams
    
    • Used for reference, updated in agile way, but low priority
    Exploring the Universe with the worlds’ largest radio telescope
    

    View Slide

45. Agile-ish Practices at SKA1
    • Consortia work-planning
    
    • Agile for SDP, TM
    
    • LFAA uses agile for software development of AAVS1 prototype
    
    • Others want to embrace SAFe for construction, just not now
    
    • Multimodal communication
    
    • Email, SMS, Confluence, JIRA, Jama comments, Slack, Vidyo, Skype…
    Exploring the Universe with the worlds’ largest radio telescope
    

    View Slide

46. Not so Agile-ish Practices at SKAO
    • Engineering Change Process (ECPs) and Change Notes (CNs)
    
    • Bug, or Feature? Still TBD for ECPs → more time to reflect
    
    • Need for formality/signature/buy-in across consortia requires time; we need to
    be able to move from as agreed to as designed before as signed… but how?
    
    • Lack of JIRA/backlog culture → starting to improve
    
    • Lack of self-awareness of performance metrics → starting to improve
    Exploring the Universe with the worlds’ largest radio telescope
    

    View Slide

47. Agile SE is not for free…
    • Coordination/communication takes time!
    
    • Self-organisation frees time from management, but needs assistance from
    monitoring → back to coordination!
    
    • Automation of monitoring and communication can help… but still the
    effectiveness of the communication needs to be monitored.
    Exploring the Universe with the worlds’ largest radio telescope
    a lot of
    But it makes it easier to
    focus on the right things
    

    View Slide

48. Conclusions
    • Agile SE can be seen as a tailoring of SE; you still need to identify which
    are the processes that provide value to your system/organisation.
    
    • It is a challenge to create teams that can work together, but making it
    on a cadence makes it easier to achieve it… through repetition.
    
    • We have to improve a lot at SKAO… but we’re working on it.
    
    • Agile for Agile’s sake is a fad; you become Agile it to build quality, and
    respond to mistakes, sooner.
    Exploring the Universe with the worlds’ largest radio telescope
    

    View Slide

49. Thanks!
    Exploring the Universe with the worlds’ largest radio telescope
    SQUARE KILOMETRE ARRAY
    

    View Slide

50. Questions?
    Exploring the Universe with the worlds’ largest radio telescope
    SQUARE KILOMETRE ARRAY
    j.santander-vela@skatelescope.org
    

    View Slide