Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Off the Shelf or Build It Ourselves?

Neil Ernst
November 04, 2022

Off the Shelf or Build It Ourselves?

A talk for ADASS 2022 on component selection in scientific software. CC-BY 4.0

Neil Ernst

November 04, 2022
Tweet

More Decks by Neil Ernst

Other Decks in Research

Transcript

  1. Neil Ernst, Software Engineering, University of Victoria - ADASS 2022

    Off the Shelf or Build It Ourselves? The Nature of Components in Scientific Software
  2. Build vs Buy 2 Bias: need to focus on system

    architecture Bias: do this systematically Bias: all decisions take place in a context: Key constraints Team and individual skills Mission goals 2001
  3. What is ‘externally developed’ software? 3 Off the shelf -

    not written for the project specifically
  4. What is ‘externally developed’ software? 3 Off the shelf -

    not written for the project specifically Derived - mostly not written for the project but somewhat adapted, eg. from other instruments
  5. What is ‘externally developed’ software? 3 Off the shelf -

    not written for the project specifically Derived - mostly not written for the project but somewhat adapted, eg. from other instruments Bespoke - written specifically for the project
  6. What is ‘externally developed’ software? 3 Off the shelf -

    not written for the project specifically Derived - mostly not written for the project but somewhat adapted, eg. from other instruments Bespoke - written specifically for the project
  7. What is ‘externally developed’ software? 3 Off the shelf -

    not written for the project specifically Derived - mostly not written for the project but somewhat adapted, eg. from other instruments Bespoke - written specifically for the project Astropy dependencies: numpy, matplotlib, pandas
  8. What is ‘externally developed’ software? 3 Off the shelf -

    not written for the project specifically Derived - mostly not written for the project but somewhat adapted, eg. from other instruments Bespoke - written specifically for the project Astropy dependencies: numpy, matplotlib, pandas CASA: openMPI, ImageMagick, Astropy
  9. What is ‘externally developed’ software? 3 Off the shelf -

    not written for the project specifically Derived - mostly not written for the project but somewhat adapted, eg. from other instruments Bespoke - written specifically for the project Astropy dependencies: numpy, matplotlib, pandas CASA: openMPI, ImageMagick, Astropy SKA: Tango messaging
  10. What is ‘externally developed’ software? 3 Off the shelf -

    not written for the project specifically Derived - mostly not written for the project but somewhat adapted, eg. from other instruments Bespoke - written specifically for the project Astropy dependencies: numpy, matplotlib, pandas CASA: openMPI, ImageMagick, Astropy SKA: Tango messaging ESO CPL: FFTW, WCSLib
  11. What is ‘externally developed’ software? 3 Off the shelf -

    not written for the project specifically Derived - mostly not written for the project but somewhat adapted, eg. from other instruments Bespoke - written specifically for the project Astropy dependencies: numpy, matplotlib, pandas CASA: openMPI, ImageMagick, Astropy SKA: Tango messaging ESO CPL: FFTW, WCSLib
  12. What is ‘externally developed’ software? 3 Off the shelf -

    not written for the project specifically Derived - mostly not written for the project but somewhat adapted, eg. from other instruments Bespoke - written specifically for the project Astropy dependencies: numpy, matplotlib, pandas CASA: openMPI, ImageMagick, Astropy SKA: Tango messaging ESO CPL: FFTW, WCSLib (And don’t forget operating systems, compilers, Ci/CD tools, networking systems)
  13. 4

  14. Big Question 6 Many libraries, frameworks, projects — how to

    choose, and when to build your own? 1. Know the quality attribute scenarios for your system. 2. Build and maintain a list of design concepts: architecture tactics, libraries, design patterns. 3. Continually measure and evaluate system components.
  15. Understanding Architecture Drivers 8 Before we can answer (most) questions

    about the system, need to know what we care about. Start with mission goals and technical context. Mission goals: why are we doing this? “Examine epoch of reionization” “Understand fast radio bursts” “Keep government funding”
  16. Learning Technical Context 9 Technical context: Need to know what

    we already have, and what works. What constraints do we face? What legacy code and technical debt are we dealing with? Need to ensure we talk to people who know these questions and have answers (or could figure it out).
  17. Quality Attribute Scenarios 11 Now we have two prioritization dimensions

    (mission/technical). Let’s generate “test cases” for what the system should be doing. From whom? People on either side (head of software, chief scientist/PI, data engineers, end users…) Where to start?
  18. Quality Attribute Utility Tree 12 Most architecturally relevant scenarios come

    from non-functional requirements, the “ilities”: Usability, maintainability, reliability, accuracy, performance, scalability Start with a few of those and drill in to a specific, measurable test E.g. (SKA): “when the telescope manager schedules a SchedulingBlock, the new schedule is updated within 1 minute” must be able to validate this is the case (often analytically, not data-oriented) Then prioritize on mission importance and technical challenge
  19. 13

  20. Quality Attribute Scenarios: So What? 14 Now you have your

    test cases by which to evaluate buy/build decisions. Should we use Tango, or write our own message passing infrastructure? → How well does each solution do on the different QAS? Use attribute-driven design approach to iteratively examine each piece.
  21. Building A Design Concept Library 17 Integration Messaging Data Collector

    Flume
 Logstash
 FluentD Distributed messaging broker RabbitMQ
 Amazon SQS
 ActiveMQ ETL/ELT ETL/Data integration Talend
 Informatica Data Storage Distributed File System Key-Value Riak Redis Berkeley DB NoSQL db Column-family HBase
 Cassandra Designing Software Architectures, Kazman and Cervantes, 2019
  22. ML components - another twist 19 Often ML components are

    a combination of “reuse model” and “add custom fine-tuning” Must include ML models (BERT, DALL-E) and training data in list of external components Sculley, “ML: The High Interest Credit Card of TD”
  23. Attributes include: Problem addressed Cost/Cost of Changes Licence Support Learning

    curve External software is an instantiated design concept 20 Maturity Popularity OSS health indicators Compatibility/integratability Support for key QA Size
  24. Health Indicators 22 Most instruments very long-lived - e.g., SKA

    50+ years Need to assess “health” of community (or company) building software CHAOSS Project: → Measure EDI, Value, Risk, Evolution https://chaoss.community/about/
  25. Measuring and Assessing: Key Ideas 23 Last Responsible Moment: do

    not take decisions until absolutely necessary Cost of Delay: missing deadlines has a cost we often ignore. Technical Debt: beware creating a short-term solution at the expense of long- term maintainability
  26. Example 1: BursaTec Mexico 24 BursaTec: Mexican stock exchange Constraints:

    speed, time to market, new features Key architectural driver: how to deliver trades < 1sec?
  27. Example 1: BursaTec Mexico 24 BursaTec: Mexican stock exchange Constraints:

    speed, time to market, new features Key architectural driver: how to deliver trades < 1sec? Design concept: enterprise message bus pattern
  28. Example 1: BursaTec Mexico 24 BursaTec: Mexican stock exchange Constraints:

    speed, time to market, new features Key architectural driver: how to deliver trades < 1sec? Design concept: enterprise message bus pattern
  29. Example 1: BursaTec Mexico 24 BursaTec: Mexican stock exchange Constraints:

    speed, time to market, new features Key architectural driver: how to deliver trades < 1sec? Design concept: enterprise message bus pattern Question: buy externally or build internally
  30. Example 1: BursaTec Mexico 24 BursaTec: Mexican stock exchange Constraints:

    speed, time to market, new features Key architectural driver: how to deliver trades < 1sec? Design concept: enterprise message bus pattern Question: buy externally or build internally Solution: evaluate in parallel and make decision with more data (Last responsible moment)
  31. Example 2: SKA 25 SKA: Square Kilometre Array - skao.int

    Constraints: longevity, latency, data volumes
  32. Example 2: SKA 25 SKA: Square Kilometre Array - skao.int

    Constraints: longevity, latency, data volumes Key architectural driver: balance maintainability and speed
  33. Example 2: SKA 25 SKA: Square Kilometre Array - skao.int

    Constraints: longevity, latency, data volumes Key architectural driver: balance maintainability and speed Design concept: low-level instrument control
  34. Example 2: SKA 25 SKA: Square Kilometre Array - skao.int

    Constraints: longevity, latency, data volumes Key architectural driver: balance maintainability and speed Design concept: low-level instrument control
  35. Example 2: SKA 25 SKA: Square Kilometre Array - skao.int

    Constraints: longevity, latency, data volumes Key architectural driver: balance maintainability and speed Design concept: low-level instrument control Question: which control software to buy?
  36. Example 2: SKA 25 SKA: Square Kilometre Array - skao.int

    Constraints: longevity, latency, data volumes Key architectural driver: balance maintainability and speed Design concept: low-level instrument control Question: which control software to buy? Solution: Tango controls - after extensive prototyping → 2015 SKA Local Monitoring and Control (LMC) Standardisation Workshop → key low-level software that will be hard to change out → derived as SKA discovers edge cases - hire Tango expert to contribute to Tango Kernel
  37. Summary: the Big Question 26 So how to choose, and

    when to build your own? 1. Know the quality attribute scenarios for your system. 2. Build and maintain a list of design concepts: architecture tactics, libraries, design patterns. 3. Continually measure and evaluate system components.