Off the Shelf or Build It Ourselves?

A talk for ADASS 2022 on component selection in scientific software. CC-BY 4.0

1. Neil Ernst, Software Engineering, University of Victoria - ADASS 2022 Off the Shelf or Build It Ourselves? The Nature of Components in Scientific Software
2. Build vs Buy 2 Bias: need to focus on system architecture Bias: do this systematically Bias: all decisions take place in a context: Key constraints Team and individual skills Mission goals 2001
3. What is ‘externally developed’ software? 3
4. What is ‘externally developed’ software? 3 Off the shelf - not written for the project specifically
5. What is ‘externally developed’ software? 3 Off the shelf - not written for the project specifically Derived - mostly not written for the project but somewhat adapted, eg. from other instruments
6. What is ‘externally developed’ software? 3 Off the shelf - not written for the project specifically Derived - mostly not written for the project but somewhat adapted, eg. from other instruments Bespoke - written specifically for the project
7. What is ‘externally developed’ software? 3 Off the shelf - not written for the project specifically Derived - mostly not written for the project but somewhat adapted, eg. from other instruments Bespoke - written specifically for the project
8. What is ‘externally developed’ software? 3 Off the shelf - not written for the project specifically Derived - mostly not written for the project but somewhat adapted, eg. from other instruments Bespoke - written specifically for the project Astropy dependencies: numpy, matplotlib, pandas
9. What is ‘externally developed’ software? 3 Off the shelf - not written for the project specifically Derived - mostly not written for the project but somewhat adapted, eg. from other instruments Bespoke - written specifically for the project Astropy dependencies: numpy, matplotlib, pandas CASA: openMPI, ImageMagick, Astropy
10. What is ‘externally developed’ software? 3 Off the shelf - not written for the project specifically Derived - mostly not written for the project but somewhat adapted, eg. from other instruments Bespoke - written specifically for the project Astropy dependencies: numpy, matplotlib, pandas CASA: openMPI, ImageMagick, Astropy SKA: Tango messaging
11. What is ‘externally developed’ software? 3 Off the shelf - not written for the project specifically Derived - mostly not written for the project but somewhat adapted, eg. from other instruments Bespoke - written specifically for the project Astropy dependencies: numpy, matplotlib, pandas CASA: openMPI, ImageMagick, Astropy SKA: Tango messaging ESO CPL: FFTW, WCSLib
12. What is ‘externally developed’ software? 3 Off the shelf - not written for the project specifically Derived - mostly not written for the project but somewhat adapted, eg. from other instruments Bespoke - written specifically for the project Astropy dependencies: numpy, matplotlib, pandas CASA: openMPI, ImageMagick, Astropy SKA: Tango messaging ESO CPL: FFTW, WCSLib
13. What is ‘externally developed’ software? 3 Off the shelf - not written for the project specifically Derived - mostly not written for the project but somewhat adapted, eg. from other instruments Bespoke - written specifically for the project Astropy dependencies: numpy, matplotlib, pandas CASA: openMPI, ImageMagick, Astropy SKA: Tango messaging ESO CPL: FFTW, WCSLib (And don’t forget operating systems, compilers, Ci/CD tools, networking systems)
15. 5 https://xkcd.com/2224/
16. Big Question 6 Many libraries, frameworks, projects — how to choose, and when to build your own? 1. Know the quality attribute scenarios for your system. 2. Build and maintain a list of design concepts: architecture tactics, libraries, design patterns. 3. Continually measure and evaluate system components.
17. Quality Attribute Scenarios
18. Understanding Architecture Drivers 8 Before we can answer (most) questions about the system, need to know what we care about. Start with mission goals and technical context. Mission goals: why are we doing this? “Examine epoch of reionization” “Understand fast radio bursts” “Keep government funding”
19. Learning Technical Context 9 Technical context: Need to know what we already have, and what works. What constraints do we face? What legacy code and technical debt are we dealing with? Need to ensure we talk to people who know these questions and have answers (or could figure it out).
20. Architect cartoon from DSSE 10
21. Quality Attribute Scenarios 11 Now we have two prioritization dimensions (mission/technical). Let’s generate “test cases” for what the system should be doing. From whom? People on either side (head of software, chief scientist/PI, data engineers, end users…) Where to start?
22. Quality Attribute Utility Tree 12 Most architecturally relevant scenarios come from non-functional requirements, the “ilities”: Usability, maintainability, reliability, accuracy, performance, scalability Start with a few of those and drill in to a specific, measurable test E.g. (SKA): “when the telescope manager schedules a SchedulingBlock, the new schedule is updated within 1 minute” must be able to validate this is the case (often analytically, not data-oriented) Then prioritize on mission importance and technical challenge
24. Quality Attribute Scenarios: So What? 14 Now you have your test cases by which to evaluate buy/build decisions. Should we use Tango, or write our own message passing infrastructure? → How well does each solution do on the different QAS? Use attribute-driven design approach to iteratively examine each piece.
25. Design Concepts
26. 16 Designing Software Architectures, Kazman and Cervantes, 2019 Attribute -driven design
27. Building A Design Concept Library 17 Integration Messaging Data Collector Flume
 Logstash
 FluentD Distributed messaging broker RabbitMQ
 Amazon SQS
 ActiveMQ ETL/ELT ETL/Data integration Talend
 Informatica Data Storage Distributed File System Key-Value Riak Redis Berkeley DB NoSQL db Column-family HBase
 Cassandra Designing Software Architectures, Kazman and Cervantes, 2019
28. 18 Designing Software Architectures, Kazman and Cervantes, 2019
29. ML components - another twist 19 Often ML components are a combination of “reuse model” and “add custom fine-tuning” Must include ML models (BERT, DALL-E) and training data in list of external components Sculley, “ML: The High Interest Credit Card of TD”
30. Attributes include: Problem addressed Cost/Cost of Changes Licence Support Learning curve External software is an instantiated design concept 20 Maturity Popularity OSS health indicators Compatibility/integratability Support for key QA Size
31. Measure and Assess
32. Health Indicators 22 Most instruments very long-lived - e.g., SKA 50+ years Need to assess “health” of community (or company) building software CHAOSS Project: → Measure EDI, Value, Risk, Evolution https://chaoss.community/about/
33. Measuring and Assessing: Key Ideas 23 Last Responsible Moment: do not take decisions until absolutely necessary Cost of Delay: missing deadlines has a cost we often ignore. Technical Debt: beware creating a short-term solution at the expense of long- term maintainability
34. Example 1: BursaTec Mexico 24
35. Example 1: BursaTec Mexico 24 BursaTec: Mexican stock exchange
36. Example 1: BursaTec Mexico 24 BursaTec: Mexican stock exchange
37. Example 1: BursaTec Mexico 24 BursaTec: Mexican stock exchange Constraints: speed, time to market, new features
38. Example 1: BursaTec Mexico 24 BursaTec: Mexican stock exchange Constraints: speed, time to market, new features Key architectural driver: how to deliver trades < 1sec?
39. Example 1: BursaTec Mexico 24 BursaTec: Mexican stock exchange Constraints: speed, time to market, new features Key architectural driver: how to deliver trades < 1sec? Design concept: enterprise message bus pattern
40. Example 1: BursaTec Mexico 24 BursaTec: Mexican stock exchange Constraints: speed, time to market, new features Key architectural driver: how to deliver trades < 1sec? Design concept: enterprise message bus pattern
41. Example 1: BursaTec Mexico 24 BursaTec: Mexican stock exchange Constraints: speed, time to market, new features Key architectural driver: how to deliver trades < 1sec? Design concept: enterprise message bus pattern Question: buy externally or build internally
42. Example 1: BursaTec Mexico 24 BursaTec: Mexican stock exchange Constraints: speed, time to market, new features Key architectural driver: how to deliver trades < 1sec? Design concept: enterprise message bus pattern Question: buy externally or build internally Solution: evaluate in parallel and make decision with more data (Last responsible moment)
43. Example 2: SKA 25
44. Example 2: SKA 25 SKA: Square Kilometre Array - skao.int
45. Example 2: SKA 25 SKA: Square Kilometre Array - skao.int
46. Example 2: SKA 25 SKA: Square Kilometre Array - skao.int Constraints: longevity, latency, data volumes
47. Example 2: SKA 25 SKA: Square Kilometre Array - skao.int Constraints: longevity, latency, data volumes Key architectural driver: balance maintainability and speed
48. Example 2: SKA 25 SKA: Square Kilometre Array - skao.int Constraints: longevity, latency, data volumes Key architectural driver: balance maintainability and speed Design concept: low-level instrument control
49. Example 2: SKA 25 SKA: Square Kilometre Array - skao.int Constraints: longevity, latency, data volumes Key architectural driver: balance maintainability and speed Design concept: low-level instrument control
50. Example 2: SKA 25 SKA: Square Kilometre Array - skao.int Constraints: longevity, latency, data volumes Key architectural driver: balance maintainability and speed Design concept: low-level instrument control Question: which control software to buy?
51. Example 2: SKA 25 SKA: Square Kilometre Array - skao.int Constraints: longevity, latency, data volumes Key architectural driver: balance maintainability and speed Design concept: low-level instrument control Question: which control software to buy? Solution: Tango controls - after extensive prototyping → 2015 SKA Local Monitoring and Control (LMC) Standardisation Workshop → key low-level software that will be hard to change out → derived as SKA discovers edge cases - hire Tango expert to contribute to Tango Kernel
52. Summary: the Big Question 26 So how to choose, and when to build your own? 1. Know the quality attribute scenarios for your system. 2. Build and maintain a list of design concepts: architecture tactics, libraries, design patterns. 3. Continually measure and evaluate system components.
53. Neil Ernst
 nernst@uvic.ca @neilernst