Towards Rapid Composition with Confidence in Robotics Software

ERA talk at Int Workshop on Robotics in Software Engineering at ICSE, 2018

Transcript

1. Towards Rapid Composition With Confidence in Robotics Software Neil Ernst’, Rick Kazman, Phil Bianco⸙
   ’6OJWFSTJUZ
   PG
   7JDUPSJB
   6OJWFSTJUZ
   PG
   )BXBJJ
   ⸙4PGUXBSF
   &OHJOFFSJOH
   *OTUJUVUF
2. Problem Modern software development is a world of rapid release and compositional development. This challenges developers to rapidly deploy trusted systems that include increasing numbers of untrusted components. Bad decisions are easy to make and have long reach. For example, decisions based on outdated documentation or decisions skewed to one criterion (e.g., performance).
3. AoA and Assurance Conformance Us Marketing material and vendor telecons Heuristic, off the cuff decisions Confidence Assessment Speed Project Goal
4. Context: ROS-M • ROS-M(ilitary) envisions a “component” approach to assembling DoD software for uncrewed vehicles. • Vision: special enclave with software tools, support services (e.g. docs and metadata), and a registry of available (approved) ROS components. Developer dilemma: Which components do I use? What information do I need to avoid a bad decision? How do I get reliable data?
5. Working Usage Scenario A developer is working on a UGV using ROS-M Sergey Demushkin, noun project
6. Approach 1. Determine typical indicators, plus tools and data for each indicator. 2. Score each component. 3. Aggregate indicators: use expert input for weights (e.g., peak load, design hotspots, vulnerability collection). 4. Validate on open source corpus and with stakeholders.
7. Sample Project Health Indicators
8. Quality Attributes
9. Sample Results
10. Evaluation Criteria Tool Correct • Internal tests against known component measures. • “Cross-fold”: pilot on 1 component, then test on new ROS components. Increased Confidence • Ask ROS experts for past problematic indicators and measures (e.g. use of O(n2) algorithm). • Inject these and measure detection rate. • Survey stakeholders in practice.
11. Evaluation Criteria (2) Reduced Decision Time • Baseline current approaches • Scorecard result (post-setup) achieved 1 SD faster than developer acting without our tool Operational Validity • Collaborator pilot of tool on their system with their sample components
12. Critiques and Open Questions • What indicators ought to be included? • We use existing architecture analyses • Iterative assessment clearly important • How to contextualize analysis for particular system? • Ignores mixed-criticality community, other open architecture standards (FACE) • More component selection than composition • Just pick the best component (it isn’t that hard?)
13. Neil Ernst nernst@uvic.ca Introduced ROS-M, with different focus for building ROS based robots Argued for scoreboard approach to rating ROS components Scoreboard combines indicators and potential components to rank candidates Rick Kazman kazman@hawaii.edu
14. Aggregation of Values Possible choices: • equal weighting: sum all indicators, normalize to 0..1 • normalize to industry baselines and categorize as high, medium, low Approach: 1. Each indicator gets a weight w based on our model of stakeholder priority 2. The model is derived from interviews and short AHP prioritization exercises 3. Normalize as necessary for different denominators such as SLOC, programming language 4. We create customizable templates, based on stakeholder interviews. - E.g. (wM1 * wM2 ) + 2(wP1 * log wP2 ) + 3(wS 1 ) (some more important) (some highly correlated)