Why Did the System Do That? Explaining how software works in terms of your intentions and its design

Research job talk

Transcript

1. Why Did the System Do That? Explaining how software works in terms of your intentions and its design Neil Ernst Software Engineering Institute Carnegie Mellon University neil@neilernst.net
2. You are here SEI
3. You are here SEI
4. You are here SEI
5. You are here SEI
6. Talk overview Motivation → why ‘explanation’? SHORT → quickly find optimal design/intention mappings TD study → what happens when people do not know what software should be doing ODDA → understand design and configuration Future work → challenges in explaining intelligent, connected systems MOTIVATION
7. Why ‘explanation’? Moving towards “intelligent connected” software systems Intelligent: adaptive and continually evolving Connected: emergent structure built with “other people’s software” MOTIVATION
8. Example: Smart transportation US DoT
9. ‘Explaining’ software Research challenges in intelligent connected systems: MOTIVATION
10. ‘Explaining’ software Research challenges in intelligent connected systems: Understand what happens when software is ‘built’ and evolves MOTIVATION
11. ‘Explaining’ software Research challenges in intelligent connected systems: Understand what happens when software is ‘built’ and evolves Managing software ‘brownfields’ with hidden design problems MOTIVATION
12. ‘Explaining’ software Research challenges in intelligent connected systems: Understand what happens when software is ‘built’ and evolves Managing software ‘brownfields’ with hidden design problems Dealing with little or no documentation MOTIVATION
13. ‘Explaining’ software Research challenges in intelligent connected systems: Understand what happens when software is ‘built’ and evolves Managing software ‘brownfields’ with hidden design problems Dealing with little or no documentation Analysing trade-offs of control for capability MOTIVATION
14. pgh left pic Explaining the “Pittsburgh Left”
15. pgh left pic Explaining the “Pittsburgh Left”
16. An intelligent, connected system
17. Explanation: connecting intentions and design
18. Lack of explanation means: We only see a fraction of the component source Build-time dependency resolution is complex Run-time adaptations are made with uninterpretable - unexplainable - outcomes MOTIVATION
19. Why did the system do that? Why Intentions How Design MOTIVATION Explaining software systems
20. Why did the system do that? Why Intentions How Design What Mappings MOTIVATION Explaining software systems
21. Mapping intentions to design MOTIVATION
22. Mapping intentions to design MOTIVATION
23. Mapping intentions to design Intentions MOTIVATION
24. Mapping intentions to design Intentions Design MOTIVATION
25. Mapping intentions to design Intentions Design MOTIVATION
26. Mapping intentions to design Intentions Design MOTIVATION
27. Explaining software systems MOTIVATION
28. Explaining software systems Intentions should be modeled: MOTIVATION
29. Explaining software systems Intentions should be modeled: → understand quality attributes with respect to business/mission goals MOTIVATION
30. Explaining software systems Intentions should be modeled: → understand quality attributes with respect to business/mission goals → is “it” (ours+theirs) doing what we need MOTIVATION
31. Explaining software systems Intentions should be modeled: → understand quality attributes with respect to business/mission goals → is “it” (ours+theirs) doing what we need → define policies and constraints MOTIVATION
32. Explaining software systems Design should be understood: → matches our intentions → not violating security/assurance constraints → emergent architectural properties MOTIVATION
33. 2010 2011 2012 2013 Automated quality topic naming [ESE13] RE as knowledge base [RE11] Paraconsistent Requirements [CAISE12] Techne Requirements Models [RE10] Intentions Design MOTIVATION My research
34. Intentions Design 2014 2015 2016 2017 Dealing with Technical Debt [FSE15] Mapping Intentions with SHORT[RE17] Evolving architecture [ICSME14] Design Roadmaps [WICSA16] TD Design Rules [ICSA17] ODDA Dynamic Design Analysis […] MOTIVATION My research
35. Intentions Design 2014 2015 2016 2017 Dealing with Technical Debt [FSE15] Mapping Intentions with SHORT[RE17] Evolving architecture [ICSME14] Design Roadmaps [WICSA16] TD Design Rules [ICSA17] ODDA Dynamic Design Analysis […] MOTIVATION My research
36. Talk overview Motivation → why ‘explanation’? SHORT → quickly find optimal design/intention mappings TD study → what happens when people do not know what software should be doing ODDA → understand design and configuration Future work → challenges in explaining intelligent, connected systems SHORT → quickly find optimal design/intention mappings OVERVIEW
37. Industry anecdote: optimal design roadmaps
38. Industry anecdote: optimal design roadmaps 452. million. dollars.
39. Industry anecdote: optimal design roadmaps 452. million. dollars. (US)
40. Industry anecdote: optimal design roadmaps
41. Quickly find optimal mappings from intentions to design Documentation Quick Feature Delivery Choose Doc Tool Documentation Tool Regression Test Big Bang Rewrite Incremental Rewrite Support Existing Apps Modernize Number Tiers Service Layer 2 Tier 3 Tier Tiers w/ Service Layer Sequence New Database PnP Framework Hard goal Softgoal Task/ service pref conflict LEGEND Data Service Data Model App Framework Monitor Data Service Comprehensive Data Model Extensible Data Model Specific Data Model App Framework Monitor Create Test Environment DB Vendor Test Env General Test Env Bakeoff Result Data Model Pilot Data Service Spec Data Service Pilot J2EE Specification Monitoring Pilot Access Control Assessed Access Control Pilot Good example of Agile Government development Easily Share Data Internally Easily Share Data W/ Partners and or Provide logical data schema internally Define data model for all shared data Define ext. mandatory data std Build internal extensible data model External data model can be extended External clients get exactly what they request XXX coordinates and internal client does whatever XXX coordinates and external client does whatever Choose Candidate System Svc layer w/ existing biz logic in DB Svc layer w/ extracted biz logic SHORT
42. Quickly find optimal mappings from intentions to design Documentation Quick Feature Delivery Choose Doc Tool Documentation Tool Regression Test Big Bang Rewrite Incremental Rewrite Support Existing Apps Modernize Number Tiers Service Layer 2 Tier 3 Tier Tiers w/ Service Layer Sequence New Database PnP Framework Hard goal Softgoal Task/ service pref conflict LEGEND Data Service Data Model App Framework Monitor Data Service Comprehensive Data Model Extensible Data Model Specific Data Model App Framework Monitor Create Test Environment DB Vendor Test Env General Test Env Bakeoff Result Data Model Pilot Data Service Spec Data Service Pilot J2EE Specification Monitoring Pilot Access Control Assessed Access Control Pilot Good example of Agile Government development Easily Share Data Internally Easily Share Data W/ Partners and or Provide logical data schema internally Define data model for all shared data Define ext. mandatory data std Build internal extensible data model External data model can be extended External clients get exactly what they request XXX coordinates and internal client does whatever XXX coordinates and external client does whatever Choose Candidate System Svc layer w/ existing biz logic in DB Svc layer w/ extracted biz logic SHORT
43. Quickly find optimal mappings from intentions to design Documentation Quick Feature Delivery Choose Doc Tool Documentation Tool Regression Test Big Bang Rewrite Incremental Rewrite Support Existing Apps Modernize Number Tiers Service Layer 2 Tier 3 Tier Tiers w/ Service Layer Sequence New Database PnP Framework Hard goal Softgoal Task/ service pref conflict LEGEND Data Service Data Model App Framework Monitor Data Service Comprehensive Data Model Extensible Data Model Specific Data Model App Framework Monitor Create Test Environment DB Vendor Test Env General Test Env Bakeoff Result Data Model Pilot Data Service Spec Data Service Pilot J2EE Specification Monitoring Pilot Access Control Assessed Access Control Pilot Good example of Agile Government development Easily Share Data Internally Easily Share Data W/ Partners and or Provide logical data schema internally Define data model for all shared data Define ext. mandatory data std Build internal extensible data model External data model can be extended External clients get exactly what they request XXX coordinates and internal client does whatever XXX coordinates and external client does whatever Choose Candidate System Svc layer w/ existing biz logic in DB Svc layer w/ extracted biz logic SHORT
44. Past work on intention-design mapping → express the models as SAT problems → add paraconsistency → use abductive search Problem: Bylander showed this was NP-hard and intractable for most pragmatic cases SHORT
45. Solution: sample, optimize, rank and test SHORT: multi-objective evolutionary search for optimal decisions. → Find optimal solutions → Rank decisions in order of importance. → Find “keys”, bottle-neck decisions. SHORT
46. Searching for decisions Goals linked via and/or edges Objectives: minimize cost, maximize benefit, maximize soft-goals satisfied For N iterations, randomly set goals to “ON/OFF” and find objective values Use Differential Evolution to intelligently evaluate the search space SHORT
47. Quickly find optimal mapping from intentions to design SHORT Documentation Quick Feature Delivery Choose Doc Tool Documentation Tool Regression Test Big Bang Rewrite Incremental Rewrite Support Existing Apps Modernize Number Tiers Service Layer 2 Tier 3 Tier Tiers w/ Service Layer Sequence New Database PnP Framework Hard goal Softgoal Task/ service pref conflict LEGEND Data Service Data Model App Framework Monitor Data Service Comprehensive Data Model Extensible Data Model Specific Data Model App Framework Monitor Create Test Environment DB Vendor Test Env General Test Env Bakeoff Result Data Model Pilot Data Service Spec Data Service Pilot J2EE Specification Monitoring Pilot Access Control Assessed Access Control Pilot Good example of Agile Government development Easily Share Data Internally Easily Share Data W/ Partners and or Provide logical data schema internally Define data model for all shared data Define ext. mandatory data std Build internal extensible data model External data model can be extended External clients get exactly what they request XXX coordinates and internal client does whatever XXX coordinates and external client does whatever Choose Candidate System Svc layer w/ existing biz logic in DB Svc layer w/ extracted biz logic Decision structure Cost/benefit values Differential Evolution (search) “what are the best combinations of decisions?” Bayesian Support (rank) “which decisions appear in the best solutions?” SHORT
48. Quickly find optimal mapping from intentions to design SHORT Documentation Quick Feature Delivery Choose Doc Tool Documentation Tool Regression Test Big Bang Rewrite Incremental Rewrite Support Existing Apps Modernize Number Tiers Service Layer 2 Tier 3 Tier Tiers w/ Service Layer Sequence New Database PnP Framework Hard goal Softgoal Task/ service pref conflict LEGEND Data Service Data Model App Framework Monitor Data Service Comprehensive Data Model Extensible Data Model Specific Data Model App Framework Monitor Create Test Environment DB Vendor Test Env General Test Env Bakeoff Result Data Model Pilot Data Service Spec Data Service Pilot J2EE Specification Monitoring Pilot Access Control Assessed Access Control Pilot Good example of Agile Government development Easily Share Data Internally Easily Share Data W/ Partners and or Provide logical data schema internally Define data model for all shared data Define ext. mandatory data std Build internal extensible data model External data model can be extended External clients get exactly what they request XXX coordinates and internal client does whatever XXX coordinates and external client does whatever Choose Candidate System Svc layer w/ existing biz logic in DB Svc layer w/ extracted biz logic Decision structure Cost/benefit values Differential Evolution (search) “what are the best combinations of decisions?” Bayesian Support (rank) “which decisions appear in the best solutions?” SHORT R Node S Su 1 J2EE O0.129 2 PnP O0.124 3 New O0.115 4 Documen O0.114 5 Access O0.113 6 Monitorin O0.112 7 General O0.110 8 Bakeoff O0.110 9 Access O0.108 10 DB O0.105 Key Decisions (the ones that matter)
49. Ranking decisions to find ‘keys’ Rank Node Status Support 1 J2EE spec ON 0.129 2 PnP framework OFF 0.124 3 New database OFF 0.115 4 Documentation tool ON 0.114 5 Access controls ON 0.113 6 Monitoring pilot ON 0.112 7 General test env ON 0.110 8 Bakeoff results ON 0.110 9 Access control pilot ON 0.108 10 DB vendor test env ON 0.105 .. … … … SHORT
50. Convergence after key decisions
51. Finding key decisions SHORT
52. Lessons learned Many models contain “keys” “Keys” can be found quickly “Keys” dramatically reduce the decision space SHORT
53. Talk overview Motivation → why ‘explanation’? SHORT → quickly find optimal design/intention mappings TD study → what happens when people do not know what software should be doing ODDA → understand design and configuration Future work → challenges in explaining intelligent, connected systems OVERVIEW TD study → what happens when people do not know what software should be doing
54. Industry anecdote: preventing technical debt
55. Conway’s law creates long-term risk organizations which design systems ... are constrained to produce designs which are copies of the communication structures of these organizations. — M. Conway TECHNICAL DEBT “
56. Central intentions and distributed design TECHNICAL DEBT Data Processor Array - Aus. Array - S.A. Data Transport Science Management Signal Processor Design Design Design Design Design Design
57. Central intentions and distributed design TECHNICAL DEBT Data Processor Array - Aus. Array - S.A. Data Transport Science Management Signal Processor Design Design Design Design Design Design System HQ Intentions
58. Problem investigation: technical debt Technical debt is when what we built (assembled) doesn’t match what we wanted. Does our design satisfy our intentions? Do our intentions match the design? TECHNICAL DEBT
59. Technical debt is … Cunningham, 1992: “Shipping first time code is like going into debt. A little debt speeds development so long as it is paid back promptly with a rewrite... The danger occurs when the debt is not repaid” TECHNICAL DEBT
60. Research Question Are issues with architectural elements among the most significant sources of technical debt? TECHNICAL DEBT
61. Methodology 1. Pilot Survey 2. Full survey Multiple choice background questions Open-ended answers Likert agreement 3. Follow-up interviews TECHNICAL DEBT
62. Demographics Type Description # Surveys Multinational for profit American Fortune 500 government contractor 3,500+ Mutinational for profit European F500 hardware organization 15,000+ US Government R&D American research software lab 200+ TECHNICAL DEBT
63. Demographics Developers (42%) and managers (32%) Web systems (24%) or embedded (31%) 29% > 10 years old Systems >100KLOC and < 1MLOC in size TECHNICAL DEBT
64. Ranking: Architecture choices key TECHNICAL DEBT # of top 3 appearances 0 75 150 225 300 Bad architecture choices O verly com plex code O bsolete technology Inter-m odule dependencies Inefficient builds
65. Coding: Architecture choices key % of Responses with code 0 20 40 60 80 Arhcitecture C hoice Aw areness Tim e Pressure R equirem ents Shortfall C ode Problem s Lack of D ocs TECHNICAL DEBT
66. Open-ended responses “decisions that could have been done at an earlier stage if we had had more time” “platform was not designed with scalability in mind” “In retrospect we put messaging/communication ... in the wrong place” TECHNICAL DEBT
67. Software entropy Lehman/Parnas concept of entropy or aging: “over the years, other sites would begin using the system and would require changes to how the workflow operated” 89% of longer-lived systems saw architectural issues as a significant source of debt. TECHNICAL DEBT
68. Lessons learned TECHNICAL DEBT
69. Lessons learned Leading sources of technical debt are (expedient) architectural choices TECHNICAL DEBT
70. Lessons learned Leading sources of technical debt are (expedient) architectural choices Managing design drift is vital in managing technical debt. TECHNICAL DEBT
71. Lessons learned Leading sources of technical debt are (expedient) architectural choices Managing design drift is vital in managing technical debt. Problem: gap between intentions and current design TECHNICAL DEBT
72. Talk overview Motivation → why ‘explanation’? SHORT → quickly find optimal design/intention mappings TD study → what happens when people do not know what software should be doing ODDA → understand design and configuration Future work → challenges in explaining intelligent, connected systems ODDA → understand design and configuration OVERVIEW
73. Industry anecdote: explaining configurable systems
74. http://blog.cleverelephant.ca/2017/02/on-transformation.html
75. “The call stack of despair” https://twitter.com/FioraAeterna/status/689908645476184068 ODDA
76. “The call stack of despair” https://twitter.com/FioraAeterna/status/689908645476184068 “Our” Code ODDA
77. A C B static dynamic A B C A 1 B 1 C 1 Understanding design-intention mapping in intelligent connected systems ODDA
78. A C B static dynamic D A B C A 1 B 1 C 1 Understanding design-intention mapping in intelligent connected systems ODDA
79. A C B static dynamic D A B C D A 1 B 1 C 1 1 D A B C A 1 B 1 C 1 Understanding design-intention mapping in intelligent connected systems ODDA
80. Explaining designs from configuration A B C D A 1 B 1 C 1 1 D C.java import com.google.guice.* . . . @inject public DbConn() { 
 } bind(Conn.class). to(SqlConn.class); ODDA
81. Explaining designs from configuration A B C D A 1 B 1 C 1 1 D C.java import com.google.guice.* . . . @inject public DbConn() { 
 } bind(Conn.class). to(SqlConn.class); ODDA X1
82. Explaining designs from configuration A B C D A 1 B 1 C 1 1 D C.java import com.google.guice.* . . . @inject public DbConn() { 
 } bind(Conn.class). to(SqlConn.class); X1 X1 - config. point ODDA X1
83. A C B static dynamic Understanding design-intention mapping in intelligent connected systems A B C A B C C1 B B ODDA
84. A C B static dynamic Understanding design-intention mapping in intelligent connected systems Quantify how many dynamic dependencies are in connected systems. A B C A B C C1 B B ODDA
85. A C B static dynamic Understanding design-intention mapping in intelligent connected systems Quantify how many dynamic dependencies are in connected systems. A B C A B C C1 Create mechanism to capture dynamic aspect and where it comes from. B B ODDA
86. A C B static dynamic Understanding design-intention mapping in intelligent connected systems Quantify how many dynamic dependencies are in connected systems. A B C A B C C1 Analyze results incorporating dynamic information and compare to compile-time baseline. Create mechanism to capture dynamic aspect and where it comes from. B B ODDA
87. Lessons learn(ing) Code level analysis vital Static design problems not identical to dynamic ones Nature of ‘dependency’ changes ODDA
88. Hinterland Who’s Who
89. Talk overview Motivation → why ‘explanation’? SHORT → quickly find optimal design/intention mappings TD study → what happens when people do not know what software should be doing ODDA → understand design and configuration Future work → challenges in explaining intelligent, connected systems Future work → challenges in explaining intelligent, connected systems OVERVIEW
90. Intentions Design 2014 2015 2016 2017 Dealing with Technical Debt [FSE15] Mapping Intentions with SHORT[RE17] Evolving architecture [ICSME14] Design Roadmaps [WICSA16] TD Design Rules [ICSA17] ODDA Dynamic Design Analysis [?] RESEARCH VISION Research vision
91. Short-term research challenge RESEARCH VISION config
92. Short-term research challenge Identify/learn system boundaries automatically RESEARCH VISION config
93. Short-term research challenge Identify/learn system boundaries automatically RESEARCH VISION config
94. Short-term research challenge RESEARCH VISION config
95. Short-term research challenge RESEARCH VISION config
96. Short-term research challenge Find dynamic design problems and technical debt RESEARCH VISION config
97. Mining system artifacts to automatically derive explanations Mid-term research challenge RESEARCH VISION Intentions Architecture Models Design Discussions Code Bug Reports History Stack Overflow “QUERY” “EXPLANATION”
98. Long-term vision Automatically configure intelligent, connected systems from high-level intentions and constraints Explanation with decision dashboards: why the software did that (and what decision to make next) RESEARCH VISION
99. Why did the system do that?
100. Why did the system do that? SHORT: Here’s why it should do it, and quick discovery of optimal ways to make it happen
101. Why did the system do that? SHORT: Here’s why it should do it, and quick discovery of optimal ways to make it happen TD Study: I’m not even sure what the system should be doing.
102. Why did the system do that? SHORT: Here’s why it should do it, and quick discovery of optimal ways to make it happen TD Study: I’m not even sure what the system should be doing. ODDA: What are my pieces of the system and how are they connected?
103. Why did the system do that? Intentions Design Mappings MOTIVATION Explaining software systems 0 40 80 A C B static dynamic D
104. Why did the system do that? neilernst.net @neilernst Intentions Design Mappings MOTIVATION Explaining software systems 0 40 80 A C B static dynamic D