SATURN 2013 - All Architecture Evaluation is not the Same

Transcript

1. © Fraunhofer IESE 1 All Architecture Evaluation is not the Same Lessons Learned from more than 50 Architecture Evaluations in Industry Matthias Naab May 2, 2013 SATURN 2013, Minneapolis
2. © Fraunhofer IESE 2 Fraunhofer IESE
3. © Fraunhofer IESE 3 Architecture Evaluation is a Mature Discipline … … but only few Practical Examples Published
4. © Fraunhofer IESE 4 Fraunhofer IESE > 50 Architecture Evaluations 2004 - 2013
5. © Fraunhofer IESE 5 Goals of this Talk Share experiences with other practitioners Complement literature Give empirical evidences
6. © Fraunhofer IESE 6 Companies in these Countries Involved
7. © Fraunhofer IESE 7 Industries Involved
8. © Fraunhofer IESE 8 Systems under Evaluation: Code Base Size: 10 KLoC – 10 MLoC Languages: Java, C, C++, C#, Delphi, Fortran, Cobol, Gen Age: 0 years – 30 years
9. © Fraunhofer IESE 9 Initiation of Architecture Evaluations
10. © Fraunhofer IESE 10 Initiator of Architecture Evaluation Initiator in same company Initiator in other company Top management Development management Method support group Development team Current customer Potential customer Disappointed customer Cautious customer
11. © Fraunhofer IESE 11 Typical Goals and Evaluation Questions How adequate are the solutions for my requirements? What is the impact of changing to paradigm SOA, Cloud, OO, … How adequate is our architecture as a basis for our future product portfolio? How different are two architectures? How feasible is the migration path? How can our system be modularized? How can we improve performance, reuse, maintainability, …? Which framework / platform / technology fits our needs best? How compliant is our solution to a reference architecture?
12. © Fraunhofer IESE 12 Emergency Rescue (Evolution vs. Revolution) Clash Quality Management Risk Management Initial Situation of Architecture Evaluation Project “out of hand” Project “on plan” Evaluation & Improvement Evaluation only Project Type Criticality ~ 20 Projects ~ 5 Projects ~ 5 Projects ~ 15 Projects ~ 5 Projects
13. © Fraunhofer IESE 14 Setup of Architecture Evaluation Projects
14. © Fraunhofer IESE 15 Organizational Constellation of Evaluations 2 (1-6) Initiating Organization 3-15 Organization with product under evaluation Organization with product under evaluation … 3-15 3-15 [Number of People involved]
15. © Fraunhofer IESE 16 Effort Spent on the Evaluation 4-200 Initiating Organization 1-10 Organization with product under evaluation Organization with product under evaluation … 2-60 2-60 [Person Days]
16. © Fraunhofer IESE 17 Factors Driving Effort for Architecture Evaluation Need for fast results Overall Effort Number of stakeholders Organizational complexity System size and complexity Evaluation questions Required confidence Criticality of situation
17. © Fraunhofer IESE 18 Overview on Architecture Evaluation Approach Implementation/system Architecture Stakeholder Concerns Knowledge Models Documents Source code Code metrics 0110 01 Concern elicitation check  Scenario Rating  Solution adequacy assessment Documentation assessment Compliance / distance assessment 0110 01 0110 01 Code quality assessment Interpretation
18. © Fraunhofer IESE 20 Outcome of Architecture Evaluations
19. © Fraunhofer IESE 21 Findings: Requirements that are Often Neglected Runtime Quality Attributes Devtime Quality Attributes Operation Quality Attributes Typically known Partially missing quantification Often not explicitly known Often hard to quantify Typically not explicitly known Often not addressed well Often not addressed well Often addressed well Partially missing quantification
20. © Fraunhofer IESE 22 Findings: Adequacy of Architectures under Evaluation [33 solution adequacy assessments] 6 11 16 Architecture typically not thoroughly defined Architecture often not fully adequate any more (older systems) Architecture thoroughly defined and maintained Often Emergency or Rescue projects Often Risk Management or Quality Management projects
21. © Fraunhofer IESE 23 Findings: Aspects that are „Over-Elaborated“ Technical Architecture Business Architecture Specification of general architectural styles Selection of technologies Definition of concrete components or guidelines how to define them Mapping of concrete functionality to technologies Over-Elaborated Neglected OSGi ESB …
22. © Fraunhofer IESE 24 Findings: Architecture Documentation Architectural Requirements Architecture Implementation Often not available Often not available Often very good knowledge Often very good knowledge → Missing uniformity, lack of compliance, quality problems D. Rost, M. Naab: Architecture Documentation for Developers: A Survey, ECSA 2013 Reconstruction is essential as basis for evaluation
23. © Fraunhofer IESE 25 Findings: Architecture Compliance [26 compliance checks] 9 7 10 Implementation chaotic Typically very high overall quality Strong adverse impact on quality of systems Impact often perceived by users and developers High cost for rework
24. © Fraunhofer IESE 26 Interpretation of Evaluation Results Architecture Evaluation often not fully objective and quantitative No standard interpretation possible Interpretation has to consider evaluation questions + many context factors Even quantitative data (e.g. number of incompliant relationships) often hard to interpret Representation of results for management is challenging (→ actions?) Tool-based reverse engineering often leads to nice but useless visualizations Stakeholders partially try to influence the interpretation for their goals
25. © Fraunhofer IESE 27 Follow-Up of Architecture Evaluations
26. © Fraunhofer IESE 28 Which Problems can be Fixed, which can‘t? Problems that often can be fixed Problems that often can’t be fixed Missing documentation Missing support for several new scenarios High degree of incompliance in code Missing thoroughness in definition of initial architecture Strong degree of degeneration of architecture over time Lower degree of incompliance in code Missing commitment in the fixing phase
27. © Fraunhofer IESE 29 Actions Taken after the Architecture Evaluations Project stopped New project for future architecture Initiative for improvement of architecture capabilities Selection of one of the candidate systems / technologies None Project for removing architecture violations Project for improvement of architecture
28. © Fraunhofer IESE 30 Evaluate your Architecture – early and regularly! Effort and method strongly depend on goals and context Interpretation of results is challenging Thorough and continuous architecting is key to success Key take aways of this talk
29. © Fraunhofer IESE 31
30. © Fraunhofer IESE 32 architecture@iese.fraunhofer.de architecture.iese.fraunhofer.de Dr. Matthias Naab Phone: +49 631 6800 2249 matthias.naab@iese.fraunhofer.de