When Architecture Meets Data

A talk given at IEEE Software Experts Summit, October 2015 in Beijing, exploring how software architecture is likely to change in the future as a result of software defined platforms and on-demand analytics, and the implications of this.

1. endava.com QUALITY. PRODUCTIVITY. INNOVATION. When Architecture Meets Data Self-Aware Systems for an Imperfect World Eoin Woods IEEE Software Experts Summit Beijing, October 2015 20151019.2
2. 2 When Architecture Meets Data Self-Aware Systems for an Imperfect World  Defining Architecture  Architecture Today  Where Architecture is Going  Self-Adaptive Systems  Examples  Implications For Us All QUALITY. PRODUCTIVITY. INNOVATION.
3. 3 3 To begin: what is “architecture”? Many uses of the term “architecture” For our purposes: Software Architecture is the set of software related elements that comprise a software-intensive system, their relationships, and the set of design decisions that allow the system to meet its system quality requirements (“non-functionals”) Software architecture tends to be the decisions that are difficult to change once implemented [Fowler] QUALITY. PRODUCTIVITY. INNOVATION.
4. 4 4 Architecture Today Software architecture today can be characterised as: • Static yet Ephemeral • Set in stone, difficult to discern in the running system • Depends on “Up-Front” Platform Design • The least agile part is the runtime platform design • Difficult to change • Architectural change is a manual, time-consuming affair QUALITY. PRODUCTIVITY. INNOVATION.
5. 5 5 Architecture – Static yet Ephemeral QUALITY. PRODUCTIVITY. INNOVATION. uml-diagrams.org http://msdn.microsoft.com
6. 6 6 Architecture – Big Platform Up Front QUALITY. PRODUCTIVITY. INNOVATION. http://www.clipartpanda.com/ http://kenwoodrecords.com/ http://www.aberdeeninc.com/ http://www.tbngconsulting.com/
7. 7 7 Architecture – Difficult to Change QUALITY. PRODUCTIVITY. INNOVATION. http://pubs.opengroup.org https://www.thehouseshop.com
8. 8 8 Where Architecture is Going Where do current trends point for architecture? QUALITY. PRODUCTIVITY. INNOVATION. Data Driven Dynamic http://daily.captaindash.com/
9. 9 9 Learning from Infrastructure Infrastructure is undergoing a revolution • Static, device centric, manual change, request based … to dynamic, virtualised, automated, policy based • Enabled by the “API for Infrastructure” – “SDx” QUALITY. PRODUCTIVITY. INNOVATION. http://www.cxotoday.com/story/cisco-vmware-deliver-next-generation-cloud-infrastructure/ http://www.bbc.co.uk/news/business-22879160
10. 10 10 Democratisation of Data Data and analytics are also undergoing major change • Centralised very expensive warehouses … to scalable open source and SaaS analysis platforms • Platform and algorithms becoming accessible commodities • Allow predictive analytics on large data sets QUALITY. PRODUCTIVITY. INNOVATION. http://www.iconshut.com/ http://www.dbtekpro.com/2014/10/21/scaling-data-apps-cloud/
11. 11 11 Analytics Providing System Insight Analytics possibilities • Descriptive – what has happened so far? • Predictive – what is likely in the future? • Prescriptive – how do we optimise in response? Example using Predictive Analytics • Transaction profiling of system activity metrics • Machine learning to “teach” the system what peaks in activity look like • Resulting model predicting future activity peaks QUALITY. PRODUCTIVITY. INNOVATION. https://azure. microsoft.com
12. 12 12 Analytics Providing System Insight QUALITY. PRODUCTIVITY. INNOVATION. 0 50 100 150 200 Batch Time Batch Time 0 10 20 30 40 50 60 70 Memory Memory 0 10000 20000 30000 40000 50000 60000 Volume ? ? ? http://commons.wikimedia.org
13. 13 13 Impact on Software Architecture? What could the effect be on software architecture? • SDx gives measurement and control of platform • Analytics provides insight into system operation • SDx + Analytics == dynamic platform • Where does this leave the software architecture? • The weak link? QUALITY. PRODUCTIVITY. INNOVATION.
14. 14 14 Impact on Our Systems • SDx + Analytics == dynamic platform • Software architecture is still static – left behind? QUALITY. PRODUCTIVITY. INNOVATION. Dynamic Infrastructure Platform Analytics Based Controller Observations Changes http://msdn.microsoft.com
15. 15 15 • What if we had an API for our architecture? • Needs first class architecture in the code • SDI + Analytics + SDA == dynamic system Architecture API Control The Missing API - Architecture QUALITY. PRODUCTIVITY. INNOVATION. Platform API Analytics Based Controller Control Dynamic Infrastructure Platform Measurements
16. 16 Not a new idea 1999 – IEEE Intelligent Systems Article • An Architecture Based Approach to Self-Adaptive Systems • Oreizy, Gorlick, Taylor, Heimbigner, Johnson, Medvidovic, Quilici, Rosenblum and Wolf 2009 – QoSA Conference theme • Architectures for Self-Adaptive Software Systems Research still going on today • Recent survey paper “A survey on engineering approaches for self-adaptive systems” by Krupitzer, Roth, VanSyckel, Schiele, and Becker Self-Adaptive Systems QUALITY. PRODUCTIVITY. INNOVATION.
17. 17 17 Self-Adaptive Systems QUALITY. PRODUCTIVITY. INNOVATION. Conditional Statements Online Algorithms Generic & Parameterised Algorithms Algorithm Selection Evolutionary Algorithms & ML
18. 18 18 Self-Adaptive Systems QUALITY. PRODUCTIVITY. INNOVATION. Evolution Maintain Integrity Make changes; collect observations Adaptation Evaluate Operation Plan Change Apply Change Model (Adapted from Oreizy 99) System
19. 19 19 What problems might this solve? We could have systems that … • Auto-scale reactively • Add capacity when load increases (we have this today) • Intelligently auto-scale predictively • Adjust capacity for predicted peaks and troughs • Change their functional behaviour • Predictively replace processing components to meet predicted requirements QUALITY. PRODUCTIVITY. INNOVATION.
20. 20 20 Performance Analytics Risk Estimator An Example QUALITY. PRODUCTIVITY. INNOVATION. Transaction Processor Request Handler Demand Message Bus Risk Calculator Risk Dashboard Predictive Controller Risk Estimator Risk Calculator
21. 21 21 An API for Architecture Controlling our architecture would involve: • Knowing the current structure (a model) • Adding & removing elements and connectors • Being able to “rewire” elements & connectors • Patterns for allowable transformations • Structural constraints to reject invalid structures • Policy based constraints for unacceptable systems • API implemented by the platform (PaaS) Programming language extension or an external language? • In the codebase & “compiled” • Maybe related to existing “ADLs”? QUALITY. PRODUCTIVITY. INNOVATION.
22. 22 22 An API for Architecture Our example using an imaginary API QUALITY. PRODUCTIVITY. INNOVATION. arch = Platform.getArchitecture("MyArch") riskCalc = arch.getElement("RiskCalculator") msgBus = calc.getConnector("inboundMsg") dashBrd = calc.getConnector("dashboard") riskEst = arch.getElement("RiskEstimator") riskEst.setConnector("dashboard", dashBrd) riskEst.setConnector("inboundMsg", msgBus) riskCalc.setConnector("dashboard", null) riskCalc.setConnector("inboundMsg", null) riskCalc.deactivate() riskEst.activate() arch.commitChanges()
23. 23 23 Implications What does this mean for research and practice? • More cross disciplinary research than is normal • Data, machine learning, adaptive systems, middleware, … • More industry and academic cooperation • Industry has the systems, academia the models • Crossing roles including architecture, infrastructure & testing • First class runtime software architecture • You need a model of the architecture to change it • That model should be in the implementation • No (general) runtime architecture models in use today • The challenge Software Architecture research needs? QUALITY. PRODUCTIVITY. INNOVATION.
24. 24 24 Our Future Systems Architectures QUALITY. PRODUCTIVITY. INNOVATION. Handler Processor Calculator Store SDx Infrastructure System Analytics Architecture API Self-Adaptive Systems Dynamic Data-Driven Systems
25. 25 Thank you QUALITY. PRODUCTIVITY. INNOVATION. Eoin Woods Endava eoin.woods@endava.com +44 207 367 1000 en_ewoods