The Rise, the ruin and the rescue

Developers and architects spend most of their time adding features to their products (so-called maintenance) and are often annoyed about the many deficits of these systems: even supposedly simple things are becoming incredibly difficult with these legacy systems, and the time to market is getting worse and worse as business calls for more and more features. It’s rare to find time to reduce technical debt and clean up increasingly messy dependencies.

Gernot Starke explores possibilities to systematically escape legacy hell and reduce technical and other debt in your systems. You’ll learn strategical and tactical improvement approaches you can scale to fit your actual situation. Gernot conducts a breadth-first search for existing problems, issues, and risks within your system and clearly identifies technical, organizational, and communicative debts and determines their severity in order to concentrate on the worst of them. You’ll start to improve the situation using a number of strategic approaches, including brain size, where you systematically simplify and reduce and migrate toward self-contained systems or microservices; change by split and change by extraction to reduce dependencies; improve domain focus and incrementally introduce domain-driven design practices in legacy systems (restructure to domain); and improve modularization. Gernot explains each of the approaches based on a (not very hypothetical) large-scale ecommerce system. You’ll hear about the rise, decline, and rescue of that system.

Transcript

1. The Rise, the Ruin & the Rescue. Improving a large eCommerce system Dr. Gernot Starke
2. Dr. Gernot Starke INNOQ Fellow • Architecture-Improver • Coach, Trainer • Founder arc42, aim42 • iSAQB e.V.
3. Similarities … • to existing companies are desired. • Everything (except 3 slides) has really happened... • All persons and companies mentioned here are fictitious © Dr. Gernot Starke.
4. about the company. Internationally operating sales company, administration located in Colorado (USA), headquarter in Berlin (Germany). Bundles the expertise of over 500 specialists across the entire sales and technical spectrum: from efficient procurement and partner support to the design and introduction of forward-looking configuration and sales processes to customer-oriented sales and service. With more than 1,900,000 customers worldwide, 900,000 of them in Europe, SAMM Inc. is internationally and innovatively positioned, at the same time traditionally and focused on sustainable customer and partner orientation.
5. 1992 1999 2004 2009 2015 Atlas Software GmbH & Co. KG Dr. Blue & Partner Red Inc. Hodor KG Engineering WebDev Inc. (London + München) Red Holding + Red Europa SAMM International Yellow Finance Inc Subsidaries in Hungary& Pakistan Grey Inc 2019
6. Heart of the company… Not related to the 2019 horror-movie. Although... venom42 1997 2019
7. The Rise … 22
8. 23 Anna & Mark
9. Wie groß soll Ihr Regal sein? Breite: Höhe: Tiefe: Böden: weiter… Maße Material Farben Bestellung cm cm cm 60 60 25 1 Beratung http:/ /samm24.com/de/shop/furniture/config
10. 25 Franz-Georg v.S. Head of Administration, German Embassy, Brasil
12. Doris Pharmacist
13. http://www.scope-online.de/upload_weka/Knapp%20Apothekenroboter_Apostore-908_1191372.jpg
14. 29 Peter Coordinator NPO
15. interactive configuration, AI-supported proposals, planning, execution, training, customs & export control, regulatory approvals, safety checks
16. The Ruin …
17. 1992 1997 2003 2008 2013 „Atlas 1“ (Host) „Atlas 2“ (AS/400, Cobol) Backoffice- catalogue (Java + Host) eGov (Python) Web-Catalogue (Java) Pricemaster (Smalltalk) ComSuite (Java & Python) SAMM Sales (Java & Co) VENOM (Java & Co) Campaigner (Java, PHP) various competitors „WaWi“ (Host, Cobol)
18. Example: 1992 1997 2003 2008 2013 „Atlas 1“ (Host) „Atlas 2“ (AS/400, Cobol) Backoffice- Katalog (Java + Host) eGov (Python) Web-Katalog (Java) Pricemaster (Smalltalk) ComSuite (Java & Python) SAMM Sales (Java & Co) VENOM (Java & Co) Campaigner (Java, PHP) various competitors „WaWi“ (Host, Cobol)
19. Integration project /-desaster • Fixed end date, fixed budget • Requirements only roughly defined • Management only pursues project goals „Atlas 2“ (AS/400, Cobol) SAMM Sales (Java & Co)
20. „Atlas 2“ (AS/400, Cobol) SAMM Sales (Java & Co)
21. Instead of … „Clean“ integration based upon current domain-/data model...
22. … Quick‘n Dirty … Wrapper-Code for two-way transformations. „Atlas 2“ (AS/400, Cobol) SAMM Sales (Java & Co) Atlas2SAMM Mapping SAMM2Atlas Mapping Atlas2SAMM Glue SAMM2Atlas Glue
23. Technical Debt… „Atlas 2“ (AS/400, Cobol) SAMM Sales (Java & Co) Atlas2SAMM Mapping SAMM2Atlas Mapping Atlas2SAMM Glue SAMM2Atlas Glue
24. Donald. • CTO • Formerly: M&A Board of Directors (Vorstand), CEO Dr. Taler Finance & Controlling Delivery Research Information Technology External Relations Supply Sales HR Internal Systems Customer Systems Accounting Marketing Partner Relations Innovation Market Research BER BER DUR DUR 5 2 4 4 35 5 15 4 2 5 6 BER BER DUR BER DY IT FC ER RS
25. The situation. • strong decline in revenue from private customers • Time-to-Market unacceptable • so far no market entry in mobile internet • Hardly and innovation in products • Unused portfolio of patents
26. Conclusion from Donald: Current system makes innovation impossible: • Code generelly too bad • Clear know-how deficits in VENOM team
27. 42 Doris, Apothekerin Lean, agile IT Donald proposes „bimodal IT“
28. Found a startup: 43 • Located in Berlin – like SAMM Inc. • Donald made CEO
29. 44 Big Bang ... Requirem ents specification Data- m igration Architecture + Development Plan: approx 30 people, 3+ Scrum-Teams individual employees
30. 2 Month later… 45 Cheap rent, but: difficult to reach.
31. 100 Days • Highly promising prototypes • Ramp-Up delayed • so far only 6 (planned: 16) employees • Workaround: more external consultants
32. 6 Month • VENOM: Strong decline in sales and earnings • Complax clarification of central requirements delays features in new system • byzz.io: first employees quit to find better jobs
33. 12 Month 48 Management decides: • New system (prematurely) going live • Data migration declared „finished“
34. Hours later • New system shows disastrous errors in operation • byzz.io Administration massively overwhelmed Results lead to: • Rollback to previous system (VENOM) • Rollback of data migration
35. Supervisory Board intervenes...
36. Breaking News. Donald H. becomes Minister of Technology Donald H., long-time CIO of SAMM Inc., will take over the leadership of the new Ministry of Technology and Digitization at the beginning of the year. The spokeswoman of SAMM Inc. praised the good cooperation and the successful implementation of even critical projects – with Donald H., SAMM Inc. lost an extraordinary leader. Fake news #1
37. 52 Aaron S.
38. 53 aim42.org e valuate analyze improve Systematic Modernisation
39. aim42... 54 • identifies real problems and pains in systems • helps priorize („cost of pain)“ • suggests remedies
40. Improving systematically. Remedies Problems (prioritised)
41. Analyze. www.aim42.org breadth search Stakeholder Architecture Context Quality Data Processes Iteration1
42. Analyze: Stakeholder ... Interview breadth search Stakeholder Architecture Context Quality Data Processes
43. Stakeholder know many probleme (+ solutions).
44. Cathy, Developer • „DRAG“ – too many dependencies • Build and deploy: horror • Increasing extra-work-hours • Lack of overview • Hardly any innovation
45. Michael, UI-Developer • Small changes take increasingly longer to finish • Increasing number of (self-introduced) bugs • Too much technical debt
46. Stakeholder know many problems. Wrong data in archive Releases require manual work Releases depend on certain people Failing Communication with marketing departement Build process slow and unreliable Too many dependencies between subsystems
47. Analyze: Architecture ... • Internal structure • Code: • Implemenentation • Metrics • Concepts Stakeholder Architecture Context Quality Data Processes breadth search
48. Statical (Code) Analysis • Different languages complicate analysis... • Conventional metrics fail! Price Management Campaigns Pricing Engine Hodor Optical Archive Legend: Java PHP Python C/C++ Hask ell Cobol PL/ SQL Flash HTML/ JS Sales Frontend Private + Corporate Configurator Shell Client Contracts UDS (User Data Service) Order & Fullfillment Vouchers & Rebates eGov Shop Sales & Contracts Archive External Partners Price Management Data Warehouse Marketing & Sales Campaigns Atlas Customs & Logistics Pricing Engine Sales Backend Private+Corp Hodor Optical Archive Post-Sales Services Security Extensions Legend: Java PHP Python C/C++ Hask ell Cobol PL/ SQL Flash HTML/ JS Sales Backend eGovernment
49. Meet John Ousterhout • TCL/TK • RAFT • Electric-Cloud https://web.stanford.edu/~ouster/cgi-bin/cs190-winter18/lecture.php?topic=working
50. Analyze: Tactical Tornado. UDS (User Data Service) Order & Fullfillment Vouchers & Rebates Sales & Contracts Archive Price Management Marketing & Sales Campaigns Atlas Customs & Logistics Pricing Engine Sales Backend Private+Corp Sales Backend eGovernment Haskell, complicated domain logic Middleware (Message-Queue)
51. ... in price calculation. UDS (User Data Service) Order & Fullfillment Vouchers & Rebates Sales & Contracts Archive Price Management Marketing & Sales Campaigns Atlas Customs & Logistics Pricing Engine Sales Backend Private+Corp Sales Backend eGovernment Haskell, complicated domain logic Middleware (Message-Queue) Pricing calculations (partially) implemented within middleware L
52. Level-1: Component Disorder Sales Frontend Private + Corporate Configurator Shell Client Contracts UDS (User Data Service) Order & Fullfillment Vouchers & Rebates eGov Shop Sales & Contracts Archive External Partners Price Management Data Warehouse Marketing & Sales Campaigns Atlas Customs & Logistics Pricing Engine Sales Backend Private+Corp Hodor Optical Archive Post-Sales Services Security Extensions Legend: Java PHP Python C/C++ Hask ell Cobol PL/ SQL Flash HTML/ JS Sales Backend eGovernment
53. Level-1: Component Disorder „Heisenbugs“ in price calculation Haskell developer rarely present Overly complicated interaction between validation and calculation Highly volatile external interfaces NGO, pharmacy & government pricing overly dependend Self-made middleware
54. Analyze - Concepts ... Configurator Shell Meta Configurator Configuration Data Akquisition Legend: Java Prolog Python Drools Flash HTML/ JS Configuration Expert Configuration Validator Specific Configurator Sales Frontend Voucher generate use (compile-time) Pricing Engine User Mgnmt Sales Backend use (runtime) Client Contracts breadth search Stakeholder Architecture Context Quality Data Processes Fake news #2
55. Analyze: Quality ... How well are quality requirements achieved? • Performance • Flexibility • Security • etc... i.e. ATAM Software Product Quality Functional Suitability Reliability Performance efficiency Operability Security Compatibility Maintain- ability Transfer- ability Appropriate- ness Accuracy Compliance Availability Fault tolerance Recover- ability Compliance Time- behaviour Resource- utilisation Compliance Appropriate- ness Recognise- ability Learnability Ease-of-use Helpfulness Attractiveness Technical accessibility Compliance Confidential- ity Integrity Non- repudiation Account- ability Authenticity Compliance Replace- ability Co- existence Inter- operability Compliance Modularity Reusability Analyzability Changeability Modification stability Testability Compliance Portability Adaptability Installability Compliance breadth search Stakeholder Architecture Context Quality Data Processes
56. Analyze: Quality … Prio Attribute Szenario 1 Performance configurator display potential add-ons< 5 sec 1 Operability Configurator (private customers) usable on iOS & Android 2 Performance Price calculation finished < 10 sec 2 Changeability New product category < 30d live ... <etc> Architectural Approach on-demand loading of add-ons Multiple data sources for add-ons, only serialized queries possible Add-on queries partially hindered by wrong data in optical archive Configuration implemented Flash Parts of pricing data need to be retrieved from optical archive <u.v.a.m.> Software Product Quality Functional Suitability Reliability Performance efficiency Operability Security Compatibility Maintain- ability Transfer- ability Appropriate- ness Accuracy Compliance Availability Fault tolerance Recover- ability Compliance Time- behaviour Resource- utilisation Compliance Appropriate- ness Recognise- ability Learnability Ease-of-use Helpfulness Attractiveness Technical accessibility Compliance Confidential- ity Integrity Non- repudiation Account- ability Authenticity Compliance Replace- ability Co- existence Inter- operability Compliance Modularity Reusability Analyzability Changeability Modification stability Testability Compliance Portability Adaptability Installability Compliance Add-on calculation 10- 180 sec due to time- consuming queries Adobe-Flash not usable on iOS and Android Data required for pricing located in >4 different data sources
57. Analyze: Data ... • Structure • Data model • Content • Distribution / Replication • Security / Privacys Stakeholder Architecture Context Quality Data Processes breadth search
58. Data model horror • Massive performance issues • Data model (based upon former AS/400 DB2) • 5 tables, >400 columns (!!) each • Massively (!) coupled ... (500) ... (400) ... (400) ... (300) ... (400) Highly inperformant and complicated data model Row-count exceeds standard-DB capabilities (re-load required every 6 month)
59. Analyze: Processes ... Requirements/ Business-Analyse • Development • Test • Rollout • Operations Stakeholder Architecture Context Quality Data Processes breadth search
60. Organizational Disorder • 40% Development Inhouse, • 30% (external) Contractors • 30% Near-/Offshore with external support Heterogeneous: • Contracting • Development and release processes • Environments for sources, build, test + deploy Intransparent decision processes Lack of consistency within source code Need for coordination across teams & companies Chaos
61. Requirements chaos: Conflicts between business departments Requirements chaos: Always changing priorities Management optimizing „projects“ instead of product
62. Analysis identified problems… Flash-Konfigurator zu aufwändig in der Pflege falsche Daten im Archiv Enge Kopplung der Pricing Engine Releases dauern zu lange Releases benötigen viele manuelle Eingriffe zu hoher Anteil an manuellem Test Betriebsübergabe von einzelnen Personen abhängig Haskell Entwicklerin zu selten anwesend konkrete Preise hängen von zu vielen Parametern ab Know-How Flaschenhälse in Entwicklung und Betrieb Einkauf & Produktdesign komplett un-agil Scrum in Entwicklung kollidiert mit Planung in Fachbereichen Produktdaten verteilt auf zwei Sales-Backends mangelnde Qualität (Performance, Robustheit, Verfügbarkeit, Sicherheit) bei Verkaufs- und Vertragsdaten „Heisenbugs“ bei Preisberechnung „komplizierte“ Interaktion zwischen Validierung und Preisbestimmung Expertenwissen über Konfiguration verteilt auf Prolog und Drools Flash als Sicherheitsrisiko zu viele Datenquellen Pricing- Schnittstelle(n) sehr volatil optisches Archiv enthält falsche Daten breadth search
63. Conclusion: • Many parts are „good enough“ • Overall structure („big picture“) really bad • Code quality partially bad • High competence in development teams • Data migration extremely risky -> postpone
64. Analyse identifies problems & solution ideas Flash-Konfigurator zu aufwändig in der Pflege falsche Daten im Archiv Enge Kopplung der Pricing Engine Releases dauern zu lange Releases benötigen viele manuelle Eingriffe zu hoher Anteil an manuellem Test Betriebsübergabe von einzelnen Personen abhängig Haskell Entwicklerin zu selten anwesend konkrete Preise hängen von zu vielen Parametern ab Know-How Flaschenhälse in Entwicklung und Betrieb Einkauf & Produktdesign komplett un-agil Scrum in Entwicklung kollidiert mit Planung in Fachbereichen Produktdaten verteilt auf zwei Sales-Backends mangelnde Qualität (Performance, Robustheit, Verfügbarkeit, Sicherheit) bei Verkaufs- und Vertragsdaten „Heisenbugs“ bei Preisberechnung „komplizierte“ Interaktion zwischen Validierung und Preisbestimmung Expertenwissen über Konfiguration verteilt auf Prolog und Drools Flash als Sicherheitsrisiko zu viele Datenquellen Pricing- Schnittstelle(n) sehr volatil optisches Archiv enthält falsche Daten Pricing engine into one single component Introduce PO (product owner) as role factor-out SCS for NPOs create executable specifications/ tests with Cucumber
65. Systematic Modernisation. • Keep treasures: continous restructure • Rapid removal of critical hot spots • Transformation to SCS („High-Level Modularization“) • Transition to agile organization
66. Integrate improvements into day-to-day development... day-to-day development time tactical improvement long-term improvement tactical improvement tactical improvement tactical improvement
67. Bea Change Facilitator find the right people for the job!
68. Agile Practices. gently but stringently introduce „agile“ in: • Development • IT-Operations • Sales • Organization • ... ✓
69. Further organizational change. • Specification-by-Example • Fully automated acceptance tests for „domains“: • Pharmacy • Government / Embassies • Private customers • NGO/NPO
70. Architecture-Modernisation (1). Change-by- Extract. Client Flawed (incohesive) System Client „other“ other features 2 Client Flawed System Client „other“ other features 1 3 Better other features Client (reduced) Flawed System Client „other“
71. Architecture-Modernisation (2). Change-by- Split. Client Type 1 Flawed System Client Type 2 1 Client Type 1 Reduced to Type 1 Client Type 2 Reduced to Type 2 2 New Type 1 System Client Type 1 Client Type 2 New Type 2 System 3
72. Venom-Splits (1). Venom, Split-1 NGOs, User Groups Venom, Split-2 Private User Corporate Users Government Users Operations Internal Users Operations Internal Users
73. Venom-Splits (2): NGO-Spezifics. NGOs, User Groups Operations Internal Users kann entfallen Security Extensions Sales Frontend Private + Corporate Configurator Shell Client Contracts UDS (User Data Service) Order & Fullfillment Voucher s & Rebates eGov Shop Sales & Contracts Archive External Partners Price Manageme nt Data Warehouse Marketing & Sales Campaigns Atlas Customs & Logistics Pricing Engine Sales Backend Private+Corp Hodor Optical Archive Post-Sales Services Sales Backend eGovernme nt
74. Venom-Splits (3): Commons. Estimated reduction: from 2 Mio LOC (VENOM) down to <200 kLOC in NGO-Split common NGOs, User Groups Operations Internal Users Sales Frontend Client Data NGO User Management Inventory Sales Order & Contracts Archive Pricing Engine Sales Backend Security Extensions DB Commons Client Contract Common Price Mngmt Commons Common User Data verkleinert
75. 3 Month later... Retrospective for NGO Split: • time-to-market: 5 days (instead of >30 in VENOM) • Production-Bugs reduced to <2/Woche (formerly >10) • Developer-Happyness: ++ • Inter-Team coordination required more effort • Scrum-of-Scrum for Common Services
76. in parallel ... prepare for additional splits : • Government • Pharmacy • Private
77. SW-Architecture... As role in development: • Embedded within teams • Default-skill for dev‘s
78. Preserve Know-How ... pragmatical architecture documentation https://arc42.org
79. 9 Month later... Re-Architecture Pricing-Engine (Java replaces Haskell) • JBoss Drools with modular rule sets • Merge Price-Management with Pricing-Engine
80. 18 Month later... • Stable increase of business revenue and turnover • MVP „VENOM2Go“ very successful • (nearly) zero employee fluctuation
81. From the press. Apple brings in expert for software modernization Aaron Schwarz, until recently responsible for the internal VENOm system of SAMM Inc. disclosed his immediate move to Apple Inc. in Cupertino, California. Schwarz helped SAMM Inc. out of a severe crisis within only 18 month by re-architecting the (huge) internal e-commerc system. The VENOM system is now regarded as a model for successful modernization strategies in e-business. Schwarz writes that the VENOM-Teams are now mature enough and didn‘t need his ongoing support, so he could move on to new endavours. At Apple Schwarz will be responsible for the software architecture of the Apple iBank, based upon various legacy systems of some international banks Apple recently acquired. Fake news #3
82. Rate my session, please!
83. Thank you. Gernot Starke gernot.starke@innoq.com Twitter: @gernotstarke www.innoq.com