The Rise, the Ruin & the Rescue. Improving a large eCommerce system Dr. Gernot Starke

Dr. Gernot Starke INNOQ Fellow • Architecture-Improver • Coach, Trainer • Founder arc42, aim42 • iSAQB e.V.

Similarities … • to existing companies are desired. • Everything (except 3 slides) has really happened... • All persons and companies mentioned here are fictitious © Dr. Gernot Starke.

about the company. Internationally operating sales company, administration located in Colorado (USA), headquarter in Berlin (Germany). Bundles the expertise of over 500 specialists across the entire sales and technical spectrum: from efficient procurement and partner support to the design and introduction of forward-looking configuration and sales processes to customer-oriented sales and service. With more than 1,900,000 customers worldwide, 900,000 of them in Europe, SAMM Inc. is internationally and innovatively positioned, at the same time traditionally and focused on sustainable customer and partner orientation.

1992 1999 2004 2009 2015 Atlas Software GmbH & Co. KG Dr. Blue & Partner Red Inc. Hodor KG Engineering WebDev Inc. (London + München) Red Holding + Red Europa SAMM International Yellow Finance Inc Subsidaries in Hungary& Pakistan Grey Inc 2019

Heart of the company… Not related to the 2019 horror-movie. Although... venom42 1997 2019

The Rise … 22

23 Anna & Mark

Wie groß soll Ihr Regal sein? Breite: Höhe: Tiefe: Böden: weiter… Maße Material Farben Bestellung cm cm cm 60 60 25 1 Beratung http:/ /samm24.com/de/shop/furniture/config

25 Franz-Georg v.S. Head of Administration, German Embassy, Brasil

No content

Doris Pharmacist

http://www.scope-online.de/upload_weka/Knapp%20Apothekenroboter_Apostore-908_1191372.jpg

29 Peter Coordinator NPO

interactive configuration, AI-supported proposals, planning, execution, training, customs & export control, regulatory approvals, safety checks

The Ruin …

1992 1997 2003 2008 2013 „Atlas 1“ (Host) „Atlas 2“ (AS/400, Cobol) Backoffice- catalogue (Java + Host) eGov (Python) Web-Catalogue (Java) Pricemaster (Smalltalk) ComSuite (Java & Python) SAMM Sales (Java & Co) VENOM (Java & Co) Campaigner (Java, PHP) various competitors „WaWi“ (Host, Cobol)

Example: 1992 1997 2003 2008 2013 „Atlas 1“ (Host) „Atlas 2“ (AS/400, Cobol) Backoffice- Katalog (Java + Host) eGov (Python) Web-Katalog (Java) Pricemaster (Smalltalk) ComSuite (Java & Python) SAMM Sales (Java & Co) VENOM (Java & Co) Campaigner (Java, PHP) various competitors „WaWi“ (Host, Cobol)

Integration project /-desaster • Fixed end date, fixed budget • Requirements only roughly defined • Management only pursues project goals „Atlas 2“ (AS/400, Cobol) SAMM Sales (Java & Co)

„Atlas 2“ (AS/400, Cobol) SAMM Sales (Java & Co)

Instead of … „Clean“ integration based upon current domain-/data model...

… Quick‘n Dirty … Wrapper-Code for two-way transformations. „Atlas 2“ (AS/400, Cobol) SAMM Sales (Java & Co) Atlas2SAMM Mapping SAMM2Atlas Mapping Atlas2SAMM Glue SAMM2Atlas Glue

Technical Debt… „Atlas 2“ (AS/400, Cobol) SAMM Sales (Java & Co) Atlas2SAMM Mapping SAMM2Atlas Mapping Atlas2SAMM Glue SAMM2Atlas Glue

Donald. • CTO • Formerly: M&A Board of Directors (Vorstand), CEO Dr. Taler Finance & Controlling Delivery Research Information Technology External Relations Supply Sales HR Internal Systems Customer Systems Accounting Marketing Partner Relations Innovation Market Research BER BER DUR DUR 5 2 4 4 35 5 15 4 2 5 6 BER BER DUR BER DY IT FC ER RS

The situation. • strong decline in revenue from private customers • Time-to-Market unacceptable • so far no market entry in mobile internet • Hardly and innovation in products • Unused portfolio of patents

Conclusion from Donald: Current system makes innovation impossible: • Code generelly too bad • Clear know-how deficits in VENOM team

42 Doris, Apothekerin Lean, agile IT Donald proposes „bimodal IT“

Found a startup: 43 • Located in Berlin – like SAMM Inc. • Donald made CEO

44 Big Bang ... Requirem ents specification Data- m igration Architecture + Development Plan: approx 30 people, 3+ Scrum-Teams individual employees

2 Month later… 45 Cheap rent, but: difficult to reach.

100 Days • Highly promising prototypes • Ramp-Up delayed • so far only 6 (planned: 16) employees • Workaround: more external consultants

6 Month • VENOM: Strong decline in sales and earnings • Complax clarification of central requirements delays features in new system • byzz.io: first employees quit to find better jobs

12 Month 48 Management decides: • New system (prematurely) going live • Data migration declared „finished“

Hours later • New system shows disastrous errors in operation • byzz.io Administration massively overwhelmed Results lead to: • Rollback to previous system (VENOM) • Rollback of data migration

Supervisory Board intervenes...

Breaking News. Donald H. becomes Minister of Technology Donald H., long-time CIO of SAMM Inc., will take over the leadership of the new Ministry of Technology and Digitization at the beginning of the year. The spokeswoman of SAMM Inc. praised the good cooperation and the successful implementation of even critical projects – with Donald H., SAMM Inc. lost an extraordinary leader. Fake news #1

52 Aaron S.

53 aim42.org e valuate analyze improve Systematic Modernisation

aim42... 54 • identifies real problems and pains in systems • helps priorize („cost of pain)“ • suggests remedies

Improving systematically. Remedies Problems (prioritised)

Analyze. www.aim42.org breadth search Stakeholder Architecture Context Quality Data Processes Iteration1

Analyze: Stakeholder ... Interview breadth search Stakeholder Architecture Context Quality Data Processes

Stakeholder know many probleme (+ solutions).

Cathy, Developer • „DRAG“ – too many dependencies • Build and deploy: horror • Increasing extra-work-hours • Lack of overview • Hardly any innovation

Michael, UI-Developer • Small changes take increasingly longer to finish • Increasing number of (self-introduced) bugs • Too much technical debt

Stakeholder know many problems. Wrong data in archive Releases require manual work Releases depend on certain people Failing Communication with marketing departement Build process slow and unreliable Too many dependencies between subsystems

Analyze: Architecture ... • Internal structure • Code: • Implemenentation • Metrics • Concepts Stakeholder Architecture Context Quality Data Processes breadth search

Statical (Code) Analysis • Different languages complicate analysis... • Conventional metrics fail! Price Management Campaigns Pricing Engine Hodor Optical Archive Legend: Java PHP Python C/C++ Hask ell Cobol PL/ SQL Flash HTML/ JS Sales Frontend Private + Corporate Configurator Shell Client Contracts UDS (User Data Service) Order & Fullfillment Vouchers & Rebates eGov Shop Sales & Contracts Archive External Partners Price Management Data Warehouse Marketing & Sales Campaigns Atlas Customs & Logistics Pricing Engine Sales Backend Private+Corp Hodor Optical Archive Post-Sales Services Security Extensions Legend: Java PHP Python C/C++ Hask ell Cobol PL/ SQL Flash HTML/ JS Sales Backend eGovernment

Meet John Ousterhout • TCL/TK • RAFT • Electric-Cloud https://web.stanford.edu/~ouster/cgi-bin/cs190-winter18/lecture.php?topic=working

Analyze: Tactical Tornado. UDS (User Data Service) Order & Fullfillment Vouchers & Rebates Sales & Contracts Archive Price Management Marketing & Sales Campaigns Atlas Customs & Logistics Pricing Engine Sales Backend Private+Corp Sales Backend eGovernment Haskell, complicated domain logic Middleware (Message-Queue)

... in price calculation. UDS (User Data Service) Order & Fullfillment Vouchers & Rebates Sales & Contracts Archive Price Management Marketing & Sales Campaigns Atlas Customs & Logistics Pricing Engine Sales Backend Private+Corp Sales Backend eGovernment Haskell, complicated domain logic Middleware (Message-Queue) Pricing calculations (partially) implemented within middleware L

Level-1: Component Disorder Sales Frontend Private + Corporate Configurator Shell Client Contracts UDS (User Data Service) Order & Fullfillment Vouchers & Rebates eGov Shop Sales & Contracts Archive External Partners Price Management Data Warehouse Marketing & Sales Campaigns Atlas Customs & Logistics Pricing Engine Sales Backend Private+Corp Hodor Optical Archive Post-Sales Services Security Extensions Legend: Java PHP Python C/C++ Hask ell Cobol PL/ SQL Flash HTML/ JS Sales Backend eGovernment

Level-1: Component Disorder „Heisenbugs“ in price calculation Haskell developer rarely present Overly complicated interaction between validation and calculation Highly volatile external interfaces NGO, pharmacy & government pricing overly dependend Self-made middleware

Analyze - Concepts ... Configurator Shell Meta Configurator Configuration Data Akquisition Legend: Java Prolog Python Drools Flash HTML/ JS Configuration Expert Configuration Validator Specific Configurator Sales Frontend Voucher generate use (compile-time) Pricing Engine User Mgnmt Sales Backend use (runtime) Client Contracts breadth search Stakeholder Architecture Context Quality Data Processes Fake news #2

Analyze: Quality ... How well are quality requirements achieved? • Performance • Flexibility • Security • etc... i.e. ATAM Software Product Quality Functional Suitability Reliability Performance efficiency Operability Security Compatibility Maintain- ability Transfer- ability Appropriate- ness Accuracy Compliance Availability Fault tolerance Recover- ability Compliance Time- behaviour Resource- utilisation Compliance Appropriate- ness Recognise- ability Learnability Ease-of-use Helpfulness Attractiveness Technical accessibility Compliance Confidential- ity Integrity Non- repudiation Account- ability Authenticity Compliance Replace- ability Co- existence Inter- operability Compliance Modularity Reusability Analyzability Changeability Modification stability Testability Compliance Portability Adaptability Installability Compliance breadth search Stakeholder Architecture Context Quality Data Processes

Analyze: Quality … Prio Attribute Szenario 1 Performance configurator display potential add-ons< 5 sec 1 Operability Configurator (private customers) usable on iOS & Android 2 Performance Price calculation finished < 10 sec 2 Changeability New product category < 30d live ... Architectural Approach on-demand loading of add-ons Multiple data sources for add-ons, only serialized queries possible Add-on queries partially hindered by wrong data in optical archive Configuration implemented Flash Parts of pricing data need to be retrieved from optical archive Software Product Quality Functional Suitability Reliability Performance efficiency Operability Security Compatibility Maintain- ability Transfer- ability Appropriate- ness Accuracy Compliance Availability Fault tolerance Recover- ability Compliance Time- behaviour Resource- utilisation Compliance Appropriate- ness Recognise- ability Learnability Ease-of-use Helpfulness Attractiveness Technical accessibility Compliance Confidential- ity Integrity Non- repudiation Account- ability Authenticity Compliance Replace- ability Co- existence Inter- operability Compliance Modularity Reusability Analyzability Changeability Modification stability Testability Compliance Portability Adaptability Installability Compliance Add-on calculation 10- 180 sec due to time- consuming queries Adobe-Flash not usable on iOS and Android Data required for pricing located in >4 different data sources

Analyze: Data ... • Structure • Data model • Content • Distribution / Replication • Security / Privacys Stakeholder Architecture Context Quality Data Processes breadth search

Data model horror • Massive performance issues • Data model (based upon former AS/400 DB2) • 5 tables, >400 columns (!!) each • Massively (!) coupled ... (500) ... (400) ... (400) ... (300) ... (400) Highly inperformant and complicated data model Row-count exceeds standard-DB capabilities (re-load required every 6 month)

Analyze: Processes ... Requirements/ Business-Analyse • Development • Test • Rollout • Operations Stakeholder Architecture Context Quality Data Processes breadth search

Organizational Disorder • 40% Development Inhouse, • 30% (external) Contractors • 30% Near-/Offshore with external support Heterogeneous: • Contracting • Development and release processes • Environments for sources, build, test + deploy Intransparent decision processes Lack of consistency within source code Need for coordination across teams & companies Chaos

Requirements chaos: Conflicts between business departments Requirements chaos: Always changing priorities Management optimizing „projects“ instead of product

Analysis identified problems… Flash-Konfigurator zu aufwändig in der Pflege falsche Daten im Archiv Enge Kopplung der Pricing Engine Releases dauern zu lange Releases benötigen viele manuelle Eingriffe zu hoher Anteil an manuellem Test Betriebsübergabe von einzelnen Personen abhängig Haskell Entwicklerin zu selten anwesend konkrete Preise hängen von zu vielen Parametern ab Know-How Flaschenhälse in Entwicklung und Betrieb Einkauf & Produktdesign komplett un-agil Scrum in Entwicklung kollidiert mit Planung in Fachbereichen Produktdaten verteilt auf zwei Sales-Backends mangelnde Qualität (Performance, Robustheit, Verfügbarkeit, Sicherheit) bei Verkaufs- und Vertragsdaten „Heisenbugs“ bei Preisberechnung „komplizierte“ Interaktion zwischen Validierung und Preisbestimmung Expertenwissen über Konfiguration verteilt auf Prolog und Drools Flash als Sicherheitsrisiko zu viele Datenquellen Pricing- Schnittstelle(n) sehr volatil optisches Archiv enthält falsche Daten breadth search

Conclusion: • Many parts are „good enough“ • Overall structure („big picture“) really bad • Code quality partially bad • High competence in development teams • Data migration extremely risky -> postpone

Analyse identifies problems & solution ideas Flash-Konfigurator zu aufwändig in der Pflege falsche Daten im Archiv Enge Kopplung der Pricing Engine Releases dauern zu lange Releases benötigen viele manuelle Eingriffe zu hoher Anteil an manuellem Test Betriebsübergabe von einzelnen Personen abhängig Haskell Entwicklerin zu selten anwesend konkrete Preise hängen von zu vielen Parametern ab Know-How Flaschenhälse in Entwicklung und Betrieb Einkauf & Produktdesign komplett un-agil Scrum in Entwicklung kollidiert mit Planung in Fachbereichen Produktdaten verteilt auf zwei Sales-Backends mangelnde Qualität (Performance, Robustheit, Verfügbarkeit, Sicherheit) bei Verkaufs- und Vertragsdaten „Heisenbugs“ bei Preisberechnung „komplizierte“ Interaktion zwischen Validierung und Preisbestimmung Expertenwissen über Konfiguration verteilt auf Prolog und Drools Flash als Sicherheitsrisiko zu viele Datenquellen Pricing- Schnittstelle(n) sehr volatil optisches Archiv enthält falsche Daten Pricing engine into one single component Introduce PO (product owner) as role factor-out SCS for NPOs create executable specifications/ tests with Cucumber

Systematic Modernisation. • Keep treasures: continous restructure • Rapid removal of critical hot spots • Transformation to SCS („High-Level Modularization“) • Transition to agile organization

Integrate improvements into day-to-day development... day-to-day development time tactical improvement long-term improvement tactical improvement tactical improvement tactical improvement

Bea Change Facilitator find the right people for the job!

Agile Practices. gently but stringently introduce „agile“ in: • Development • IT-Operations • Sales • Organization • ... ✓

Further organizational change. • Specification-by-Example • Fully automated acceptance tests for „domains“: • Pharmacy • Government / Embassies • Private customers • NGO/NPO

Architecture-Modernisation (1). Change-by- Extract. Client Flawed (incohesive) System Client „other“ other features 2 Client Flawed System Client „other“ other features 1 3 Better other features Client (reduced) Flawed System Client „other“

Architecture-Modernisation (2). Change-by- Split. Client Type 1 Flawed System Client Type 2 1 Client Type 1 Reduced to Type 1 Client Type 2 Reduced to Type 2 2 New Type 1 System Client Type 1 Client Type 2 New Type 2 System 3

Venom-Splits (1). Venom, Split-1 NGOs, User Groups Venom, Split-2 Private User Corporate Users Government Users Operations Internal Users Operations Internal Users

Venom-Splits (2): NGO-Spezifics. NGOs, User Groups Operations Internal Users kann entfallen Security Extensions Sales Frontend Private + Corporate Configurator Shell Client Contracts UDS (User Data Service) Order & Fullfillment Voucher s & Rebates eGov Shop Sales & Contracts Archive External Partners Price Manageme nt Data Warehouse Marketing & Sales Campaigns Atlas Customs & Logistics Pricing Engine Sales Backend Private+Corp Hodor Optical Archive Post-Sales Services Sales Backend eGovernme nt

Venom-Splits (3): Commons. Estimated reduction: from 2 Mio LOC (VENOM) down to <200 kLOC in NGO-Split common NGOs, User Groups Operations Internal Users Sales Frontend Client Data NGO User Management Inventory Sales Order & Contracts Archive Pricing Engine Sales Backend Security Extensions DB Commons Client Contract Common Price Mngmt Commons Common User Data verkleinert

3 Month later... Retrospective for NGO Split: • time-to-market: 5 days (instead of >30 in VENOM) • Production-Bugs reduced to <2/Woche (formerly >10) • Developer-Happyness: ++ • Inter-Team coordination required more effort • Scrum-of-Scrum for Common Services

in parallel ... prepare for additional splits : • Government • Pharmacy • Private

SW-Architecture... As role in development: • Embedded within teams • Default-skill for dev‘s

Preserve Know-How ... pragmatical architecture documentation https://arc42.org

9 Month later... Re-Architecture Pricing-Engine (Java replaces Haskell) • JBoss Drools with modular rule sets • Merge Price-Management with Pricing-Engine

18 Month later... • Stable increase of business revenue and turnover • MVP „VENOM2Go“ very successful • (nearly) zero employee fluctuation

From the press. Apple brings in expert for software modernization Aaron Schwarz, until recently responsible for the internal VENOm system of SAMM Inc. disclosed his immediate move to Apple Inc. in Cupertino, California. Schwarz helped SAMM Inc. out of a severe crisis within only 18 month by re-architecting the (huge) internal e-commerc system. The VENOM system is now regarded as a model for successful modernization strategies in e-business. Schwarz writes that the VENOM-Teams are now mature enough and didn‘t need his ongoing support, so he could move on to new endavours. At Apple Schwarz will be responsible for the software architecture of the Apple iBank, based upon various legacy systems of some international banks Apple recently acquired. Fake news #3

Rate my session, please!

Thank you. Gernot Starke [email protected] Twitter: @gernotstarke www.innoq.com