Architecture War Stories

In this session, I will talk about entertaining examples of architectural disasters in software projects. We will see how excellent ideas can turn into nightmares, how one can slowly but thoroughly introduce incredible complexity, and how a merge between organizational and technical failures can grind productivity to a halt. Names and irrelevant details have been changed to protect the somewhat innocent, but everything is based on actual things I had to experience – and sometimes helped create – in the real world.

Transcript

1. Stefan Tilkov | @stilkov | innoQ Architecture War Stories CRAFT Budapest 04/25/2014
2. Ground Rules No offense – just lessons learned Most systems are in production Still, many {managers | users | admins} are happy
3. Less can be more
4. Context Customer care, accounting & billing Large-scale big-bang replacement effort > 150 developers > 20 architects/framework devs Java, Swing, CORBA, Oracle Custom persistence framework
5. Server Client DB
6. Server Identity Map 1 7 3 9 Tx 1 Tx2 Transaction Map 1 9 1 2 -
7. Client DB Server
8. Server Server Object Warehouse Object Warehouse
9. Global synchronization? 2 PC? Cache notifications? Events? Transaction locality? Client Affinity?
10. Server Server Object Warehouse Object Warehouse
11. Server Server Cache Cache DB Cache
12. Lessons learned Don’t cache the cache In distributed systems, application state is your enemy If it’s “sophisticated”, you probably shouldn’t be doing it Development ≠ production
13. Data Complications
14. Context Insurance system Maintenance 1000 person months/year Smalltalk, COBOL, C, IMS, DB2 Homegrown COBOL OO system
15. Client (Smalltalk) Broker Gateway Transaction Manager Server (COBOL)
16. Online Transactions Online DB Access Layer Interface DB Batch External Module External Module External Module Tables Foreign Keys Indexes OR/M Events Deltas Blobs
17. Migration
19. Lessons learned Data should be free of code dependencies The longer you store it, the simpler it needs to be Beware of smelly blobs
20. Evaluating Yourself to Death
21. Context Strategic health care platform Architecture team 20+10 people Build foundation for 800+ person year effort
22. Vendor Selection Collect and agree on requirements Week 0 Conduct market research Week 8 Send out RFP to selected vendors Week 10 Evaluate responses, create shortlist, start PoC Week 14 Evaluate PoC results, recommend vendor X Week 20 Accept your CEO picked vendor Y Week 26
23. Vendor Selection Collect and agree on requirements Week 0 Conduct market research Week 8 Send out RFP to selected vendors Week 10 Evaluate responses, create shortlist, start PoC Week 14 Evaluate PoC results, recommend vendor X Week 20 Shutdown company Week 26
24. Lessons learned Just do it Better ask for forgiveness than permission
25. Your system WILL be dynamic
26. Context Large-scale insurance system Model-driven development > 100 developers 2 Releases/year
27. Modeling Business Concept 0 3 6 4 5 2 1 Technical Concept Implementation Module Test Integration Test Rollout Vision What if you miss your slot? Modeling
28. Policy Holder Clause - id - value - dueDate - name - address - status - text - validity - taxClass * * regionCode Model Name New Name (Meaning) Description Release Introduced taxClass regionCode … 10.3 …
29. Lessons learned Centralized responsibility hurts Faced with too much rigidity, a way around the rules will be found Just because you’re used to it doesn’t mean it’s acceptable
30. Formalism and Common Sense as Orthogonal Concepts
31. Context Telecommunications OSS platform Support new devices without code changes > 100 developers > 3 countries
32. Warning ((Meta) meta) models) are a perfect means to waste time, on a highly inspiring intellectual level, without any results for years Highly complex methodology, tool support, agreement processes Consultants with exclusive pricing strategies
33. Upload «servlet» DB Model XML XML XML XML XML Transformation «slsb» .pl .sh Schema .sql Interface .idl Call «ant task» Quartz Transformer «servlet» Objects «entity» Generic CRUD UI Access «slsb»
34. Lessons learned If it makes you want to scratch your eyes out, you might not want to do it Sometimes constraints need to be fought Beware all things “meta”
35. The Perils of Being Smart
36. Context Complex enterprise-wide SOA effort Custom-built “lightweight” middleware
37. Consumer/Provider Middleware Lib Transport Abstraction Messaging API Business Logic Configuration Service Registry Service
38. Consumer/Provider Middleware Lib Transport Abstraction Messaging API Business Logic XML Obj Consumer/Provider Middleware Lib Transport Abstraction Messaging API Business Logic Message XML
39. “The Product” Consumer/Provider Middleware Lib Transport Abstraction Messaging API Business Logic XML Obj OO Mapper Obj XML Transport Abstraction HTTP CORBA JMS Object API
41. “The Product” Transport Abstraction OO Mapper Obj XML Object API Messaging API Interceptor Interceptor SOAP Message Header Body XML Message XML Obj null SOAP Message Header Body XML
42. Lessons learned Fight the madness Tunneling is the root of all evil Don’t be too smart
43. Non-extensible Extensibility
44. Context E-Commerce (retail) provider Global customer base Catalog/CMS/Shop/Fulfillment Multi-tenant Highly customizable
45. Large customers (“strategic”) Small customers (“long tail”) Specific General High Low (Acceptable) Cost Customization (Need) The Solution
46. Lessons learned Don’t be all things to all people Sometimes specific code is preferable Saying “no” might help
47. Summary
48. Never-ending supply of {war stories | catastrophes | disasters} seems guaranteed
49. Advice:
50. 1. Give Feedback
51. 2. Reflect
52. 3. Iterate
53. Q&A Stefan Tilkov, @stilkov stefan.tilkov@innoq.com Phone: +49 170 471 2625 innoQ Deutschland GmbH Krischerstr. 100 40789 Monheim am Rhein Germany Phone: +49 2173 3366-0 innoQ Schweiz GmbH info@innoq.com Gewerbestr. 11 CH-6330 Cham Switzerland Phone: +41 41 743 0116 www.innoq.com Ohlauer Straße 43 10999 Berlin Germany Phone: +49 2173 3366-0 Robert-Bosch-Straße 7 64293 Darmstadt Germany Phone: +49 2173 3366-0 Radlkoferstraße 2 D-81373 München Germany Telefon +49 (0) 89 741185-270