8 years of product: From Monolith to Microservices

Presentation at Software Architecture Gathering 2020 (https://www.software-architecture-gathering.de/).

LeanIX is a Software-as-a-Service from Bonn, Germany, which started in 2012 and now has over 300 customers. In product and engineering, the solution is currently being further developed by more than 80 employees. Co-founder and CEO André Christ gives a look behind the scenes: From the transition to an API-based service structure, frequent deployments thanks to Kubernetes, to changes in the organization of product management. Among other things, he shares "lessons learned" in the development of a generic Java integration framework, the appropriate choice between GraphQL and REST API, the migration to Kubernetes and the plan to replace the monolithic front end.

Transcript

1. 8 Years Product @ LeanIX: From Monolith to Microservices André Christ Co-Founder & CEO Oct 14th 2020
2. 2 2 Years later… Did we really survive?
3. 3 Caution: My view might be biased 1st SW Engineer 1st Product Manager 1st CTO CEO à à à 2012 2020
4. 4 3 x 5 Lessons Learned 5 Lessons Learned: How we survived a microservices migration 5 Lessons Learned: Our biggest architecture challenges today 5 Lessons Learned: How we grew our product organization
5. What problem does LeanIX solve?
6. 6 Missing transparency blocks IT Transformation Air Ocean + 100 more IT System App App App App App App App App App App App App App App App App App App App App App App App App App App App App App App App > 100 Global IT Systems > 2000 Local IT Systems Interface
7. 7 LeanIX is like the “Google Maps for the IT”
8. 8 LeanIX helps companies to map out their IT Your Product Custom Development Custom Development Custom Development Custom Development Marketing Sales Production Customer Success Finance & IT BI & Collaboration Your product
9. 9 Bonn Boston Hyderabad > 270 Employees 24 Nationalities > 100% Y-o-Y Growth > 330 Customers 120M$ Funding Munich Utrecht
10. Customers Employees 10 Product, Engineering & UX in context of growth 2 2 2 15 20 66 45 67 119 188 39 2020 2 275 79 6 2018 2017 2014 45 20 13 2016 166 2012 2015 2013 2019 India US Germany 1 1 1 7 10 2 23 37 64 12 2 4 13 80 3 28 14 2020 2019 2012 2016 2014 2017 2013 2018 1 5 44 2015 UX Product Mgmt Engineering 5 11 30 52 99 157 251 330 2012 2020 2019 2013 2016 2015 2014 2017 1 2018 Bootstrapping Startup Scaleup Employees in R&D Series A $ 2.5M Series B $ 7.5M Series C $ 30M Series D $ 80M Microservices Migration Funding Azure Migration
11. How did our journey to Microservices look like?
12. 12 We started in 2012 with a strong MVP focus
14. 14 First product version used by customer in 2012
15. 15 Customers demanded more flexibility from us New Fact Sheets New Relations Attributes* A B C Needed Levels of Configuration Authorization & Reporting B Hard-Coded Data Model A Product C
16. 16 Our Journey to Microservices from 2015 to 2018 2 Developers 15 Developers Fixed Data Model Flexible Data Model MVC Model View Controller API First + SPA 10 Customers 120 Customers MVP Minimal Viable Product Enterprise Ready 2015 2018 Scale! Functionality! Robustness! Innovation! Scale! CEO wants …
17. 17 Technical View: Loosely coupled services View Controller Model Monolith 2012 - 2014 Microservices 2015 - 2018 Single Page App REST API REST API REST API REST API REST API REST API Pathfinder MTM Webhooks Survey Metrics Synclog REST API Export REST API Images REST API Integra- tions
18. 18 How we survived a Microservices migration 1 Create self-service staging environment for every version Avoid behaving like Netflix: # Developers > # Microservices Build Green-Blue container deployment to allow frequent updates React quickly on issues in production discovered with good tools Switch from Scrum to Kanban during Migration to react on Customer feedback 2 3 4 5
19. 19 Self-service staging for every version /beta Version 3 Version 4 - “Pathfinder” 1
20. 20 You are not Netflix – Find your definition! # Developers > # Microservices API first Swagger API Docs One Concern Containerized No shared Database Configurable Adrian Cockcroft ex-CTO Netflix 2
21. 21 Green-Blue deployment for frequent updates Load-Balancer (default = BLUE) Svc A 1.0 Svc A 1.1 Svc B 1.0 Svc C 1.0 DB, Index, Queue DB, Elastic Test Svc B 1.1 Svc C 1.1 3 Own deployment of docker containers
22. 22 React quickly on issues in production Instance Instance Instances GET /healthcheck Servers CPU / Mem / Disk / Errors / JVM Logfiles Incidents Incidents DISABLED OCE1 on duty Incident auto-generated status.leanix.net Escalation to management Incident manual Engineering Customer Success Bug classifies as EC1 EC1 Bug support@leanix.net 4
23. 23 Hypercare during Migration: Now KANBAN! • ”War”-Room • Every morning: 60 min Session Dev-Leads & Migration-Team • For ~ 6 months: KANBAN 5
24. How does our architecture look like today?
25. 25 LeanIX is running on Azure in multiple regions us-2.leanix.net us.leanix.net ca.leanix.net eu.leanix.net app.leanix.net au.leanix.net de.leanix.net uk.leanix.net Existing Region Planned Region
26. 26 Multi-Tenancy on different levels Instance eu-1.leanix.net Customer 1 Customer 2 Customer 3 Instance eu-2.leanix.net Customer 4 Platform Services westeurope Docker / K8s Docker / K8s Customer 5
27. 27 Biggest architecture challenges today 1 Fast & frequent deployment with K8s – but, non-native K8s services require high rework Event-bus architecture good in theory – but, increasing event payloads kills performance Teams better aligned to concerns – but, single frontend app becomes bottleneck Re-use across services sounds good – but, “copy-paste” is better in Microservices world GraphQL is a great addition for frontend – but, GraphQL is not a silver bullet for data loading 2 3 4 5
28. 28 Non-native K8s services require high rework pfapi Postgres Kafka Elasticsearch ArangoDB pfweb admin auth mtm webhooks images export (v2) integrations poll metrics synclog Backend Frontend Resource Postgres apptio connector Postgres cloudockit connector k8s connector impacts torg (store) torg (store) Postgres workflow diagrams Postgres Redis Camunda Postgres Postgres Instance Original Platform- Services New native K8s Services portal integration api Postgres Postgres Postgres Kafka InfluxDB Postgres Postgres Files Postgres Postgres Postgres Elasticsearch ltls cicd connector jira cloud connector 3+ data science services watchdog Misc High Medium Low Migration Complexity 1 auth CDN CDN Biggest Hurdles • Latency of managed DBs • State in local caches • Long-running jobs reports CDN < 2018 > 2018
29. 29 Increasing event payloads kill performance pfapi opportunity postgres kafka elastic arango pfweb admin auth self- service mtm webhooks integrations poll postgres kafka postgres elastic postgres idp postfix rabbitmq elastic transaction factsheet postgres postgres workspace user instance … subscription delivery event event event … event factsheet … factsheet poll result poll … event workspace KPIs … Techno- pedia jobs configs Update FS 1 2 3 4 6 5 7 7 7 8 8 12 10 9 11 • Every change is propagated as a self- contained event • Services and Customers can subscribe to webhooks • Event payload contains the change and the new object • Customers add more data & dependencies in LeanIX • Event payloads get bigger and bigger à Back-wards compatible re-design necessary 2
30. 30 Single frontend app becomes bottleneck • Teams separated by different concerns, but Single-Page-App limits autonomy of teams • Increasing build & test time, each deploy 45 min • Currently: o Reports in IFrame, deployed to CDN o Angular Mono-Repo (subprojects in single Code-repository) • Next: o Further shift to IFrame where possible o Evaluation of Web Components Single Page App REST API REST API REST API REST API REST API REST API REST API REST API Dashboard Inventory Fact Sheet Survey Reports Diagrams … Team 1 Team 2 Team 3 Team 4 Team 5 3
31. 31 “Copy-paste” is better in Microservices world 4 A) Re-usable sync framework B) Centralize bootstrapping code C) Generated SDKs via Swagger Generic Syncer Signavio Syncer ServiceNow Syncer Dropwizard LeanIX “Dropkit” Svc 1 Svc 2 … Svc N Svc 1 API Svc 2 API … Svc N API Svc 1 SDK Svc 2 SDK … SDK Svc 3 SDK Svc 1 Svc 2 … • High abstraction level created big complexity • Every change risks issues à Extract into Integration API • Maintaining own library for simple code creates overhead • Version updates time consuming à Include Dropwizard directly • High overhead for re-generating SDKs for API changes • Long roundtrip for simple changes à Use light HTTP-Client (retrofit) Svc 3
32. 32 GraphQL is not a silver bullet for data loading 5 query { allBusinessCapabilities { factSheets { id displayName relToChild {factSheet{id}} } } allApplications { factSheets { id displayName relToChild {factSheet{id}} relApplToBusCapability {factSheet{id}} } } } • Strongly typed queries for our flexible data model • Works well in frontend / reports to reduce payload Connector LeanIX Data Interchange Format (LDIF) Integration API LeanIX GraphQL API Json- Format Create Update Delete GraphQL API Integration API • Abstracts CRUD-logic needed for syncs behind API • Powerful expression-based processors
33. How did the Product & Engineering organization evolve?
34. 34 Different organization in each phase Bootstrapping Startup Scaleup • 1 PM (Co-Founder) • 1 Team • < 15 Engineers ”One Team” • < 5 PM + CTO • < 3 UX • < 8 Teams • < 50 Engineers ”Dedicated PM” Product Management (PM) • 1-2 PM (Co-Founder) • < 5 Teams • < 25 Engineers ”First Teams” • 1 CTO + 1 Prod. Ops • > 10 PMs • > 5 UX • > 10 Teams • > 50 Engineers ”Product Tribes” Engineers User Experience Pivotal Tracker
35. 35 How we grew our product organization 1 Start with full-stack teams to train autonomy Create On Call Duty in team to establish ownership Have feature toggles mechanism to keep deploying Stay light in your tools until you can afford it Build frontend library to stay consistent 2 3 4 5
36. 36 Start with full-stack teams to train autonomy • P. (FS) • M. (FS) • U. (FS) + U. (Ops) • D. (BE) • A. (FE) • A. (FE) • B. (BE) • G. (FE) • R. (BE) • R. (FE) • F. (Prod) + R. (Ops) • R. Ops • U. Ops Platform Ops FS = Full Stack BE = Back End FE = Front End Ops = Operations 1 2 3 4 • During Microservice- Migration: 1 team in Kanban • After migration: First split into teams, return to Scrum • Every team cross- functional • Hard to persist with small team size 1
37. 37 Create On Call Duty to establish ownership Instance Instance Instances GET /healthcheck Servers CPU / Mem / Disk / Errors / JVM Logfiles Incidents Incidents DISABLED OCE1 on duty Incident auto-generated status.leanix.net Escalation to management Incident manual Engineering Customer Success Bug classifies as EC1 EC1 Bug support@leanix.net • Rotating OCE (On-Call Engineer) for a week • All experienced engineers (> 9-12 month with LeanIX) take part • At > 10 Services hard to sustain knowledge • New SRE approach in Development (higher standardization, monitoring) 2
38. 38 Feature toggles for continuous deployment • Own solution for product feature flags: o Beta Features o Paid Features • Can be configured per workspace (customer) • Can be grouped into an edition (e.g. Smart, Professional, Ultimate) • Good tools available now, e.g. LaunchDarkly 3
39. 39 Stay light in your tools until you can afford it • Pivotal Tracker: Single line stories, simple grouping to epics • Forced first engineers to think in product • Move to Jira creates more structure, workflows, … and overhead • Makes only sense if Product Management is established and Admin exists 4
40. 40 Build Frontend-Library to stay consistent • Storybook to create library for small atomic UI components • Document use and visually test without loading complex dev stack • Supports various UI frameworks (e.g. Angular, React, …) 5
41. 41 Summary: 3 x 5 Lessons Learned 5 Lessons Learned: How we survived a microservices migration 5 Lessons Learned: Our biggest architecture challenges today 5 Lessons Learned: How we grew our product organization
42. 42 Thank you!