How not to lose your mind with too many microservices - Architecture Gathering 2016

If you have too many microservices (which might be something > 30) you need a structured way to know what's going on and where to get more information about your system. We show two approaches how E-POST and Zalando going to solve this problem.

Transcript

1. HOW NOT TO LOSE YOUR MIND WITH TOO MANY MICROSERVICES OLIVER WEHRENS - E-POST - @OWEHRENS FELIX MÜLLER - ZALANDO SE - @FMUELLER_BLN
2. ONCE THERE WAS A MONOLITH. ▸ One (big) codebase ▸ Many teams working on it ▸ Lots of communication overhead T H E MONOLITH = Team
3. THEN YOU TRY TO SCALE YOUR ORGANIZATION… T H E MONOLITH
4. THEN YOU TRY TO SCALE YOUR ORGANIZATION… T H E MONOLITH UPS!
5. YOU DECIDE TO GO FOR MICROSERVICE ARCHITECTURE. ▸ Only one team works on service codebase ▸ Independent deployment of business functionality ▸ Less blocking communication = Microservice
6. EVERYTHING IS AWESOME !
7. EVERYTHING IS AWESOME ?
8. MICROSERVICES MEAN … ▸ More Services. ▸ More Teams. ▸ More Communication. ▸ More Documentation. ▸ More of everything.
9. PROBLEMS TO SOLVE ▸ What is available on the platform ? ▸ What does the whole platform look like ? ▸ Who is responsible for a service ? ▸ How to get more information about a service ? ▸ Which Software versions and licenses do we use ?
10. STANDARDS OR DIE.
11. … OR HAVE METADATA (IN ONE PLACE).
12. W I K I HERE NFORMATION ILLS TSELF
13. WIKI (RANT) ▸ Created once ▸ Rarely updated ▸ Nothing can be found ▸ If updated, nobody knows if this is up to date ▸ Developers just don’t like update Wikis ▸ Use the source Luke.
14. … OR COLLECT METADATA.
15. MANUAL AUTOMATED BUILD TIME (SHOULD) RUNTIME (ACTUAL)
16. MANUAL THINGS THAT DON'T CHANGE OFTEN
17. AUTOMATED EVERYTHING ELSE (AS MUCH AS YOU CAN)
18. BUILD TIME ▸ Everything available at Code Level & CI System ▸ VCS information ▸ License & Dependency information ▸ Build chain information ▸ Code Stats (Age, Committer, Language)
19. RUNTIME ▸ VM Level ▸ Network connections ▸ Setup Level ▸ Sizing
20. META DATA @ ZALANDO SE
22. RADICAL AGILITY ORGANISATIONAL CHANGE
23. RADICAL AGILITY ▸ Autonomous teams working towards purpose ▸ Individuals striving for mastery ▸ Technical: ▸ Microservices ▸ API First ▸ REST, Cloud, Software as a Service…
24. STUPS
25. OVERVIEW
26. STUPS COMES WITH AN APPLICATION REGISTRY ▸ Kio as application registry ▸ each deployed application has to be registered there ▸ every version also has to be registered there ▸ information like vcs root, owning team, ticket system…
27. ZALANDO’S FUTURE SYSTEM WILL CONSIST OF THOUSANDS OF MICROSERVICES.
28. API DISCOVERY
29. API DISCOVERY ▸ A system to crawl and curate all deployed APIs. TWINTIP CRAWLER TWINTIP STORAGE SWAGGER UI APIS
30. SWAGGER UI
31. THAT’S NICE BUT MICROSERVICES ARE MORE THAN APIS
32. LEANIX
33. FULL-TEXT SEARCH ON ALL TECH DATA
34. INFORMATION MANAGEMENT
35. ZALANDO’S TOOLCHAIN ▸ Aggregate metadata automatically ▸ Top-Down approach ▸ API Discovery via Twintip ▸ https://github.com/zalando-stups/twintip-crawler ▸ IT Portfolio Management via LeanIX
36. META DATA @ E-POST DEV
38. SITUATION ▸ ~ 250 Source code repositories ▸ 40 Services ▸ 12 Teams
39. STATUS QUO SERVICE REGISTRY BACK THEN … Klaus
40. OUR REQUIREMENTS ▸ Every VCS root needs documentation ▸ Description ▸ Type ▸ Team name ▸ Owner ▸ VCS & CI Information
41. IN THE BEGINNING (Q4/2014) - THE GOOD ▸ Started with Wiki ▸ Description in yaml file in the Source Code ▸ Executed during CI Run ▸ Automated Code Dependencies via Maven, Gradle, SBT ▸ Formatted to HTML and uploaded to Wiki
42. IN THE BEGINNING (Q4/2015) - THE BAD ▸ Search was limited ▸ Data could not be queried for additional benefit ▸ We had a couple of other places where we distributed information about services, what they do, how they get deployed etc. ▸ No immediate benefit, no problem when outdated
43. WE NEED SOMETHING BETTER.
44. OUR REQUIREMENTS ▸ General: Team name, Owner, a short name, description, type ▸ Runtime: memory needs, cpu, machine type ▸ Service: what do I provide, which port, protocol, private/ public ▸ Dependencies to other services ▸ Software Dependencies and Licenses ▸ Query DSL
45. WHAT’S OUT THERE? System-Z Services Directory NOT OPEN SOURCE
46. PIVIO HTTP://PIVIO.IO
47. PIVIO ▸ A (simple) system to describe service meta data CLIENT SERVER (WRAPPER) WEB DB (ELASTIC- SEARCH) JSON JSON YAML
48. WHAT DOES IT LOOK LIKE ?
49. PIVIO YAML ▸ One pivio.yaml file in root vcs directory ▸ Format at: http://pivio.io/docs/ #section-dataformat ▸ Extendable to your own needs
50. OVERVIEW
51. DETAIL
52. DETAIL (II)
53. FEED
54. ELASTIC SEARCH BASED QUERY
55. OUR REQUIREMENTS ▸ General: Team name, Owner, a short name, description, type ▸ Runtime: memory needs, cpu, machine type ▸ Service: what do I provide, which port, protocol, private/ public ▸ Dependencies to other services ▸ Software Dependencies and Licenses ▸ Query DSL ✅
56. DATA QUALITY ▸ … is key! ▸ Organisational changes not reflected sometimes ▸ Owners don’t benefit from quality ▸ Make use of the data that are relevant to the creator! ▸ You will have dirty data.
57. USE CASES FOR E-POST DEVELOPMENT ▸ Machine sizing for Open Nebula ▸ Service names for Consul ▸ TBD: Firewall rules generation ▸ General information about all services ▸ Visualize dependencies of teams and bounded contexts ▸ Impact analysis of changing APIs
58. SOFTWARE VERSION DEPENDENCY CHECK (60 LINES OF JS)
59. SERVICE DEPENDENCY CHECK (284 LINES OF JS)
61. CONCLUSION ▸ Big Picture with micro services is hard ▸ Metadata helps to understand the system ▸ Needs to be easily editable (e.g. in the IDE) ▸ Needs to be useful to the creator ▸ Metadata will be dirty ▸ Link build time and runtime information ▸ Build tools on top of it, have a Query Language!
62. THANKS. QUESTIONS? HTTP://PIVIO.IO @FMUELLER_BLN @OWEHRENS