Service Discovery For Machines And Humans - OOP conference 2017

Combining Continuous Deployment and a Microservice architecture brings new challenges to develop and operate your platform. A service discovery enables you to build a flexible system. Developers need to have an up to date view on the deployed services as well. A human readable registry with relevant information is needed.
I will outline what we solved with a Service Registry and what impact it had on our architecture. Furthermore I will show what we needed to give our developers to get an up to date view on the whole platform.

Transcript

1. SERVICE DISCOVERY FOR MACHINES AND HUMANS OLIVER WEHRENS @OWEHRENS E-POST DEVELOPMENT GMBH
2. SERVICE DISCOVERY FOR MACHINES AND HUMANS BACKGROUND ▸ Chief Architect E-POST Development GmbH ▸ Building E-POST System with > 80 Services ▸ Building Applications for DHL ▸ ~ 100 developers
3. WHAT DID WE LEARN RUNNING A SYSTEM WITH > 30 (SOMETIMES MICRO) SERVICES
4. ONCE THERE WAS A MONOLITH. ▸ One (big) codebase ▸ Few teams working on it ▸ Some communication overhead T H E MONOLITH
5. THEN YOU TRY TO SCALE YOUR ORGANIZATION… T H E MONOLITH
6. THEN YOU TRY TO SCALE YOUR ORGANIZATION… T H E MONOLITH ?????
7. YOU DECIDE TO GO FOR MICROSERVICE ARCHITECTURE. ▸ Only one team works on service codebase ▸ Independent deployment of business functionality ▸ Less blocking communication = Microservice
8. EVERYTHING IS AWESOME !
9. EVERYTHING IS AWESOME ?
10. MICROSERVICE ARCHITECTURE
11. MICROSERVICE ARCHITECTURE - WHERE ARE ALL MY SERVICES ?
12. MICROSERVICE ARCHITECTURE - WHO WRITES ALL THIS ?
13. SERVICE DISCOVERY FOR MACHINES AND HUMANS
14. MACHINES
15. MANY SERVICES - MOVING PROBLEMS TO A DIFFERENT LEVEL ▸ How do services find each other? ▸ How do they find all instances? ▸ Fail over? ▸ Resilience ?
16. TALKING TO OTHER SERVICES ▸ HA-Proxy with config in tools like puppet ▸ Configure your service to talk to static list of local HA-Proxy Services ▸ Done.
17. PROBLEMS ▸ Static list of services in e.g. Hiera ▸ Changes in service configuration is painful ▸ Changes often need a puppet run & restart ▸ Always machines only updated, never create new and throw away old one (sometimes at sizing hosts) ▸ All hosts are health checking each other all the time
18. GOALS ▸ We had use cases for client side load bancing (in code) not haproxy ▸ Immutability of service, updateing services would also make manual work obsolete ▸ Get rid of service restart at sizing
19. CATALOG OF AVAILABLE SERVICES
20. HOW WOULD IT WORK? ▸ Service announces it self to a directory (self registration or third party registration) ▸ Modify own vs. install just another piece of software ▸ Directory Interval checks if service is still there ▸ Either dead or alive ▸ Client asks directory for a specific service and gets answer ▸ Resilience is still important
21. SERVICE DISCOVERY FOR MACHINES ▸ Several Solutions ▸ Roll your own - You ▸ etcd - CoreOS ▸ Zookeeper - Apache ▸ Heureka - Netflix ▸ Consul - Hashicorp
22. CONSUL BY HASHICORP - SERVER ▸ Cluster of at least 3 ▸ Needs to be up all the time - more complexity ▸ Connect data centers - for data centers or very strict network/firewall rules ▸ Can also act as DNS (.consul) ▸ Restricting access read/write with ACLs
23. CONSUL BY HASHICORP - CLIENT ▸ Installed on each Client locally ▸ Checks service health on given url ▸ Propagates status with gossip protocol (https:// en.wikipedia.org/wiki/Gossip_protocol)
24. USAGE CONSUL AT E-POST ▸ Client roll out on all hosts automatically ▸ Registration ▸ HA-Proxy Template ▸ Lightweight Wrapper around Netflix Ribbon with Consul integration ▸ DNS implementation ▸ Keys under which services will be found defined by teams
25. CONSUL AT E-POST (II) ▸ Took about 120 days to get infrastructure right ▸ Zones ▸ Templates ▸ Networking ▸ ACLs ▸ Takes very long when > 90 Developers are involved to eliminate legacy base
26. SERVICE DISCOVERY FOR MACHINES - CONCLUSION ▸ If you have > 30 Services it is worthwhile investigating ▸ No static distribution of service addresses ▸ Easier to automate, less effort in the mid/long run ▸ Use existing software, don’t write your own
27. MICROSERVICE ARCHITECTURE - WHERE ARE ALL MY SERVICES ?
28. MICROSERVICE ARCHITECTURE - WHO WRITES ALL THIS ?
29. HUMANS
30. WE HAVE A LOT OF SERVICES AND MACHINES CAN FIND IT … HOW ABOUT HUMANS? Architecture Board - 2015
31. MICROSERVICES MEAN … ▸ More Services. ▸ More Teams. ▸ More Communication. ▸ More Documentation. ▸ More of everything.
32. PROBLEMS TO SOLVE ▸ What is available on the platform ? ▸ What does the whole platform look like ? ▸ Who is responsible for a service ? ▸ How to get more information about a service ? ▸ Which Software versions and licenses do we use ?
33. STANDARDS OR DIE.
34. … OR HAVE METADATA (IN ONE PLACE).
35. W I K I HERE NFORMATION ILLS TSELF
36. WIKI (RANT) ▸ Created once ▸ Rarely updated ▸ Nothing can be found ▸ Developers just don’t like update Wikis ▸ If updated, nobody knows if this is up to date ▸ Use the source Luke.
37. … OR COLLECT METADATA.
38. MANUAL AUTOMATED BUILD TIME (SHOULD) RUNTIME (ACTUAL)
39. MANUAL THINGS THAT DON'T CHANGE OFTEN
40. AUTOMATED EVERYTHING ELSE (AS MUCH AS YOU CAN)
41. BUILD TIME ▸ Everything available at Code Level & CI System ▸ VCS information ▸ License & Dependency information ▸ Build chain information ▸ Code Stats (Age, Committer, Language)
42. RUNTIME ▸ VM Level ▸ Network connections ▸ Setup Level ▸ Sizing
44. SITUATION ▸ ~ 250 Source code repositories ▸ 40 Services ▸ 12 Teams
45. STATUS QUO SERVICE REGISTRY BACK THEN … Klaus
46. OUR REQUIREMENTS ▸ Every VCS root needs documentation ▸ Description ▸ Type ▸ Team name ▸ Owner ▸ VCS & CI Information
47. IN THE BEGINNING (Q4/2014) - THE GOOD ▸ Started with Wiki ▸ Description in yaml file in the Source Code ▸ Executed during CI Run ▸ Automated Code Dependencies via Maven, Gradle, SBT ▸ Formatted to HTML and uploaded to Wiki
48. IN THE BEGINNING (Q4/2015) - THE BAD ▸ Search was limited ▸ Data could not be queried for additional benefit ▸ We had a couple of other places where we distributed information about services, what they do, how they get deployed etc. ▸ No immediate benefit, no problem when outdated
49. WE NEED SOMETHING BETTER.
50. OUR REQUIREMENTS ▸ General: Team name, Owner, a short name, description, type ▸ Runtime: memory needs, cpu, machine type ▸ Service: what do I provide, which port, protocol, private/ public ▸ Dependencies to other services ▸ Software Dependencies and Licenses ▸ Query DSL
51. WHAT’S OUT THERE? System-Z Services Directory NOT OPEN SOURCE
52. PIVIO HTTP://PIVIO.IO
53. PIVIO ▸ A (very simple) system to describe service meta data CLIENT SERVER (WRAPPER) WEB DB (ELASTIC- SEARCH) JSON JSON YAML
54. WHAT DOES IT LOOK LIKE ?
55. PIVIO YAML ▸ One pivio.yaml file in root vcs directory ▸ Format at: http://pivio.io/docs/ #section-dataformat ▸ Extendable to your own needs
56. OVERVIEW
57. DETAIL
58. DETAIL (II)
59. FEED
60. ELASTIC SEARCH BASED QUERY
61. OUR REQUIREMENTS ▸ General: Team name, Owner, a short name, description, type ▸ Runtime: memory needs, cpu, machine type ▸ Service: what do I provide, which port, protocol, private/ public ▸ Dependencies to other services ▸ Software Dependencies and Licenses ▸ Query DSL ✅
62. DATA QUALITY ▸ … is key! ▸ Organisational changes not reflected sometimes ▸ Owners don’t benefit from quality ▸ Make use of the data that are relevant to the creator! ▸ You will have dirty data.
63. USE CASES FOR E-POST DEVELOPMENT ▸ Machine sizing for Open Nebula ▸ Service names for Consul ▸ Generating Documentation ▸ General information about all services ▸ Visualize dependencies of teams and bounded contexts ▸ Impact analysis of changing APIs
64. SOFTWARE VERSION DEPENDENCY CHECK (60 LINES OF JS)
65. SERVICE DEPENDENCY CHECK (284 LINES OF JS)
67. SERVICE DISCOVERY FOR HUMANS - CONCLUSION ▸ Big Picture with micro services is hard ▸ Metadata helps to understand the system ▸ Needs to be easily editable (e.g. in the IDE) ▸ Needs to be useful to the creator ▸ Metadata will be dirty ▸ Link build time and runtime information ▸ Build tools on top of it, have a Query Language!
68. IN A SYSTEM WITH > 30 SERVICES INVEST IN SERVICE DISCOVERY FOR MACHINES AND HUMANS.
69. THANKS. QUESTIONS? HTTP://PIVIO.IO @OWEHRENS HTTP://SPEAKERDECK.COM/OWEHRENS
70. GRAPHICAL RECORDING OF THE TALK BONUS FEATURE