Lock in $30 Savings on PRO—Offer Ends Soon! ⏳

Service Discovery For Machines And Humans - OOP...

Service Discovery For Machines And Humans - OOP conference 2017

Combining Continuous Deployment and a Microservice architecture brings new challenges to develop and operate your platform. A service discovery enables you to build a flexible system. Developers need to have an up to date view on the deployed services as well. A human readable registry with relevant information is needed.
I will outline what we solved with a Service Registry and what impact it had on our architecture. Furthermore I will show what we needed to give our developers to get an up to date view on the whole platform.

Oliver Wehrens

February 01, 2017
Tweet

More Decks by Oliver Wehrens

Other Decks in Technology

Transcript

  1. SERVICE DISCOVERY FOR MACHINES AND HUMANS BACKGROUND ▸ Chief Architect

    E-POST Development GmbH ▸ Building E-POST System with > 80 Services ▸ Building Applications for DHL ▸ ~ 100 developers
  2. ONCE THERE WAS A MONOLITH. ▸ One (big) codebase ▸

    Few teams working on it ▸ Some communication overhead T H E MONOLITH
  3. YOU DECIDE TO GO FOR MICROSERVICE ARCHITECTURE. ▸ Only one

    team works on service codebase ▸ Independent deployment of business functionality ▸ Less blocking communication = Microservice
  4. MANY SERVICES - MOVING PROBLEMS TO A DIFFERENT LEVEL ▸

    How do services find each other? ▸ How do they find all instances? ▸ Fail over? ▸ Resilience ?
  5. TALKING TO OTHER SERVICES ▸ HA-Proxy with config in tools

    like puppet ▸ Configure your service to talk to static list of local HA-Proxy Services ▸ Done.
  6. PROBLEMS ▸ Static list of services in e.g. Hiera ▸

    Changes in service configuration is painful ▸ Changes often need a puppet run & restart ▸ Always machines only updated, never create new and throw away old one (sometimes at sizing hosts) ▸ All hosts are health checking each other all the time
  7. GOALS ▸ We had use cases for client side load

    bancing (in code) not haproxy ▸ Immutability of service, updateing services would also make manual work obsolete ▸ Get rid of service restart at sizing
  8. HOW WOULD IT WORK? ▸ Service announces it self to

    a directory (self registration or third party registration) ▸ Modify own vs. install just another piece of software ▸ Directory Interval checks if service is still there ▸ Either dead or alive ▸ Client asks directory for a specific service and gets answer ▸ Resilience is still important
  9. SERVICE DISCOVERY FOR MACHINES ▸ Several Solutions ▸ Roll your

    own - You ▸ etcd - CoreOS ▸ Zookeeper - Apache ▸ Heureka - Netflix ▸ Consul - Hashicorp
  10. CONSUL BY HASHICORP - SERVER ▸ Cluster of at least

    3 ▸ Needs to be up all the time - more complexity ▸ Connect data centers - for data centers or very strict network/firewall rules ▸ Can also act as DNS (.consul) ▸ Restricting access read/write with ACLs
  11. CONSUL BY HASHICORP - CLIENT ▸ Installed on each Client

    locally ▸ Checks service health on given url ▸ Propagates status with gossip protocol (https:// en.wikipedia.org/wiki/Gossip_protocol)
  12. USAGE CONSUL AT E-POST ▸ Client roll out on all

    hosts automatically ▸ Registration ▸ HA-Proxy Template ▸ Lightweight Wrapper around Netflix Ribbon with Consul integration ▸ DNS implementation ▸ Keys under which services will be found defined by teams
  13. CONSUL AT E-POST (II) ▸ Took about 120 days to

    get infrastructure right ▸ Zones ▸ Templates ▸ Networking ▸ ACLs ▸ Takes very long when > 90 Developers are involved to eliminate legacy base
  14. SERVICE DISCOVERY FOR MACHINES - CONCLUSION ▸ If you have

    > 30 Services it is worthwhile investigating ▸ No static distribution of service addresses ▸ Easier to automate, less effort in the mid/long run ▸ Use existing software, don’t write your own
  15. WE HAVE A LOT OF SERVICES AND MACHINES CAN FIND

    IT … HOW ABOUT HUMANS? Architecture Board - 2015
  16. MICROSERVICES MEAN … ▸ More Services. ▸ More Teams. ▸

    More Communication. ▸ More Documentation. ▸ More of everything.
  17. PROBLEMS TO SOLVE ▸ What is available on the platform

    ? ▸ What does the whole platform look like ? ▸ Who is responsible for a service ? ▸ How to get more information about a service ? ▸ Which Software versions and licenses do we use ?
  18. WIKI (RANT) ▸ Created once ▸ Rarely updated ▸ Nothing

    can be found ▸ Developers just don’t like update Wikis ▸ If updated, nobody knows if this is up to date ▸ Use the source Luke.
  19. BUILD TIME ▸ Everything available at Code Level & CI

    System ▸ VCS information ▸ License & Dependency information ▸ Build chain information ▸ Code Stats (Age, Committer, Language)
  20. OUR REQUIREMENTS ▸ Every VCS root needs documentation ▸ Description

    ▸ Type ▸ Team name ▸ Owner ▸ VCS & CI Information
  21. IN THE BEGINNING (Q4/2014) - THE GOOD ▸ Started with

    Wiki ▸ Description in yaml file in the Source Code ▸ Executed during CI Run ▸ Automated Code Dependencies via Maven, Gradle, SBT ▸ Formatted to HTML and uploaded to Wiki
  22. IN THE BEGINNING (Q4/2015) - THE BAD ▸ Search was

    limited ▸ Data could not be queried for additional benefit ▸ We had a couple of other places where we distributed information about services, what they do, how they get deployed etc. ▸ No immediate benefit, no problem when outdated
  23. OUR REQUIREMENTS ▸ General: Team name, Owner, a short name,

    description, type ▸ Runtime: memory needs, cpu, machine type ▸ Service: what do I provide, which port, protocol, private/ public ▸ Dependencies to other services ▸ Software Dependencies and Licenses ▸ Query DSL
  24. PIVIO ▸ A (very simple) system to describe service meta

    data CLIENT SERVER (WRAPPER) WEB DB (ELASTIC- SEARCH) JSON JSON YAML
  25. PIVIO YAML ▸ One pivio.yaml file in root vcs directory

    ▸ Format at: http://pivio.io/docs/ #section-dataformat ▸ Extendable to your own needs
  26. OUR REQUIREMENTS ▸ General: Team name, Owner, a short name,

    description, type ▸ Runtime: memory needs, cpu, machine type ▸ Service: what do I provide, which port, protocol, private/ public ▸ Dependencies to other services ▸ Software Dependencies and Licenses ▸ Query DSL ✅
  27. DATA QUALITY ▸ … is key! ▸ Organisational changes not

    reflected sometimes ▸ Owners don’t benefit from quality ▸ Make use of the data that are relevant to the creator! ▸ You will have dirty data.
  28. USE CASES FOR E-POST DEVELOPMENT ▸ Machine sizing for Open

    Nebula ▸ Service names for Consul ▸ Generating Documentation ▸ General information about all services ▸ Visualize dependencies of teams and bounded contexts ▸ Impact analysis of changing APIs
  29. SERVICE DISCOVERY FOR HUMANS - CONCLUSION ▸ Big Picture with

    micro services is hard ▸ Metadata helps to understand the system ▸ Needs to be easily editable (e.g. in the IDE) ▸ Needs to be useful to the creator ▸ Metadata will be dirty ▸ Link build time and runtime information ▸ Build tools on top of it, have a Query Language!
  30. IN A SYSTEM WITH > 30 SERVICES INVEST IN SERVICE

    DISCOVERY FOR MACHINES AND HUMANS.