Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Service Discovery For Machines And Humans - OOP conference 2017

Service Discovery For Machines And Humans - OOP conference 2017

Combining Continuous Deployment and a Microservice architecture brings new challenges to develop and operate your platform. A service discovery enables you to build a flexible system. Developers need to have an up to date view on the deployed services as well. A human readable registry with relevant information is needed.
I will outline what we solved with a Service Registry and what impact it had on our architecture. Furthermore I will show what we needed to give our developers to get an up to date view on the whole platform.

Oliver Wehrens

February 01, 2017
Tweet

More Decks by Oliver Wehrens

Other Decks in Technology

Transcript

  1. SERVICE DISCOVERY FOR
    MACHINES AND HUMANS
    OLIVER WEHRENS @OWEHRENS
    E-POST DEVELOPMENT GMBH

    View full-size slide

  2. SERVICE DISCOVERY FOR MACHINES AND HUMANS
    BACKGROUND
    ▸ Chief Architect E-POST Development GmbH
    ▸ Building E-POST System with > 80 Services
    ▸ Building Applications for DHL
    ▸ ~ 100 developers

    View full-size slide

  3. WHAT DID WE LEARN RUNNING
    A SYSTEM WITH > 30
    (SOMETIMES MICRO) SERVICES

    View full-size slide

  4. ONCE THERE WAS A MONOLITH.
    ▸ One (big) codebase
    ▸ Few teams working on it
    ▸ Some communication
    overhead
    T H E MONOLITH

    View full-size slide

  5. THEN YOU TRY TO SCALE YOUR ORGANIZATION…
    T H E MONOLITH

    View full-size slide

  6. THEN YOU TRY TO SCALE YOUR ORGANIZATION…
    T H E MONOLITH
    ?????

    View full-size slide

  7. YOU DECIDE TO GO FOR MICROSERVICE ARCHITECTURE.
    ▸ Only one team works on
    service codebase
    ▸ Independent deployment of
    business functionality
    ▸ Less blocking communication
    = Microservice

    View full-size slide

  8. EVERYTHING IS
    AWESOME !

    View full-size slide

  9. EVERYTHING IS
    AWESOME ?

    View full-size slide

  10. MICROSERVICE ARCHITECTURE

    View full-size slide

  11. MICROSERVICE ARCHITECTURE - WHERE ARE ALL MY SERVICES ?

    View full-size slide

  12. MICROSERVICE ARCHITECTURE - WHO WRITES ALL THIS ?

    View full-size slide

  13. SERVICE DISCOVERY
    FOR MACHINES
    AND
    HUMANS

    View full-size slide

  14. MANY SERVICES - MOVING PROBLEMS TO A DIFFERENT LEVEL
    ▸ How do services find each other?
    ▸ How do they find all instances?
    ▸ Fail over?
    ▸ Resilience ?

    View full-size slide

  15. TALKING TO OTHER SERVICES
    ▸ HA-Proxy with config in tools like puppet
    ▸ Configure your service to talk to static list of local HA-Proxy
    Services
    ▸ Done.

    View full-size slide

  16. PROBLEMS
    ▸ Static list of services in e.g. Hiera
    ▸ Changes in service configuration is painful
    ▸ Changes often need a puppet run & restart
    ▸ Always machines only updated, never create new and
    throw away old one (sometimes at sizing hosts)
    ▸ All hosts are health checking each other all the time

    View full-size slide

  17. GOALS
    ▸ We had use cases for client side load bancing (in code) not
    haproxy
    ▸ Immutability of service, updateing services would also
    make manual work obsolete
    ▸ Get rid of service restart at sizing

    View full-size slide

  18. CATALOG OF
    AVAILABLE SERVICES

    View full-size slide

  19. HOW WOULD IT WORK?
    ▸ Service announces it self to a directory (self registration or
    third party registration)
    ▸ Modify own vs. install just another piece of software
    ▸ Directory Interval checks if service is still there
    ▸ Either dead or alive
    ▸ Client asks directory for a specific service and gets answer
    ▸ Resilience is still important

    View full-size slide

  20. SERVICE DISCOVERY FOR MACHINES
    ▸ Several Solutions
    ▸ Roll your own - You
    ▸ etcd - CoreOS
    ▸ Zookeeper - Apache
    ▸ Heureka - Netflix
    ▸ Consul - Hashicorp

    View full-size slide

  21. CONSUL BY HASHICORP - SERVER
    ▸ Cluster of at least 3
    ▸ Needs to be up all the time - more complexity
    ▸ Connect data centers - for data centers or very strict
    network/firewall rules
    ▸ Can also act as DNS (.consul)
    ▸ Restricting access read/write with ACLs

    View full-size slide

  22. CONSUL BY HASHICORP - CLIENT
    ▸ Installed on each Client locally
    ▸ Checks service health on given url
    ▸ Propagates status with gossip protocol (https://
    en.wikipedia.org/wiki/Gossip_protocol)

    View full-size slide

  23. USAGE CONSUL AT E-POST
    ▸ Client roll out on all hosts automatically
    ▸ Registration
    ▸ HA-Proxy Template
    ▸ Lightweight Wrapper around Netflix Ribbon with Consul
    integration
    ▸ DNS implementation
    ▸ Keys under which services will be found defined by teams

    View full-size slide

  24. CONSUL AT E-POST (II)
    ▸ Took about 120 days to get infrastructure right
    ▸ Zones
    ▸ Templates
    ▸ Networking
    ▸ ACLs
    ▸ Takes very long when > 90 Developers are involved to
    eliminate legacy base

    View full-size slide

  25. SERVICE DISCOVERY FOR MACHINES - CONCLUSION
    ▸ If you have > 30 Services it is worthwhile investigating
    ▸ No static distribution of service addresses
    ▸ Easier to automate, less effort in the mid/long run
    ▸ Use existing software, don’t write your own

    View full-size slide

  26. MICROSERVICE ARCHITECTURE - WHERE ARE ALL MY SERVICES ?

    View full-size slide

  27. MICROSERVICE ARCHITECTURE - WHO WRITES ALL THIS ?

    View full-size slide

  28. WE HAVE A LOT OF SERVICES
    AND MACHINES CAN FIND IT
    … HOW ABOUT HUMANS?
    Architecture Board - 2015

    View full-size slide

  29. MICROSERVICES MEAN …
    ▸ More Services.
    ▸ More Teams.
    ▸ More Communication.
    ▸ More Documentation.
    ▸ More of everything.

    View full-size slide

  30. PROBLEMS TO SOLVE
    ▸ What is available on the platform ?
    ▸ What does the whole platform look like ?
    ▸ Who is responsible for a service ?
    ▸ How to get more information about a service ?
    ▸ Which Software versions and licenses do we use ?

    View full-size slide

  31. STANDARDS OR
    DIE.

    View full-size slide

  32. … OR HAVE METADATA
    (IN ONE PLACE).

    View full-size slide

  33. W
    I
    K
    I
    HERE
    NFORMATION
    ILLS
    TSELF

    View full-size slide

  34. WIKI (RANT)
    ▸ Created once
    ▸ Rarely updated
    ▸ Nothing can be found
    ▸ Developers just don’t like update Wikis
    ▸ If updated, nobody knows if this is up to date
    ▸ Use the source Luke.

    View full-size slide

  35. … OR COLLECT
    METADATA.

    View full-size slide

  36. MANUAL
    AUTOMATED
    BUILD TIME (SHOULD)
    RUNTIME (ACTUAL)

    View full-size slide

  37. MANUAL
    THINGS THAT DON'T
    CHANGE OFTEN

    View full-size slide

  38. AUTOMATED
    EVERYTHING ELSE (AS
    MUCH AS YOU CAN)

    View full-size slide

  39. BUILD TIME
    ▸ Everything available at Code Level & CI System
    ▸ VCS information
    ▸ License & Dependency information
    ▸ Build chain information
    ▸ Code Stats (Age, Committer, Language)

    View full-size slide

  40. RUNTIME
    ▸ VM Level
    ▸ Network connections
    ▸ Setup Level
    ▸ Sizing

    View full-size slide

  41. SITUATION
    ▸ ~ 250 Source code repositories
    ▸ 40 Services
    ▸ 12 Teams

    View full-size slide

  42. STATUS QUO
    SERVICE REGISTRY
    BACK THEN …
    Klaus

    View full-size slide

  43. OUR REQUIREMENTS
    ▸ Every VCS root needs documentation
    ▸ Description
    ▸ Type
    ▸ Team name
    ▸ Owner
    ▸ VCS & CI Information

    View full-size slide

  44. IN THE BEGINNING (Q4/2014) - THE GOOD
    ▸ Started with Wiki
    ▸ Description in yaml file in the Source Code
    ▸ Executed during CI Run
    ▸ Automated Code Dependencies via Maven, Gradle, SBT
    ▸ Formatted to HTML and uploaded to Wiki

    View full-size slide

  45. IN THE BEGINNING (Q4/2015) - THE BAD
    ▸ Search was limited
    ▸ Data could not be queried for additional benefit
    ▸ We had a couple of other places where we distributed
    information about services, what they do, how they get
    deployed etc.
    ▸ No immediate benefit, no problem when outdated

    View full-size slide

  46. WE NEED
    SOMETHING BETTER.

    View full-size slide

  47. OUR REQUIREMENTS
    ▸ General: Team name, Owner, a short name, description, type
    ▸ Runtime: memory needs, cpu, machine type
    ▸ Service: what do I provide, which port, protocol, private/
    public
    ▸ Dependencies to other services
    ▸ Software Dependencies and Licenses
    ▸ Query DSL

    View full-size slide

  48. WHAT’S OUT THERE?
    System-Z
    Services
    Directory
    NOT OPEN SOURCE

    View full-size slide

  49. PIVIO
    HTTP://PIVIO.IO

    View full-size slide

  50. PIVIO
    ▸ A (very simple) system to describe service meta data
    CLIENT SERVER
    (WRAPPER)
    WEB
    DB
    (ELASTIC-
    SEARCH)
    JSON JSON
    YAML

    View full-size slide

  51. WHAT DOES IT
    LOOK LIKE ?

    View full-size slide

  52. PIVIO YAML
    ▸ One pivio.yaml file in root vcs
    directory
    ▸ Format at: http://pivio.io/docs/
    #section-dataformat
    ▸ Extendable to your own needs

    View full-size slide

  53. ELASTIC SEARCH BASED QUERY

    View full-size slide

  54. OUR REQUIREMENTS
    ▸ General: Team name, Owner, a short name, description, type
    ▸ Runtime: memory needs, cpu, machine type
    ▸ Service: what do I provide, which port, protocol, private/
    public
    ▸ Dependencies to other services
    ▸ Software Dependencies and Licenses
    ▸ Query DSL

    View full-size slide

  55. DATA QUALITY
    ▸ … is key!
    ▸ Organisational changes not reflected sometimes
    ▸ Owners don’t benefit from quality
    ▸ Make use of the data that are relevant to the creator!
    ▸ You will have dirty data.

    View full-size slide

  56. USE CASES FOR E-POST DEVELOPMENT
    ▸ Machine sizing for Open Nebula
    ▸ Service names for Consul
    ▸ Generating Documentation
    ▸ General information about all services
    ▸ Visualize dependencies of teams and bounded contexts
    ▸ Impact analysis of changing APIs

    View full-size slide

  57. SOFTWARE VERSION DEPENDENCY CHECK (60 LINES OF JS)

    View full-size slide

  58. SERVICE DEPENDENCY CHECK (284 LINES OF JS)

    View full-size slide

  59. SERVICE DISCOVERY FOR HUMANS - CONCLUSION
    ▸ Big Picture with micro services is hard
    ▸ Metadata helps to understand the system
    ▸ Needs to be easily editable (e.g. in the IDE)
    ▸ Needs to be useful to the creator
    ▸ Metadata will be dirty
    ▸ Link build time and runtime information
    ▸ Build tools on top of it, have a Query Language!

    View full-size slide

  60. IN A SYSTEM WITH > 30 SERVICES
    INVEST IN SERVICE DISCOVERY
    FOR MACHINES AND HUMANS.

    View full-size slide

  61. THANKS.
    QUESTIONS?
    HTTP://PIVIO.IO
    @OWEHRENS
    HTTP://SPEAKERDECK.COM/OWEHRENS

    View full-size slide

  62. GRAPHICAL RECORDING OF
    THE TALK
    BONUS FEATURE

    View full-size slide