Upgrade to Pro — share decks privately, control downloads, hide ads and more …

A Journey into a Microservice World

mattheath
January 29, 2015

A Journey into a Microservice World

Presented at Vilnius JUG #37

As Hailo expanded to a global presence, we needed to re-evaluate our approach to technology. This talk follows Hailo's journey from a monolithic application to a new microservice platform, running on three continents, built in both Java and Go.

mattheath

January 29, 2015
Tweet

More Decks by mattheath

Other Decks in Programming

Transcript

  1. A JOURNEY
    INTO A
    MICROSERVICE
    WORLD
    @mattheath

    View full-size slide

  2. Redis
    Load Balancer
    Java

    App
    Load Balancer
    MySQL
    Slave
    MySQL
    aws
    PHP

    API
    PHP

    API

    View full-size slide

  3. Redis
    Load Balancer
    Java

    App
    Load Balancer
    MySQL
    Slave
    MySQL
    aws
    PHP

    API
    PHP

    API

    View full-size slide

  4. Redis
    Load Balancer
    Java

    App
    Load Balancer
    MySQL
    Slave
    MySQL
    aws
    PHP

    API
    Services
    PHP

    API

    View full-size slide

  5. PHP

    API
    MySQL
    PHP
    Service
    Java
    Service
    LB
    per city per-region
    Load Balancer
    Cassandra
    PHP

    API
    PHP
    Service
    MySQL
    MySQL
    LB
    LB
    PHP

    API
    PHP

    API
    Java
    App
    Java
    App
    Java
    App

    View full-size slide

  6. PHP

    API
    MySQL
    PHP
    Service
    Java
    Service
    LB
    per city per-region
    Load Balancer
    Cassandra
    PHP
    Service
    MySQL
    MySQL
    LB
    LB
    PHP

    API
    PHP

    API
    Java
    App
    Java
    App
    Java
    App
    PHP

    API

    View full-size slide

  7. SLOW FAILOVER

    View full-size slide

  8. LACK OF

    AUTOMATION

    View full-size slide

  9. PHP

    API
    MySQL
    PHP
    Service
    Java
    Service
    LB
    per city per-region
    Load Balancer
    Cassandra
    PHP
    Service
    MySQL
    MySQL
    LB
    LB
    PHP

    API
    PHP

    API
    Java
    App
    Java
    App
    Java
    App
    PHP

    API

    View full-size slide

  10. UNCLEAR
    RESPONSIBILITIES

    View full-size slide

  11. SLOW

    DEVELOPMENT

    View full-size slide

  12. Communication paths
    with two programmers
    Communication paths
    with three programmers
    Communication paths
    with four programmers
    Communication paths
    with five programmers
    Communication paths
    with ten programmers

    View full-size slide

  13. A CLOUDY

    BEGINNING

    View full-size slide

  14. SCALE

    BREAKS
    HARDWARE

    View full-size slide

  15. SPEED

    BREAKS
    SOFTWARE

    View full-size slide

  16. “construct a highly agile and
    highly available service from
    ephemeral and assumed broken
    components”
    - Adrian Cockcroſt

    View full-size slide

  17. region 1 region n

    View full-size slide

  18. Cassandra
    region 1 region n
    Cassandra

    View full-size slide

  19. Cassandra
    region 1
    Cassandra
    Go

    Service
    Java
    Service
    Java

    Service
    Java
    Service
    Go

    Service
    Go

    Service
    region n

    View full-size slide

  20. Cassandra
    region 1
    Cassandra
    Go

    Service
    Java
    Service
    Java

    Service
    Java
    Service
    RabbitMQ Message Bus

    (federated clusters per AZ)
    RabbitMQ Message Bus

    (federated clusters per AZ)
    Go

    Service
    Go

    Service
    region n

    View full-size slide

  21. Cassandra
    Go

    Service
    Go

    Service
    Java
    Service
    Load Balancer
    API / Routing Layer
    region 1
    Go

    Service
    Java

    Service
    Java
    Service
    Load Balancer
    API / Routing Layer
    RabbitMQ Message Bus

    (federated clusters per AZ)
    RabbitMQ Message Bus

    (federated clusters per AZ)
    Cassandra
    region n

    View full-size slide

  22. Logic
    Service
    Handler
    Storage

    View full-size slide

  23. Logic
    Storage
    Handler
    Library for building services that talk Protobuf via RMQ
    Service
    platform-layer

    View full-size slide

  24. Logic
    service-layer
    Storage
    Handler
    Library for building services that talk Protobuf via RMQ
    Self-configuring external service adapters
    Service
    platform-layer

    View full-size slide

  25. Logic
    service-layer
    Storage
    platform-layer
    Handler
    Library for building services that talk Protobuf via RMQ
    Self-configuring external service adapters
    Services get for free:
    • Provisioning
    • Service discovery
    • Configuration
    • Monitoring
    • Authentication/authorisation
    • AB testing
    • Self configuring connectivity 

    to third-party services
    Service

    View full-size slide

  26. Provisioning Service Provisioning Service Provisioning Service
    CI Pipeline (Janky/Jenkins)
    Amazon S3
    Provisioning Manager / Scheduler

    View full-size slide

  27. Provisioning Service Provisioning Service Provisioning Service
    CI Pipeline (Janky/Jenkins)
    Amazon S3
    Provisioning Manager / Scheduler

    View full-size slide

  28. Provisioning Service
    Rabbit MQ
    Discovery
    Service
    Provisioning Service
    New

    Service

    View full-size slide

  29. Provisioning Service
    Rabbit MQ
    Discovery
    Service
    Binding

    Service
    Provisioning Service
    New

    Service

    View full-size slide

  30. DEALING WITH
    COMPLEXITY

    View full-size slide

  31. LOAD
    FAILURE
    DEGRADATION

    View full-size slide

  32. 15,000 JOBS/HOUR
    50,000 DRIVERS
    30,000+ REQ/S

    View full-size slide

  33. CONTINUOUS
    PRODUCTION
    TESTING

    View full-size slide

  34. Provisioning Service
    Rabbit MQ
    Monitoring

    Service
    Provisioning Service

    Service
    Publish
    Healthchecks

    View full-size slide

  35. DISTRIBUTED

    TRACING

    View full-size slide

  36. hailo~2~api api.v1.customer service.customer
    hailo~2~api api.v1.customer service.customer

    View full-size slide

  37. hailo~2~api api.v1.customer service.customer
    hailo~2~api api.v1.customer service.customer
    REQ
    REP
    REQ
    REP
    IN
    OUT
    IN
    OUT

    View full-size slide

  38. {
    "timestamp": 1410262798427145176,
    "traceId": "d30479b8-1491-4390-7cf5-4cd14bc4b765",
    "type": "OUT",
    "messageId": "a661f9ef-774c-49b2-6e74-cfed65f7d120",
    "parentMessageId": "",
    "from": "com.hailocab.hshell",
    "to": "com.hailocab.service.nearest-driver.search",
    "hostname": "ip-10-13-2-251",
    "az": "eu-west-1a",
    "handlerInstanceId": “server-com.hailocab.service.nearest-driver-18bd089e-8ef1-4ca1-75cb-8...c”,
    "duration": 11222094
    }
    {
    "timestamp": 1410262798416053450,
    "traceId": "d30479b8-1491-4390-7cf5-4cd14bc4b765",
    "type": "REQ",
    "messageId": "6404dd1e-c995-48a9-73dc-9edb1380f0bf",
    "parentMessageId": "a661f9ef-774c-49b2-6e74-cfed65f7d120",
    "from": "com.hailocab.service.nearest-driver",
    "to": "com.hailocab.service.zoning.search",
    "hostname": "ip-10-13-2-251",
    "az": "eu-west-1a"
    }

    View full-size slide

  39. Phosphor
    Host Instances
    Publish
    Service A
    Trace Library
    worker
    chan
    UDP
    Service B
    Trace Library
    worker
    chan
    UDP
    Trace
    Service
    In-memory
    Aggregates
    Optional

    persistant
    storage
    Dashboards
    Monitoring

    View full-size slide

  40. Tracing: 33eda743-f124-435c-71fc-3c872bbc98e6
    2014-09-07 02:20:19.867 [/] [START] → -
    2014-09-07 02:20:19.867 [eu-west-1a/ip-10-11-3-51] [REQ] com.hailocab.hailo-2-api → com.hailocab.api.v1.customer.neardrivers -
    2014-09-07 02:20:19.867 [eu-west-1a/ip-10-11-2-203] [IN] com.hailocab.hailo-2-api → com.hailocab.api.v1.customer.neardrivers -
    2014-09-07 02:20:19.868 [eu-west-1a/ip-10-11-2-203] [REQ] com.hailocab.api.v1.customer → com.hailocab.service.feature-flags.features -
    2014-09-07 02:20:19.869 [eu-west-1a/ip-10-11-3-111] [IN] com.hailocab.api.v1.customer → com.hailocab.service.feature-flags.features -
    2014-09-07 02:20:19.876 [eu-west-1a/ip-10-11-3-111] [REQ] com.hailocab.service.feature-flags → com.hailocab.service.hob.list -
    2014-09-07 02:20:19.877 [eu-west-1a/ip-10-11-3-168] [IN] com.hailocab.service.hob → com.hailocab.service.config.compile -
    2014-09-07 02:20:19.877 [eu-west-1a/ip-10-11-3-111] [IN] com.hailocab.service.feature-flags → com.hailocab.service.hob.list -
    2014-09-07 02:20:19.877 [eu-west-1a/ip-10-11-3-111] [REQ] com.hailocab.service.hob → com.hailocab.service.config.compile -
    2014-09-07 02:20:19.883 [eu-west-1a/ip-10-11-3-168] [OUT] com.hailocab.service.hob → com.hailocab.service.config.compile - 5.59 ms
    2014-09-07 02:20:19.886 [eu-west-1a/ip-10-11-3-111] [REP] com.hailocab.service.hob → com.hailocab.service.config.compile - 8.40 ms
    2014-09-07 02:20:19.887 [eu-west-1a/ip-10-11-3-111] [OUT] com.hailocab.service.feature-flags → com.hailocab.service.hob.list - 9.72 ms
    2014-09-07 02:20:19.889 [eu-west-1a/ip-10-11-3-111] [REP] com.hailocab.service.feature-flags → com.hailocab.service.hob.list - 13.23 ms
    2014-09-07 02:20:19.889 [eu-west-1a/ip-10-11-3-111] [OUT] com.hailocab.api.v1.customer → com.hailocab.service.feature-flags.features - 20.58 ms
    2014-09-07 02:20:19.890 [eu-west-1a/ip-10-11-2-203] [REP] com.hailocab.api.v1.customer → com.hailocab.service.feature-flags.features - 22.59 ms
    2014-09-07 02:20:19.902 [eu-west-1a/ip-10-11-2-203] [REQ] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare -
    2014-09-07 02:20:19.903 [eu-west-1a/ip-10-11-2-203] [REQ] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare -
    2014-09-07 02:20:19.903 [eu-west-1a/ip-10-11-2-203] [REQ] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare -
    2014-09-07 02:20:19.904 [eu-west-1a/ip-10-11-3-111] [IN] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare -
    2014-09-07 02:20:19.904 [eu-west-1a/ip-10-11-3-111] [OUT] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare - 0.36 ms
    2014-09-07 02:20:19.905 [eu-west-1a/ip-10-11-2-203] [REP] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare - 1.97 ms
    2014-09-07 02:20:19.905 [eu-west-1a/ip-10-11-2-214] [IN] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare -
    2014-09-07 02:20:19.905 [eu-west-1a/ip-10-11-2-203] [REQ] com.hailocab.api.v1.customer → com.hailocab.service.nearest-driver.search -
    2014-09-07 02:20:19.905 [eu-west-1a/ip-10-11-2-214] [OUT] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare - 0.10 ms

    ERR - com.hailocab.service.fare.basefare: Missing config at xxx
    2014-09-07 02:20:19.906 [eu-west-1a/ip-10-11-2-214] [IN] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare -
    2014-09-07 02:20:19.906 [eu-west-1a/ip-10-11-2-214] [OUT] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare - 0.06 ms 

    ERR - com.hailocab.service.fare.basefare: Missing config at xxx
    2014-09-07 02:20:19.907 [eu-west-1a/ip-10-11-3-58] [IN] com.hailocab.api.v1.customer → com.hailocab.service.nearest-driver.search -
    2014-09-07 02:20:19.907 [eu-west-1a/ip-10-11-3-58] [REQ] com.hailocab.service.nearest-driver → com.hailocab.service.zoning.search -
    2014-09-07 02:20:19.908 [eu-west-1a/ip-10-11-3-58] [IN] com.hailocab.service.nearest-driver → com.hailocab.service.zoning.search -
    2014-09-07 02:20:19.908 [eu-west-1a/ip-10-11-3-58] [OUT] com.hailocab.service.nearest-driver → com.hailocab.service.zoning.search - 0.20 ms
    2014-09-07 02:20:19.909 [eu-west-1a/ip-10-11-3-58] [REP] com.hailocab.service.nearest-driver → com.hailocab.service.zoning.search - 2.25 ms
    2014-09-07 02:20:19.909 [eu-west-1a/ip-10-11-3-58] [REQ] com.hailocab.service.nearest-driver → com.hailocab.service.raziel.multisearch -
    2014-09-07 02:20:19.912 [eu-west-1a/ip-10-11-3-227] [IN] com.hailocab.service.nearest-driver → com.hailocab.service.raziel.multisearch -
    2014-09-07 02:20:19.919 [eu-west-1a/ip-10-11-3-58] [REP] com.hailocab.service.nearest-driver → com.hailocab.service.raziel.multisearch - 9.46 ms
    2014-09-07 02:20:19.919 [eu-west-1a/ip-10-11-3-58] [REQ] com.hailocab.service.nearest-driver → com.hailocab.service.eta.multitraveltime -
    2014-09-07 02:20:19.919 [eu-west-1a/ip-10-11-3-227] [OUT] com.hailocab.service.nearest-driver → com.hailocab.service.raziel.multisearch - 7.58 ms
    2014-09-07 02:20:19.920 [eu-west-1a/ip-10-11-3-58] [IN] com.hailocab.service.nearest-driver → com.hailocab.service.eta.multitraveltime -
    2014-09-07 02:20:19.920 [eu-west-1a/ip-10-11-3-58] [OUT] com.hailocab.service.nearest-driver → com.hailocab.service.eta.multitraveltime - 0.06 ms
    2014-09-07 02:20:19.921 [eu-west-1a/ip-10-11-3-58] [REP] com.hailocab.service.nearest-driver → com.hailocab.service.eta.multitraveltime - 1.77 ms
    2014-09-07 02:20:19.921 [eu-west-1a/ip-10-11-3-58] [OUT] com.hailocab.api.v1.customer → com.hailocab.service.nearest-driver.search - 14.02 ms
    2014-09-07 02:20:19.921 [eu-west-1a/ip-10-11-2-203] [REP] com.hailocab.api.v1.customer → com.hailocab.service.nearest-driver.search - 15.48 ms
    2014-09-07 02:20:19.941 [eu-west-1a/ip-10-11-2-203] [REQ] com.hailocab.api.v1.customer → com.hailocab.service.experiment.readlastupdated -
    2014-09-07 02:20:19.945 [eu-west-1a/ip-10-11-2-214] [IN] com.hailocab.api.v1.customer → com.hailocab.service.experiment.readlastupdated -
    2014-09-07 02:20:19.947 [eu-west-1a/ip-10-11-2-214] [OUT] com.hailocab.api.v1.customer → com.hailocab.service.experiment.readlastupdated - 1.82 ms
    2014-09-07 02:20:19.947 [eu-west-1a/ip-10-11-2-203] [REP] com.hailocab.api.v1.customer → com.hailocab.service.experiment.readlastupdated - 6.01 ms
    2014-09-07 02:20:19.948 [eu-west-1a/ip-10-11-2-203] [OUT] com.hailocab.hailo-2-api → com.hailocab.api.v1.customer.neardrivers - 80.46 ms
    2014-09-07 02:20:19.950 [eu-west-1a/ip-10-11-3-51] [REP] com.hailocab.hailo-2-api → com.hailocab.api.v1.customer.neardrivers - 82.71 ms

    View full-size slide

  41. WHY BUILD
    MICROSERVICES?

    View full-size slide

  42. SIMPLE

    INDEPENDENT
    UNITS

    View full-size slide

  43. SIMPLE

    SCALABILITY

    View full-size slide

  44. EXPECT

    FAILURE

    View full-size slide

  45. AUTOMATE

    EVERYTHING

    View full-size slide

  46. CLOUD NATIVE

    ANTIFRAGILITY

    View full-size slide

  47. AČIŪ!
    Image Credits

    HMS President: Roger Marks
    Orbital Ion Cannon: www.rom.ac
    Go Gophers: Renee French
    Duopia: Jonny Hughes
    Microchips: S4nt1
    Control Room: NASA

    View full-size slide