$30 off During Our Annual Pro Sale. View Details »

A Journey into a Microservice World

mattheath
January 29, 2015

A Journey into a Microservice World

Presented at Vilnius JUG #37

As Hailo expanded to a global presence, we needed to re-evaluate our approach to technology. This talk follows Hailo's journey from a monolithic application to a new microservice platform, running on three continents, built in both Java and Go.

mattheath

January 29, 2015
Tweet

More Decks by mattheath

Other Decks in Programming

Transcript

  1. A JOURNEY
    INTO A
    MICROSERVICE
    WORLD
    @mattheath

    View Slide

  2. View Slide

  3. View Slide

  4. View Slide

  5. Redis
    Load Balancer
    Java

    App
    Load Balancer
    MySQL
    Slave
    MySQL
    aws
    PHP

    API
    PHP

    API

    View Slide

  6. View Slide

  7. View Slide

  8. View Slide

  9. View Slide

  10. Redis
    Load Balancer
    Java

    App
    Load Balancer
    MySQL
    Slave
    MySQL
    aws
    PHP

    API
    PHP

    API

    View Slide

  11. Redis
    Load Balancer
    Java

    App
    Load Balancer
    MySQL
    Slave
    MySQL
    aws
    PHP

    API
    Services
    PHP

    API

    View Slide

  12. PHP

    API
    MySQL
    PHP
    Service
    Java
    Service
    LB
    per city per-region
    Load Balancer
    Cassandra
    PHP

    API
    PHP
    Service
    MySQL
    MySQL
    LB
    LB
    PHP

    API
    PHP

    API
    Java
    App
    Java
    App
    Java
    App

    View Slide

  13. PHP

    API
    MySQL
    PHP
    Service
    Java
    Service
    LB
    per city per-region
    Load Balancer
    Cassandra
    PHP
    Service
    MySQL
    MySQL
    LB
    LB
    PHP

    API
    PHP

    API
    Java
    App
    Java
    App
    Java
    App
    PHP

    API

    View Slide

  14. SLOW FAILOVER

    View Slide

  15. LACK OF

    AUTOMATION

    View Slide

  16. PHP

    API
    MySQL
    PHP
    Service
    Java
    Service
    LB
    per city per-region
    Load Balancer
    Cassandra
    PHP
    Service
    MySQL
    MySQL
    LB
    LB
    PHP

    API
    PHP

    API
    Java
    App
    Java
    App
    Java
    App
    PHP

    API

    View Slide

  17. UNCLEAR
    RESPONSIBILITIES

    View Slide

  18. SLOW

    DEVELOPMENT

    View Slide

  19. Communication paths
    with two programmers
    Communication paths
    with three programmers
    Communication paths
    with four programmers
    Communication paths
    with five programmers
    Communication paths
    with ten programmers

    View Slide

  20. A CLOUDY

    BEGINNING

    View Slide

  21. SCALE

    BREAKS
    HARDWARE

    View Slide

  22. SPEED

    BREAKS
    SOFTWARE

    View Slide

  23. View Slide

  24. “construct a highly agile and
    highly available service from
    ephemeral and assumed broken
    components”
    - Adrian Cockcroſt

    View Slide

  25. region 1 region n

    View Slide

  26. Cassandra
    region 1 region n
    Cassandra

    View Slide

  27. Cassandra
    region 1
    Cassandra
    Go

    Service
    Java
    Service
    Java

    Service
    Java
    Service
    Go

    Service
    Go

    Service
    region n

    View Slide

  28. Cassandra
    region 1
    Cassandra
    Go

    Service
    Java
    Service
    Java

    Service
    Java
    Service
    RabbitMQ Message Bus

    (federated clusters per AZ)
    RabbitMQ Message Bus

    (federated clusters per AZ)
    Go

    Service
    Go

    Service
    region n

    View Slide

  29. Cassandra
    Go

    Service
    Go

    Service
    Java
    Service
    Load Balancer
    API / Routing Layer
    region 1
    Go

    Service
    Java

    Service
    Java
    Service
    Load Balancer
    API / Routing Layer
    RabbitMQ Message Bus

    (federated clusters per AZ)
    RabbitMQ Message Bus

    (federated clusters per AZ)
    Cassandra
    region n

    View Slide

  30. Logic
    Service
    Handler
    Storage

    View Slide

  31. Logic
    Storage
    Handler
    Library for building services that talk Protobuf via RMQ
    Service
    platform-layer

    View Slide

  32. Logic
    service-layer
    Storage
    Handler
    Library for building services that talk Protobuf via RMQ
    Self-configuring external service adapters
    Service
    platform-layer

    View Slide

  33. Logic
    service-layer
    Storage
    platform-layer
    Handler
    Library for building services that talk Protobuf via RMQ
    Self-configuring external service adapters
    Services get for free:
    • Provisioning
    • Service discovery
    • Configuration
    • Monitoring
    • Authentication/authorisation
    • AB testing
    • Self configuring connectivity 

    to third-party services
    Service

    View Slide

  34. View Slide

  35. View Slide

  36. View Slide

  37. Provisioning Service Provisioning Service Provisioning Service
    CI Pipeline (Janky/Jenkins)
    Amazon S3
    Provisioning Manager / Scheduler

    View Slide

  38. Provisioning Service Provisioning Service Provisioning Service
    CI Pipeline (Janky/Jenkins)
    Amazon S3
    Provisioning Manager / Scheduler

    View Slide

  39. Provisioning Service
    Rabbit MQ
    Discovery
    Service
    Provisioning Service
    New

    Service

    View Slide

  40. Provisioning Service
    Rabbit MQ
    Discovery
    Service
    Binding

    Service
    Provisioning Service
    New

    Service

    View Slide

  41. DEALING WITH
    COMPLEXITY

    View Slide

  42. TESTING

    View Slide

  43. LOAD
    FAILURE
    DEGRADATION

    View Slide

  44. 15,000 JOBS/HOUR
    50,000 DRIVERS
    30,000+ REQ/S

    View Slide

  45. CONTINUOUS
    PRODUCTION
    TESTING

    View Slide

  46. MONITORING

    View Slide

  47. Provisioning Service
    Rabbit MQ
    Monitoring

    Service
    Provisioning Service

    Service
    Publish
    Healthchecks

    View Slide

  48. View Slide

  49. View Slide

  50. View Slide

  51. DISTRIBUTED

    TRACING

    View Slide

  52. hailo~2~api api.v1.customer service.customer
    hailo~2~api api.v1.customer service.customer

    View Slide

  53. hailo~2~api api.v1.customer service.customer
    hailo~2~api api.v1.customer service.customer
    REQ
    REP
    REQ
    REP
    IN
    OUT
    IN
    OUT

    View Slide

  54. {
    "timestamp": 1410262798427145176,
    "traceId": "d30479b8-1491-4390-7cf5-4cd14bc4b765",
    "type": "OUT",
    "messageId": "a661f9ef-774c-49b2-6e74-cfed65f7d120",
    "parentMessageId": "",
    "from": "com.hailocab.hshell",
    "to": "com.hailocab.service.nearest-driver.search",
    "hostname": "ip-10-13-2-251",
    "az": "eu-west-1a",
    "handlerInstanceId": “server-com.hailocab.service.nearest-driver-18bd089e-8ef1-4ca1-75cb-8...c”,
    "duration": 11222094
    }
    {
    "timestamp": 1410262798416053450,
    "traceId": "d30479b8-1491-4390-7cf5-4cd14bc4b765",
    "type": "REQ",
    "messageId": "6404dd1e-c995-48a9-73dc-9edb1380f0bf",
    "parentMessageId": "a661f9ef-774c-49b2-6e74-cfed65f7d120",
    "from": "com.hailocab.service.nearest-driver",
    "to": "com.hailocab.service.zoning.search",
    "hostname": "ip-10-13-2-251",
    "az": "eu-west-1a"
    }

    View Slide

  55. Phosphor
    Host Instances
    Publish
    Service A
    Trace Library
    worker
    chan
    UDP
    Service B
    Trace Library
    worker
    chan
    UDP
    Trace
    Service
    In-memory
    Aggregates
    Optional

    persistant
    storage
    Dashboards
    Monitoring

    View Slide

  56. Tracing: 33eda743-f124-435c-71fc-3c872bbc98e6
    2014-09-07 02:20:19.867 [/] [START] → -
    2014-09-07 02:20:19.867 [eu-west-1a/ip-10-11-3-51] [REQ] com.hailocab.hailo-2-api → com.hailocab.api.v1.customer.neardrivers -
    2014-09-07 02:20:19.867 [eu-west-1a/ip-10-11-2-203] [IN] com.hailocab.hailo-2-api → com.hailocab.api.v1.customer.neardrivers -
    2014-09-07 02:20:19.868 [eu-west-1a/ip-10-11-2-203] [REQ] com.hailocab.api.v1.customer → com.hailocab.service.feature-flags.features -
    2014-09-07 02:20:19.869 [eu-west-1a/ip-10-11-3-111] [IN] com.hailocab.api.v1.customer → com.hailocab.service.feature-flags.features -
    2014-09-07 02:20:19.876 [eu-west-1a/ip-10-11-3-111] [REQ] com.hailocab.service.feature-flags → com.hailocab.service.hob.list -
    2014-09-07 02:20:19.877 [eu-west-1a/ip-10-11-3-168] [IN] com.hailocab.service.hob → com.hailocab.service.config.compile -
    2014-09-07 02:20:19.877 [eu-west-1a/ip-10-11-3-111] [IN] com.hailocab.service.feature-flags → com.hailocab.service.hob.list -
    2014-09-07 02:20:19.877 [eu-west-1a/ip-10-11-3-111] [REQ] com.hailocab.service.hob → com.hailocab.service.config.compile -
    2014-09-07 02:20:19.883 [eu-west-1a/ip-10-11-3-168] [OUT] com.hailocab.service.hob → com.hailocab.service.config.compile - 5.59 ms
    2014-09-07 02:20:19.886 [eu-west-1a/ip-10-11-3-111] [REP] com.hailocab.service.hob → com.hailocab.service.config.compile - 8.40 ms
    2014-09-07 02:20:19.887 [eu-west-1a/ip-10-11-3-111] [OUT] com.hailocab.service.feature-flags → com.hailocab.service.hob.list - 9.72 ms
    2014-09-07 02:20:19.889 [eu-west-1a/ip-10-11-3-111] [REP] com.hailocab.service.feature-flags → com.hailocab.service.hob.list - 13.23 ms
    2014-09-07 02:20:19.889 [eu-west-1a/ip-10-11-3-111] [OUT] com.hailocab.api.v1.customer → com.hailocab.service.feature-flags.features - 20.58 ms
    2014-09-07 02:20:19.890 [eu-west-1a/ip-10-11-2-203] [REP] com.hailocab.api.v1.customer → com.hailocab.service.feature-flags.features - 22.59 ms
    2014-09-07 02:20:19.902 [eu-west-1a/ip-10-11-2-203] [REQ] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare -
    2014-09-07 02:20:19.903 [eu-west-1a/ip-10-11-2-203] [REQ] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare -
    2014-09-07 02:20:19.903 [eu-west-1a/ip-10-11-2-203] [REQ] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare -
    2014-09-07 02:20:19.904 [eu-west-1a/ip-10-11-3-111] [IN] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare -
    2014-09-07 02:20:19.904 [eu-west-1a/ip-10-11-3-111] [OUT] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare - 0.36 ms
    2014-09-07 02:20:19.905 [eu-west-1a/ip-10-11-2-203] [REP] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare - 1.97 ms
    2014-09-07 02:20:19.905 [eu-west-1a/ip-10-11-2-214] [IN] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare -
    2014-09-07 02:20:19.905 [eu-west-1a/ip-10-11-2-203] [REQ] com.hailocab.api.v1.customer → com.hailocab.service.nearest-driver.search -
    2014-09-07 02:20:19.905 [eu-west-1a/ip-10-11-2-214] [OUT] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare - 0.10 ms

    ERR - com.hailocab.service.fare.basefare: Missing config at xxx
    2014-09-07 02:20:19.906 [eu-west-1a/ip-10-11-2-214] [IN] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare -
    2014-09-07 02:20:19.906 [eu-west-1a/ip-10-11-2-214] [OUT] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare - 0.06 ms 

    ERR - com.hailocab.service.fare.basefare: Missing config at xxx
    2014-09-07 02:20:19.907 [eu-west-1a/ip-10-11-3-58] [IN] com.hailocab.api.v1.customer → com.hailocab.service.nearest-driver.search -
    2014-09-07 02:20:19.907 [eu-west-1a/ip-10-11-3-58] [REQ] com.hailocab.service.nearest-driver → com.hailocab.service.zoning.search -
    2014-09-07 02:20:19.908 [eu-west-1a/ip-10-11-3-58] [IN] com.hailocab.service.nearest-driver → com.hailocab.service.zoning.search -
    2014-09-07 02:20:19.908 [eu-west-1a/ip-10-11-3-58] [OUT] com.hailocab.service.nearest-driver → com.hailocab.service.zoning.search - 0.20 ms
    2014-09-07 02:20:19.909 [eu-west-1a/ip-10-11-3-58] [REP] com.hailocab.service.nearest-driver → com.hailocab.service.zoning.search - 2.25 ms
    2014-09-07 02:20:19.909 [eu-west-1a/ip-10-11-3-58] [REQ] com.hailocab.service.nearest-driver → com.hailocab.service.raziel.multisearch -
    2014-09-07 02:20:19.912 [eu-west-1a/ip-10-11-3-227] [IN] com.hailocab.service.nearest-driver → com.hailocab.service.raziel.multisearch -
    2014-09-07 02:20:19.919 [eu-west-1a/ip-10-11-3-58] [REP] com.hailocab.service.nearest-driver → com.hailocab.service.raziel.multisearch - 9.46 ms
    2014-09-07 02:20:19.919 [eu-west-1a/ip-10-11-3-58] [REQ] com.hailocab.service.nearest-driver → com.hailocab.service.eta.multitraveltime -
    2014-09-07 02:20:19.919 [eu-west-1a/ip-10-11-3-227] [OUT] com.hailocab.service.nearest-driver → com.hailocab.service.raziel.multisearch - 7.58 ms
    2014-09-07 02:20:19.920 [eu-west-1a/ip-10-11-3-58] [IN] com.hailocab.service.nearest-driver → com.hailocab.service.eta.multitraveltime -
    2014-09-07 02:20:19.920 [eu-west-1a/ip-10-11-3-58] [OUT] com.hailocab.service.nearest-driver → com.hailocab.service.eta.multitraveltime - 0.06 ms
    2014-09-07 02:20:19.921 [eu-west-1a/ip-10-11-3-58] [REP] com.hailocab.service.nearest-driver → com.hailocab.service.eta.multitraveltime - 1.77 ms
    2014-09-07 02:20:19.921 [eu-west-1a/ip-10-11-3-58] [OUT] com.hailocab.api.v1.customer → com.hailocab.service.nearest-driver.search - 14.02 ms
    2014-09-07 02:20:19.921 [eu-west-1a/ip-10-11-2-203] [REP] com.hailocab.api.v1.customer → com.hailocab.service.nearest-driver.search - 15.48 ms
    2014-09-07 02:20:19.941 [eu-west-1a/ip-10-11-2-203] [REQ] com.hailocab.api.v1.customer → com.hailocab.service.experiment.readlastupdated -
    2014-09-07 02:20:19.945 [eu-west-1a/ip-10-11-2-214] [IN] com.hailocab.api.v1.customer → com.hailocab.service.experiment.readlastupdated -
    2014-09-07 02:20:19.947 [eu-west-1a/ip-10-11-2-214] [OUT] com.hailocab.api.v1.customer → com.hailocab.service.experiment.readlastupdated - 1.82 ms
    2014-09-07 02:20:19.947 [eu-west-1a/ip-10-11-2-203] [REP] com.hailocab.api.v1.customer → com.hailocab.service.experiment.readlastupdated - 6.01 ms
    2014-09-07 02:20:19.948 [eu-west-1a/ip-10-11-2-203] [OUT] com.hailocab.hailo-2-api → com.hailocab.api.v1.customer.neardrivers - 80.46 ms
    2014-09-07 02:20:19.950 [eu-west-1a/ip-10-11-3-51] [REP] com.hailocab.hailo-2-api → com.hailocab.api.v1.customer.neardrivers - 82.71 ms

    View Slide

  57. View Slide

  58. View Slide

  59. View Slide

  60. View Slide

  61. View Slide

  62. WHY BUILD
    MICROSERVICES?

    View Slide

  63. SIMPLE

    INDEPENDENT
    UNITS

    View Slide

  64. SIMPLE

    SCALABILITY

    View Slide

  65. EXPECT

    FAILURE

    View Slide

  66. AUTOMATE

    EVERYTHING

    View Slide

  67. CLOUD NATIVE

    ANTIFRAGILITY

    View Slide

  68. AČIŪ!
    Image Credits

    HMS President: Roger Marks
    Orbital Ion Cannon: www.rom.ac
    Go Gophers: Renee French
    Duopia: Jonny Hughes
    Microchips: S4nt1
    Control Room: NASA

    View Slide