A Journey into a Microservice World

Presented at Vilnius JUG #37

As Hailo expanded to a global presence, we needed to re-evaluate our approach to technology. This talk follows Hailo's journey from a monolithic application to a new microservice platform, running on three continents, built in both Java and Go.

Transcript

1. A JOURNEY
   INTO A
   MICROSERVICE
   WORLD
   @mattheath
   

   View Slide

2. View Slide

3. View Slide

4. View Slide

5. Redis
   Load Balancer
   Java

   App
   Load Balancer
   MySQL
   Slave
   MySQL
   aws
   PHP

   API
   PHP

   API
   

   View Slide

6. View Slide

7. View Slide

8. View Slide

9. View Slide

10. Redis
    Load Balancer
    Java

    App
    Load Balancer
    MySQL
    Slave
    MySQL
    aws
    PHP

    API
    PHP

    API
    

    View Slide

11. Redis
    Load Balancer
    Java

    App
    Load Balancer
    MySQL
    Slave
    MySQL
    aws
    PHP

    API
    Services
    PHP

    API
    

    View Slide

12. PHP

    API
    MySQL
    PHP
    Service
    Java
    Service
    LB
    per city per-region
    Load Balancer
    Cassandra
    PHP

    API
    PHP
    Service
    MySQL
    MySQL
    LB
    LB
    PHP

    API
    PHP

    API
    Java
    App
    Java
    App
    Java
    App
    

    View Slide

13. PHP

    API
    MySQL
    PHP
    Service
    Java
    Service
    LB
    per city per-region
    Load Balancer
    Cassandra
    PHP
    Service
    MySQL
    MySQL
    LB
    LB
    PHP

    API
    PHP

    API
    Java
    App
    Java
    App
    Java
    App
    PHP

    API
    

    View Slide

14. SLOW FAILOVER
    

    View Slide

15. LACK OF

    AUTOMATION
    

    View Slide

16. PHP

    API
    MySQL
    PHP
    Service
    Java
    Service
    LB
    per city per-region
    Load Balancer
    Cassandra
    PHP
    Service
    MySQL
    MySQL
    LB
    LB
    PHP

    API
    PHP

    API
    Java
    App
    Java
    App
    Java
    App
    PHP

    API
    

    View Slide

17. UNCLEAR
    RESPONSIBILITIES
    

    View Slide

18. SLOW

    DEVELOPMENT
    

    View Slide

19. Communication paths
    with two programmers
    Communication paths
    with three programmers
    Communication paths
    with four programmers
    Communication paths
    with five programmers
    Communication paths
    with ten programmers
    

    View Slide

20. A CLOUDY

    BEGINNING
    

    View Slide

21. SCALE

    BREAKS
    HARDWARE
    

    View Slide

22. SPEED

    BREAKS
    SOFTWARE
    

    View Slide

23. View Slide

24. “construct a highly agile and
    highly available service from
    ephemeral and assumed broken
    components”
    - Adrian Cockcroſt
    

    View Slide

25. region 1 region n
    

    View Slide

26. Cassandra
    region 1 region n
    Cassandra
    

    View Slide

27. Cassandra
    region 1
    Cassandra
    Go

    Service
    Java
    Service
    Java

    Service
    Java
    Service
    Go

    Service
    Go

    Service
    region n
    

    View Slide

28. Cassandra
    region 1
    Cassandra
    Go

    Service
    Java
    Service
    Java

    Service
    Java
    Service
    RabbitMQ Message Bus

    (federated clusters per AZ)
    RabbitMQ Message Bus

    (federated clusters per AZ)
    Go

    Service
    Go

    Service
    region n
    

    View Slide

29. Cassandra
    Go

    Service
    Go

    Service
    Java
    Service
    Load Balancer
    API / Routing Layer
    region 1
    Go

    Service
    Java

    Service
    Java
    Service
    Load Balancer
    API / Routing Layer
    RabbitMQ Message Bus

    (federated clusters per AZ)
    RabbitMQ Message Bus

    (federated clusters per AZ)
    Cassandra
    region n
    

    View Slide

30. Logic
    Service
    Handler
    Storage
    

    View Slide

31. Logic
    Storage
    Handler
    Library for building services that talk Protobuf via RMQ
    Service
    platform-layer
    

    View Slide

32. Logic
    service-layer
    Storage
    Handler
    Library for building services that talk Protobuf via RMQ
    Self-configuring external service adapters
    Service
    platform-layer
    

    View Slide

33. Logic
    service-layer
    Storage
    platform-layer
    Handler
    Library for building services that talk Protobuf via RMQ
    Self-configuring external service adapters
    Services get for free:
    • Provisioning
    • Service discovery
    • Configuration
    • Monitoring
    • Authentication/authorisation
    • AB testing
    • Self configuring connectivity 

    to third-party services
    Service
    

    View Slide

34. View Slide

35. View Slide

36. View Slide

37. Provisioning Service Provisioning Service Provisioning Service
    CI Pipeline (Janky/Jenkins)
    Amazon S3
    Provisioning Manager / Scheduler
    

    View Slide

38. Provisioning Service Provisioning Service Provisioning Service
    CI Pipeline (Janky/Jenkins)
    Amazon S3
    Provisioning Manager / Scheduler
    

    View Slide

39. Provisioning Service
    Rabbit MQ
    Discovery
    Service
    Provisioning Service
    New

    Service
    

    View Slide

40. Provisioning Service
    Rabbit MQ
    Discovery
    Service
    Binding

    Service
    Provisioning Service
    New

    Service
    

    View Slide

41. DEALING WITH
    COMPLEXITY
    

    View Slide

42. TESTING
    

    View Slide

43. LOAD
    FAILURE
    DEGRADATION
    

    View Slide

44. 15,000 JOBS/HOUR
    50,000 DRIVERS
    30,000+ REQ/S
    

    View Slide

45. CONTINUOUS
    PRODUCTION
    TESTING
    

    View Slide

46. MONITORING
    

    View Slide

47. Provisioning Service
    Rabbit MQ
    Monitoring

    Service
    Provisioning Service
    

    Service
    Publish
    Healthchecks
    

    View Slide

48. View Slide

49. View Slide

50. View Slide

51. DISTRIBUTED

    TRACING
    

    View Slide

52. hailo~2~api api.v1.customer service.customer
    hailo~2~api api.v1.customer service.customer
    

    View Slide

53. hailo~2~api api.v1.customer service.customer
    hailo~2~api api.v1.customer service.customer
    REQ
    REP
    REQ
    REP
    IN
    OUT
    IN
    OUT
    

    View Slide

54. {
    "timestamp": 1410262798427145176,
    "traceId": "d30479b8-1491-4390-7cf5-4cd14bc4b765",
    "type": "OUT",
    "messageId": "a661f9ef-774c-49b2-6e74-cfed65f7d120",
    "parentMessageId": "",
    "from": "com.hailocab.hshell",
    "to": "com.hailocab.service.nearest-driver.search",
    "hostname": "ip-10-13-2-251",
    "az": "eu-west-1a",
    "handlerInstanceId": “server-com.hailocab.service.nearest-driver-18bd089e-8ef1-4ca1-75cb-8...c”,
    "duration": 11222094
    }
    {
    "timestamp": 1410262798416053450,
    "traceId": "d30479b8-1491-4390-7cf5-4cd14bc4b765",
    "type": "REQ",
    "messageId": "6404dd1e-c995-48a9-73dc-9edb1380f0bf",
    "parentMessageId": "a661f9ef-774c-49b2-6e74-cfed65f7d120",
    "from": "com.hailocab.service.nearest-driver",
    "to": "com.hailocab.service.zoning.search",
    "hostname": "ip-10-13-2-251",
    "az": "eu-west-1a"
    }
    

    View Slide

55. Phosphor
    Host Instances
    Publish
    Service A
    Trace Library
    worker
    chan
    UDP
    Service B
    Trace Library
    worker
    chan
    UDP
    Trace
    Service
    In-memory
    Aggregates
    Optional

    persistant
    storage
    Dashboards
    Monitoring
    

    View Slide

56. Tracing: 33eda743-f124-435c-71fc-3c872bbc98e6
    2014-09-07 02:20:19.867 [/] [START] → -
    2014-09-07 02:20:19.867 [eu-west-1a/ip-10-11-3-51] [REQ] com.hailocab.hailo-2-api → com.hailocab.api.v1.customer.neardrivers -
    2014-09-07 02:20:19.867 [eu-west-1a/ip-10-11-2-203] [IN] com.hailocab.hailo-2-api → com.hailocab.api.v1.customer.neardrivers -
    2014-09-07 02:20:19.868 [eu-west-1a/ip-10-11-2-203] [REQ] com.hailocab.api.v1.customer → com.hailocab.service.feature-flags.features -
    2014-09-07 02:20:19.869 [eu-west-1a/ip-10-11-3-111] [IN] com.hailocab.api.v1.customer → com.hailocab.service.feature-flags.features -
    2014-09-07 02:20:19.876 [eu-west-1a/ip-10-11-3-111] [REQ] com.hailocab.service.feature-flags → com.hailocab.service.hob.list -
    2014-09-07 02:20:19.877 [eu-west-1a/ip-10-11-3-168] [IN] com.hailocab.service.hob → com.hailocab.service.config.compile -
    2014-09-07 02:20:19.877 [eu-west-1a/ip-10-11-3-111] [IN] com.hailocab.service.feature-flags → com.hailocab.service.hob.list -
    2014-09-07 02:20:19.877 [eu-west-1a/ip-10-11-3-111] [REQ] com.hailocab.service.hob → com.hailocab.service.config.compile -
    2014-09-07 02:20:19.883 [eu-west-1a/ip-10-11-3-168] [OUT] com.hailocab.service.hob → com.hailocab.service.config.compile - 5.59 ms
    2014-09-07 02:20:19.886 [eu-west-1a/ip-10-11-3-111] [REP] com.hailocab.service.hob → com.hailocab.service.config.compile - 8.40 ms
    2014-09-07 02:20:19.887 [eu-west-1a/ip-10-11-3-111] [OUT] com.hailocab.service.feature-flags → com.hailocab.service.hob.list - 9.72 ms
    2014-09-07 02:20:19.889 [eu-west-1a/ip-10-11-3-111] [REP] com.hailocab.service.feature-flags → com.hailocab.service.hob.list - 13.23 ms
    2014-09-07 02:20:19.889 [eu-west-1a/ip-10-11-3-111] [OUT] com.hailocab.api.v1.customer → com.hailocab.service.feature-flags.features - 20.58 ms
    2014-09-07 02:20:19.890 [eu-west-1a/ip-10-11-2-203] [REP] com.hailocab.api.v1.customer → com.hailocab.service.feature-flags.features - 22.59 ms
    2014-09-07 02:20:19.902 [eu-west-1a/ip-10-11-2-203] [REQ] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare -
    2014-09-07 02:20:19.903 [eu-west-1a/ip-10-11-2-203] [REQ] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare -
    2014-09-07 02:20:19.903 [eu-west-1a/ip-10-11-2-203] [REQ] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare -
    2014-09-07 02:20:19.904 [eu-west-1a/ip-10-11-3-111] [IN] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare -
    2014-09-07 02:20:19.904 [eu-west-1a/ip-10-11-3-111] [OUT] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare - 0.36 ms
    2014-09-07 02:20:19.905 [eu-west-1a/ip-10-11-2-203] [REP] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare - 1.97 ms
    2014-09-07 02:20:19.905 [eu-west-1a/ip-10-11-2-214] [IN] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare -
    2014-09-07 02:20:19.905 [eu-west-1a/ip-10-11-2-203] [REQ] com.hailocab.api.v1.customer → com.hailocab.service.nearest-driver.search -
    2014-09-07 02:20:19.905 [eu-west-1a/ip-10-11-2-214] [OUT] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare - 0.10 ms

    ERR - com.hailocab.service.fare.basefare: Missing config at xxx
    2014-09-07 02:20:19.906 [eu-west-1a/ip-10-11-2-214] [IN] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare -
    2014-09-07 02:20:19.906 [eu-west-1a/ip-10-11-2-214] [OUT] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare - 0.06 ms 

    ERR - com.hailocab.service.fare.basefare: Missing config at xxx
    2014-09-07 02:20:19.907 [eu-west-1a/ip-10-11-3-58] [IN] com.hailocab.api.v1.customer → com.hailocab.service.nearest-driver.search -
    2014-09-07 02:20:19.907 [eu-west-1a/ip-10-11-3-58] [REQ] com.hailocab.service.nearest-driver → com.hailocab.service.zoning.search -
    2014-09-07 02:20:19.908 [eu-west-1a/ip-10-11-3-58] [IN] com.hailocab.service.nearest-driver → com.hailocab.service.zoning.search -
    2014-09-07 02:20:19.908 [eu-west-1a/ip-10-11-3-58] [OUT] com.hailocab.service.nearest-driver → com.hailocab.service.zoning.search - 0.20 ms
    2014-09-07 02:20:19.909 [eu-west-1a/ip-10-11-3-58] [REP] com.hailocab.service.nearest-driver → com.hailocab.service.zoning.search - 2.25 ms
    2014-09-07 02:20:19.909 [eu-west-1a/ip-10-11-3-58] [REQ] com.hailocab.service.nearest-driver → com.hailocab.service.raziel.multisearch -
    2014-09-07 02:20:19.912 [eu-west-1a/ip-10-11-3-227] [IN] com.hailocab.service.nearest-driver → com.hailocab.service.raziel.multisearch -
    2014-09-07 02:20:19.919 [eu-west-1a/ip-10-11-3-58] [REP] com.hailocab.service.nearest-driver → com.hailocab.service.raziel.multisearch - 9.46 ms
    2014-09-07 02:20:19.919 [eu-west-1a/ip-10-11-3-58] [REQ] com.hailocab.service.nearest-driver → com.hailocab.service.eta.multitraveltime -
    2014-09-07 02:20:19.919 [eu-west-1a/ip-10-11-3-227] [OUT] com.hailocab.service.nearest-driver → com.hailocab.service.raziel.multisearch - 7.58 ms
    2014-09-07 02:20:19.920 [eu-west-1a/ip-10-11-3-58] [IN] com.hailocab.service.nearest-driver → com.hailocab.service.eta.multitraveltime -
    2014-09-07 02:20:19.920 [eu-west-1a/ip-10-11-3-58] [OUT] com.hailocab.service.nearest-driver → com.hailocab.service.eta.multitraveltime - 0.06 ms
    2014-09-07 02:20:19.921 [eu-west-1a/ip-10-11-3-58] [REP] com.hailocab.service.nearest-driver → com.hailocab.service.eta.multitraveltime - 1.77 ms
    2014-09-07 02:20:19.921 [eu-west-1a/ip-10-11-3-58] [OUT] com.hailocab.api.v1.customer → com.hailocab.service.nearest-driver.search - 14.02 ms
    2014-09-07 02:20:19.921 [eu-west-1a/ip-10-11-2-203] [REP] com.hailocab.api.v1.customer → com.hailocab.service.nearest-driver.search - 15.48 ms
    2014-09-07 02:20:19.941 [eu-west-1a/ip-10-11-2-203] [REQ] com.hailocab.api.v1.customer → com.hailocab.service.experiment.readlastupdated -
    2014-09-07 02:20:19.945 [eu-west-1a/ip-10-11-2-214] [IN] com.hailocab.api.v1.customer → com.hailocab.service.experiment.readlastupdated -
    2014-09-07 02:20:19.947 [eu-west-1a/ip-10-11-2-214] [OUT] com.hailocab.api.v1.customer → com.hailocab.service.experiment.readlastupdated - 1.82 ms
    2014-09-07 02:20:19.947 [eu-west-1a/ip-10-11-2-203] [REP] com.hailocab.api.v1.customer → com.hailocab.service.experiment.readlastupdated - 6.01 ms
    2014-09-07 02:20:19.948 [eu-west-1a/ip-10-11-2-203] [OUT] com.hailocab.hailo-2-api → com.hailocab.api.v1.customer.neardrivers - 80.46 ms
    2014-09-07 02:20:19.950 [eu-west-1a/ip-10-11-3-51] [REP] com.hailocab.hailo-2-api → com.hailocab.api.v1.customer.neardrivers - 82.71 ms
    

    View Slide

57. View Slide

58. View Slide

59. View Slide

60. View Slide

61. View Slide

62. WHY BUILD
    MICROSERVICES?
    

    View Slide

63. SIMPLE

    INDEPENDENT
    UNITS
    

    View Slide

64. SIMPLE

    SCALABILITY
    

    View Slide

65. EXPECT

    FAILURE
    

    View Slide

66. AUTOMATE

    EVERYTHING
    

    View Slide

67. CLOUD NATIVE

    ANTIFRAGILITY
    

    View Slide

68. AČIŪ!
    Image Credits

    HMS President: Roger Marks
    Orbital Ion Cannon: www.rom.ac
    Go Gophers: Renee French
    Duopia: Jonny Hughes
    Microchips: S4nt1
    Control Room: NASA
    

    View Slide