Scaling Microservices in Go - High Load Strategy 2015

67f4a8f2a209a38d7242829947b26ba3?s=47 mattheath
January 30, 2015

Scaling Microservices in Go - High Load Strategy 2015

Presented at High Load Strategy 2015 in Vilnius, Lithuania

Moving to a microservice architecture is a complex challenge, which often requires re-evaluating our approach to technology. Having previously helped transition Hailo to a new microservice platform, built almost entirely in Go, Matt is now working with Starling, using the latest technologies to build a new breed of digital bank. This talk will cover how to develop and migrate to a microservice architecture, the benefits of this approach, common pitfalls to avoid, and lessons learnt when developing high volume, low latency, distributed Go applications.

67f4a8f2a209a38d7242829947b26ba3?s=128

mattheath

January 30, 2015
Tweet

Transcript

  1. SCALING MICRO- SERVICES WITH GO HIGH LOAD STRATEGY 2015 @mattheath

  2. S TA R L I N G

  3. monoliths traditional dev

  4. Database Application

  5. Database App App App Database App Load Balancer App App

  6. SLOW
 DEVELOPMENT

  7. SLOW
 RELEASE CYCLES

  8. COMPLEX
 SOFTWARE

  9. HARD
 TO SCALE

  10. SCALE
 BREAKS
 HARDWARE

  11. SPEED
 BREAKS
 SOFTWARE

  12. Communication paths with two programmers Communication paths with three programmers

    Communication paths with four programmers Communication paths with five programmers Communication paths with ten programmers
  13. None
  14. “construct a highly agile and highly available service from ephemeral

    and assumed broken components”- Adrian Cockcroſt
  15. A CLOUDY
 FUTURE

  16. MICROSERVICES

  17. transport layers COMMUNICATION

  18. Database Service Service Service Load Balancer API / Routing Layer

    datacentre 1 Service Service Service Load Balancer API / Routing Layer RabbitMQ Message Bus RabbitMQ Message Bus Database datacentre n
  19. None
  20. package handler import ( "..." ) func Foo(req *server.Request) (protobuf.Message,

    error) { }
  21. service.Register( &service.Endpoint{ Name: "foo", Handler: handler.Foo, } )

  22. deliveries, err := server.Listen() for delivery := range deliveries {

    go handleDelivery(delivery) }
  23. func handleDelivery(delivery amqp.Delivery) { // Do something for this request

    response, err := executeRequest(delivery) // Send Response server.Respond(response) }
  24. var tokens = make(chan bool, 1000) 
 func handleDelivery(delivery amqp.Delivery)

    { // Throttle requests // error handling omitted... select { case <-tokens: response, err := executeRequest(delivery) tokens <- true // replace token case <-time.After(500 * time.Millisecond): response, err := timeoutResponse() } // Send Response server.Respond(response) }
  25. Logic Handler Storage Service

  26. platform-library Library for building services that talk Protobuf via RMQ

    Logic Handler Storage Service
  27. service-library Library for building services that talk Protobuf via RMQ

    Self-configuring external service adapters platform-library Logic Handler Storage Service
  28. Services get for free: • Provisioning • Service discovery •

    Configuration • Monitoring • Authentication/authorisation • AB testing • Self configuring connectivity 
 to third-party services Service Library for building services that talk Protobuf via RMQ service-library platform-library Logic Handler Storage Self-configuring external service adapters
  29. None
  30. Provisioning Daemon Provisioning Daemon Provisioning Daemon CI Pipeline (Janky/Jenkins) Amazon

    S3 Provisioning Scheduler - eg. Marathon on Mesos
  31. DEALING WITH COMPLEXITY

  32. LOOSE
 COUPLING

  33. ASYNC EVENT
 DRIVEN

  34. TESTING

  35. LOAD FAILURE DEGRADATION

  36. MONITORING

  37. DISTRIBUTED
 TRACING

  38. api api.customer service.customer api api.customer service.customer

  39. api api.customer service.customer api api.customer service.customer REQ REP REQ REP

    IN OUT IN OUT
  40. { "timestamp": 1410262798427145176, "traceId": "d30479b8-1491-4390-7cf5-4cd14bc4b765", "type": "OUT", "messageId": "a661f9ef-774c-49b2-6e74-cfed65f7d120", "parentMessageId":

    "", "from": “uk.co.starlingbank.webui”, "to": “uk.co.starlingbank.api.customer", "hostname": "ip-10-13-2-251", "az": "eu-west-1a", "handlerInstanceId": “server-18bd089e-8ef1-4ca1-75cb-8...c”, "duration": 11222094 } { "timestamp": 1410262798416053450, "traceId": "d30479b8-1491-4390-7cf5-4cd14bc4b765", "type": "REQ", "messageId": "6404dd1e-c995-48a9-73dc-9edb1380f0bf", "parentMessageId": "a661f9ef-774c-49b2-6e74-cfed65f7d120", "from": "uk.co.starlingbank.api.customer", "to": "uk.co.starlingbank.service.customer", "hostname": "ip-10-13-2-251", "az": "eu-west-1a" }
  41. Tracing: 33eda743-f124-435c-71fc-3c872bbc98e6 2014-09-07 02:20:19.867 [/] [START] → - 2014-09-07 02:20:19.867

    [eu-west-1a/ip-10-11-3-51] [REQ] uk.co.starlingbank.api → uk.co.starlingbank.api.customer.neardrivers - 2014-09-07 02:20:19.867 [eu-west-1a/ip-10-11-2-203] [IN] uk.co.starlingbank.api → uk.co.starlingbank.api.customer.neardrivers - 2014-09-07 02:20:19.868 [eu-west-1a/ip-10-11-2-203] [REQ] uk.co.starlingbank.api.customer → uk.co.starlingbank.service.feature-flags.features - 2014-09-07 02:20:19.869 [eu-west-1a/ip-10-11-3-111] [IN] uk.co.starlingbank.api.customer → uk.co.starlingbank.service.feature-flags.features - 2014-09-07 02:20:19.876 [eu-west-1a/ip-10-11-3-111] [REQ] uk.co.starlingbank.service.feature-flags → uk.co.starlingbank.service.hob.list - 2014-09-07 02:20:19.877 [eu-west-1a/ip-10-11-3-168] [IN] uk.co.starlingbank.service.hob → uk.co.starlingbank.service.config.compile - 2014-09-07 02:20:19.877 [eu-west-1a/ip-10-11-3-111] [IN] uk.co.starlingbank.service.feature-flags → uk.co.starlingbank.service.hob.list - 2014-09-07 02:20:19.877 [eu-west-1a/ip-10-11-3-111] [REQ] uk.co.starlingbank.service.hob → uk.co.starlingbank.service.config.compile - 2014-09-07 02:20:19.883 [eu-west-1a/ip-10-11-3-168] [OUT] uk.co.starlingbank.service.hob → uk.co.starlingbank.service.config.compile - 5.59 ms 2014-09-07 02:20:19.886 [eu-west-1a/ip-10-11-3-111] [REP] uk.co.starlingbank.service.hob → uk.co.starlingbank.service.config.compile - 8.40 ms 2014-09-07 02:20:19.887 [eu-west-1a/ip-10-11-3-111] [OUT] uk.co.starlingbank.service.feature-flags → uk.co.starlingbank.service.hob.list - 9.72 ms 2014-09-07 02:20:19.889 [eu-west-1a/ip-10-11-3-111] [REP] uk.co.starlingbank.service.feature-flags → uk.co.starlingbank.service.hob.list - 13.23 ms 2014-09-07 02:20:19.889 [eu-west-1a/ip-10-11-3-111] [OUT] uk.co.starlingbank.api.customer → uk.co.starlingbank.service.feature-flags.features - 20.58 ms 2014-09-07 02:20:19.890 [eu-west-1a/ip-10-11-2-203] [REP] uk.co.starlingbank.api.customer → uk.co.starlingbank.service.feature-flags.features - 22.59 ms 2014-09-07 02:20:19.902 [eu-west-1a/ip-10-11-2-203] [REQ] uk.co.starlingbank.api.customer → uk.co.starlingbank.service.fare.basefare - 2014-09-07 02:20:19.903 [eu-west-1a/ip-10-11-2-203] [REQ] uk.co.starlingbank.api.customer → uk.co.starlingbank.service.fare.basefare - 2014-09-07 02:20:19.903 [eu-west-1a/ip-10-11-2-203] [REQ] uk.co.starlingbank.api.customer → uk.co.starlingbank.service.fare.basefare - 2014-09-07 02:20:19.904 [eu-west-1a/ip-10-11-3-111] [IN] uk.co.starlingbank.api.customer → uk.co.starlingbank.service.fare.basefare - 2014-09-07 02:20:19.904 [eu-west-1a/ip-10-11-3-111] [OUT] uk.co.starlingbank.api.customer → uk.co.starlingbank.service.fare.basefare - 0.36 ms 2014-09-07 02:20:19.905 [eu-west-1a/ip-10-11-2-203] [REP] uk.co.starlingbank.api.customer → uk.co.starlingbank.service.fare.basefare - 1.97 ms 2014-09-07 02:20:19.905 [eu-west-1a/ip-10-11-2-214] [IN] uk.co.starlingbank.api.customer → uk.co.starlingbank.service.fare.basefare - 2014-09-07 02:20:19.905 [eu-west-1a/ip-10-11-2-203] [REQ] uk.co.starlingbank.api.customer → uk.co.starlingbank.service.nearest-driver.search - 2014-09-07 02:20:19.905 [eu-west-1a/ip-10-11-2-214] [OUT] uk.co.starlingbank.api.customer → uk.co.starlingbank.service.fare.basefare - 0.10 ms
 ERR - uk.co.starlingbank.service.fare.basefare: Missing config at xxx 2014-09-07 02:20:19.906 [eu-west-1a/ip-10-11-2-214] [IN] uk.co.starlingbank.api.customer → uk.co.starlingbank.service.fare.basefare - 2014-09-07 02:20:19.906 [eu-west-1a/ip-10-11-2-214] [OUT] uk.co.starlingbank.api.customer → uk.co.starlingbank.service.fare.basefare - 0.06 ms 
 ERR - uk.co.starlingbank.service.fare.basefare: Missing config at xxx 2014-09-07 02:20:19.907 [eu-west-1a/ip-10-11-3-58] [IN] uk.co.starlingbank.api.customer → uk.co.starlingbank.service.nearest-driver.search - 2014-09-07 02:20:19.907 [eu-west-1a/ip-10-11-3-58] [REQ] uk.co.starlingbank.service.nearest-driver → uk.co.starlingbank.service.zoning.search - 2014-09-07 02:20:19.908 [eu-west-1a/ip-10-11-3-58] [IN] uk.co.starlingbank.service.nearest-driver → uk.co.starlingbank.service.zoning.search - 2014-09-07 02:20:19.908 [eu-west-1a/ip-10-11-3-58] [OUT] uk.co.starlingbank.service.nearest-driver → uk.co.starlingbank.service.zoning.search - 0.20 ms 2014-09-07 02:20:19.909 [eu-west-1a/ip-10-11-3-58] [REP] uk.co.starlingbank.service.nearest-driver → uk.co.starlingbank.service.zoning.search - 2.25 ms 2014-09-07 02:20:19.909 [eu-west-1a/ip-10-11-3-58] [REQ] uk.co.starlingbank.service.nearest-driver → uk.co.starlingbank.service.raziel.multisearch - 2014-09-07 02:20:19.912 [eu-west-1a/ip-10-11-3-227] [IN] uk.co.starlingbank.service.nearest-driver → uk.co.starlingbank.service.raziel.multisearch - 2014-09-07 02:20:19.919 [eu-west-1a/ip-10-11-3-58] [REP] uk.co.starlingbank.service.nearest-driver → uk.co.starlingbank.service.raziel.multisearch - 9.46 ms 2014-09-07 02:20:19.919 [eu-west-1a/ip-10-11-3-58] [REQ] uk.co.starlingbank.service.nearest-driver → uk.co.starlingbank.service.eta.multitraveltime - 2014-09-07 02:20:19.919 [eu-west-1a/ip-10-11-3-227] [OUT] uk.co.starlingbank.service.nearest-driver → uk.co.starlingbank.service.raziel.multisearch - 7.58 ms 2014-09-07 02:20:19.920 [eu-west-1a/ip-10-11-3-58] [IN] uk.co.starlingbank.service.nearest-driver → uk.co.starlingbank.service.eta.multitraveltime - 2014-09-07 02:20:19.920 [eu-west-1a/ip-10-11-3-58] [OUT] uk.co.starlingbank.service.nearest-driver → uk.co.starlingbank.service.eta.multitraveltime - 0.06 ms 2014-09-07 02:20:19.921 [eu-west-1a/ip-10-11-3-58] [REP] uk.co.starlingbank.service.nearest-driver → uk.co.starlingbank.service.eta.multitraveltime - 1.77 ms 2014-09-07 02:20:19.921 [eu-west-1a/ip-10-11-3-58] [OUT] uk.co.starlingbank.api.customer → uk.co.starlingbank.service.nearest-driver.search - 14.02 ms 2014-09-07 02:20:19.921 [eu-west-1a/ip-10-11-2-203] [REP] uk.co.starlingbank.api.customer → uk.co.starlingbank.service.nearest-driver.search - 15.48 ms 2014-09-07 02:20:19.941 [eu-west-1a/ip-10-11-2-203] [REQ] uk.co.starlingbank.api.customer → uk.co.starlingbank.service.experiment.readlastupdated - 2014-09-07 02:20:19.945 [eu-west-1a/ip-10-11-2-214] [IN] uk.co.starlingbank.api.customer → uk.co.starlingbank.service.experiment.readlastupdated - 2014-09-07 02:20:19.947 [eu-west-1a/ip-10-11-2-214] [OUT] uk.co.starlingbank.api.customer → uk.co.starlingbank.service.experiment.readlastupdated - 1.82 ms 2014-09-07 02:20:19.947 [eu-west-1a/ip-10-11-2-203] [REP] uk.co.starlingbank.api.customer → uk.co.starlingbank.service.experiment.readlastupdated - 6.01 ms 2014-09-07 02:20:19.948 [eu-west-1a/ip-10-11-2-203] [OUT] uk.co.starlingbank.api → uk.co.starlingbank.api.customer.neardrivers - 80.46 ms 2014-09-07 02:20:19.950 [eu-west-1a/ip-10-11-3-51] [REP] uk.co.starlingbank.api → uk.co.starlingbank.api.customer.neardrivers - 82.71 ms
  42. None
  43. None
  44. None
  45. None
  46. None
  47. WHY BUILD MICROSERVICES?

  48. SIMPLE
 INDEPENDENT UNITS

  49. SIMPLE
 SCALABILITY

  50. EXPECT
 FAILURE

  51. AUTOMATE
 EVERYTHING

  52. CLOUD NATIVE
 ANTIFRAGILITY

  53. GO?

  54. SMALL SIMPLE EASY TO READ EASY TO LEARN

  55. CONCURRENCY PRIMITIVES INTERFACE SUPPORT

  56. EASY DEPLOYMENT & MANAGEMENT IN PRODUCTION AND AT SCALE

  57. THANKS! Image Credits
 IBM System/360: IBM Orbital Ion Cannon: www.rom.ac

    Go Gophers: Renee French Duopia: Jonny Hughes Control Room: NASA Microchips: Santi Antennas: Alex Weimer