Scaling Micro-Services in Go - HighLoad++ 2014

67f4a8f2a209a38d7242829947b26ba3?s=47 mattheath
October 31, 2014

Scaling Micro-Services in Go - HighLoad++ 2014

Presented at HighLoad++ 2014 in Moscow

As Hailo expanded to a global presence, we needed to re-evaluate our approach to technology. Moving from a monolithic PHP & Java application, and embracing a cloud native approach, Hailo transitioned to a new micro-service platform, running on three continents, built almost entirely in Go. This talk will cover how we developed and migrated to a micro service architecture, common pitfalls to avoid, and lessons learnt when developing high volume, low latency, distributed Go applications.

Поскольку рост проекта Hailo обеспечил ему глобальное присутствие, нам пришлось пересмотреть наш подход к технологиям. Мы решили уйти от монолитного приложения на PHP и Java и внедрить нативную поддержку «облаков», и проект Hailo перешёл на новую платформу микросервисов, работающую на трех континентах и почти полностью построенную на Go. В данном докладе я расскажу, как мы разработали архитектуру микросервисов и впоследствии перешли на неё, перечислю распространенные ошибки и объясню, как их избежать, и поделюсь уроками, которые мы извлекли из разработки на Go распределенных приложений, рассчитанных на обработку больших объёмов данных с минимальной задержкой.

67f4a8f2a209a38d7242829947b26ba3?s=128

mattheath

October 31, 2014
Tweet

Transcript

  1. Scaling Micro- Services in Go Matt Heath, Hailo

  2. None
  3. None
  4. None
  5. PHP
 API Load Balancer PHP
 API PHP
 API Java
 App

    PHP
 API PHP
 API Load Balancer PHP
 API MySQL Slave Redis MySQL eu-west-1
  6. ALL HAIL THE MONOLITH

  7. None
  8. PHP
 API Load Balancer PHP
 API PHP
 API Java
 App

    PHP
 API PHP
 API Load Balancer PHP
 API MySQL Slave Redis MySQL eu-west-1
  9. PHP
 API MySQL PHP Service Java Service ELB per city

    multi-region ELB Cassandra PHP
 API PHP Service MySQL MySQL ELB ELB PHP
 API PHP
 API Java App Java App Java App
  10. PHP
 API MySQL PHP Service Java Service ELB per city

    multi-region ELB Cassandra PHP Service MySQL MySQL ELB ELB PHP
 API PHP
 API Java App Java App Java App PHP
 API
  11. PHP
 API MySQL PHP Service Java Service ELB per city

    multi-region ELB Cassandra PHP Service MySQL MySQL ELB ELB PHP
 API PHP
 API Java App Java App Java App PHP
 API
  12. Communication paths with two programmers Communication paths with three programmers

    Communication paths with four programmers Communication paths with five programmers Communication paths with ten programmers
  13. DEVELOPING NEW FEATURES UNCLEAR RESPONSIBILITIES SLOW FAILOVER LACK OF AUTOMATION

  14. None
  15. “construct a highly agile and highly available service from ephemeral

    and assumed broken components” - Adrian Cockcroſt
  16. Cassandra Go
 Service Go
 Service Java Service ELB Go “Thin”

    API us-east-1 eu-west-1 Go
 Service Go
 Service Java Service ELB Go “Thin” API RabbitMQ Message Bus
 (federated clusters per AZ) RabbitMQ Message Bus
 (federated clusters per AZ) Cassandra
  17. None
  18. Logic Service Handler Storage

  19. Logic Storage Handler Library for building services that talk Protobuf

    via RMQ Service go-platform-layer
  20. package handler import ( "..." ) func Foo(req *server.Request) (proto.Message,

    errors.Error) { }
  21. service.Register( &service.Endpoint{ Name: "foo", Mean: 100, Upper95: 200, Handler: handler.Foo,

    } )
  22. Logic go-service-layer Storage Handler Library for building services that talk

    Protobuf via RMQ Self-configuring external service adapters Service go-platform-layer
  23. package handler import ( "github.com/hailocab/gomemcache/memcache" ) func Foo(req *server.Request) (proto.Message,

    errors.Error) { item, err := memcache.Get("key") if err != nil { // handle error } // ... }
  24. Logic go-service-layer Storage go-platform-layer Handler Library for building services that

    talk Protobuf via RMQ Self-configuring external service adapters Services get for free: • Provisioning • Service discovery • Configuration • Monitoring • Authentication/authorisation • AB testing • Self configuring connectivity 
 to third-party services Service
  25. None
  26. None
  27. None
  28. Provisioning Service Provisioning Service Provisioning Service CI Pipeline (Janky/Jenkins) Amazon

    S3 Provisioning Manager / Scheduler
  29. Provisioning Service Provisioning Service Provisioning Service CI Pipeline (Janky/Jenkins) Amazon

    S3 Provisioning Manager / Scheduler
  30. Provisioning Service Rabbit MQ Discovery Service Provisioning Service New
 Service

    AUTOMATIC SERVICE DISCOVERY
  31. Provisioning Service Rabbit MQ Discovery Service Binding
 Service Provisioning Service

    New
 Service AUTOMATIC SERVICE DISCOVERY
  32. SMALL INDEPENDENT SERVICES SINGLE RESPONSIBILITY EASE PAIN, SCALE RAPIDLY

  33. CLOUD NATIVE / ANTIFRAGILE EXPECT FAILURE AUTOMATE EVERYTHING

  34. TESTING

  35. LOAD FAILURE DEGRADATION

  36. 15,000 JOBS/HOUR 50,000 DRIVERS 30,000+ REQ/S

  37. CONTINUOUS PRODUCTION TESTING

  38. MONITORING

  39. Provisioning Service Rabbit MQ Monitoring
 Service Provisioning Service 
 Service

    Publish Healthchecks AUTOMATIC HEALTHCHECK REGISTRATION
  40. None
  41. None
  42. None
  43. DISTRIBUTED
 TRACING

  44. hailo~2~api api.v1.customer service.customer hailo~2~api api.v1.customer service.customer

  45. hailo~2~api api.v1.customer service.customer hailo~2~api api.v1.customer service.customer REQ REP REQ REP

    IN OUT IN OUT
  46. { "timestamp": 1410262798427145176, "traceId": "d30479b8-1491-4390-7cf5-4cd14bc4b765", "type": "OUT", "messageId": "a661f9ef-774c-49b2-6e74-cfed65f7d120", "parentMessageId":

    "", "from": "com.hailocab.hshell", "to": "com.hailocab.service.nearest-driver.search", "hostname": "ip-10-13-2-251", "az": "eu-west-1a", "handlerInstanceId": “server-com.hailocab.service.nearest-driver-18bd089e-8ef1-4ca1-75cb-8...c”, "duration": 11222094 } { "timestamp": 1410262798416053450, "traceId": "d30479b8-1491-4390-7cf5-4cd14bc4b765", "type": "REQ", "messageId": "6404dd1e-c995-48a9-73dc-9edb1380f0bf", "parentMessageId": "a661f9ef-774c-49b2-6e74-cfed65f7d120", "from": "com.hailocab.service.nearest-driver", "to": "com.hailocab.service.zoning.search", "hostname": "ip-10-13-2-251", "az": "eu-west-1a" }
  47. Phosphor Host Instances Publish Service A Trace Library goroutine chan

    UDP Service B Trace Library goroutine chan UDP Trace Service In-memory Aggregates Optional
 persistant storage Dashboards Monitoring
  48. var traceChan chan []byte func init() { // Use a

    buffered channel traceChan = make(chan []byte, 200) // Fire off a background worker for this channel defaultClient = NewClient(traceChan) go defaultClient.publisher() } // Send, drops trace if the backend is at capacity func Send(msg []byte) { select { case traceChan <- msg: // Success default: // Default case fired if channel is full // Ensures this is non blocking } }
  49. Tracing: 33eda743-f124-435c-71fc-3c872bbc98e6 2014-09-07 02:20:19.867 [/] [START] → - 2014-09-07 02:20:19.867

    [eu-west-1a/ip-10-11-3-51] [REQ] com.hailocab.hailo-2-api → com.hailocab.api.v1.customer.neardrivers - 2014-09-07 02:20:19.867 [eu-west-1a/ip-10-11-2-203] [IN] com.hailocab.hailo-2-api → com.hailocab.api.v1.customer.neardrivers - 2014-09-07 02:20:19.868 [eu-west-1a/ip-10-11-2-203] [REQ] com.hailocab.api.v1.customer → com.hailocab.service.feature-flags.features - 2014-09-07 02:20:19.869 [eu-west-1a/ip-10-11-3-111] [IN] com.hailocab.api.v1.customer → com.hailocab.service.feature-flags.features - 2014-09-07 02:20:19.876 [eu-west-1a/ip-10-11-3-111] [REQ] com.hailocab.service.feature-flags → com.hailocab.service.hob.list - 2014-09-07 02:20:19.877 [eu-west-1a/ip-10-11-3-168] [IN] com.hailocab.service.hob → com.hailocab.service.config.compile - 2014-09-07 02:20:19.877 [eu-west-1a/ip-10-11-3-111] [IN] com.hailocab.service.feature-flags → com.hailocab.service.hob.list - 2014-09-07 02:20:19.877 [eu-west-1a/ip-10-11-3-111] [REQ] com.hailocab.service.hob → com.hailocab.service.config.compile - 2014-09-07 02:20:19.883 [eu-west-1a/ip-10-11-3-168] [OUT] com.hailocab.service.hob → com.hailocab.service.config.compile - 5.59 ms 2014-09-07 02:20:19.886 [eu-west-1a/ip-10-11-3-111] [REP] com.hailocab.service.hob → com.hailocab.service.config.compile - 8.40 ms 2014-09-07 02:20:19.887 [eu-west-1a/ip-10-11-3-111] [OUT] com.hailocab.service.feature-flags → com.hailocab.service.hob.list - 9.72 ms 2014-09-07 02:20:19.889 [eu-west-1a/ip-10-11-3-111] [REP] com.hailocab.service.feature-flags → com.hailocab.service.hob.list - 13.23 ms 2014-09-07 02:20:19.889 [eu-west-1a/ip-10-11-3-111] [OUT] com.hailocab.api.v1.customer → com.hailocab.service.feature-flags.features - 20.58 ms 2014-09-07 02:20:19.890 [eu-west-1a/ip-10-11-2-203] [REP] com.hailocab.api.v1.customer → com.hailocab.service.feature-flags.features - 22.59 ms 2014-09-07 02:20:19.902 [eu-west-1a/ip-10-11-2-203] [REQ] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare - 2014-09-07 02:20:19.903 [eu-west-1a/ip-10-11-2-203] [REQ] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare - 2014-09-07 02:20:19.903 [eu-west-1a/ip-10-11-2-203] [REQ] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare - 2014-09-07 02:20:19.904 [eu-west-1a/ip-10-11-3-111] [IN] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare - 2014-09-07 02:20:19.904 [eu-west-1a/ip-10-11-3-111] [OUT] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare - 0.36 ms 2014-09-07 02:20:19.905 [eu-west-1a/ip-10-11-2-203] [REP] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare - 1.97 ms 2014-09-07 02:20:19.905 [eu-west-1a/ip-10-11-2-214] [IN] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare - 2014-09-07 02:20:19.905 [eu-west-1a/ip-10-11-2-203] [REQ] com.hailocab.api.v1.customer → com.hailocab.service.nearest-driver.search - 2014-09-07 02:20:19.905 [eu-west-1a/ip-10-11-2-214] [OUT] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare - 0.10 ms
 ERR - com.hailocab.service.fare.basefare: Missing config at xxx 2014-09-07 02:20:19.906 [eu-west-1a/ip-10-11-2-214] [IN] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare - 2014-09-07 02:20:19.906 [eu-west-1a/ip-10-11-2-214] [OUT] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare - 0.06 ms 
 ERR - com.hailocab.service.fare.basefare: Missing config at xxx 2014-09-07 02:20:19.907 [eu-west-1a/ip-10-11-3-58] [IN] com.hailocab.api.v1.customer → com.hailocab.service.nearest-driver.search - 2014-09-07 02:20:19.907 [eu-west-1a/ip-10-11-3-58] [REQ] com.hailocab.service.nearest-driver → com.hailocab.service.zoning.search - 2014-09-07 02:20:19.908 [eu-west-1a/ip-10-11-3-58] [IN] com.hailocab.service.nearest-driver → com.hailocab.service.zoning.search - 2014-09-07 02:20:19.908 [eu-west-1a/ip-10-11-3-58] [OUT] com.hailocab.service.nearest-driver → com.hailocab.service.zoning.search - 0.20 ms 2014-09-07 02:20:19.909 [eu-west-1a/ip-10-11-3-58] [REP] com.hailocab.service.nearest-driver → com.hailocab.service.zoning.search - 2.25 ms 2014-09-07 02:20:19.909 [eu-west-1a/ip-10-11-3-58] [REQ] com.hailocab.service.nearest-driver → com.hailocab.service.raziel.multisearch - 2014-09-07 02:20:19.912 [eu-west-1a/ip-10-11-3-227] [IN] com.hailocab.service.nearest-driver → com.hailocab.service.raziel.multisearch - 2014-09-07 02:20:19.919 [eu-west-1a/ip-10-11-3-58] [REP] com.hailocab.service.nearest-driver → com.hailocab.service.raziel.multisearch - 9.46 ms 2014-09-07 02:20:19.919 [eu-west-1a/ip-10-11-3-58] [REQ] com.hailocab.service.nearest-driver → com.hailocab.service.eta.multitraveltime - 2014-09-07 02:20:19.919 [eu-west-1a/ip-10-11-3-227] [OUT] com.hailocab.service.nearest-driver → com.hailocab.service.raziel.multisearch - 7.58 ms 2014-09-07 02:20:19.920 [eu-west-1a/ip-10-11-3-58] [IN] com.hailocab.service.nearest-driver → com.hailocab.service.eta.multitraveltime - 2014-09-07 02:20:19.920 [eu-west-1a/ip-10-11-3-58] [OUT] com.hailocab.service.nearest-driver → com.hailocab.service.eta.multitraveltime - 0.06 ms 2014-09-07 02:20:19.921 [eu-west-1a/ip-10-11-3-58] [REP] com.hailocab.service.nearest-driver → com.hailocab.service.eta.multitraveltime - 1.77 ms 2014-09-07 02:20:19.921 [eu-west-1a/ip-10-11-3-58] [OUT] com.hailocab.api.v1.customer → com.hailocab.service.nearest-driver.search - 14.02 ms 2014-09-07 02:20:19.921 [eu-west-1a/ip-10-11-2-203] [REP] com.hailocab.api.v1.customer → com.hailocab.service.nearest-driver.search - 15.48 ms 2014-09-07 02:20:19.941 [eu-west-1a/ip-10-11-2-203] [REQ] com.hailocab.api.v1.customer → com.hailocab.service.experiment.readlastupdated - 2014-09-07 02:20:19.945 [eu-west-1a/ip-10-11-2-214] [IN] com.hailocab.api.v1.customer → com.hailocab.service.experiment.readlastupdated - 2014-09-07 02:20:19.947 [eu-west-1a/ip-10-11-2-214] [OUT] com.hailocab.api.v1.customer → com.hailocab.service.experiment.readlastupdated - 1.82 ms 2014-09-07 02:20:19.947 [eu-west-1a/ip-10-11-2-203] [REP] com.hailocab.api.v1.customer → com.hailocab.service.experiment.readlastupdated - 6.01 ms 2014-09-07 02:20:19.948 [eu-west-1a/ip-10-11-2-203] [OUT] com.hailocab.hailo-2-api → com.hailocab.api.v1.customer.neardrivers - 80.46 ms 2014-09-07 02:20:19.950 [eu-west-1a/ip-10-11-3-51] [REP] com.hailocab.hailo-2-api → com.hailocab.api.v1.customer.neardrivers - 82.71 ms
  50. None
  51. None
  52. None
  53. None
  54. None
  55. SO, GO?

  56. SMALL SIMPLE EASY TO READ & LEARN

  57. CONCURRENCY PRIMITIVES INTERFACE SUPPORT

  58. EASY DEPLOYMENT & MANAGEMENT IN PRODUCTION AND AT SCALE

  59. СПАСИБО!

  60. IMAGES HMS President: Roger Marks Orbital Ion Cannon: www.rom.ac Go

    Gophers: Renee French Utopia/Dystopia: Dylan Glynn Control Room: NASA