Scaling Micro-Services in Go - HighLoad++ 2014

Presented at HighLoad++ 2014 in Moscow

As Hailo expanded to a global presence, we needed to re-evaluate our approach to technology. Moving from a monolithic PHP & Java application, and embracing a cloud native approach, Hailo transitioned to a new micro-service platform, running on three continents, built almost entirely in Go. This talk will cover how we developed and migrated to a micro service architecture, common pitfalls to avoid, and lessons learnt when developing high volume, low latency, distributed Go applications.

Поскольку рост проекта Hailo обеспечил ему глобальное присутствие, нам пришлось пересмотреть наш подход к технологиям. Мы решили уйти от монолитного приложения на PHP и Java и внедрить нативную поддержку «облаков», и проект Hailo перешёл на новую платформу микросервисов, работающую на трех континентах и почти полностью построенную на Go. В данном докладе я расскажу, как мы разработали архитектуру микросервисов и впоследствии перешли на неё, перечислю распространенные ошибки и объясню, как их избежать, и поделюсь уроками, которые мы извлекли из разработки на Go распределенных приложений, рассчитанных на обработку больших объёмов данных с минимальной задержкой.

Transcript

1. Scaling Micro- Services in Go Matt Heath, Hailo
5. PHP
 API Load Balancer PHP
 API PHP
 API Java
 App PHP
 API PHP
 API Load Balancer PHP
 API MySQL Slave Redis MySQL eu-west-1
6. ALL HAIL THE MONOLITH
8. PHP
 API Load Balancer PHP
 API PHP
 API Java
 App PHP
 API PHP
 API Load Balancer PHP
 API MySQL Slave Redis MySQL eu-west-1
9. PHP
 API MySQL PHP Service Java Service ELB per city multi-region ELB Cassandra PHP
 API PHP Service MySQL MySQL ELB ELB PHP
 API PHP
 API Java App Java App Java App
10. PHP
 API MySQL PHP Service Java Service ELB per city multi-region ELB Cassandra PHP Service MySQL MySQL ELB ELB PHP
 API PHP
 API Java App Java App Java App PHP
 API
11. PHP
 API MySQL PHP Service Java Service ELB per city multi-region ELB Cassandra PHP Service MySQL MySQL ELB ELB PHP
 API PHP
 API Java App Java App Java App PHP
 API
12. Communication paths with two programmers Communication paths with three programmers Communication paths with four programmers Communication paths with five programmers Communication paths with ten programmers
13. DEVELOPING NEW FEATURES UNCLEAR RESPONSIBILITIES SLOW FAILOVER LACK OF AUTOMATION
15. “construct a highly agile and highly available service from ephemeral and assumed broken components” - Adrian Cockcroſt
16. Cassandra Go
 Service Go
 Service Java Service ELB Go “Thin” API us-east-1 eu-west-1 Go
 Service Go
 Service Java Service ELB Go “Thin” API RabbitMQ Message Bus
 (federated clusters per AZ) RabbitMQ Message Bus
 (federated clusters per AZ) Cassandra
18. Logic Service Handler Storage
19. Logic Storage Handler Library for building services that talk Protobuf via RMQ Service go-platform-layer
20. package handler import ( "..." ) func Foo(req *server.Request) (proto.Message, errors.Error) { }
21. service.Register( &service.Endpoint{ Name: "foo", Mean: 100, Upper95: 200, Handler: handler.Foo, } )
22. Logic go-service-layer Storage Handler Library for building services that talk Protobuf via RMQ Self-configuring external service adapters Service go-platform-layer
23. package handler import ( "github.com/hailocab/gomemcache/memcache" ) func Foo(req *server.Request) (proto.Message, errors.Error) { item, err := memcache.Get("key") if err != nil { // handle error } // ... }
24. Logic go-service-layer Storage go-platform-layer Handler Library for building services that talk Protobuf via RMQ Self-configuring external service adapters Services get for free: • Provisioning • Service discovery • Configuration • Monitoring • Authentication/authorisation • AB testing • Self configuring connectivity 
 to third-party services Service
28. Provisioning Service Provisioning Service Provisioning Service CI Pipeline (Janky/Jenkins) Amazon S3 Provisioning Manager / Scheduler
29. Provisioning Service Provisioning Service Provisioning Service CI Pipeline (Janky/Jenkins) Amazon S3 Provisioning Manager / Scheduler
30. Provisioning Service Rabbit MQ Discovery Service Provisioning Service New
 Service AUTOMATIC SERVICE DISCOVERY
31. Provisioning Service Rabbit MQ Discovery Service Binding
 Service Provisioning Service New
 Service AUTOMATIC SERVICE DISCOVERY
32. SMALL INDEPENDENT SERVICES SINGLE RESPONSIBILITY EASE PAIN, SCALE RAPIDLY
33. CLOUD NATIVE / ANTIFRAGILE EXPECT FAILURE AUTOMATE EVERYTHING
34. TESTING
35. LOAD FAILURE DEGRADATION
36. 15,000 JOBS/HOUR 50,000 DRIVERS 30,000+ REQ/S
37. CONTINUOUS PRODUCTION TESTING
38. MONITORING
39. Provisioning Service Rabbit MQ Monitoring
 Service Provisioning Service 
 Service Publish Healthchecks AUTOMATIC HEALTHCHECK REGISTRATION
43. DISTRIBUTED
 TRACING
44. hailo~2~api api.v1.customer service.customer hailo~2~api api.v1.customer service.customer
45. hailo~2~api api.v1.customer service.customer hailo~2~api api.v1.customer service.customer REQ REP REQ REP IN OUT IN OUT
46. { "timestamp": 1410262798427145176, "traceId": "d30479b8-1491-4390-7cf5-4cd14bc4b765", "type": "OUT", "messageId": "a661f9ef-774c-49b2-6e74-cfed65f7d120", "parentMessageId": "", "from": "com.hailocab.hshell", "to": "com.hailocab.service.nearest-driver.search", "hostname": "ip-10-13-2-251", "az": "eu-west-1a", "handlerInstanceId": “server-com.hailocab.service.nearest-driver-18bd089e-8ef1-4ca1-75cb-8...c”, "duration": 11222094 } { "timestamp": 1410262798416053450, "traceId": "d30479b8-1491-4390-7cf5-4cd14bc4b765", "type": "REQ", "messageId": "6404dd1e-c995-48a9-73dc-9edb1380f0bf", "parentMessageId": "a661f9ef-774c-49b2-6e74-cfed65f7d120", "from": "com.hailocab.service.nearest-driver", "to": "com.hailocab.service.zoning.search", "hostname": "ip-10-13-2-251", "az": "eu-west-1a" }
47. Phosphor Host Instances Publish Service A Trace Library goroutine chan UDP Service B Trace Library goroutine chan UDP Trace Service In-memory Aggregates Optional
 persistant storage Dashboards Monitoring
48. var traceChan chan []byte func init() { // Use a buffered channel traceChan = make(chan []byte, 200) // Fire off a background worker for this channel defaultClient = NewClient(traceChan) go defaultClient.publisher() } // Send, drops trace if the backend is at capacity func Send(msg []byte) { select { case traceChan <- msg: // Success default: // Default case fired if channel is full // Ensures this is non blocking } }
49. Tracing: 33eda743-f124-435c-71fc-3c872bbc98e6 2014-09-07 02:20:19.867 [/] [START] → - 2014-09-07 02:20:19.867 [eu-west-1a/ip-10-11-3-51] [REQ] com.hailocab.hailo-2-api → com.hailocab.api.v1.customer.neardrivers - 2014-09-07 02:20:19.867 [eu-west-1a/ip-10-11-2-203] [IN] com.hailocab.hailo-2-api → com.hailocab.api.v1.customer.neardrivers - 2014-09-07 02:20:19.868 [eu-west-1a/ip-10-11-2-203] [REQ] com.hailocab.api.v1.customer → com.hailocab.service.feature-flags.features - 2014-09-07 02:20:19.869 [eu-west-1a/ip-10-11-3-111] [IN] com.hailocab.api.v1.customer → com.hailocab.service.feature-flags.features - 2014-09-07 02:20:19.876 [eu-west-1a/ip-10-11-3-111] [REQ] com.hailocab.service.feature-flags → com.hailocab.service.hob.list - 2014-09-07 02:20:19.877 [eu-west-1a/ip-10-11-3-168] [IN] com.hailocab.service.hob → com.hailocab.service.config.compile - 2014-09-07 02:20:19.877 [eu-west-1a/ip-10-11-3-111] [IN] com.hailocab.service.feature-flags → com.hailocab.service.hob.list - 2014-09-07 02:20:19.877 [eu-west-1a/ip-10-11-3-111] [REQ] com.hailocab.service.hob → com.hailocab.service.config.compile - 2014-09-07 02:20:19.883 [eu-west-1a/ip-10-11-3-168] [OUT] com.hailocab.service.hob → com.hailocab.service.config.compile - 5.59 ms 2014-09-07 02:20:19.886 [eu-west-1a/ip-10-11-3-111] [REP] com.hailocab.service.hob → com.hailocab.service.config.compile - 8.40 ms 2014-09-07 02:20:19.887 [eu-west-1a/ip-10-11-3-111] [OUT] com.hailocab.service.feature-flags → com.hailocab.service.hob.list - 9.72 ms 2014-09-07 02:20:19.889 [eu-west-1a/ip-10-11-3-111] [REP] com.hailocab.service.feature-flags → com.hailocab.service.hob.list - 13.23 ms 2014-09-07 02:20:19.889 [eu-west-1a/ip-10-11-3-111] [OUT] com.hailocab.api.v1.customer → com.hailocab.service.feature-flags.features - 20.58 ms 2014-09-07 02:20:19.890 [eu-west-1a/ip-10-11-2-203] [REP] com.hailocab.api.v1.customer → com.hailocab.service.feature-flags.features - 22.59 ms 2014-09-07 02:20:19.902 [eu-west-1a/ip-10-11-2-203] [REQ] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare - 2014-09-07 02:20:19.903 [eu-west-1a/ip-10-11-2-203] [REQ] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare - 2014-09-07 02:20:19.903 [eu-west-1a/ip-10-11-2-203] [REQ] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare - 2014-09-07 02:20:19.904 [eu-west-1a/ip-10-11-3-111] [IN] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare - 2014-09-07 02:20:19.904 [eu-west-1a/ip-10-11-3-111] [OUT] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare - 0.36 ms 2014-09-07 02:20:19.905 [eu-west-1a/ip-10-11-2-203] [REP] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare - 1.97 ms 2014-09-07 02:20:19.905 [eu-west-1a/ip-10-11-2-214] [IN] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare - 2014-09-07 02:20:19.905 [eu-west-1a/ip-10-11-2-203] [REQ] com.hailocab.api.v1.customer → com.hailocab.service.nearest-driver.search - 2014-09-07 02:20:19.905 [eu-west-1a/ip-10-11-2-214] [OUT] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare - 0.10 ms
 ERR - com.hailocab.service.fare.basefare: Missing config at xxx 2014-09-07 02:20:19.906 [eu-west-1a/ip-10-11-2-214] [IN] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare - 2014-09-07 02:20:19.906 [eu-west-1a/ip-10-11-2-214] [OUT] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare - 0.06 ms 
 ERR - com.hailocab.service.fare.basefare: Missing config at xxx 2014-09-07 02:20:19.907 [eu-west-1a/ip-10-11-3-58] [IN] com.hailocab.api.v1.customer → com.hailocab.service.nearest-driver.search - 2014-09-07 02:20:19.907 [eu-west-1a/ip-10-11-3-58] [REQ] com.hailocab.service.nearest-driver → com.hailocab.service.zoning.search - 2014-09-07 02:20:19.908 [eu-west-1a/ip-10-11-3-58] [IN] com.hailocab.service.nearest-driver → com.hailocab.service.zoning.search - 2014-09-07 02:20:19.908 [eu-west-1a/ip-10-11-3-58] [OUT] com.hailocab.service.nearest-driver → com.hailocab.service.zoning.search - 0.20 ms 2014-09-07 02:20:19.909 [eu-west-1a/ip-10-11-3-58] [REP] com.hailocab.service.nearest-driver → com.hailocab.service.zoning.search - 2.25 ms 2014-09-07 02:20:19.909 [eu-west-1a/ip-10-11-3-58] [REQ] com.hailocab.service.nearest-driver → com.hailocab.service.raziel.multisearch - 2014-09-07 02:20:19.912 [eu-west-1a/ip-10-11-3-227] [IN] com.hailocab.service.nearest-driver → com.hailocab.service.raziel.multisearch - 2014-09-07 02:20:19.919 [eu-west-1a/ip-10-11-3-58] [REP] com.hailocab.service.nearest-driver → com.hailocab.service.raziel.multisearch - 9.46 ms 2014-09-07 02:20:19.919 [eu-west-1a/ip-10-11-3-58] [REQ] com.hailocab.service.nearest-driver → com.hailocab.service.eta.multitraveltime - 2014-09-07 02:20:19.919 [eu-west-1a/ip-10-11-3-227] [OUT] com.hailocab.service.nearest-driver → com.hailocab.service.raziel.multisearch - 7.58 ms 2014-09-07 02:20:19.920 [eu-west-1a/ip-10-11-3-58] [IN] com.hailocab.service.nearest-driver → com.hailocab.service.eta.multitraveltime - 2014-09-07 02:20:19.920 [eu-west-1a/ip-10-11-3-58] [OUT] com.hailocab.service.nearest-driver → com.hailocab.service.eta.multitraveltime - 0.06 ms 2014-09-07 02:20:19.921 [eu-west-1a/ip-10-11-3-58] [REP] com.hailocab.service.nearest-driver → com.hailocab.service.eta.multitraveltime - 1.77 ms 2014-09-07 02:20:19.921 [eu-west-1a/ip-10-11-3-58] [OUT] com.hailocab.api.v1.customer → com.hailocab.service.nearest-driver.search - 14.02 ms 2014-09-07 02:20:19.921 [eu-west-1a/ip-10-11-2-203] [REP] com.hailocab.api.v1.customer → com.hailocab.service.nearest-driver.search - 15.48 ms 2014-09-07 02:20:19.941 [eu-west-1a/ip-10-11-2-203] [REQ] com.hailocab.api.v1.customer → com.hailocab.service.experiment.readlastupdated - 2014-09-07 02:20:19.945 [eu-west-1a/ip-10-11-2-214] [IN] com.hailocab.api.v1.customer → com.hailocab.service.experiment.readlastupdated - 2014-09-07 02:20:19.947 [eu-west-1a/ip-10-11-2-214] [OUT] com.hailocab.api.v1.customer → com.hailocab.service.experiment.readlastupdated - 1.82 ms 2014-09-07 02:20:19.947 [eu-west-1a/ip-10-11-2-203] [REP] com.hailocab.api.v1.customer → com.hailocab.service.experiment.readlastupdated - 6.01 ms 2014-09-07 02:20:19.948 [eu-west-1a/ip-10-11-2-203] [OUT] com.hailocab.hailo-2-api → com.hailocab.api.v1.customer.neardrivers - 80.46 ms 2014-09-07 02:20:19.950 [eu-west-1a/ip-10-11-3-51] [REP] com.hailocab.hailo-2-api → com.hailocab.api.v1.customer.neardrivers - 82.71 ms
55. SO, GO?
56. SMALL SIMPLE EASY TO READ & LEARN
57. CONCURRENCY PRIMITIVES INTERFACE SUPPORT
58. EASY DEPLOYMENT & MANAGEMENT IN PRODUCTION AND AT SCALE
59. СПАСИБО!
60. IMAGES HMS President: Roger Marks Orbital Ion Cannon: www.rom.ac Go Gophers: Renee French Utopia/Dystopia: Dylan Glynn Control Room: NASA