Scaling Micro-Services in Go - HighLoad++ 2014

Presented at HighLoad++ 2014 in Moscow

As Hailo expanded to a global presence, we needed to re-evaluate our approach to technology. Moving from a monolithic PHP & Java application, and embracing a cloud native approach, Hailo transitioned to a new micro-service platform, running on three continents, built almost entirely in Go. This talk will cover how we developed and migrated to a micro service architecture, common pitfalls to avoid, and lessons learnt when developing high volume, low latency, distributed Go applications.

Поскольку рост проекта Hailo обеспечил ему глобальное присутствие, нам пришлось пересмотреть наш подход к технологиям. Мы решили уйти от монолитного приложения на PHP и Java и внедрить нативную поддержку «облаков», и проект Hailo перешёл на новую платформу микросервисов, работающую на трех континентах и почти полностью построенную на Go. В данном докладе я расскажу, как мы разработали архитектуру микросервисов и впоследствии перешли на неё, перечислю распространенные ошибки и объясню, как их избежать, и поделюсь уроками, которые мы извлекли из разработки на Go распределенных приложений, рассчитанных на обработку больших объёмов данных с минимальной задержкой.

Transcript

1. Scaling Micro-
   Services in Go
   Matt Heath, Hailo
   

   View Slide

2. View Slide

3. View Slide

4. View Slide

5. PHP

   API
   Load Balancer
   PHP

   API
   PHP

   API
   Java

   App
   PHP

   API
   PHP

   API
   Load Balancer
   PHP

   API
   MySQL
   Slave
   Redis MySQL
   eu-west-1
   

   View Slide

6. ALL
   HAIL
   THE
   MONOLITH
   

   View Slide

7. View Slide

8. PHP

   API
   Load Balancer
   PHP

   API
   PHP

   API
   Java

   App
   PHP

   API
   PHP

   API
   Load Balancer
   PHP

   API
   MySQL
   Slave
   Redis MySQL
   eu-west-1
   

   View Slide

9. PHP

   API
   MySQL
   PHP
   Service
   Java
   Service
   ELB
   per city multi-region
   ELB
   Cassandra
   PHP

   API
   PHP
   Service
   MySQL
   MySQL
   ELB
   ELB
   PHP

   API
   PHP

   API
   Java
   App
   Java
   App
   Java
   App
   

   View Slide

10. PHP

    API
    MySQL
    PHP
    Service
    Java
    Service
    ELB
    per city multi-region
    ELB
    Cassandra
    PHP
    Service
    MySQL
    MySQL
    ELB
    ELB
    PHP

    API
    PHP

    API
    Java
    App
    Java
    App
    Java
    App
    PHP

    API
    

    View Slide

11. PHP

    API
    MySQL
    PHP
    Service
    Java
    Service
    ELB
    per city multi-region
    ELB
    Cassandra
    PHP
    Service
    MySQL
    MySQL
    ELB
    ELB
    PHP

    API
    PHP

    API
    Java
    App
    Java
    App
    Java
    App
    PHP

    API
    

    View Slide

12. Communication paths
    with two programmers
    Communication paths
    with three programmers
    Communication paths
    with four programmers
    Communication paths
    with five programmers
    Communication paths
    with ten programmers
    

    View Slide

13. DEVELOPING NEW FEATURES
    UNCLEAR RESPONSIBILITIES
    SLOW FAILOVER
    LACK OF AUTOMATION
    

    View Slide

14. View Slide

15. “construct a highly agile and
    highly available service from
    ephemeral and assumed broken
    components”
    - Adrian Cockcroſt
    

    View Slide

16. Cassandra
    Go

    Service
    Go

    Service
    Java
    Service
    ELB
    Go “Thin” API
    us-east-1 eu-west-1
    Go

    Service
    Go

    Service
    Java
    Service
    ELB
    Go “Thin” API
    RabbitMQ Message Bus

    (federated clusters per AZ)
    RabbitMQ Message Bus

    (federated clusters per AZ)
    Cassandra
    

    View Slide

17. View Slide

18. Logic
    Service
    Handler
    Storage
    

    View Slide

19. Logic
    Storage
    Handler
    Library for building services that talk Protobuf via RMQ
    Service
    go-platform-layer
    

    View Slide

20. package handler
    import (
    "..."
    )
    func Foo(req *server.Request) (proto.Message, errors.Error) {
    }
    

    View Slide

21. service.Register(
    &service.Endpoint{
    Name: "foo",
    Mean: 100,
    Upper95: 200,
    Handler: handler.Foo,
    }
    )
    

    View Slide

22. Logic
    go-service-layer
    Storage
    Handler
    Library for building services that talk Protobuf via RMQ
    Self-configuring external service adapters
    Service
    go-platform-layer
    

    View Slide

23. package handler
    import (
    "github.com/hailocab/gomemcache/memcache"
    )
    func Foo(req *server.Request) (proto.Message, errors.Error) {
    item, err := memcache.Get("key")
    if err != nil {
    // handle error
    }
    // ...
    }
    

    View Slide

24. Logic
    go-service-layer
    Storage
    go-platform-layer
    Handler
    Library for building services that talk Protobuf via RMQ
    Self-configuring external service adapters
    Services get for free:
    • Provisioning
    • Service discovery
    • Configuration
    • Monitoring
    • Authentication/authorisation
    • AB testing
    • Self configuring connectivity 

    to third-party services
    Service
    

    View Slide

25. View Slide

26. View Slide

27. View Slide

28. Provisioning Service Provisioning Service Provisioning Service
    CI Pipeline (Janky/Jenkins)
    Amazon S3
    Provisioning Manager / Scheduler
    

    View Slide

29. Provisioning Service Provisioning Service Provisioning Service
    CI Pipeline (Janky/Jenkins)
    Amazon S3
    Provisioning Manager / Scheduler
    

    View Slide

30. Provisioning Service
    Rabbit MQ
    Discovery
    Service
    Provisioning Service
    New

    Service
    AUTOMATIC SERVICE DISCOVERY
    

    View Slide

31. Provisioning Service
    Rabbit MQ
    Discovery
    Service
    Binding

    Service
    Provisioning Service
    New

    Service
    AUTOMATIC SERVICE DISCOVERY
    

    View Slide

32. SMALL INDEPENDENT SERVICES
    SINGLE RESPONSIBILITY
    EASE PAIN, SCALE RAPIDLY
    

    View Slide

33. CLOUD NATIVE / ANTIFRAGILE
    EXPECT FAILURE
    AUTOMATE EVERYTHING
    

    View Slide

34. TESTING
    

    View Slide

35. LOAD
    FAILURE
    DEGRADATION
    

    View Slide

36. 15,000 JOBS/HOUR
    50,000 DRIVERS
    30,000+ REQ/S
    

    View Slide

37. CONTINUOUS
    PRODUCTION
    TESTING
    

    View Slide

38. MONITORING
    

    View Slide

39. Provisioning Service
    Rabbit MQ
    Monitoring

    Service
    Provisioning Service
    

    Service
    Publish
    Healthchecks
    AUTOMATIC HEALTHCHECK REGISTRATION
    

    View Slide

40. View Slide

41. View Slide

42. View Slide

43. DISTRIBUTED

    TRACING
    

    View Slide

44. hailo~2~api api.v1.customer service.customer
    hailo~2~api api.v1.customer service.customer
    

    View Slide

45. hailo~2~api api.v1.customer service.customer
    hailo~2~api api.v1.customer service.customer
    REQ
    REP
    REQ
    REP
    IN
    OUT
    IN
    OUT
    

    View Slide

46. {
    "timestamp": 1410262798427145176,
    "traceId": "d30479b8-1491-4390-7cf5-4cd14bc4b765",
    "type": "OUT",
    "messageId": "a661f9ef-774c-49b2-6e74-cfed65f7d120",
    "parentMessageId": "",
    "from": "com.hailocab.hshell",
    "to": "com.hailocab.service.nearest-driver.search",
    "hostname": "ip-10-13-2-251",
    "az": "eu-west-1a",
    "handlerInstanceId": “server-com.hailocab.service.nearest-driver-18bd089e-8ef1-4ca1-75cb-8...c”,
    "duration": 11222094
    }
    {
    "timestamp": 1410262798416053450,
    "traceId": "d30479b8-1491-4390-7cf5-4cd14bc4b765",
    "type": "REQ",
    "messageId": "6404dd1e-c995-48a9-73dc-9edb1380f0bf",
    "parentMessageId": "a661f9ef-774c-49b2-6e74-cfed65f7d120",
    "from": "com.hailocab.service.nearest-driver",
    "to": "com.hailocab.service.zoning.search",
    "hostname": "ip-10-13-2-251",
    "az": "eu-west-1a"
    }
    

    View Slide

47. Phosphor
    Host Instances
    Publish
    Service A
    Trace Library
    goroutine
    chan
    UDP
    Service B
    Trace Library
    goroutine
    chan
    UDP
    Trace
    Service
    In-memory
    Aggregates
    Optional

    persistant
    storage
    Dashboards
    Monitoring
    

    View Slide

48. var traceChan chan []byte
    func init() {
    // Use a buffered channel
    traceChan = make(chan []byte, 200)
    // Fire off a background worker for this channel
    defaultClient = NewClient(traceChan)
    go defaultClient.publisher()
    }
    // Send, drops trace if the backend is at capacity
    func Send(msg []byte) {
    select {
    case traceChan <- msg:
    // Success
    default:
    // Default case fired if channel is full
    // Ensures this is non blocking
    }
    }
    

    View Slide

49. Tracing: 33eda743-f124-435c-71fc-3c872bbc98e6
    2014-09-07 02:20:19.867 [/] [START] → -
    2014-09-07 02:20:19.867 [eu-west-1a/ip-10-11-3-51] [REQ] com.hailocab.hailo-2-api → com.hailocab.api.v1.customer.neardrivers -
    2014-09-07 02:20:19.867 [eu-west-1a/ip-10-11-2-203] [IN] com.hailocab.hailo-2-api → com.hailocab.api.v1.customer.neardrivers -
    2014-09-07 02:20:19.868 [eu-west-1a/ip-10-11-2-203] [REQ] com.hailocab.api.v1.customer → com.hailocab.service.feature-flags.features -
    2014-09-07 02:20:19.869 [eu-west-1a/ip-10-11-3-111] [IN] com.hailocab.api.v1.customer → com.hailocab.service.feature-flags.features -
    2014-09-07 02:20:19.876 [eu-west-1a/ip-10-11-3-111] [REQ] com.hailocab.service.feature-flags → com.hailocab.service.hob.list -
    2014-09-07 02:20:19.877 [eu-west-1a/ip-10-11-3-168] [IN] com.hailocab.service.hob → com.hailocab.service.config.compile -
    2014-09-07 02:20:19.877 [eu-west-1a/ip-10-11-3-111] [IN] com.hailocab.service.feature-flags → com.hailocab.service.hob.list -
    2014-09-07 02:20:19.877 [eu-west-1a/ip-10-11-3-111] [REQ] com.hailocab.service.hob → com.hailocab.service.config.compile -
    2014-09-07 02:20:19.883 [eu-west-1a/ip-10-11-3-168] [OUT] com.hailocab.service.hob → com.hailocab.service.config.compile - 5.59 ms
    2014-09-07 02:20:19.886 [eu-west-1a/ip-10-11-3-111] [REP] com.hailocab.service.hob → com.hailocab.service.config.compile - 8.40 ms
    2014-09-07 02:20:19.887 [eu-west-1a/ip-10-11-3-111] [OUT] com.hailocab.service.feature-flags → com.hailocab.service.hob.list - 9.72 ms
    2014-09-07 02:20:19.889 [eu-west-1a/ip-10-11-3-111] [REP] com.hailocab.service.feature-flags → com.hailocab.service.hob.list - 13.23 ms
    2014-09-07 02:20:19.889 [eu-west-1a/ip-10-11-3-111] [OUT] com.hailocab.api.v1.customer → com.hailocab.service.feature-flags.features - 20.58 ms
    2014-09-07 02:20:19.890 [eu-west-1a/ip-10-11-2-203] [REP] com.hailocab.api.v1.customer → com.hailocab.service.feature-flags.features - 22.59 ms
    2014-09-07 02:20:19.902 [eu-west-1a/ip-10-11-2-203] [REQ] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare -
    2014-09-07 02:20:19.903 [eu-west-1a/ip-10-11-2-203] [REQ] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare -
    2014-09-07 02:20:19.903 [eu-west-1a/ip-10-11-2-203] [REQ] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare -
    2014-09-07 02:20:19.904 [eu-west-1a/ip-10-11-3-111] [IN] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare -
    2014-09-07 02:20:19.904 [eu-west-1a/ip-10-11-3-111] [OUT] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare - 0.36 ms
    2014-09-07 02:20:19.905 [eu-west-1a/ip-10-11-2-203] [REP] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare - 1.97 ms
    2014-09-07 02:20:19.905 [eu-west-1a/ip-10-11-2-214] [IN] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare -
    2014-09-07 02:20:19.905 [eu-west-1a/ip-10-11-2-203] [REQ] com.hailocab.api.v1.customer → com.hailocab.service.nearest-driver.search -
    2014-09-07 02:20:19.905 [eu-west-1a/ip-10-11-2-214] [OUT] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare - 0.10 ms

    ERR - com.hailocab.service.fare.basefare: Missing config at xxx
    2014-09-07 02:20:19.906 [eu-west-1a/ip-10-11-2-214] [IN] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare -
    2014-09-07 02:20:19.906 [eu-west-1a/ip-10-11-2-214] [OUT] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare - 0.06 ms 

    ERR - com.hailocab.service.fare.basefare: Missing config at xxx
    2014-09-07 02:20:19.907 [eu-west-1a/ip-10-11-3-58] [IN] com.hailocab.api.v1.customer → com.hailocab.service.nearest-driver.search -
    2014-09-07 02:20:19.907 [eu-west-1a/ip-10-11-3-58] [REQ] com.hailocab.service.nearest-driver → com.hailocab.service.zoning.search -
    2014-09-07 02:20:19.908 [eu-west-1a/ip-10-11-3-58] [IN] com.hailocab.service.nearest-driver → com.hailocab.service.zoning.search -
    2014-09-07 02:20:19.908 [eu-west-1a/ip-10-11-3-58] [OUT] com.hailocab.service.nearest-driver → com.hailocab.service.zoning.search - 0.20 ms
    2014-09-07 02:20:19.909 [eu-west-1a/ip-10-11-3-58] [REP] com.hailocab.service.nearest-driver → com.hailocab.service.zoning.search - 2.25 ms
    2014-09-07 02:20:19.909 [eu-west-1a/ip-10-11-3-58] [REQ] com.hailocab.service.nearest-driver → com.hailocab.service.raziel.multisearch -
    2014-09-07 02:20:19.912 [eu-west-1a/ip-10-11-3-227] [IN] com.hailocab.service.nearest-driver → com.hailocab.service.raziel.multisearch -
    2014-09-07 02:20:19.919 [eu-west-1a/ip-10-11-3-58] [REP] com.hailocab.service.nearest-driver → com.hailocab.service.raziel.multisearch - 9.46 ms
    2014-09-07 02:20:19.919 [eu-west-1a/ip-10-11-3-58] [REQ] com.hailocab.service.nearest-driver → com.hailocab.service.eta.multitraveltime -
    2014-09-07 02:20:19.919 [eu-west-1a/ip-10-11-3-227] [OUT] com.hailocab.service.nearest-driver → com.hailocab.service.raziel.multisearch - 7.58 ms
    2014-09-07 02:20:19.920 [eu-west-1a/ip-10-11-3-58] [IN] com.hailocab.service.nearest-driver → com.hailocab.service.eta.multitraveltime -
    2014-09-07 02:20:19.920 [eu-west-1a/ip-10-11-3-58] [OUT] com.hailocab.service.nearest-driver → com.hailocab.service.eta.multitraveltime - 0.06 ms
    2014-09-07 02:20:19.921 [eu-west-1a/ip-10-11-3-58] [REP] com.hailocab.service.nearest-driver → com.hailocab.service.eta.multitraveltime - 1.77 ms
    2014-09-07 02:20:19.921 [eu-west-1a/ip-10-11-3-58] [OUT] com.hailocab.api.v1.customer → com.hailocab.service.nearest-driver.search - 14.02 ms
    2014-09-07 02:20:19.921 [eu-west-1a/ip-10-11-2-203] [REP] com.hailocab.api.v1.customer → com.hailocab.service.nearest-driver.search - 15.48 ms
    2014-09-07 02:20:19.941 [eu-west-1a/ip-10-11-2-203] [REQ] com.hailocab.api.v1.customer → com.hailocab.service.experiment.readlastupdated -
    2014-09-07 02:20:19.945 [eu-west-1a/ip-10-11-2-214] [IN] com.hailocab.api.v1.customer → com.hailocab.service.experiment.readlastupdated -
    2014-09-07 02:20:19.947 [eu-west-1a/ip-10-11-2-214] [OUT] com.hailocab.api.v1.customer → com.hailocab.service.experiment.readlastupdated - 1.82 ms
    2014-09-07 02:20:19.947 [eu-west-1a/ip-10-11-2-203] [REP] com.hailocab.api.v1.customer → com.hailocab.service.experiment.readlastupdated - 6.01 ms
    2014-09-07 02:20:19.948 [eu-west-1a/ip-10-11-2-203] [OUT] com.hailocab.hailo-2-api → com.hailocab.api.v1.customer.neardrivers - 80.46 ms
    2014-09-07 02:20:19.950 [eu-west-1a/ip-10-11-3-51] [REP] com.hailocab.hailo-2-api → com.hailocab.api.v1.customer.neardrivers - 82.71 ms
    

    View Slide

50. View Slide

51. View Slide

52. View Slide

53. View Slide

54. View Slide

55. SO, GO?
    

    View Slide

56. SMALL
    SIMPLE
    EASY TO READ & LEARN
    

    View Slide

57. CONCURRENCY PRIMITIVES
    INTERFACE SUPPORT
    

    View Slide

58. EASY DEPLOYMENT &
    MANAGEMENT IN PRODUCTION
    AND AT SCALE
    

    View Slide

59. СПАСИБО!
    

    View Slide

60. IMAGES
    HMS President: Roger Marks
    Orbital Ion Cannon: www.rom.ac
    Go Gophers: Renee French
    Utopia/Dystopia: Dylan Glynn
    Control Room: NASA
    

    View Slide