Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Scaling Micro-Services in Go - HighLoad++ 2014

mattheath
October 31, 2014

Scaling Micro-Services in Go - HighLoad++ 2014

Presented at HighLoad++ 2014 in Moscow

As Hailo expanded to a global presence, we needed to re-evaluate our approach to technology. Moving from a monolithic PHP & Java application, and embracing a cloud native approach, Hailo transitioned to a new micro-service platform, running on three continents, built almost entirely in Go. This talk will cover how we developed and migrated to a micro service architecture, common pitfalls to avoid, and lessons learnt when developing high volume, low latency, distributed Go applications.

Поскольку рост проекта Hailo обеспечил ему глобальное присутствие, нам пришлось пересмотреть наш подход к технологиям. Мы решили уйти от монолитного приложения на PHP и Java и внедрить нативную поддержку «облаков», и проект Hailo перешёл на новую платформу микросервисов, работающую на трех континентах и почти полностью построенную на Go. В данном докладе я расскажу, как мы разработали архитектуру микросервисов и впоследствии перешли на неё, перечислю распространенные ошибки и объясню, как их избежать, и поделюсь уроками, которые мы извлекли из разработки на Go распределенных приложений, рассчитанных на обработку больших объёмов данных с минимальной задержкой.

mattheath

October 31, 2014
Tweet

More Decks by mattheath

Other Decks in Programming

Transcript

  1. Scaling Micro-
    Services in Go
    Matt Heath, Hailo

    View Slide

  2. View Slide

  3. View Slide

  4. View Slide

  5. PHP

    API
    Load Balancer
    PHP

    API
    PHP

    API
    Java

    App
    PHP

    API
    PHP

    API
    Load Balancer
    PHP

    API
    MySQL
    Slave
    Redis MySQL
    eu-west-1

    View Slide

  6. ALL
    HAIL
    THE
    MONOLITH

    View Slide

  7. View Slide

  8. PHP

    API
    Load Balancer
    PHP

    API
    PHP

    API
    Java

    App
    PHP

    API
    PHP

    API
    Load Balancer
    PHP

    API
    MySQL
    Slave
    Redis MySQL
    eu-west-1

    View Slide

  9. PHP

    API
    MySQL
    PHP
    Service
    Java
    Service
    ELB
    per city multi-region
    ELB
    Cassandra
    PHP

    API
    PHP
    Service
    MySQL
    MySQL
    ELB
    ELB
    PHP

    API
    PHP

    API
    Java
    App
    Java
    App
    Java
    App

    View Slide

  10. PHP

    API
    MySQL
    PHP
    Service
    Java
    Service
    ELB
    per city multi-region
    ELB
    Cassandra
    PHP
    Service
    MySQL
    MySQL
    ELB
    ELB
    PHP

    API
    PHP

    API
    Java
    App
    Java
    App
    Java
    App
    PHP

    API

    View Slide

  11. PHP

    API
    MySQL
    PHP
    Service
    Java
    Service
    ELB
    per city multi-region
    ELB
    Cassandra
    PHP
    Service
    MySQL
    MySQL
    ELB
    ELB
    PHP

    API
    PHP

    API
    Java
    App
    Java
    App
    Java
    App
    PHP

    API

    View Slide

  12. Communication paths
    with two programmers
    Communication paths
    with three programmers
    Communication paths
    with four programmers
    Communication paths
    with five programmers
    Communication paths
    with ten programmers

    View Slide

  13. DEVELOPING NEW FEATURES
    UNCLEAR RESPONSIBILITIES
    SLOW FAILOVER
    LACK OF AUTOMATION

    View Slide

  14. View Slide

  15. “construct a highly agile and
    highly available service from
    ephemeral and assumed broken
    components”
    - Adrian Cockcroſt

    View Slide

  16. Cassandra
    Go

    Service
    Go

    Service
    Java
    Service
    ELB
    Go “Thin” API
    us-east-1 eu-west-1
    Go

    Service
    Go

    Service
    Java
    Service
    ELB
    Go “Thin” API
    RabbitMQ Message Bus

    (federated clusters per AZ)
    RabbitMQ Message Bus

    (federated clusters per AZ)
    Cassandra

    View Slide

  17. View Slide

  18. Logic
    Service
    Handler
    Storage

    View Slide

  19. Logic
    Storage
    Handler
    Library for building services that talk Protobuf via RMQ
    Service
    go-platform-layer

    View Slide

  20. package handler
    import (
    "..."
    )
    func Foo(req *server.Request) (proto.Message, errors.Error) {
    }

    View Slide

  21. service.Register(
    &service.Endpoint{
    Name: "foo",
    Mean: 100,
    Upper95: 200,
    Handler: handler.Foo,
    }
    )

    View Slide

  22. Logic
    go-service-layer
    Storage
    Handler
    Library for building services that talk Protobuf via RMQ
    Self-configuring external service adapters
    Service
    go-platform-layer

    View Slide

  23. package handler
    import (
    "github.com/hailocab/gomemcache/memcache"
    )
    func Foo(req *server.Request) (proto.Message, errors.Error) {
    item, err := memcache.Get("key")
    if err != nil {
    // handle error
    }
    // ...
    }

    View Slide

  24. Logic
    go-service-layer
    Storage
    go-platform-layer
    Handler
    Library for building services that talk Protobuf via RMQ
    Self-configuring external service adapters
    Services get for free:
    • Provisioning
    • Service discovery
    • Configuration
    • Monitoring
    • Authentication/authorisation
    • AB testing
    • Self configuring connectivity 

    to third-party services
    Service

    View Slide

  25. View Slide

  26. View Slide

  27. View Slide

  28. Provisioning Service Provisioning Service Provisioning Service
    CI Pipeline (Janky/Jenkins)
    Amazon S3
    Provisioning Manager / Scheduler

    View Slide

  29. Provisioning Service Provisioning Service Provisioning Service
    CI Pipeline (Janky/Jenkins)
    Amazon S3
    Provisioning Manager / Scheduler

    View Slide

  30. Provisioning Service
    Rabbit MQ
    Discovery
    Service
    Provisioning Service
    New

    Service
    AUTOMATIC SERVICE DISCOVERY

    View Slide

  31. Provisioning Service
    Rabbit MQ
    Discovery
    Service
    Binding

    Service
    Provisioning Service
    New

    Service
    AUTOMATIC SERVICE DISCOVERY

    View Slide

  32. SMALL INDEPENDENT SERVICES
    SINGLE RESPONSIBILITY
    EASE PAIN, SCALE RAPIDLY

    View Slide

  33. CLOUD NATIVE / ANTIFRAGILE
    EXPECT FAILURE
    AUTOMATE EVERYTHING

    View Slide

  34. TESTING

    View Slide

  35. LOAD
    FAILURE
    DEGRADATION

    View Slide

  36. 15,000 JOBS/HOUR
    50,000 DRIVERS
    30,000+ REQ/S

    View Slide

  37. CONTINUOUS
    PRODUCTION
    TESTING

    View Slide

  38. MONITORING

    View Slide

  39. Provisioning Service
    Rabbit MQ
    Monitoring

    Service
    Provisioning Service

    Service
    Publish
    Healthchecks
    AUTOMATIC HEALTHCHECK REGISTRATION

    View Slide

  40. View Slide

  41. View Slide

  42. View Slide

  43. DISTRIBUTED

    TRACING

    View Slide

  44. hailo~2~api api.v1.customer service.customer
    hailo~2~api api.v1.customer service.customer

    View Slide

  45. hailo~2~api api.v1.customer service.customer
    hailo~2~api api.v1.customer service.customer
    REQ
    REP
    REQ
    REP
    IN
    OUT
    IN
    OUT

    View Slide

  46. {
    "timestamp": 1410262798427145176,
    "traceId": "d30479b8-1491-4390-7cf5-4cd14bc4b765",
    "type": "OUT",
    "messageId": "a661f9ef-774c-49b2-6e74-cfed65f7d120",
    "parentMessageId": "",
    "from": "com.hailocab.hshell",
    "to": "com.hailocab.service.nearest-driver.search",
    "hostname": "ip-10-13-2-251",
    "az": "eu-west-1a",
    "handlerInstanceId": “server-com.hailocab.service.nearest-driver-18bd089e-8ef1-4ca1-75cb-8...c”,
    "duration": 11222094
    }
    {
    "timestamp": 1410262798416053450,
    "traceId": "d30479b8-1491-4390-7cf5-4cd14bc4b765",
    "type": "REQ",
    "messageId": "6404dd1e-c995-48a9-73dc-9edb1380f0bf",
    "parentMessageId": "a661f9ef-774c-49b2-6e74-cfed65f7d120",
    "from": "com.hailocab.service.nearest-driver",
    "to": "com.hailocab.service.zoning.search",
    "hostname": "ip-10-13-2-251",
    "az": "eu-west-1a"
    }

    View Slide

  47. Phosphor
    Host Instances
    Publish
    Service A
    Trace Library
    goroutine
    chan
    UDP
    Service B
    Trace Library
    goroutine
    chan
    UDP
    Trace
    Service
    In-memory
    Aggregates
    Optional

    persistant
    storage
    Dashboards
    Monitoring

    View Slide

  48. var traceChan chan []byte
    func init() {
    // Use a buffered channel
    traceChan = make(chan []byte, 200)
    // Fire off a background worker for this channel
    defaultClient = NewClient(traceChan)
    go defaultClient.publisher()
    }
    // Send, drops trace if the backend is at capacity
    func Send(msg []byte) {
    select {
    case traceChan <- msg:
    // Success
    default:
    // Default case fired if channel is full
    // Ensures this is non blocking
    }
    }

    View Slide

  49. Tracing: 33eda743-f124-435c-71fc-3c872bbc98e6
    2014-09-07 02:20:19.867 [/] [START] → -
    2014-09-07 02:20:19.867 [eu-west-1a/ip-10-11-3-51] [REQ] com.hailocab.hailo-2-api → com.hailocab.api.v1.customer.neardrivers -
    2014-09-07 02:20:19.867 [eu-west-1a/ip-10-11-2-203] [IN] com.hailocab.hailo-2-api → com.hailocab.api.v1.customer.neardrivers -
    2014-09-07 02:20:19.868 [eu-west-1a/ip-10-11-2-203] [REQ] com.hailocab.api.v1.customer → com.hailocab.service.feature-flags.features -
    2014-09-07 02:20:19.869 [eu-west-1a/ip-10-11-3-111] [IN] com.hailocab.api.v1.customer → com.hailocab.service.feature-flags.features -
    2014-09-07 02:20:19.876 [eu-west-1a/ip-10-11-3-111] [REQ] com.hailocab.service.feature-flags → com.hailocab.service.hob.list -
    2014-09-07 02:20:19.877 [eu-west-1a/ip-10-11-3-168] [IN] com.hailocab.service.hob → com.hailocab.service.config.compile -
    2014-09-07 02:20:19.877 [eu-west-1a/ip-10-11-3-111] [IN] com.hailocab.service.feature-flags → com.hailocab.service.hob.list -
    2014-09-07 02:20:19.877 [eu-west-1a/ip-10-11-3-111] [REQ] com.hailocab.service.hob → com.hailocab.service.config.compile -
    2014-09-07 02:20:19.883 [eu-west-1a/ip-10-11-3-168] [OUT] com.hailocab.service.hob → com.hailocab.service.config.compile - 5.59 ms
    2014-09-07 02:20:19.886 [eu-west-1a/ip-10-11-3-111] [REP] com.hailocab.service.hob → com.hailocab.service.config.compile - 8.40 ms
    2014-09-07 02:20:19.887 [eu-west-1a/ip-10-11-3-111] [OUT] com.hailocab.service.feature-flags → com.hailocab.service.hob.list - 9.72 ms
    2014-09-07 02:20:19.889 [eu-west-1a/ip-10-11-3-111] [REP] com.hailocab.service.feature-flags → com.hailocab.service.hob.list - 13.23 ms
    2014-09-07 02:20:19.889 [eu-west-1a/ip-10-11-3-111] [OUT] com.hailocab.api.v1.customer → com.hailocab.service.feature-flags.features - 20.58 ms
    2014-09-07 02:20:19.890 [eu-west-1a/ip-10-11-2-203] [REP] com.hailocab.api.v1.customer → com.hailocab.service.feature-flags.features - 22.59 ms
    2014-09-07 02:20:19.902 [eu-west-1a/ip-10-11-2-203] [REQ] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare -
    2014-09-07 02:20:19.903 [eu-west-1a/ip-10-11-2-203] [REQ] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare -
    2014-09-07 02:20:19.903 [eu-west-1a/ip-10-11-2-203] [REQ] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare -
    2014-09-07 02:20:19.904 [eu-west-1a/ip-10-11-3-111] [IN] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare -
    2014-09-07 02:20:19.904 [eu-west-1a/ip-10-11-3-111] [OUT] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare - 0.36 ms
    2014-09-07 02:20:19.905 [eu-west-1a/ip-10-11-2-203] [REP] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare - 1.97 ms
    2014-09-07 02:20:19.905 [eu-west-1a/ip-10-11-2-214] [IN] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare -
    2014-09-07 02:20:19.905 [eu-west-1a/ip-10-11-2-203] [REQ] com.hailocab.api.v1.customer → com.hailocab.service.nearest-driver.search -
    2014-09-07 02:20:19.905 [eu-west-1a/ip-10-11-2-214] [OUT] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare - 0.10 ms

    ERR - com.hailocab.service.fare.basefare: Missing config at xxx
    2014-09-07 02:20:19.906 [eu-west-1a/ip-10-11-2-214] [IN] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare -
    2014-09-07 02:20:19.906 [eu-west-1a/ip-10-11-2-214] [OUT] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare - 0.06 ms 

    ERR - com.hailocab.service.fare.basefare: Missing config at xxx
    2014-09-07 02:20:19.907 [eu-west-1a/ip-10-11-3-58] [IN] com.hailocab.api.v1.customer → com.hailocab.service.nearest-driver.search -
    2014-09-07 02:20:19.907 [eu-west-1a/ip-10-11-3-58] [REQ] com.hailocab.service.nearest-driver → com.hailocab.service.zoning.search -
    2014-09-07 02:20:19.908 [eu-west-1a/ip-10-11-3-58] [IN] com.hailocab.service.nearest-driver → com.hailocab.service.zoning.search -
    2014-09-07 02:20:19.908 [eu-west-1a/ip-10-11-3-58] [OUT] com.hailocab.service.nearest-driver → com.hailocab.service.zoning.search - 0.20 ms
    2014-09-07 02:20:19.909 [eu-west-1a/ip-10-11-3-58] [REP] com.hailocab.service.nearest-driver → com.hailocab.service.zoning.search - 2.25 ms
    2014-09-07 02:20:19.909 [eu-west-1a/ip-10-11-3-58] [REQ] com.hailocab.service.nearest-driver → com.hailocab.service.raziel.multisearch -
    2014-09-07 02:20:19.912 [eu-west-1a/ip-10-11-3-227] [IN] com.hailocab.service.nearest-driver → com.hailocab.service.raziel.multisearch -
    2014-09-07 02:20:19.919 [eu-west-1a/ip-10-11-3-58] [REP] com.hailocab.service.nearest-driver → com.hailocab.service.raziel.multisearch - 9.46 ms
    2014-09-07 02:20:19.919 [eu-west-1a/ip-10-11-3-58] [REQ] com.hailocab.service.nearest-driver → com.hailocab.service.eta.multitraveltime -
    2014-09-07 02:20:19.919 [eu-west-1a/ip-10-11-3-227] [OUT] com.hailocab.service.nearest-driver → com.hailocab.service.raziel.multisearch - 7.58 ms
    2014-09-07 02:20:19.920 [eu-west-1a/ip-10-11-3-58] [IN] com.hailocab.service.nearest-driver → com.hailocab.service.eta.multitraveltime -
    2014-09-07 02:20:19.920 [eu-west-1a/ip-10-11-3-58] [OUT] com.hailocab.service.nearest-driver → com.hailocab.service.eta.multitraveltime - 0.06 ms
    2014-09-07 02:20:19.921 [eu-west-1a/ip-10-11-3-58] [REP] com.hailocab.service.nearest-driver → com.hailocab.service.eta.multitraveltime - 1.77 ms
    2014-09-07 02:20:19.921 [eu-west-1a/ip-10-11-3-58] [OUT] com.hailocab.api.v1.customer → com.hailocab.service.nearest-driver.search - 14.02 ms
    2014-09-07 02:20:19.921 [eu-west-1a/ip-10-11-2-203] [REP] com.hailocab.api.v1.customer → com.hailocab.service.nearest-driver.search - 15.48 ms
    2014-09-07 02:20:19.941 [eu-west-1a/ip-10-11-2-203] [REQ] com.hailocab.api.v1.customer → com.hailocab.service.experiment.readlastupdated -
    2014-09-07 02:20:19.945 [eu-west-1a/ip-10-11-2-214] [IN] com.hailocab.api.v1.customer → com.hailocab.service.experiment.readlastupdated -
    2014-09-07 02:20:19.947 [eu-west-1a/ip-10-11-2-214] [OUT] com.hailocab.api.v1.customer → com.hailocab.service.experiment.readlastupdated - 1.82 ms
    2014-09-07 02:20:19.947 [eu-west-1a/ip-10-11-2-203] [REP] com.hailocab.api.v1.customer → com.hailocab.service.experiment.readlastupdated - 6.01 ms
    2014-09-07 02:20:19.948 [eu-west-1a/ip-10-11-2-203] [OUT] com.hailocab.hailo-2-api → com.hailocab.api.v1.customer.neardrivers - 80.46 ms
    2014-09-07 02:20:19.950 [eu-west-1a/ip-10-11-3-51] [REP] com.hailocab.hailo-2-api → com.hailocab.api.v1.customer.neardrivers - 82.71 ms

    View Slide

  50. View Slide

  51. View Slide

  52. View Slide

  53. View Slide

  54. View Slide

  55. SO, GO?

    View Slide

  56. SMALL
    SIMPLE
    EASY TO READ & LEARN

    View Slide

  57. CONCURRENCY PRIMITIVES
    INTERFACE SUPPORT

    View Slide

  58. EASY DEPLOYMENT &
    MANAGEMENT IN PRODUCTION
    AND AT SCALE

    View Slide

  59. СПАСИБО!

    View Slide

  60. IMAGES
    HMS President: Roger Marks
    Orbital Ion Cannon: www.rom.ac
    Go Gophers: Renee French
    Utopia/Dystopia: Dylan Glynn
    Control Room: NASA

    View Slide