Hailo 2.0 Internal Tech Talk

This is similar to the H2 board presentation, but includes more depth aimed at selling the platform to the internal tech teams. The section on "Designing new things" is a good example of how we were thinking about these internal customers of our platform.

Transcript

1. Hailo 2.0
   David Gardner, Architect at Hailo
   

   View Slide

2. H1: November 2011
   

   View Slide

3. PHP
   Cust
   API
   eu-west-1
   PHP
   Driver
   API
   Java
   Hailo
   Engine
   MySQL
   PHP
   Driver
   API
   PHP
   Driver
   API
   PHP
   Cust
   API
   PHP
   Cust
   API
   Load Balancer Load Balancer
   MySQL
   Redis
   

   View Slide

4. H1.5: 2012
   

   View Slide

5. PHP
   Cust
   API
   eu-west-1
   Java
   Hailo
   Engine
   MySQL
   PHP
   Driver
   API
   PHP
   Cust
   API
   PHP
   Cust
   API
   ELB ELB
   Java
   Hailo
   Engine
   PHP
   Driver
   API
   ELB
   MySQL
   Java
   Hailo
   Engine
   MySQL
   PHP
   Driver
   API
   ELB
   C* C* C*
   PHP
   Cust
   service
   PHP
   Credits
   service
   Java
   Pay
   service
   eu-west-1
   us-east-1
   

   View Slide

6. Why H2?
   

   View Slide

7. Original ambitions
   •  Provide a simple framework for us to build an efficient, resilient,
   second generation Hailo
   •  Allow Hailo to scale the business along three axis: adding features
   to our current business, adding cities and brand new stuff
   •  Solve pain points in our current architecture
   •  Be productive
   

   View Slide

8. + features + cities
   + brand new + productive
   - pain
   

   View Slide

9. PHP
   Cust
   API
   eu-west-1
   Java
   Hailo
   Engine
   MySQL
   PHP
   Driver
   API
   PHP
   Cust
   API
   PHP
   Cust
   API
   ELB ELB
   Java
   Hailo
   Engine
   PHP
   Driver
   API
   ELB
   MySQL
   Java
   Hailo
   Engine
   MySQL
   PHP
   Driver
   API
   ELB
   C* C* C*
   PHP
   Cust
   service
   PHP
   Credits
   service
   Java
   Pay
   service
   eu-west-1
   us-east-1
   Servers needed per-city + cities
   

   View Slide

10. PHP
    Cust
    API
    eu-west-1
    Java
    Hailo
    Engine
    MySQL
    PHP
    Driver
    API
    PHP
    Cust
    API
    PHP
    Cust
    API
    ELB ELB
    Java
    Hailo
    Engine
    PHP
    Driver
    API
    ELB
    MySQL
    Java
    Hailo
    Engine
    MySQL
    PHP
    Driver
    API
    ELB
    C* C* C*
    PHP
    Cust
    service
    PHP
    Credits
    service
    Java
    Pay
    service
    eu-west-1
    us-east-1
    City configuration in many places + cities
    PHP array
    YAML
    Config service
    plists built
    into app
    plists built
    into app
    XML XML
    Config
    service
    PHP array
    YAML
    

    View Slide

11. PHP
    Cust
    API
    eu-west-1
    Java
    Hailo
    Engine
    MySQL
    PHP
    Driver
    API
    PHP
    Cust
    API
    PHP
    Cust
    API
    ELB ELB
    Java
    Hailo
    Engine
    PHP
    Driver
    API
    ELB
    MySQL
    Java
    Hailo
    Engine
    MySQL
    PHP
    Driver
    API
    ELB
    C* C* C*
    PHP
    Cust
    service
    PHP
    Credits
    service
    Java
    Pay
    service
    eu-west-1
    us-east-1
    Coordination changing three apps + features
    

    View Slide

12. PHP
    Cust
    API
    eu-west-1
    Java
    Hailo
    Engine
    MySQL
    PHP
    Driver
    API
    PHP
    Cust
    API
    PHP
    Cust
    API
    ELB ELB
    Java
    Hailo
    Engine
    PHP
    Driver
    API
    ELB
    MySQL
    Java
    Hailo
    Engine
    MySQL
    PHP
    Driver
    API
    ELB
    C* C* C*
    PHP
    Cust
    service
    PHP
    Credits
    service
    Java
    Pay
    service
    eu-west-1
    us-east-1
    Unclear responsibilities + features
    Eg: payment
    or cancellation
    

    View Slide

13. PHP
    Cust
    API
    eu-west-1
    Java
    Hailo
    Engine
    MySQL
    PHP
    Driver
    API
    PHP
    Cust
    API
    PHP
    Cust
    API
    ELB ELB
    Java
    Hailo
    Engine
    PHP
    Driver
    API
    ELB
    MySQL
    Java
    Hailo
    Engine
    MySQL
    PHP
    Driver
    API
    ELB
    C* C* C*
    PHP
    Cust
    service
    PHP
    Credits
    service
    Java
    Pay
    service
    eu-west-1
    us-east-1
    Broad but inflexible services + features, + brand new
    

    View Slide

14. PHP
    Cust
    API
    eu-west-1
    Java
    Hailo
    Engine
    MySQL
    PHP
    Driver
    API
    PHP
    Cust
    API
    PHP
    Cust
    API
    ELB ELB
    Java
    Hailo
    Engine
    PHP
    Driver
    API
    ELB
    MySQL
    Java
    Hailo
    Engine
    MySQL
    PHP
    Driver
    API
    ELB
    C* C* C*
    PHP
    Cust
    service
    PHP
    Credits
    service
    Java
    Pay
    service
    eu-west-1
    us-east-1
    Separate push deployment models + productive
    rsync
    conan/cap
    rsync
    conan/cap
    conan/cap
    

    View Slide

15. PHP
    Cust
    API
    eu-west-1
    Java
    Hailo
    Engine
    MySQL
    PHP
    Driver
    API
    PHP
    Cust
    API
    PHP
    Cust
    API
    ELB ELB
    Java
    Hailo
    Engine
    PHP
    Driver
    API
    ELB
    MySQL
    Java
    Hailo
    Engine
    MySQL
    PHP
    Driver
    API
    ELB
    C* C* C*
    PHP
    Cust
    service
    PHP
    Credits
    service
    Java
    Pay
    service
    eu-west-1
    us-east-1
    Different auth models - pain
    none none
    IP
    whitelist
    plus token
    turned off
    IP
    whitelist
    plus login
    service
    

    View Slide

16. PHP
    Cust
    API
    eu-west-1
    Java
    Hailo
    Engine
    MySQL
    PHP
    Driver
    API
    PHP
    Cust
    API
    PHP
    Cust
    API
    ELB ELB
    Java
    Hailo
    Engine
    PHP
    Driver
    API
    ELB
    MySQL
    Java
    Hailo
    Engine
    MySQL
    PHP
    Driver
    API
    ELB
    C* C* C*
    PHP
    Cust
    service
    PHP
    Credits
    service
    Java
    Pay
    service
    eu-west-1
    us-east-1
    SPOFs - pain
    

    View Slide

17. PHP
    Cust
    API
    PHP
    Cust
    API
    PHP
    Cust
    API
    ELB
    ELB
    Load balancing broken and complex - pain
    PHP
    Cust
    HAP
    ELB
    PHP
    Credits
    HAP
    ELB
    Java
    Pay
    HAP
    Phone
    ELB
    API
    ELB
    HAProxy
    Service
    ELB
    HAProxy
    Service
    us-east-1
    eu-west-1
    us-east-1
    

    View Slide

18. H2: 2013
    

    View Slide

19. Utopia Dystopia
    Static
    Better
    Cheaper
    Sooner
    Dynamic
    Broken
    Inefficient
    

    View Slide

20. Key features
    •  Lose PHP, adopt Go – gains in efficiency of developer time and
    compute resource
    •  Eliminate all SPOFs – adopt a cloud native approach to build a
    working whole out of ephemeral and often broken parts
    •  Scale engineering output in line with additional resource – services
    with few, clearly defined responsibilities reduce friction
    •  Increase reusability – develop features by composing fine-grained
    services that are agnostic to Hailo’s current operation
    

    View Slide

21. Design for global
    

    View Slide

22. us-east-1
    C* C* C*
    eu-west-1
    ELB
    Go “Thin” API
    RabbitMQ Message Bus
    (federated clusters per AZ)
    Go
    Service
    Go
    Service
    Java
    Service
    C* C* C*
    ELB
    Go “Thin” API
    RabbitMQ Message Bus
    (federated clusters per AZ)
    Go
    Service
    Go
    Service
    Java
    Service
    1.
    2.
    3.
    

    View Slide

23. ELB
    Go “Thin” API
    1.
    H1
    Driver
    API
    H2
    “API”
    Service
    H2
    Orch.
    Service
    H2
    Core
    Service
    v1-api-driver-london.elasticride.com
    api-driver-london.elasticride.com
    http RMQ
    RMQ = Hailo’s RabbitMQ Message Bus
    RMQ
    RMQ
    {
    "match":{
    "regulatoryArea":"LON,DUB",
    "path":"/v1/point",
    "proportion”:0.5
    },
    "action":2
    },
    {
    "match":{
    "path":"/v1/experiment",
    "proportion”:1.0
    },
    "action":2
    },
    We borrow the idea of “specificity” from CSS and use this concept to sort
    the rules and thus define how they are evaluated.
    Need to build better tools around this. Danger Zone!
    

    View Slide

24. 2.
    RMQ RMQ
    cluster
    RMQ RMQ
    cluster
    RMQ RMQ
    cluster
    Service
    haproxy
    Services always connect to localhost.
    HAProxy sends to the same AZ, unless that AZ is
    down, in which case it “fails over” to a different AZ.
    RMQ runs in clusters of 2, within each AZ
    Each exchange is federated to the other AZs
    

    View Slide

25. 2.
    H2 RabbitMQ Exchange
    •  Bespoke exchange for RabbitMQ, written in Erlang
    •  Based on the headers exchange
    •  Main difference is random distribution of messages to a single
    bound queue (rather than delivering messages matched on headers
    to every matched bound queue)
    

    View Slide

26. Handler
    3.
    Logic
    Storage
    go-platform-layer
    go-service-layer
    Self-configuring external service adapters
    Library for building services that talk via RMQ
    // Clone makes a new city by cloning
    // an existing one
    func Clone(req *server.Request)
    (proto.Message, errors.Error) {
    // do stuff
    return &clone.Response{}, nil
    }
    

    View Slide

27. Discovery
    Service
    Binding
    Service
    Config
    Service
    Login
    Service
    •  Keeps track of every running instance of a service
    within a single region
    •  Stores this information as ephemeral nodes in
    Zookeeper, keeping a watch to ensure strong
    consistency between all instances of the discovery
    service within a region
    •  Sends heartbeats to services periodically via RMQ
    and removes dead instances
    •  Self-healing system because instances that don’t
    receive heartbeats will try to reconnect and failing
    that die
    

    View Slide

28. Discovery
    Service
    Binding
    Service
    Config
    Service
    Login
    Service
    •  Creates bindings within RabbitMQ for all running
    services, leveraging information in the discovery
    service
    •  Reacts to services coming up and down by
    creating and destroying bindings
    •  Bindings establish a connection between an
    exchange and a queue
    •  Stores and manages control plane data in order to
    provide advanced bindings such as “send 10% of
    traffic to this particular version of the service”
    

    View Slide

29. Discovery
    Service
    Binding
    Service
    Config
    Service
    Login
    Service
    •  Stores application configuration data as JSON
    •  Able to store JSON under any arbitrary key
    •  Can combine many keys, on request, to serve up
    “compiled” config
    

    View Slide

30. Discovery
    Service
    Binding
    Service
    Config
    Service
    Login
    Service
    •  Credential and session/token store for all
    applications
    •  The only thing that is able to issue and sign (with
    private key) tokens
    •  Applications can exchange a session ID for a
    token, which they can then use to establish
    authorisation
    

    View Slide

31. Designing new things
    

    View Slide

32. The RII of design
    •  Responsibility
    Design what service(s) are needed to implement a feature. Define
    what the responsibility is for each service. Responsibilities
    shouldn’t overlap.
    •  Interface
    Design the inputs and outputs of each service.
    •  Implementation
    Build the service.
    R > I > I: responsibility is king
    

    View Slide

33. ETA
    Service
    Routing
    Service
    Phone
    Service
    Profile
    Service
    State
    Service
    Charge
    Service
    Near
    Drivers
    Service
    Tow
    Truck
    Service
    Restau-
    rant
    Service
    Place
    Service
    /v1/customer/neardrivers
    API TIER
    ORCHESTRATION
    TIER
    CORE
    TIER
    

    View Slide

34. Tooling
    

    View Slide

35. Move fast: build a service in 30 minutes
    

    View Slide

36. Move fast: simple global deployment
    

    View Slide

37. Move fast: automatic monitoring of services
    

    View Slide

38. Move fast: automatic service health metrics
    

    View Slide

39. Move fast: powerful tools for developer insight
    

    View Slide

40. The future
    

    View Slide

41. Priorities
    •  Stability
    •  Test failure
    •  Test, benchmark and optimise the core platform
    •  Make us 100% certain we can cope with the demands of large markets
    •  Efficiency
    •  Use less AWS – make better use of what we have
    •  Build out tooling to make developers lives easier
    

    View Slide

42. Automate end-to-end testing
    •  Run tests against any environment (anticipating N environment
    future)
    •  Share knowledge so others can write end-to-end tests
    •  Aggregate results to get a clear and stable picture of performance
    This means that we have a stable and reliable picture of end-to-end
    performance and correctness that we can observe whenever we make
    changes.
    

    View Slide

43. Build environments on-demand
    •  Single button press to launch something that will run Hailo –
    something you can connect driver and customer apps to
    •  Focus on complete automation of Hailo, to provide a high level of
    repeatability
    •  Unlocks many advantages including:
    •  ability to failure test on virgin environments and then throw them away
    •  ability to develop and QA features in isolation
    •  continual assertion that we haven’t cut any corners with automation
    

    View Slide

44. Add release tooling
    •  Package up services and configuration into a “release”
    •  Environment automatically built to run a clone of production plus
    your changes – you can connect apps directly to this
    •  End-to-end tests kicked off automatically
    •  UI showing results of tests, summary of changes, and a huge
    DEPLOY button for someone in QA to “sign off” the release and
    push it into production
    

    View Slide