Upgrade to Pro — share decks privately, control downloads, hide ads and more …

H2O Aunt Sally

H2O Aunt Sally

This was the first deck I put together to socialise the core ideas of building a new version of the Hailo software stack.

Our CTO (Rorie) had asked me to think about this with the brief: "Allow Hailo to scale the business along three axis: adding features to our current business, adding cities and brand new stuff."

The last modified date on this is 15th May 2013. I gave an update to the board in November 2013 where I talked about key milestones: first commit was May 31st, by November 8th we had entire job flow working integrated with a new driver app, and powered by around 60 microservices!

Dave Gardner

May 15, 2013

More Decks by Dave Gardner

Other Decks in Technology


  1. Aunt Sally is a traditional throwing game.
 The term is

    often used metaphorically to mean something that is a target for criticism.
  2. 0 / Ambitions Provide a simple framework for us to

    build an efficient, resilient, second generation Hailo (aka h20) Allow Hailo to scale the business along three axis: adding features to our current business, adding cities and brand new stuff Solve pain points in our current architecture 
 Be productive
  3. 0 / Logical design API tier Orchestration tier Services tier

    Data storage tier • Exposed to the outside world • All clients talk to this layer, via HTTP • Permitted to talk to orchestration and services tiers
  4. 0 / Logical design API tier Orchestration tier Services tier

    Data storage tier • Services that coordinate flows across multiple services, in a reusable way • An example would be registration where a record is created via one service and then a confirmation email sent via another • Permitted to talk to the service tier and able to leverage messaging solutions • Should never talk to data storage and API tiers
  5. 0 / Logical design API tier Orchestration tier Services tier

    Data storage tier • Core “leaf” services • An example would be a customer service that does CRUD on customer records • Permitted to talk to the data storage tier • Should never talk to any other services in any tier (including other core services)
  6. 0 / Logical design API tier Orchestration tier Services tier

    Data storage tier • Storage engines such as Cassandra • Only accessible from core services • Services never share state (database), so each core service is solely responsible for its own data access
  7. 1 / Feature Feature: retrieve details on customer 12313500143 name

    service.customer.retrieve request message CustomerId { required uint64 id = 1; } response message Customer { required unit64 id = 1; required locale string = 2; }
  8. 1 / Feature import( “github.com/hailocab/goservice” msg “path/to/compiled/protobuf/retrieve” ) func main()

    { goservice.Register(ns, reqDef, rspDef, func(req *msg.CustomerId) *msg.Customer { // do stuff here and build a *msg.Customer to return } }
  9. 1 / Feature • Global “endpoint” namespace in the form

    /{tier}/{service}/{foo}/{bar}… • Action-oriented namespace; there is no REST here thank you • Every feature endpoint must define a single request/response pair, in Protobuf • Interface is a simple function which accepts request object and returns response object
  10. 2 / Package A small collection of features; a micro-service.

    • create new customer • retrieve details on customer • update customer • delete customer
  11. 3 / Provision • Fully automated from start to finish;

    “push-button” system • Same binary that goes through QA is deployed to production • Push-based deployment, uneven distribution (not every package has to run on every server)
  12. 4 / Register pkg A pkg A pkg A Register

    with message bus:
 “I am here and ready for messages” Server 1 Server 2 Server 3
  13. 4 / Register Register with ZK, by creating an Ephemeral

 “I am running on this server” Zookeeper pkg A pkg A pkg A Server 1 Server 2 Server 3
  14. 4 / Register • Zookeeper keeps track of every package

    running • Registration via ZK ephemeral node, so failure will automatically remove • Information in ZK includes version number • Message bus keeps track of connected “workers” (packages)
  15. 5 / Use import( “github.com/hailocab/goservice” msg “path/to/compiled/protobuf/retrieve” ) func main()

    { rsp := goservice.Sync(ns, req) rspChan := make(chan *goservice.Response) goservice.Async(ns, req, rspChan) goservice.Async(otherNs, otherReq, rspChan) rsp1 := <- rspChan rsp2 := <- rspChan }
  16. 5 / Use • Local broker/router running on every server

    • Clients (APIs, services) send their requests to the local router; it is then up to the router to decide which worker to send it to for processing • Router can make decisions based on: • properties of the request (mapping request namespace to the right service) • external configuration such as “send more traffic to version A of this package” • health and speed of connected workers
  17. 6 / Monitor • Each server exposes a single HTTP

    web service for querying health of packages deployed on that box, plus the health of the router/broker • Focus on pulling metrics and graphs together to form actionable intelligence, presented in the same place as fully automated “actions” (buttons to make changes)
  18. 7 / Debug • Unified logging format and full log

    aggregation • Inspect metrics and activity on individual deployed binaries via common tooling (CLI interface) `h20 list foo/bar` list all active packages matching name foo/bar `h20 stats <some id>` get stats of a single specific package running on a single server
  19. 8 / Control • Real-time control over the network of

    routers/brokers • Manual traffic shaping to avoid a specific machine • A/B test two versions of the same package • Gradual rollout of a new version of a package, gradually shifting all traffic to a new version and then retiring the old • Automatic monitoring of provisioning and launching of binaries
  20. 9 / Summary Simple concepts, reused everywhere, all the way

    down the stack Make my request 
 protobuf Send it to this thing
 apps to single API, everything else to local broker Get back my response
 protobuf Request Broker Response