H2O Aunt Sally

This was the first deck I put together to socialise the core ideas of building a new version of the Hailo software stack.

Our CTO (Rorie) had asked me to think about this with the brief: "Allow Hailo to scale the business along three axis: adding features to our current business, adding cities and brand new stuff."

The last modified date on this is 15th May 2013. I gave an update to the board in November 2013 where I talked about key milestones: first commit was May 31st, by November 8th we had entire job flow working integrated with a new driver app, and powered by around 60 microservices!

Transcript

1. H20 Aunt Sally
2. Aunt Sally is a traditional throwing game.
 The term is often used metaphorically to mean something that is a target for criticism.
3. 0 / Ambitions Provide a simple framework for us to build an efficient, resilient, second generation Hailo (aka h20) Allow Hailo to scale the business along three axis: adding features to our current business, adding cities and brand new stuff Solve pain points in our current architecture 
 Be productive
4. 0 / Logical design API tier Orchestration tier Services tier Data storage tier • Exposed to the outside world • All clients talk to this layer, via HTTP • Permitted to talk to orchestration and services tiers
5. 0 / Logical design API tier Orchestration tier Services tier Data storage tier • Services that coordinate flows across multiple services, in a reusable way • An example would be registration where a record is created via one service and then a confirmation email sent via another • Permitted to talk to the service tier and able to leverage messaging solutions • Should never talk to data storage and API tiers
6. 0 / Logical design API tier Orchestration tier Services tier Data storage tier • Core “leaf” services • An example would be a customer service that does CRUD on customer records • Permitted to talk to the data storage tier • Should never talk to any other services in any tier (including other core services)
7. 0 / Logical design API tier Orchestration tier Services tier Data storage tier • Storage engines such as Cassandra • Only accessible from core services • Services never share state (database), so each core service is solely responsible for its own data access
8. 0 / Physical design Client Service Messaging system protobuf protobuf
9. 1 / Feature Feature: retrieve details on customer 12313500143 name service.customer.retrieve request message CustomerId { required uint64 id = 1; } response message Customer { required unit64 id = 1; required locale string = 2; }
10. 1 / Feature import( “github.com/hailocab/goservice” msg “path/to/compiled/protobuf/retrieve” ) func main() { goservice.Register(ns, reqDef, rspDef, func(req *msg.CustomerId) *msg.Customer { // do stuff here and build a *msg.Customer to return } }
11. 1 / Feature • Global “endpoint” namespace in the form /{tier}/{service}/{foo}/{bar}… • Action-oriented namespace; there is no REST here thank you • Every feature endpoint must define a single request/response pair, in Protobuf • Interface is a simple function which accepts request object and returns response object
12. 2 / Package A small collection of features; a micro-service. • create new customer • retrieve details on customer • update customer • delete customer
13. 3 / Provision Github Build binary Stage QA Deploy Tag
14. 3 / Provision • Fully automated from start to finish; “push-button” system • Same binary that goes through QA is deployed to production • Push-based deployment, uneven distribution (not every package has to run on every server)
15. 4 / Register pkg A pkg A pkg A Register with message bus:
 
 “I am here and ready for messages” Server 1 Server 2 Server 3
16. 4 / Register Register with ZK, by creating an Ephemeral node:
 
 “I am running on this server” Zookeeper pkg A pkg A pkg A Server 1 Server 2 Server 3
17. 4 / Register • Zookeeper keeps track of every package running • Registration via ZK ephemeral node, so failure will automatically remove • Information in ZK includes version number • Message bus keeps track of connected “workers” (packages)
18. 5 / Use Client Local router and broker protobuf protobuf (Most things are a client)
19. 5 / Use import( “github.com/hailocab/goservice” msg “path/to/compiled/protobuf/retrieve” ) func main() { rsp := goservice.Sync(ns, req) rspChan := make(chan *goservice.Response) goservice.Async(ns, req, rspChan) goservice.Async(otherNs, otherReq, rspChan) rsp1 := <- rspChan rsp2 := <- rspChan }
20. 5 / Use • Local broker/router running on every server • Clients (APIs, services) send their requests to the local router; it is then up to the router to decide which worker to send it to for processing • Router can make decisions based on: • properties of the request (mapping request namespace to the right service) • external configuration such as “send more traffic to version A of this package” • health and speed of connected workers
21. 6 / Monitor • Each server exposes a single HTTP web service for querying health of packages deployed on that box, plus the health of the router/broker • Focus on pulling metrics and graphs together to form actionable intelligence, presented in the same place as fully automated “actions” (buttons to make changes)
22. 6 / Monitor
23. 7 / Debug • Unified logging format and full log aggregation • Inspect metrics and activity on individual deployed binaries via common tooling (CLI interface) `h20 list foo/bar` list all active packages matching name foo/bar `h20 stats <some id>` get stats of a single specific package running on a single server
24. 8 / Control • Real-time control over the network of routers/brokers • Manual traffic shaping to avoid a specific machine • A/B test two versions of the same package • Gradual rollout of a new version of a package, gradually shifting all traffic to a new version and then retiring the old • Automatic monitoring of provisioning and launching of binaries
25. 9 / Summary Simple concepts, reused everywhere, all the way down the stack Make my request 
 
 protobuf Send it to this thing
 
 apps to single API, everything else to local broker Get back my response
 
 protobuf Request Broker Response