$30 off During Our Annual Pro Sale. View Details »

Micro services architectures using Gilmour

Piyush Verma
February 23, 2016

Micro services architectures using Gilmour

Piyush Verma

February 23, 2016
Tweet

More Decks by Piyush Verma

Other Decks in Programming

Transcript

  1. SOA (VS SERVERS) SERVICES ▸ Logical Entities ▸ Independent ▸

    Comprised of more services ▸ Multiple servers for load and tolerance ▸ Scalability ▸ Flexibility
  2. TO ERR IS HUMAN, TO BLAME IT ON SOMEONE IS

    MANAGEMENT POTENTIAL. - Someone on Internet WHY?
  3. WHO DUN IT? WHY? ▸ Software Problem ▸ Function Point

    Scalability ▸ Management Problem ▸ Authority ▸ Responsibility.
  4. ASYNC: SIGNALS AND SLOTS Slots Signal item.buy.500 item.buy.* item.buy.* item.buy.*

    group: push_msg FP scaling Push Message Server Push Message Server Web GILMOUR Email notifier SMS notifier Push Message server Item purchased VIP Item purchased item.buy.420
  5. ASYNC: SLOTS func main() { redis := backends.MakeRedis("127.0.0.1:6379", "pass") g

    := G.Get(redis) g.Slot("example.log", func(req *G.Request, _ *G.Message) { var msg string req.Data(&msg) log.Println(req.Sender(), "->", msg) }, nil) g.Start() } line := "Filtering out stop words" g.Signal("example.log", G.NewMessage().SetData(line)) ASYNC: SIGNALS
  6. REQUEST RESPONSE Caller Service Hello? Aloha! Wrong Number type Message

    struct { Data interface{} `json:"data"` //Actual Payload Code int `json:"code"` //Response code if this is a response. This uses HTTP error codes Sender string `json:"sender"` //Origin of the request (unique for each request) } SENDER_ID CODE
  7. REQUEST data := G.NewMessage().SetData("hello?") opts := G.NewRequestOpts().SetTimeout(200) g.Request("echo", data, func(req

    *G.Request, resp *G.Message) { var msg string req.Data(&msg); fmt.Println("Echoclient: received", msg) }, opts) opts := G.NewHandlerOpts().SetGroup("exclusive") g.ReplyTo("echo", func(req *G.Request, resp *G.Message) { var msg string req.Data(&msg) resp.SetData(fmt.Sprintf("Pong %v", msg)) }, opts) RESPONSE
  8. GILMOUR ERRORS ERROR DETECTION ▸ timeouts ▸ server side -

    execution timed out. ▸ client side - response too late ▸ confirm_subscribers
  9. ERRORS: SIGNALS & SLOTS Web GILMOUR Email notifier SMS notifier

    Push message notifier Slots Error reporter Signal
  10. GILMOUR ERRORS Caller Service Hello? Wrong Number SENDER_ID CODE Error

    reporter type gilmourError struct { topic string //topic to which the request was sent requestData string //request payload userData string //implementation dependent information sender string //sender from the request timestamp string //timestamp of error backtrace interface{} //backtrace of the error code int //response code (non-200) }
  11. GILMOUR ERRORS ERROR HANDLING ▸ Ignore ▸ Publish ▸ Queue

    func main() { redis := backends.MakeRedis("127.0.0.1:6379", "pass") g := G.Get(redis) //g.SetErrorPolicy(protocol.ErrorPolicyQueue) //g.SetErrorPolicy(protocol.ErrorPolicyIgnore) g.SetErrorPolicy(protocol.ErrorPolicyPublish) }
  12. OPS: MONITORING SERVERS ▸ Log forwarding & aggregation ▸ Error

    Reporting ▸ Server and Health Monitoring
  13. OPS: MONITORING SERVICES ▸ Are the services servicing? ▸ Do

    services have spare capacity? ▸ Do services respond in time?
  14. OPS: MONITORING HEALTH BULLETIN Feb 23 12:25:14 qa-next-backend-manager0.datascale.io gilmour_health: D,

    [2016-02-23T06:55:12.930435 #22111] DEBUG -- : Ensuring topics have at least one subscriber Feb 23 12:25:14 qa-next-backend-manager0.datascale.io gilmour_health: D, [2016-02-23T06:55:12.931160 #22111] DEBUG -- : Ensuring that gilmour servers respond to health ping Feb 23 12:26:14 qa-next-backend-manager0.datascale.io backend_manager: D, [2016-02-23T06:56:12.932775 #9605] DEBUG -- : Executing handler for gilmour.health.ip-10-0-0-146-pid-9605- uuid-9d0484dd-08e5-4ec3-a3e4-9a109634712b from 2881867d41f24228b462b9eff0b7e88e
  15. COMPOSITION ▸ compose - service1 | service2 | composition3 ▸

    andand - service1 && service2 && composition3 ▸ oror - service1 || composition2 || service3 ▸ batch ▸ (service1; service2; service3) > out ▸ service1; composition2; service3 > out ▸ parallel
  16. COMPOSITION popular := g.NewParallel( g.NewRequestComposition("example.popular3"), g.NewRequestComposition("example.popular4"), ) batch := g.NewPipe(

    g.NewRequestComposition("example.fetch"), g.NewRequestComposition("example.words"), g.NewRequestComposition("example.count"), popular, ) data := G.NewMessage() data.SetData("https://s3.amazonaws.com/ds-data-sample/test.txt") msg := <-batch.Execute(data)
  17. COMPOSITION popular := g.NewParallel( g.NewRequestComposition("example.popular3"), g.NewRequestComposition(“example.popular4"), g.NewRequestComposition("example.popular5"), ) batch :=

    g.NewPipe( g.NewRequestComposition("example.fetch"), g.NewRequestComposition("example.words"), g.NewRequestComposition("example.stopfilter"), g.NewRequestComposition("example.count"), popular, &MyComposer{}, ) 
 data := G.NewMessage() data.SetData("https://s3.amazonaws.com/ds-data-sample/test.txt") msg := <-batch.Execute(data)
  18. GILMOUR RECAP ▸ Library only; no external process. ▸ No

    Load Balancer ▸ No Service Discovery ▸ Async - Signal, Slots ▸ Sync - Request, Reply ▸ Scaling - Exclusion groups. ▸ Error and Health Monitoring - HealthBulletin. ▸ Failure Detection ▸ Composition - Real micro-services. ▸ No message persistence.
  19. GILMOUR CREDITS & LINKS ▸ Aditya Godbole ▸ [email protected]

    Datascale.IO ▸ datascale.io ▸ Repository ▸ github.com/gilmour-libs