Micro services architectures using Gilmour

Transcript

1. GILMOUR
   MICRO-SERVICE ARCHITECTURES USING
   

   View Slide

2. SOA (VS SERVERS)
   SERVICES
   ▸ Logical Entities
   ▸ Independent
   ▸ Comprised of more services
   ▸ Multiple servers for load and tolerance
   ▸ Scalability
   ▸ Flexibility
   

   View Slide

3. TO ERR IS HUMAN, TO BLAME
   IT ON SOMEONE IS
   MANAGEMENT POTENTIAL.
   - Someone on Internet
   WHY?
   

   View Slide

4. WHO DUN IT?
   WHY?
   ▸ Software Problem
   ▸ Function Point Scalability
   ▸ Management Problem
   ▸ Authority
   ▸ Responsibility.
   

   View Slide

5. DESIGN PATTERNS
   

   View Slide

6. ASYNC
   (SIGNALS & SLOTS)
   

   View Slide

7. ASYNC: SIGNALS AND SLOTS
   Slots
   Signal
   item.buy.500
   item.buy.*
   item.buy.*
   item.buy.*
   group: push_msg
   FP scaling
   Push
   Message
   Server
   Push
   Message
   Server
   Web GILMOUR
   Email
   notifier
   SMS
   notifier
   Push
   Message
   server
   Item purchased
   VIP Item
   purchased
   item.buy.420
   

   View Slide

8. ASYNC: SLOTS
   func main() {
   redis := backends.MakeRedis("127.0.0.1:6379", "pass")
   g := G.Get(redis)
   g.Slot("example.log", func(req *G.Request, _ *G.Message) {
   var msg string
   req.Data(&msg)
   log.Println(req.Sender(), "->", msg)
   }, nil)
   g.Start()
   }
   line := "Filtering out stop words"
   g.Signal("example.log", G.NewMessage().SetData(line))
   ASYNC: SIGNALS
   

   View Slide

9. REQUEST RESPONSE
   

   View Slide

10. REQUEST RESPONSE
    Caller Service
    Hello?
    Aloha!
    Wrong Number
    type Message struct {
    Data interface{} `json:"data"`
    //Actual Payload
    Code int `json:"code"`
    //Response code if this is a response. This uses HTTP error codes
    Sender string `json:"sender"`
    //Origin of the request (unique for each request)
    }
    SENDER_ID
    CODE
    

    View Slide

11. REQUEST
    data := G.NewMessage().SetData("hello?")
    opts := G.NewRequestOpts().SetTimeout(200)
    g.Request("echo", data, func(req *G.Request, resp *G.Message) {
    var msg string
    req.Data(&msg);
    fmt.Println("Echoclient: received", msg)
    }, opts)
    opts := G.NewHandlerOpts().SetGroup("exclusive")
    g.ReplyTo("echo", func(req *G.Request, resp *G.Message) {
    var msg string
    req.Data(&msg)
    resp.SetData(fmt.Sprintf("Pong %v", msg))
    }, opts)
    RESPONSE
    

    View Slide

12. SIGNAL-SLOT
    VS.
    REQUEST-RESPONSE
    

    View Slide

13. ERRORS:
    DETECTION
    AND
    HANDLING
    

    View Slide

14. GILMOUR ERRORS
    ERROR DETECTION
    ▸ timeouts
    ▸ server side - execution timed out.
    ▸ client side - response too late
    ▸ confirm_subscribers
    

    View Slide

15. ERRORS: SIGNALS & SLOTS
    Web GILMOUR
    Email
    notifier
    SMS
    notifier
    Push
    message
    notifier
    Slots
    Error reporter
    Signal
    

    View Slide

16. GILMOUR ERRORS
    Caller Service
    Hello?
    Wrong Number
    SENDER_ID
    CODE
    Error reporter
    type gilmourError struct {
    topic string //topic to which the request was sent
    requestData string //request payload
    userData string //implementation dependent information
    sender string //sender from the request
    timestamp string //timestamp of error
    backtrace interface{} //backtrace of the error
    code int //response code (non-200)
    }
    

    View Slide

17. GILMOUR ERRORS
    ERROR HANDLING
    ▸ Ignore
    ▸ Publish
    ▸ Queue
    func main() {
    redis := backends.MakeRedis("127.0.0.1:6379", "pass")
    g := G.Get(redis)
    //g.SetErrorPolicy(protocol.ErrorPolicyQueue)
    //g.SetErrorPolicy(protocol.ErrorPolicyIgnore)
    g.SetErrorPolicy(protocol.ErrorPolicyPublish)
    }
    

    View Slide

18. GILMOUR ERRORS
    ERROR SAMPLE
    

    View Slide

19. OPS: MONITORING
    

    View Slide

20. OPS: MONITORING
    SERVERS
    ▸ Log forwarding & aggregation
    ▸ Error Reporting
    ▸ Server and Health Monitoring
    

    View Slide

21. OPS: MONITORING
    SERVICES
    ▸ Are the services servicing?
    ▸ Do services have spare capacity?
    ▸ Do services respond in time?
    

    View Slide

22. OPS: MONITORING
    HEALTH BULLETIN
    Feb 23 12:25:14 qa-next-backend-manager0.datascale.io gilmour_health:
    D, [2016-02-23T06:55:12.930435 #22111] DEBUG -- : Ensuring topics
    have at least one subscriber
    Feb 23 12:25:14 qa-next-backend-manager0.datascale.io gilmour_health:
    D, [2016-02-23T06:55:12.931160 #22111] DEBUG -- : Ensuring that
    gilmour servers respond to health ping
    Feb 23 12:26:14 qa-next-backend-manager0.datascale.io
    backend_manager: D, [2016-02-23T06:56:12.932775 #9605] DEBUG -- :
    Executing handler for gilmour.health.ip-10-0-0-146-pid-9605-
    uuid-9d0484dd-08e5-4ec3-a3e4-9a109634712b from
    2881867d41f24228b462b9eff0b7e88e
    

    View Slide

23. OPS: MONITORING
    HEALTH BULLETIN
    

    View Slide

24. OPS: MONITORING
    HEALTH BULLETIN
    

    View Slide

25. 

    COMPOSITION
    

    View Slide

26. COMPOSITION
    ▸ compose - service1 | service2 | composition3
    ▸ andand - service1 && service2 && composition3
    ▸ oror - service1 || composition2 || service3
    ▸ batch
    ▸ (service1; service2; service3) > out
    ▸ service1; composition2; service3 > out
    ▸ parallel
    

    View Slide

27. COMPOSITION
    popular := g.NewParallel(
    g.NewRequestComposition("example.popular3"),
    g.NewRequestComposition("example.popular4"),
    )
    batch := g.NewPipe(
    g.NewRequestComposition("example.fetch"),
    g.NewRequestComposition("example.words"),
    g.NewRequestComposition("example.count"),
    popular,
    )
    data := G.NewMessage()
    data.SetData("https://s3.amazonaws.com/ds-data-sample/test.txt")
    msg := <-batch.Execute(data)
    

    View Slide

28. COMPOSITION
    popular := g.NewParallel(
    g.NewRequestComposition("example.popular3"),
    g.NewRequestComposition(“example.popular4"),
    g.NewRequestComposition("example.popular5"),
    )
    batch := g.NewPipe(
    g.NewRequestComposition("example.fetch"),
    g.NewRequestComposition("example.words"),
    g.NewRequestComposition("example.stopfilter"),
    g.NewRequestComposition("example.count"),
    popular,
    &MyComposer{},
    )
    

    data := G.NewMessage()
    data.SetData("https://s3.amazonaws.com/ds-data-sample/test.txt")
    msg := <-batch.Execute(data)
    

    View Slide

29. 

    EXAMPLES
    

    View Slide

30. EXAMPLES: ECHO SERVERS & CLIENT
    

    View Slide

31. SAMPLE TEXT
    

    View Slide

32. EXAMPLES: COMPOSITION
    

    View Slide

33. GILMOUR
    RECAP
    ▸ Library only; no external process.
    ▸ No Load Balancer
    ▸ No Service Discovery
    ▸ Async - Signal, Slots
    ▸ Sync - Request, Reply
    ▸ Scaling - Exclusion groups.
    ▸ Error and Health Monitoring - HealthBulletin.
    ▸ Failure Detection
    ▸ Composition - Real micro-services.
    ▸ No message persistence.
    

    View Slide

34. GILMOUR
    CREDITS & LINKS
    ▸ Aditya Godbole
    ▸ aa@gdbl.me
    ▸ Datascale.IO
    ▸ datascale.io
    ▸ Repository
    ▸ github.com/gilmour-libs
    

    View Slide