Contexts in Context

The context type allows deadlines, cancelation signals, and other request-scoped values, to propagate through request chains and across API boundaries, but in practice how can these features be used? In this talk Matt will talk about how Monzo use the humble context to power a variety of features in their Go based microservice platform.

Transcript

1. Contexts in Context
   Matt Heath
   

   View Slide

2. ➡ ➡ ❓ ➡
   

   View Slide

3. Hi, I’m Matt
   

   View Slide

4. @mattheath
   

   View Slide

5. View Slide

6. View Slide

7. View Slide

8. View Slide

9. View Slide

10. View Slide

11. Nov
    2015
    Apr
    2019
    

    View Slide

12. Nov
    2015
    Apr
    2019
    

    View Slide

13. Nov
    2015
    Apr
    2019
    

    View Slide

14. Nov
    2015
    Apr
    2019
    

    View Slide

15. Nov
    2015
    Fastest growing UK Bank
    1.8M Customers
    Apr
    2019
    

    View Slide

16. View Slide

17. View Slide

18. View Slide

19. View Slide

20. View Slide

21. package context
    type Context interface {
    Deadline() (deadline time.Time, ok bool)
    Done() <-chan struct{}
    Err() error
    Value(key interface{}) interface{}
    }
    

    View Slide

22. package context
    type Context interface {
    Deadline() (deadline time.Time, ok bool)
    Done() <-chan struct{}
    Err() error
    Value(key interface{}) interface{}
    }
    

    View Slide

23. package context
    type Context interface {
    Deadline() (deadline time.Time, ok bool)
    Done() <-chan struct{}
    Err() error
    Value(key interface{}) interface{}
    }
    

    View Slide

24. package context
    type Context interface {
    Deadline() (deadline time.Time, ok bool)
    Done() <-chan struct{}
    Err() error
    Value(key interface{}) interface{}
    }
    

    View Slide

25. package context
    type Context interface {
    Deadline() (deadline time.Time, ok bool)
    Done() <-chan struct{}
    Err() error
    Value(key interface{}) interface{}
    }
    

    View Slide

26. package context
    type Context interface {
    Deadline() (deadline time.Time, ok bool)
    Done() <-chan struct{}
    Err() error
    Value(key interface{}) interface{}
    }
    

    View Slide

27. ctx := context.Background()
    

    View Slide

28. ctx := context.Background()
    ctx = context.WithValue(ctx, "TraceID", "boop")
    

    View Slide

29. ctx := context.Background()
    ctx = context.WithValue(ctx, "TraceID", "boop")
    ctx.Value("TraceID") => "boop"
    

    View Slide

30. ctx := context.Background()
    ctx = context.WithValue(ctx, "TraceID", "boop")
    ctx.Value("TraceID") => "boop"
    Contexts are immutable
    

    View Slide

31. func(ctx, ...)
    

    View Slide

32. func(ctx, ...)
    func(ctx, ...)
    

    View Slide

33. func(ctx, ...)
    func(ctx, ...)
    

    View Slide

34. func(ctx, ...)
    func(ctx, ...)
    func(ctx, ...)
    func(ctx, ...)
    func(ctx, ...)
    

    View Slide

35. func(ctx, ...)
    func(ctx, ...)
    func(ctx, ...)
    func(ctx, ...)
    func(ctx, ...)
    

    View Slide

36. func(ctx, ...)
    func(ctx, ...)
    func(ctx, ...)
    func(ctx, ...)
    func(ctx, ...)
    

    View Slide

37. View Slide

38. func(ctx, ...)
    req.WithContext(ctx)
    

    View Slide

39. func(ctx, ...)
    req.WithContext(ctx) Remote API
    

    View Slide

40. func(ctx, ...)
    req.WithContext(ctx) Remote API
    ⏱
    

    View Slide

41. func(ctx, ...)
    req.WithContext(ctx) Remote API
    ⏱
    Deadline Exceeded
    err
    

    View Slide

42. package context
    type Context interface {
    Deadline() (deadline time.Time, ok bool)
    Done() <-chan struct{}
    Err() error
    Value(key interface{}) interface{}
    }
    

    View Slide

43. View Slide

44. View Slide

45. View Slide

46. View Slide

47. LICENCE WITH
    RESTRICTIONS
    WE ARE
    HERE
    A UK banking licence is authorised by the PRA and
    regulated by the PRA and FCA1, allowing deposit-
    taking and balance sheet lending. Once granted,
    it allows firms to passport across Europe, accessing
    This is followed by a “mobilisation” phase during
    which final capital is raised and IT systems are
    completed, before launching to the public.
    We received a UK banking licence in August 2016!
    FEB
    2015
    JAN
    2016
    JAN
    2017
    MAR FEB FEB
    APR MAR MAR APR MAY JUN JUL
    MAY APR
    JUN MAY
    JUL JUN
    AUG JUL
    SEP AUG
    NOV SEP
    DEC NOV DEC
    PRE APPLICATION APPLICATION MOBILISATION LAUNCH
    LICENCE WITH
    RESTRICTIONS
    50K MAX
    DEPOSIT
    WE ARE
    HERE
    A UK banking licence is authorised by the PRA and
    regulated by the PRA and FCA1, allowing deposit-
    taking and balance sheet lending. Once granted,
    it allows firms to passport across Europe, accessing
    This is followed by a “mobilisation” phase during
    which final capital is raised and IT systems are
    completed, before launching to the public.
    We received a UK banking licence in August 2016!
    JAN
    2016
    JAN
    2017
    MAR FEB FEB
    APR MAR MAR APR MAY JUN JUL
    MAY APR
    JUN MAY
    JUL JUN
    AUG JUL
    SEP AUG
    NOV SEP
    DEC NOV DEC
    E APPLICATION APPLICATION MOBILISATION LAUNCH
    LICENCE WITH
    RESTRICTIONS
    WE ARE
    HERE
    AUG

    2016
    JAN

    2016
    Feb

    2015
    APR

    2017
    A UK banking licence is authorised by the PRA and
    regulated by the PRA and FCA1, allowing deposit-
    taking and balance sheet lending. Once granted,
    it allows firms to passport across Europe, accessing
    This is followed by a “mobilisation” phase during
    which final capital is raised and IT systems are
    completed, before launching to the public.
    We received a UK banking licence in August 2016!
    FEB
    2015
    JAN
    2016
    JAN
    2017
    MAR FEB FEB
    APR MAR MAR APR
    MAY APR
    JUN MAY
    JUL JUN
    AUG JUL
    SEP AUG
    NOV SEP
    DEC NOV DEC
    PRE APPLICATION APPLICATION MOBILISATION
    LICENCE WITH
    RESTRICTIONS
    WE ARE
    HERE
    

    View Slide

48. LICENCE WITH
    RESTRICTIONS
    WE ARE
    HERE
    A UK banking licence is authorised by the PRA and
    regulated by the PRA and FCA1, allowing deposit-
    taking and balance sheet lending. Once granted,
    it allows firms to passport across Europe, accessing
    This is followed by a “mobilisation” phase during
    which final capital is raised and IT systems are
    completed, before launching to the public.
    We received a UK banking licence in August 2016!
    FEB
    2015
    JAN
    2016
    JAN
    2017
    MAR FEB FEB
    APR MAR MAR APR MAY JUN JUL
    MAY APR
    JUN MAY
    JUL JUN
    AUG JUL
    SEP AUG
    NOV SEP
    DEC NOV DEC
    PRE APPLICATION APPLICATION MOBILISATION LAUNCH
    LICENCE WITH
    RESTRICTIONS
    50K MAX
    DEPOSIT
    WE ARE
    HERE
    A UK banking licence is authorised by the PRA and
    regulated by the PRA and FCA1, allowing deposit-
    taking and balance sheet lending. Once granted,
    it allows firms to passport across Europe, accessing
    This is followed by a “mobilisation” phase during
    which final capital is raised and IT systems are
    completed, before launching to the public.
    We received a UK banking licence in August 2016!
    JAN
    2016
    JAN
    2017
    MAR FEB FEB
    APR MAR MAR APR MAY JUN JUL
    MAY APR
    JUN MAY
    JUL JUN
    AUG JUL
    SEP AUG
    NOV SEP
    DEC NOV DEC
    E APPLICATION APPLICATION MOBILISATION LAUNCH
    LICENCE WITH
    RESTRICTIONS
    WE ARE
    HERE
    AUG

    2016
    JAN

    2016
    Feb

    2015
    APR

    2017
    A UK banking licence is authorised by the PRA and
    regulated by the PRA and FCA1, allowing deposit-
    taking and balance sheet lending. Once granted,
    it allows firms to passport across Europe, accessing
    This is followed by a “mobilisation” phase during
    which final capital is raised and IT systems are
    completed, before launching to the public.
    We received a UK banking licence in August 2016!
    FEB
    2015
    JAN
    2016
    JAN
    2017
    MAR FEB FEB
    APR MAR MAR APR
    MAY APR
    JUN MAY
    JUL JUN
    AUG JUL
    SEP AUG
    NOV SEP
    DEC NOV DEC
    PRE APPLICATION APPLICATION MOBILISATION
    LICENCE WITH
    RESTRICTIONS
    WE ARE
    HERE
    2 YEARS AGO!!
    

    View Slide

49. View Slide

50. ?
    

    View Slide

51. ☁
    

    View Slide

52. ⛈
    

    View Slide

53. ⛅
    

    View Slide

54. Application
    

    View Slide

55. Application
    

    View Slide

56. APPLICATION
    Application
    

    View Slide

57. View Slide

58. View Slide

59. Feb
    2015
    Apr
    2019
    1000
    services
    

    View Slide

60. View Slide

61. View Slide

62. View Slide

63. View Slide

64. View Slide

65. View Slide

66. View Slide

67. View Slide

68. package slog

    

    func Error(ctx context.Context, msg string, params ...interface{}) {
    if l := DefaultLogger(); l != nil {
    l.Log(Eventf(DebugSeverity, ctx, msg, params...))
    }
    }
    

    View Slide

69. package slog

    

    func Error(ctx context.Context, msg string, params ...interface{}) {
    if l := DefaultLogger(); l != nil {
    l.Log(Eventf(DebugSeverity, ctx, msg, params...))
    }
    }
    

    View Slide

70. func thing(ctx context.Context, ...) {

    // do things...
    

    slog.Error(ctx, “Oh no!”, err)

    

    }
    

    View Slide

71. func thing(ctx context.Context, ...) {

    // do things...
    

    slog.Error(ctx, “Oh no!”, err)

    

    }
    

    View Slide

72. 2019-04-17 15:38:09.134539276 [DBG] Slog -> Auth request:
    

    {"auth_type": "Auth", "merchant": {"merchant_description": "Deliveroo! LONDON UK",
    "merchant_id": "XXXX"}, "network": “Mastercard" ...}

    {opentracing_id=2855967a9de91afb, trace_id=19ce030f-f652-4b4d-5f35-5d75f472f482,
    service=service.blah, service_version=d7b742fe5c03998748348fe9d7d81a6b8c7358d, hostname=s-
    blah-79f9b54887-j62d2}

    

    (handle_auth_message.go:25 handleAuthMessage)
    

    View Slide

73. 2019-04-17 15:38:09.134539276 [DBG] Slog -> Auth request:
    

    {"auth_type": "Auth", "merchant": {"merchant_description": “Deliveroo! LONDON UK",
    "merchant_id": "XXXX"}, "network": “Mastercard" ...}

    {opentracing_id=2855967a9de91afb, trace_id=19ce030f-f652-4b4d-5f35-5d75f472f482,
    service=service.blah, service_version=d7b742fe5c03998748348fe9d7d81a6b8c7358d, hostname=s-
    blah-79f9b54887-j62d2}

    

    (handle_auth_message.go:25 handleAuthMessage)
    

    View Slide

74. API
    

    View Slide

75. API
    Service
    API
    Service
    API
    Service
    API
    Service
    API
    Service
    API
    Service
    API
    Service
    API
    Service
    Edge Gateway
    

    View Slide

76. Accounts Cards Pots Feed Payments …
    Edge Gateway
    

    View Slide

77. Cards Pots Payments …
    Accounts
    New

    Exciting

    API!!
    
    
    
    
    
    Feed
    Edge Gateway
    

    View Slide

78. Cards Pots Payments …
    Accounts
    New

    Exciting

    API!!
    
    
    
    
    
    
    Feed
    Edge Gateway
    

    View Slide

79. Edge Gateway
    Accounts Cards Pots Payments …
    New

    Exciting

    API!!
    
    
    Feed
    

    View Slide

80. Load Balancer
    Edge Gateway
    

    View Slide

81. Load Balancer
    Edge Gateway Request ID: 19ce030f-f652-4b4d-5f35-5d75f472f482
    

    View Slide

82. // Generate new ID
    // Store value in context
    tid, err := NewTraceID() // v4 UUID
    if err == nil {
    req.Context = context.WithValue(req.Context, "Trace-Id", tid)
    }
    

    View Slide

83. // Generate new ID
    // Store value in context
    tid, err := NewTraceID() // v4 UUID
    if err == nil {
    req.Context = context.WithValue(req.Context, "Trace-Id", tid)
    }
    

    View Slide

84. // Generate new ID
    // Store value in context
    tid, err := NewTraceID() // v4 UUID
    if err == nil {
    req.Context = context.WithValue(req.Context, "Trace-Id", tid)
    }
    

    View Slide

85. Load Balancer
    Edge Gateway Request ID: 19ce030f-f652-4b4d-5f35-5d75f472f482
    Request ID: 19ce030f-f652-4b4d-5f35-5d75f472f482
    API
    Service
    

    View Slide

86. Service
    A
    Service
    B
    Load Balancer
    Edge Gateway Request ID: 19ce030f-f652-4b4d-5f35-5d75f472f482
    Request ID: 19ce030f-f652-4b4d-5f35-5d75f472f482
    Request ID: 19ce030f-f652-4b4d-5f35-5d75f472f482
    API
    Service
    

    View Slide

87. Service
    Service
    

    View Slide

88. Service
    Service
    Context to Headers
    Headers to Context
    HTTP
    

    View Slide

89. // Add headers previously stashed in the context
    if stash, ok := req.Context.Value(ctxKey).(http.Header); ok && stash != nil {
    for k, v := range stash {
    if req.Header[k] == nil {
    req.Header[k] = v
    }
    }
    }
    // Stash headers in the context
    stash := http.Header{}
    for k, v := range req.Header {
    for _, r := range passedHeaders {
    if r.MatchString(k) {
    stash[k] = v
    }
    }
    }
    if len(stash) > 0 {
    req.Context = context.WithValue(req.Context, ctxKey, stash)
    }
    

    View Slide

90. // Add headers previously stashed in the context
    if stash, ok := req.Context.Value(ctxKey).(http.Header); ok && stash != nil {
    for k, v := range stash {
    if req.Header[k] == nil {
    req.Header[k] = v
    }
    }
    }
    // Stash headers in the context
    stash := http.Header{}
    for k, v := range req.Header {
    for _, r := range passedHeaders {
    if r.MatchString(k) {
    stash[k] = v
    }
    }
    }
    if len(stash) > 0 {
    req.Context = context.WithValue(req.Context, ctxKey, stash)
    }
    

    View Slide

91. // Add headers previously stashed in the context
    if stash, ok := req.Context.Value(ctxKey).(http.Header); ok && stash != nil {
    for k, v := range stash {
    if req.Header[k] == nil {
    req.Header[k] = v
    }
    }
    }
    // Stash headers in the context
    stash := http.Header{}
    for k, v := range req.Header {
    for _, r := range passedHeaders {
    if r.MatchString(k) {
    stash[k] = v
    }
    }
    }
    if len(stash) > 0 {
    req.Context = context.WithValue(req.Context, ctxKey, stash)
    }
    

    View Slide

92. // Add headers previously stashed in the context
    if stash, ok := req.Context.Value(ctxKey).(http.Header); ok && stash != nil {
    for k, v := range stash {
    if req.Header[k] == nil {
    req.Header[k] = v
    }
    }
    }
    // Stash headers in the context
    stash := http.Header{}
    for k, v := range req.Header {
    for _, r := range passedHeaders {
    if r.MatchString(k) {
    stash[k] = v
    }
    }
    }
    if len(stash) > 0 {
    req.Context = context.WithValue(req.Context, ctxKey, stash)
    }
    

    View Slide

93. Service
    Service
    Context to Headers
    Headers to Context
    HTTP
    

    View Slide

94. Service
    Service
    Context to Headers
    Headers to Context
    

    View Slide

95. Service
    Service
    ✨
    ✨
    ✨
    HTTP
    HTTP
    HTTP
    

    View Slide

96. View Slide

97. View Slide

98. View Slide

99. Correlate logs
    Observe database queries
    Queue publishing
    Distributed locks
    

    View Slide

100. Correlate logs
     Observe database queries
     Queue publishing
     Distributed locks
     Understand request flow
     What’s slow?!
     What’s broken?!
     

     View Slide

101. View Slide

102. View Slide

103. Service
     Service
     

     View Slide

104. Service
     Service
     cancel() or ctx.Done()
     

     View Slide

105. Service
     Service
     x
     cancel() or ctx.Done()
     Terminate connection (http1)

     Cancel request (http2)
     

     View Slide

106. Service
     Service
     cancel() or ctx.Done()
     context cancelled
     Terminate connection (http1)

     Cancel request (http2)
     x
     

     View Slide

107. Cancel downstream requests
     

     View Slide

108. Cancel outstanding work!
     Decrease tail latencies!
     Reduce load on your database!
     

     View Slide

109. Cancel outstanding work!
     Decrease tail latencies!
     Reduce load on your database!
     
     

     View Slide

110. rspCh := make(chan Response, 1)
     go dispatchRequest(ctx, rspCh)
     

     View Slide

111. rspCh := make(chan Response, 1)
     go dispatchRequest(ctx, rspCh)
     select {
     case rsp := <-rspCh:
     return rsp
     case <-ctx.Done():
     return ctx.Error()
     }
     

     View Slide

112. rspCh := make(chan Response, 1)
     go dispatchRequest(ctx, rspCh)
     select {
     case rsp := <-rspCh:
     return rsp
     case <-ctx.Done():
     return ctx.Error()
     }
     

     View Slide

113. rspCh := make(chan Response, 1)
     go dispatchRequest(ctx, rspCh)
     select {
     case rsp := <-rspCh:
     return rsp
     case <-ctx.Done():
     return ctx.Error()
     }
     

     View Slide

114. // check if the context is cancelled!
     

     select {
     case <-ctx.Done():
     return ctx.Error()
     default:
     }
     wooooooo… let’s go!
     

     View Slide

115. Cancel outstanding work!
     Decrease tail latencies!
     Reduce load on your database!
     

     View Slide

116. Cancel outstanding work!
     Decrease tail latencies!
     Reduce load on your database!

     

     Unless operations should complete…
     
     

     View Slide

117. Service
     A
     Service
     B
     Load Balancer
     Edge Gateway
     API
     Service
     

     View Slide

118. Service
     A
     Service
     B
     Load Balancer
     Edge Gateway
     API
     Service
     Client disconnects
     

     View Slide

119. Service
     A
     Service
     B
     Load Balancer
     Edge Gateway
     API
     Service
     Client disconnects
     Cascading failure!!
     Errors everywhere!!
     Intense panic!
     

     View Slide

120. func(ctx, ...)
     func(ctx, ...)
     func(ctx, ...)
     func(ctx, ...)
     func(ctx, ...)
     

     View Slide

121. func(ctx, ...)
     func(ctx, ...)
     func(context.Background(), ...)
     func(ctx, ...)
     func(ctx, ...)
     

     View Slide

122. func(ctx, ...)
     func(ctx, ...)
     func(context.Background(), ...)
     func(ctx, ...)
     func(ctx, ...)
     DANGER ZONE
     

     View Slide

123. Request scoped variables
     Dependency Injection
     Understand your systems.
     

     View Slide

124. View Slide

125. monzo.com/careers
     

     View Slide