Message-driven application made easy with Watermill – Robert Laszczak

Transcript

1. roblaszczak MESSAGE-DRIVEN APPLICATION MADE EASY WITH WATERMILL
2. roblaszczak MESSAGE-DRIVEN ARCHITECTURE
3. roblaszczak SCALEABLE RESILIENT LOWER COUPLING
4. roblaszczak HARDER TO IMPLEMENT HARDER TO DEBUG
5. roblaszczak BUT… CAN BE BUILDING MESSAGE-DRIVEN APPLICATIONS EASIER?
6. roblaszczak PROVEN PATTERNS
7. roblaszczak WATERMILL
8. roblaszczak COMPANY X ROOM BOOKING PORTAL
9. roblaszczak
10. roblaszczak
11. roblaszczak r.Post("/book-room", func(w http.ResponseWriter, r *http.Request) { // ... if err := bookingRepo.AddBooking(booking); err != nil { panic(err) } err := paymentsInitializer.InitializePayment(p) if err != nil { panic(err) } })
12. roblaszczak
13. roblaszczak
14. roblaszczak HOW TO FIX IT?
15. roblaszczak MAYBE MESSAGE-DRIVEN?
16. roblaszczak WE DIDN’T DONE THAT BEFORE
17. roblaszczak LET’S EXPERIMENT! LET’S USE WATERMILL
18. roblaszczak
19. roblaszczak
20. roblaszczak err := bookingRepo.AddBooking(booking) //... event, err := json.Marshal(RoomBooked{ BookingUUID: booking.ID(), RoomID: req.RoomID, Price: booking.Price(), PaymentChannel: req.PaymentChannel, }) //... err := publisher.Publish( "room_bookings", message.NewMessage(watermill.NewUUID(), event), ) //...
21. roblaszczak import ( "github.com/ThreeDotsLabs/watermill/message/infrastructure/kafka" ) publisher, err := kafka.NewPublisher( []string{"localhost:9092"}, kafka.DefaultMarshaler{}, nil, // Sarama config watermillLogger, )
22. roblaszczak subscriber, err := kafka.NewSubscriber( kafka.SubscriberConfig{ Brokers: []string{"localhost:9092"}, ConsumerGroup: "some_consumer_group", }, nil, kafka.DefaultMarshaler{}, watermillLogger, )
23. roblaszczak watermillRouter, err := message.NewRouter( message.RouterConfig{}, watermillLogger, ) if err != nil { panic(err) } watermillRouter.AddMiddleware(watermillMiddleware.Recoverer)
24. roblaszczak type HandlerFunc func(msg *Message) ([]*Message, error) type HandlerMiddleware func(h HandlerFunc) HandlerFunc
25. roblaszczak watermillRouter.AddNoPublisherHandler( "book_room_handler", // handler name "room_bookings", //subscribe topic subscriber, func(msg *message.Message) ([]*message.Message, error) { event := RoomBooked{} err := json.Unmarshal(msg.Payload, &event) // ... p, err := payment.NewPayment(event.Price, event.Channel) // ... err := paymentsInitializer.InitializePayment(p) // ... return nil, nil }, )
26. roblaszczak
27. roblaszczak watermillRouter.AddMiddleware( watermillMiddleware.Recoverer, watermillMiddleware.Retry{ MaxRetries: 3, WaitTime: time.Second*10, }.Middleware, )
28. roblaszczak
29. roblaszczak
30. roblaszczak ANALOGY 400 Bad Request
31. roblaszczak POISON QUEUE
32. roblaszczak
33. roblaszczak pq, err := watermillMiddleware.NewPoisonQueue( publisher, "poison_queue" ) // ... watermillRouter.AddMiddleware( pq.Middleware, watermillMiddleware.Retry{ MaxRetries: 1, Logger: watermillLogger, }.Middleware, )
34. roblaszczak pq, err := watermillMiddleware.NewPoisonQueueWithFilter( publisher, "poison_queue", func(err error) bool { switch errors.Cause(err).(type) { case *json.InvalidUnmarshalError: return true Default: return false } }, )
35. roblaszczak FEATURE REQUEST: BOOKINGS COUNTER
36. roblaszczak JUST A SQL QUERY?
37. roblaszczak
38. roblaszczak
39. roblaszczak
40. roblaszczak CQRS COMPONENT
41. roblaszczak func (BookingsCounterGenerator) NewEvent() interface{} { return &event.RoomBooked{} } func (b BookingsCounterGenerator) Handle(e interface{}) error { event := e.(*event.RoomBooked) return b.i.IncrementBookingCounter(event.BookingUUID) }
42. roblaszczak FEATURE REQUEST: FULL TEXT SEARCH
43. roblaszczak
44. roblaszczak POLYGLOT PERSISTENCE
45. roblaszczak CONTRACT IS IMPORTANT
46. roblaszczak
47. roblaszczak DATA IS NOT UP-TO DATE
48. roblaszczak EVENTUAL CONSISTENCY
49. roblaszczak
50. roblaszczak IN MOST CASES EVENTUAL CONSISTENCY IS ACCEPTABLE
51. roblaszczak DEBUGGING
52. roblaszczak CORRELATION ID watermillRouter.AddMiddleware( watermillMiddleware.CorrelationIDWithAutogenerate( func() string { return watermill.NewShortUUID() }), )
53. roblaszczak
54. roblaszczak METRICS prometheusRegistry, closeMetricsServer := metrics.CreateRegistryAndServeHTTP(":8081") defer closeMetricsServer() metricsBuilder := metrics.NewPrometheusMetricsBuilder(prometheusRegistry, "", "") metricsBuilder.AddPrometheusRouterMetrics(watermillRouter)
55. roblaszczak
56. roblaszczak TRACING IN PROGRESS :)
57. roblaszczak
58. roblaszczak
59. roblaszczak DUPLICATED BOOKINGS
60. roblaszczak 23:09:35 [INFO] Booking 01D6P9VAFXEGXHAF6MW4ZRZD8V done 23:09:37 [INFO] Booking 01D6P9STY5CCN8RJT4YNTHFZRH done 23:09:37 [ERROR] Lost Kafka connection 23:09:38 [INFO] Connecting to Kafka 23:09:41 [INFO] Booking 01D6P9STY5CCN8RJT4YNTHFZRH done 23:09:35 [INFO] Booking 01D6P9VTE0QNCESP805MVH9Z1K done
61. roblaszczak
62. roblaszczak
63. roblaszczak
64. roblaszczak AT-LEAST-ONCE DELIVERY
65. roblaszczak 23:09:35 [INFO] Booking 01D6P9VAFXEGXHAF6MW4ZRZD8V done 23:09:37 [INFO] Booking 01D6P9STY5CCN8RJT4YNTHFZRH done 23:09:37 [ERROR] Lost Kafka connection 23:09:38 [INFO] Connecting to Kafka 23:09:41 [INFO] Booking 01D6P9STY5CCN8RJT4YNTHFZRH done 23:09:35 [INFO] Booking 01D6P9VTE0QNCESP805MVH9Z1K done
66. roblaszczak SOLUTION: DEDUPLICATION
67. roblaszczak func (m *MemoryBookings) IncrementBookingCounter(bookingUUID string) error { m.lock.Lock() defer m.lock.Unlock() if m.bookings == nil { m.bookings = map[string]struct{}{} } if _, ok := m.bookings[bookingUUID]; ok { // deduplicated return nil } m.bookings[bookingUUID] = struct{}{} m.count += 1 return nil }
68. roblaszczak WHAT’S NEXT?
69. roblaszczak DON’T BE AFRAID TO EXPERIMENT
70. roblaszczak roblaszczak https://watermill.io/ github.com/ThreeDotsLabs/watermill github.com/roblaszczak/reservation-system
71. roblaszczak roblaszczak https://watermill.io/ github.com/ThreeDotsLabs/watermill github.com/roblaszczak/reservation-system THANKS!