Fency: an idempotency barrier for RabbitMQ consumers

Even when a sender application sends a message only once, the receiver application may receive the message more than once.

The term idempotent is used in mathematics to describe a function that produces the same result if it is applied to itself: f(x) = f(f(x)). In Messaging this concepts translates into a message that has the same effect whether it is received once or multiple times. This means that a message can safely be resent without causing any problems even if the receiver receives duplicates of the same message.

The recipient can explicitly de-dupe messages by keeping track of messages that it already received. A unique message identifier simplifies this task and helps detect those cases where two legitimate messages with the same message content arrive.

In order to detect and eliminate duplicate messages based on the message identifier, the message recipient has to keep a list of already received message identifiers.

https://github.com/fencyio

Transcript

1. Fency An idempotency barrier for RabbitMQ consumers Gilles Robert @ask4gilles
2. Who am I? • Technical architect @EURid • OSS contributor:
3. Agenda • Problem definition • Idempotent consumer • Introducing Fency • Demo • Q&A
4. Problem • Even when a producer only sends a message once, the consumer may receive the message more than once.
5. From the RabbitMQ documentation « In the event of network failure (or a node crashing), messages can be duplicated, and consumers must be prepared to handle them. […] the simplest way to handle this is to ensure that your consumers handle messages in an idempotent way […]. » https://www.rabbitmq.com/reliability.html#consumer
6. How can a consumer deal with duplicate messages? • Design a consumer to be an Idempotent Consumer one that can safely receive the same message multiple times.
7. Idempotent • A function that produces the same result if it is applied to itself: f(x) = f(f(x))
8. How to design an idempotent consumer? • The consumer has to keep track of already received messages through a unique ID
9. Message should have an ID (Producer side) • Message converter does not add one by default
10. • Explicitly add it while creating bean (producer side)
11. Introducing Fency • Homegrown solution available on Github: github.com/fencyio/fency • Spring Boot starter using Redis as backend to store message metadata
12. MessageListenerContainer MessageListener Spring boot auto configuration fency-spring-boot-starter-redis spring-boot-starter-amqp RabbitMQ spring config @IdempotentConsumer Setup
13. Application startup RabbitMqBeanPostProcessor MessageListenerContainer MessageInterceptor Adds a MessageInterceptor to the advice chain fency MessageListener @IdempotentConsumer
14. Runtime Get message MessageListenerContainer MessageListener MessageInterceptor @IdempotentConsumer IdempotencyBarrier @Aspect Message Context 1 2 store message properties retrieve intercept 3 4 If found: discarded 5 If not found: save find proceed
15. What will be stored exactly? • Stored message metadata: message:7aa86931-30dd-47c8-bc2d-c37f0b7cdabf:myQueue • prefix • messageId • consumerQueueName
16. Demo
17. Fine. But my storage will explode!!! What about retention period? • A scheduled cron will delete your keys from Redis, by default, every 1st of the month • Configurable property according to your load/business
18. How can I test it? • RabbitMQ installed • Redis installed ((docker images ;)) • See sample app
19. How can I use it in my project? • Dependency: io.fency:fency-spring-boot- starter-redis:+ • Annotate your consumers with @IdempotentConsumer
20. Q&A