Microservices with RabbitMQ

Reliable communication is essential for microservice based architectures. One of the most effective patterns for microservices communication is message queue. We'll see how a Rubyist can build scalable microservices using RabbitMQ and run it on the AWS infrastructure in Docker containers.

Transcript

1. 
 Microservices with Zoran Majstorović
 github.com/zmajstor
 twitter.com/z0maj
2. Content • Intro
 from monolith decoupling to microservices integration • The Main Part
 microservices messaging with RabbitMQ/AMQP • Ruby Code Refactoring 
 from ActiveRecord Callback to Consumer as a Microservice • A Simple Microservice Deployment
 in docker container to AWS ECS
3. Let's start with a bad example
4. A better example ... or can we just publish the event and let subscribed consumers do the work? a complete to-do list for this event
5. Decoupling Publisher / Producer Subscriber / Consumer Subscriber / Consumer Subscriber / Consumer
6. Monolith vs Microservices vs Complexity in
 Interactions Code
 Complexity
7. Microservice • a small program that handle one task • independently deployable • work together by communicating so that can accomplish the larger task • integrated system on the whole provides value https://martinfowler.com/ articles/microservices.html https://youtu.be/wgdBVIX9ifA
8. App Integration Options • File Transfer • Shared Database • Remote Procedure Invocation • Messaging While all four approaches solve essentially the same problem,
 each style has its distinct advantages and disadvantages. http://www.enterpriseintegrationpatterns.com/patterns/messaging/IntegrationStylesIntro.html
9. File Transfer Consumer A Consumer B Consumer C DATA
 FILE Producer Export Import
10. Shared Database Consumer A Consumer B Consumer C Producer DB
11. Remote Procedure Invocation Consumer A Consumer B Consumer C Producer Call Result
12. Messaging Exchanges Bindings Queues Consumer A Consumer B Consumer C Producer AMQP 0-9-1 Message Broker
13. • a binary, networking protocol with minimal overhead • originated in 2003 by John O'Hara at JPMorgan Chase • version 0.9.1 was specified in 2008 • in 2011 OASIS standards group change it to a significantly different beast
14. AMQP 0-9-1 • virtual host is a logical partition within the server 
 (a multi-tenant system similar to virtual hosts in Apache/Nginx) • each of virtual hosts can have many connections, channels, queues, exchanges and some other things • bindings binds an exchange to a queue or many queues • exchange can be: direct, fanout, with topic or headers • queue is a buffer that holds messages on behalf of a consumer (or consumers)
15. The Producer • external application which creates messages and decides: • how the attributes should be configured for routing • from which exchange the messages should start from • what is the actual payload that is being sent
16. The Message • an atomic unit of processing • consists of: • content (body, bare message or payload) • attributes - metadata about the message like content type, encoding, routing key, whether the message will be persistent or not, and whether it has a priority level, etc. • is created (published) by the producer • may go through more than one exchange before landing in the right queue
17. The Exchange • Direct exchange route messages based on an exact match with the specified routing key • Fanout exchange automatically route the message to all the queues known to them (ignores the routing key) • Topic exchange pattern match on the routing key (topic) to route the messages • Header exchange use the message header attributes for matching the queue • Default (anonymous) exchange is a direct exchange (created automatically) which routes messages with empty exchange name - it compares routing key with the queue name
18. Multiple Queues and Consumers Consumer A Queue 1 Queue 2 Queue 3 Consumer B Consumer C
19. The Consumer • external application which is subscribed to one or more queues • alerted whenever a message shows up in the subscribed queue • can poll the queue at regular intervals to see which messages were added in the queue since the last time it made the request • can send acknowledgments back to RabbitMQ that the message has been: • received (known as ack), or • rejected (nack for negative acknowledgments)
20. Bindings • Routing Key • Publisher: exchange.publish(payload, routing_key: 'foo') • Queue Binding: queue.bind(exchange, routing_key: 'foo') • Headers • Publisher: exchange.publish(payload, headers: { ... }) • Queue Binding: queue.bind(exchange, arguments: { ... })
21. • an message broker implemented with Erlang/OTP • implements all the AMQP 0.9.1 concepts:
 messages, queues, exchanges, bindings, virtual hosts ... • ... and with plugins for other messaging protocols such as
 STOMP, XMPP, and more • has fantastic client support in a variety of popular languages: Ruby, Python, Java, PHP, C#, JavaScript, Go, Elixir, Objective-C, Swift, ...
22. Gems • Bunny - AMPQ client for Ruby (MRI) • March Hare - AMPQ client for JRuby • Sneakers - a background consumer/worker • Hutch - asynchronous inter-service communication • RabbitMQ HTTP API client (various management features) • Ruby RabbitMQ Clients Blog: http:// blog.rubyrabbitmq.info/
23. Features • No message loss • Persistent Messages • Publisher Confirms • Message Acknowledgment • Mirrored Queues • Message Ordering • Dead Letter Exchanges • Alternate Exchanges • Message and Queues TTLs • Consumer Priorities • Federation ... and many more
24. Back to Refactoring ... or can we just publish the event and let subscribed consumers do the work?
25. Introducing Publisher
26. Publish Message with Topic
27. Message Consumer
28. Publish Message with Headers
29. Message Consumer
30. Better Ruby Code https://github.com/jondot/sneakers https://github.com/gocardless/hutch
31. Patterns = proven solutions to recurring problems www.enterpriseintegrationpatterns.com
32. Messaging Patterns (65) Creative Commons Attribution 4.0 license. http://www.enterpriseintegrationpatterns.com/patterns/messaging/index.html
33. Microservice Messaging Benefits • language-agnostic • amplifies loose-coupling • makes scaling and rearchitecting simpler • better reliability and availability - messages will be waiting until the consumer is ready to process them • multiple communication patterns: events, message/reply, notification, async request-response, pub/sub, etc.
34. Microservice Messaging Drawbacks • introducing new complexity into the system 
 (the message broker) • a failure in the message broker can cause severe effects on the system (highly-available message broker) • big messages can cause network congestion • "eventual data consistency" (instead of immediate consistency across different microservices)
35. Message Consumer Microservice Deployment
 with Docker
 to AWS ECS
36. Installing Docker CE on Workstation • macOS: https://store.docker.com/editions/community/docker-ce- desktop-mac • Linux Ubuntu: https://store.docker.com/editions/community/docker-ce- server-ubuntu • or any other Linux distro: https://store.docker.com/search? offering=community&operating_system=linux&type=edition
37. My Ruby Project Structure
38. Dockerfile
39. docker-compose.yml docker-compose build my_consumer:latest
40. Elastic Container Registry (ECR) aws ecr create-repository --repository-name my_consumer eval $(aws ecr get-login) docker tag my_consumer:latest 123456789012.dkr.ecr.us- east-1.amazonaws.com/my_consumer:latest docker push 123456789012.dkr.ecr.us- east-1.amazonaws.com/my_consumer:latest
41. Elastic Container
 Service (ECS) • logical way to group resources (services and tasks) • currently provides two launch types: • EC2 • Fargate (abstract away EC2 instances) ECS resources collection: https://github.com/nathanpeck/awesome-ecs
42. ECS Cluster
 Building Blocks Task Definition • specify the resources for a Docker container or group of containers, such as:
 docker image (registry), ports, CPU/Mem, logs, any volumes, environment variables, etc. Task • running containers according to the Task Definition (one-off or long-running) • TaskRole allow access to S3, DynamoDB, etc. Service • manage long-running tasks (defines desired count and replace failed containers) • integrates with Elastic Load Balancer
43. ECS Task Definition
44. ECS Task aws ecs create-cluster --cluster-name staging aws ecs register-task-definition --cli-input-json file://my-consumer.json aws ecs run-task --launch-type FARGATE 
 --cluster staging 
 --task-definition my-consumer 
 --network-configuration
 "awsvpcConfiguration={subnets=[subnet-ce7c8bf2], securityGroups=[sg-08044d7e]}"
45. ECS Service • runs and maintains the requered number of tasks associated with the elastic load balancer aws ecs create-service --cluster staging 
 --service-name my-service 
 --task-definition my-consumer 
 --desired-count 2 
 --launch-type "FARGATE" 
 --network-configuration "awsvpcConfiguration={subnets=[subnet-ce7c8bf2], securityGroups=[sg-d97138af]}"
46. Deploying Updates 1. Build a new image and push it to the repository 2. Create a new revision of the Task Definition
 (revision numbers increment automatically) 3. Update Service to use new Task Definition revision or simply use: https://github.com/silinternational/ecs-deploy 
 
 ecs-deploy -c staging -n my-service -i 123456789012.dkr.ecr.us- east-1.amazonaws.com/my_consumer:latest
47. Advanced Example of AWS ECS Deployment using Terraform to spin the infrastructure: https://thecode.pub/easy-deploy-your-docker- applications-to-aws-using-ecs-and-fargate- a988a1cc842f VPC with 2 subnets 
 (1 public and 1 private)
 in each Availability Zone
48. AWS 
 CodeBuild AWS 
 CodePipeline screenshots from: https://thecode.pub/easy-deploy-your-docker-applications-to-aws-using-ecs-and-fargate-a988a1cc842f
49. AWS ECS
 vs Kubernetes 1. AWS ECS can not be run on-premise 2. AWS ECS lacks advanced features These two differences can either be seen as weakness or as strengths.
50. That's All!
 Thank You