Microservices with Zoran Majstorović
 github.com/zmajstor
 twitter.com/z0maj

Content • Intro
 from monolith decoupling to microservices integration • The Main Part
 microservices messaging with RabbitMQ/AMQP • Ruby Code Refactoring 
 from ActiveRecord Callback to Consumer as a Microservice • A Simple Microservice Deployment
 in docker container to AWS ECS

Let's start with a bad example

A better example ... or can we just publish the event and let subscribed consumers do the work? a complete to-do list for this event

Decoupling Publisher / Producer Subscriber / Consumer Subscriber / Consumer Subscriber / Consumer

Monolith vs Microservices vs Complexity in
 Interactions Code
 Complexity

Microservice • a small program that handle one task • independently deployable • work together by communicating so that can accomplish the larger task • integrated system on the whole provides value https://martinfowler.com/ articles/microservices.html https://youtu.be/wgdBVIX9ifA

App Integration Options • File Transfer • Shared Database • Remote Procedure Invocation • Messaging While all four approaches solve essentially the same problem,
 each style has its distinct advantages and disadvantages. http://www.enterpriseintegrationpatterns.com/patterns/messaging/IntegrationStylesIntro.html

File Transfer Consumer A Consumer B Consumer C DATA
 FILE Producer Export Import

Shared Database Consumer A Consumer B Consumer C Producer DB

Remote Procedure Invocation Consumer A Consumer B Consumer C Producer Call Result

Messaging Exchanges Bindings Queues Consumer A Consumer B Consumer C Producer AMQP 0-9-1 Message Broker

• a binary, networking protocol with minimal overhead • originated in 2003 by John O'Hara at JPMorgan Chase • version 0.9.1 was specified in 2008 • in 2011 OASIS standards group change it to a significantly different beast

AMQP 0-9-1 • virtual host is a logical partition within the server 
 (a multi-tenant system similar to virtual hosts in Apache/Nginx) • each of virtual hosts can have many connections, channels, queues, exchanges and some other things • bindings binds an exchange to a queue or many queues • exchange can be: direct, fanout, with topic or headers • queue is a buffer that holds messages on behalf of a consumer (or consumers)

The Producer • external application which creates messages and decides: • how the attributes should be configured for routing • from which exchange the messages should start from • what is the actual payload that is being sent

The Message • an atomic unit of processing • consists of: • content (body, bare message or payload) • attributes - metadata about the message like content type, encoding, routing key, whether the message will be persistent or not, and whether it has a priority level, etc. • is created (published) by the producer • may go through more than one exchange before landing in the right queue

The Exchange • Direct exchange route messages based on an exact match with the specified routing key • Fanout exchange automatically route the message to all the queues known to them (ignores the routing key) • Topic exchange pattern match on the routing key (topic) to route the messages • Header exchange use the message header attributes for matching the queue • Default (anonymous) exchange is a direct exchange (created automatically) which routes messages with empty exchange name - it compares routing key with the queue name

Multiple Queues and Consumers Consumer A Queue 1 Queue 2 Queue 3 Consumer B Consumer C

The Consumer • external application which is subscribed to one or more queues • alerted whenever a message shows up in the subscribed queue • can poll the queue at regular intervals to see which messages were added in the queue since the last time it made the request • can send acknowledgments back to RabbitMQ that the message has been: • received (known as ack), or • rejected (nack for negative acknowledgments)

Bindings • Routing Key • Publisher: exchange.publish(payload, routing_key: 'foo') • Queue Binding: queue.bind(exchange, routing_key: 'foo') • Headers • Publisher: exchange.publish(payload, headers: { ... }) • Queue Binding: queue.bind(exchange, arguments: { ... })

• an message broker implemented with Erlang/OTP • implements all the AMQP 0.9.1 concepts:
 messages, queues, exchanges, bindings, virtual hosts ... • ... and with plugins for other messaging protocols such as
 STOMP, XMPP, and more • has fantastic client support in a variety of popular languages: Ruby, Python, Java, PHP, C#, JavaScript, Go, Elixir, Objective-C, Swift, ...

Gems • Bunny - AMPQ client for Ruby (MRI) • March Hare - AMPQ client for JRuby • Sneakers - a background consumer/worker • Hutch - asynchronous inter-service communication • RabbitMQ HTTP API client (various management features) • Ruby RabbitMQ Clients Blog: http:// blog.rubyrabbitmq.info/

Features • No message loss • Persistent Messages • Publisher Confirms • Message Acknowledgment • Mirrored Queues • Message Ordering • Dead Letter Exchanges • Alternate Exchanges • Message and Queues TTLs • Consumer Priorities • Federation ... and many more

Back to Refactoring ... or can we just publish the event and let subscribed consumers do the work?

Introducing Publisher

Publish Message with Topic

Message Consumer

Publish Message with Headers

Message Consumer

Better Ruby Code https://github.com/jondot/sneakers https://github.com/gocardless/hutch

Patterns = proven solutions to recurring problems www.enterpriseintegrationpatterns.com

Messaging Patterns (65) Creative Commons Attribution 4.0 license. http://www.enterpriseintegrationpatterns.com/patterns/messaging/index.html

Microservice Messaging Benefits • language-agnostic • amplifies loose-coupling • makes scaling and rearchitecting simpler • better reliability and availability - messages will be waiting until the consumer is ready to process them • multiple communication patterns: events, message/reply, notification, async request-response, pub/sub, etc.

Microservice Messaging Drawbacks • introducing new complexity into the system 
 (the message broker) • a failure in the message broker can cause severe effects on the system (highly-available message broker) • big messages can cause network congestion • "eventual data consistency" (instead of immediate consistency across different microservices)

Message Consumer Microservice Deployment
 with Docker
 to AWS ECS

Installing Docker CE on Workstation • macOS: https://store.docker.com/editions/community/docker-ce- desktop-mac • Linux Ubuntu: https://store.docker.com/editions/community/docker-ce- server-ubuntu • or any other Linux distro: https://store.docker.com/search? offering=community&operating_system=linux&type=edition

My Ruby Project Structure

Dockerfile

docker-compose.yml docker-compose build my_consumer:latest

Elastic Container Registry (ECR) aws ecr create-repository --repository-name my_consumer eval $(aws ecr get-login) docker tag my_consumer:latest 123456789012.dkr.ecr.us- east-1.amazonaws.com/my_consumer:latest docker push 123456789012.dkr.ecr.us- east-1.amazonaws.com/my_consumer:latest

Elastic Container
 Service (ECS) • logical way to group resources (services and tasks) • currently provides two launch types: • EC2 • Fargate (abstract away EC2 instances) ECS resources collection: https://github.com/nathanpeck/awesome-ecs

ECS Cluster
 Building Blocks Task Definition • specify the resources for a Docker container or group of containers, such as:
 docker image (registry), ports, CPU/Mem, logs, any volumes, environment variables, etc. Task • running containers according to the Task Definition (one-off or long-running) • TaskRole allow access to S3, DynamoDB, etc. Service • manage long-running tasks (defines desired count and replace failed containers) • integrates with Elastic Load Balancer

ECS Task Definition

ECS Task aws ecs create-cluster --cluster-name staging aws ecs register-task-definition --cli-input-json file://my-consumer.json aws ecs run-task --launch-type FARGATE 
 --cluster staging 
 --task-definition my-consumer 
 --network-configuration
 "awsvpcConfiguration={subnets=[subnet-ce7c8bf2], securityGroups=[sg-08044d7e]}"

ECS Service • runs and maintains the requered number of tasks associated with the elastic load balancer aws ecs create-service --cluster staging 
 --service-name my-service 
 --task-definition my-consumer 
 --desired-count 2 
 --launch-type "FARGATE" 
 --network-configuration "awsvpcConfiguration={subnets=[subnet-ce7c8bf2], securityGroups=[sg-d97138af]}"

Deploying Updates 1. Build a new image and push it to the repository 2. Create a new revision of the Task Definition
 (revision numbers increment automatically) 3. Update Service to use new Task Definition revision or simply use: https://github.com/silinternational/ecs-deploy 
 
 ecs-deploy -c staging -n my-service -i 123456789012.dkr.ecr.us- east-1.amazonaws.com/my_consumer:latest

Advanced Example of AWS ECS Deployment using Terraform to spin the infrastructure: https://thecode.pub/easy-deploy-your-docker- applications-to-aws-using-ecs-and-fargate- a988a1cc842f VPC with 2 subnets 
 (1 public and 1 private)
 in each Availability Zone

AWS 
 CodeBuild AWS 
 CodePipeline screenshots from: https://thecode.pub/easy-deploy-your-docker-applications-to-aws-using-ecs-and-fargate-a988a1cc842f

AWS ECS
 vs Kubernetes 1. AWS ECS can not be run on-premise 2. AWS ECS lacks advanced features These two differences can either be seen as weakness or as strengths.

That's All!
 Thank You