Slide 1

Slide 1 text


 Microservices with Zoran Majstorović
 github.com/zmajstor
 twitter.com/z0maj

Slide 2

Slide 2 text

Content • Intro
 from monolith decoupling to microservices integration • The Main Part
 microservices messaging with RabbitMQ/AMQP • Ruby Code Refactoring 
 from ActiveRecord Callback to Consumer as a Microservice • A Simple Microservice Deployment
 in docker container to AWS ECS

Slide 3

Slide 3 text

Let's start with a bad example

Slide 4

Slide 4 text

A better example ... or can we just publish the event and let subscribed consumers do the work? a complete to-do list for this event

Slide 5

Slide 5 text

Decoupling Publisher / Producer Subscriber / Consumer Subscriber / Consumer Subscriber / Consumer

Slide 6

Slide 6 text

Monolith vs Microservices vs Complexity in
 Interactions Code
 Complexity

Slide 7

Slide 7 text

Microservice • a small program that handle one task • independently deployable • work together by communicating so that can accomplish the larger task • integrated system on the whole provides value https://martinfowler.com/ articles/microservices.html https://youtu.be/wgdBVIX9ifA

Slide 8

Slide 8 text

App Integration Options • File Transfer • Shared Database • Remote Procedure Invocation • Messaging While all four approaches solve essentially the same problem,
 each style has its distinct advantages and disadvantages. http://www.enterpriseintegrationpatterns.com/patterns/messaging/IntegrationStylesIntro.html

Slide 9

Slide 9 text

File Transfer Consumer A Consumer B Consumer C DATA
 FILE Producer Export Import

Slide 10

Slide 10 text

Shared Database Consumer A Consumer B Consumer C Producer DB

Slide 11

Slide 11 text

Remote Procedure Invocation Consumer A Consumer B Consumer C Producer Call Result

Slide 12

Slide 12 text

Messaging Exchanges Bindings Queues Consumer A Consumer B Consumer C Producer AMQP 0-9-1 Message Broker

Slide 13

Slide 13 text

• a binary, networking protocol with minimal overhead • originated in 2003 by John O'Hara at JPMorgan Chase • version 0.9.1 was specified in 2008 • in 2011 OASIS standards group change it to a significantly different beast

Slide 14

Slide 14 text

AMQP 0-9-1 • virtual host is a logical partition within the server 
 (a multi-tenant system similar to virtual hosts in Apache/Nginx) • each of virtual hosts can have many connections, channels, queues, exchanges and some other things • bindings binds an exchange to a queue or many queues • exchange can be: direct, fanout, with topic or headers • queue is a buffer that holds messages on behalf of a consumer (or consumers)

Slide 15

Slide 15 text

The Producer • external application which creates messages and decides: • how the attributes should be configured for routing • from which exchange the messages should start from • what is the actual payload that is being sent

Slide 16

Slide 16 text

The Message • an atomic unit of processing • consists of: • content (body, bare message or payload) • attributes - metadata about the message like content type, encoding, routing key, whether the message will be persistent or not, and whether it has a priority level, etc. • is created (published) by the producer • may go through more than one exchange before landing in the right queue

Slide 17

Slide 17 text

The Exchange • Direct exchange route messages based on an exact match with the specified routing key • Fanout exchange automatically route the message to all the queues known to them (ignores the routing key) • Topic exchange pattern match on the routing key (topic) to route the messages • Header exchange use the message header attributes for matching the queue • Default (anonymous) exchange is a direct exchange (created automatically) which routes messages with empty exchange name - it compares routing key with the queue name

Slide 18

Slide 18 text

Multiple Queues and Consumers Consumer A Queue 1 Queue 2 Queue 3 Consumer B Consumer C

Slide 19

Slide 19 text

The Consumer • external application which is subscribed to one or more queues • alerted whenever a message shows up in the subscribed queue • can poll the queue at regular intervals to see which messages were added in the queue since the last time it made the request • can send acknowledgments back to RabbitMQ that the message has been: • received (known as ack), or • rejected (nack for negative acknowledgments)

Slide 20

Slide 20 text

Bindings • Routing Key • Publisher: exchange.publish(payload, routing_key: 'foo') • Queue Binding: queue.bind(exchange, routing_key: 'foo') • Headers • Publisher: exchange.publish(payload, headers: { ... }) • Queue Binding: queue.bind(exchange, arguments: { ... })

Slide 21

Slide 21 text

• an message broker implemented with Erlang/OTP • implements all the AMQP 0.9.1 concepts:
 messages, queues, exchanges, bindings, virtual hosts ... • ... and with plugins for other messaging protocols such as
 STOMP, XMPP, and more • has fantastic client support in a variety of popular languages: Ruby, Python, Java, PHP, C#, JavaScript, Go, Elixir, Objective-C, Swift, ...

Slide 22

Slide 22 text

Gems • Bunny - AMPQ client for Ruby (MRI) • March Hare - AMPQ client for JRuby • Sneakers - a background consumer/worker • Hutch - asynchronous inter-service communication • RabbitMQ HTTP API client (various management features) • Ruby RabbitMQ Clients Blog: http:// blog.rubyrabbitmq.info/

Slide 23

Slide 23 text

Features • No message loss • Persistent Messages • Publisher Confirms • Message Acknowledgment • Mirrored Queues • Message Ordering • Dead Letter Exchanges • Alternate Exchanges • Message and Queues TTLs • Consumer Priorities • Federation ... and many more

Slide 24

Slide 24 text

Back to Refactoring ... or can we just publish the event and let subscribed consumers do the work?

Slide 25

Slide 25 text

Introducing Publisher

Slide 26

Slide 26 text

Publish Message with Topic

Slide 27

Slide 27 text

Message Consumer

Slide 28

Slide 28 text

Publish Message with Headers

Slide 29

Slide 29 text

Message Consumer

Slide 30

Slide 30 text

Better Ruby Code https://github.com/jondot/sneakers https://github.com/gocardless/hutch

Slide 31

Slide 31 text

Patterns = proven solutions to recurring problems www.enterpriseintegrationpatterns.com

Slide 32

Slide 32 text

Messaging Patterns (65) Creative Commons Attribution 4.0 license. http://www.enterpriseintegrationpatterns.com/patterns/messaging/index.html

Slide 33

Slide 33 text

Microservice Messaging Benefits • language-agnostic • amplifies loose-coupling • makes scaling and rearchitecting simpler • better reliability and availability - messages will be waiting until the consumer is ready to process them • multiple communication patterns: events, message/reply, notification, async request-response, pub/sub, etc.

Slide 34

Slide 34 text

Microservice Messaging Drawbacks • introducing new complexity into the system 
 (the message broker) • a failure in the message broker can cause severe effects on the system (highly-available message broker) • big messages can cause network congestion • "eventual data consistency" (instead of immediate consistency across different microservices)

Slide 35

Slide 35 text

Message Consumer Microservice Deployment
 with Docker
 to AWS ECS

Slide 36

Slide 36 text

Installing Docker CE on Workstation • macOS: https://store.docker.com/editions/community/docker-ce- desktop-mac • Linux Ubuntu: https://store.docker.com/editions/community/docker-ce- server-ubuntu • or any other Linux distro: https://store.docker.com/search? offering=community&operating_system=linux&type=edition

Slide 37

Slide 37 text

My Ruby Project Structure

Slide 38

Slide 38 text

Dockerfile

Slide 39

Slide 39 text

docker-compose.yml docker-compose build my_consumer:latest

Slide 40

Slide 40 text

Elastic Container Registry (ECR) aws ecr create-repository --repository-name my_consumer eval $(aws ecr get-login) docker tag my_consumer:latest 123456789012.dkr.ecr.us- east-1.amazonaws.com/my_consumer:latest docker push 123456789012.dkr.ecr.us- east-1.amazonaws.com/my_consumer:latest

Slide 41

Slide 41 text

Elastic Container
 Service (ECS) • logical way to group resources (services and tasks) • currently provides two launch types: • EC2 • Fargate (abstract away EC2 instances) ECS resources collection: https://github.com/nathanpeck/awesome-ecs

Slide 42

Slide 42 text

ECS Cluster
 Building Blocks Task Definition • specify the resources for a Docker container or group of containers, such as:
 docker image (registry), ports, CPU/Mem, logs, any volumes, environment variables, etc. Task • running containers according to the Task Definition (one-off or long-running) • TaskRole allow access to S3, DynamoDB, etc. Service • manage long-running tasks (defines desired count and replace failed containers) • integrates with Elastic Load Balancer

Slide 43

Slide 43 text

ECS Task Definition

Slide 44

Slide 44 text

ECS Task aws ecs create-cluster --cluster-name staging aws ecs register-task-definition --cli-input-json file://my-consumer.json aws ecs run-task --launch-type FARGATE 
 --cluster staging 
 --task-definition my-consumer 
 --network-configuration
 "awsvpcConfiguration={subnets=[subnet-ce7c8bf2], securityGroups=[sg-08044d7e]}"

Slide 45

Slide 45 text

ECS Service • runs and maintains the requered number of tasks associated with the elastic load balancer aws ecs create-service --cluster staging 
 --service-name my-service 
 --task-definition my-consumer 
 --desired-count 2 
 --launch-type "FARGATE" 
 --network-configuration "awsvpcConfiguration={subnets=[subnet-ce7c8bf2], securityGroups=[sg-d97138af]}"

Slide 46

Slide 46 text

Deploying Updates 1. Build a new image and push it to the repository 2. Create a new revision of the Task Definition
 (revision numbers increment automatically) 3. Update Service to use new Task Definition revision or simply use: https://github.com/silinternational/ecs-deploy 
 
 ecs-deploy -c staging -n my-service -i 123456789012.dkr.ecr.us- east-1.amazonaws.com/my_consumer:latest

Slide 47

Slide 47 text

Advanced Example of AWS ECS Deployment using Terraform to spin the infrastructure: https://thecode.pub/easy-deploy-your-docker- applications-to-aws-using-ecs-and-fargate- a988a1cc842f VPC with 2 subnets 
 (1 public and 1 private)
 in each Availability Zone

Slide 48

Slide 48 text

AWS 
 CodeBuild AWS 
 CodePipeline screenshots from: https://thecode.pub/easy-deploy-your-docker-applications-to-aws-using-ecs-and-fargate-a988a1cc842f

Slide 49

Slide 49 text

AWS ECS
 vs Kubernetes 1. AWS ECS can not be run on-premise 2. AWS ECS lacks advanced features These two differences can either be seen as weakness or as strengths.

Slide 50

Slide 50 text

That's All!
 Thank You