Voxxed Days SG 2018 - Modernize Application Development with Highly Scalable Architecture using Docker and Azure

Slidedeck from the presentation done at Voxed Days Singapore event held on 1st June 2018.

The talk demonstrated the end to end workflow of containerizing an application for managing Tech Talks. Using Docker multi stage builds and docker compose, the multi-container app was tested locally on a single node Kubernetes cluster using Minikube.

The demo was scaled to Azure cloud using managed Kubernetes instance using Azure Container Service (AKS). Scalability features were added using messaging layer with RabbitMQ. ElasticSearch with Kibana was used to build a self service UI for a denormalised representation of data and for visualisations Kibana provided a real time dashboard. e stateful persistence was handled using managed SQL server running outside of Kubernetes cluster on Azure.
The demo concluded with incorporating the Container Monitoring Solution using Prometheus and Grafana.

The video of this talk is available at Engineers.SG - https://engineers.sg/video/modernize-application-development-with-highly-scalable-architecture-using-docker-azure-cloud--2683
Alternately, same can be accessed directly from Youtube - https://youtu.be/M5mjJga1joM

Transcript

1. Modernize Application Development with Highly Scalable Architecture using Docker & Azure Nilesh Gule
2. $whoami { “name” : “Nilesh Gule”, “title” : “Architect at Credit Agricole CIB”, “website” : “https://www.HandsOnArchitect.com", “github” : “https://github.com/NileshGule" “twitter” : “@nileshgule”, “linkedin” : “https://www.linkedin.com/in/nileshgule”, “email” : “nileshgule@gmail.com", “likes” : “Technical Evangelism, Cricket” }
3. Agenda - Managed Kubernetes cluster (AKS) on Azure - State Management with Persistent Volume - Scale using Messaging - Index data using ELK - Monitor containerized workloads using Prometheus - Containerize Apps with Docker - Container Orchestration using Kubernetes (Minikube)
4. Application Overview
5. Containerize Apps
6. Inner Loop Development Workflow Source: https://docs.microsoft.com/en-us/dotnet/standard/microservices-architecture/docker-application-development-process/docker-app-development-workflow
7. Docker Multi Stage build
8. Docker-compose
9. Kubernetes
10. Kubernetes Open-source system for automating deployment, scaling, and management of containerized application across clusters of machines Basic Objects - Namespace - Pod - Volume - Service Controllers - ReplicaSet - Deployment - StatefulSet - DemonSet
11. Kubernetes - Pod - Smallest unit of deployment - Group of one or more containers - Contains specification how to run container along with its related resources (storage / networking)
12. Kubernetes - Deployment - Desired state definitions for pods and replica sets - Defines - version of container images - number of replicas - update strategy - Metadata to interact with resources
13. Node 2 Kubernetes - Service - Acts as Load Balancers for pods - Provides stable IP for consumers Node 1 TechTalksWeb Pod1 TechTalksAPI Pod2 Pod1
14. Minikube local cluster
15. Demo - Minikube
16. Managed Kubernetes Cluster (AKS) on Azure
17. Provision AKS cluster
18. Kubernetes cluster resources Resources - Network Security Group - Virtual Machine - Virtual Network - Public IP - Load balancer
19. State Management with Persistent Volume
20. Persistent Volumes Node1 Node 2 Node 3 Node1 Node 2 Node 3 Node1 Node 2 Node 3 Node1 Node 2 Node 3 Without PV With PV
21. Demo - AKS Deployment
22. Scale using Messaging
23. Queue based load levelling Source: https://azureinteractives.azurewebsites.net/CloudDesignPatterns/default.html nileshgule/ techtalksprocessor nileshgule/ techtalksweb nileshgule/ sqlclient techtalksapi pivotalrabbitmq/ rabbitmq-autocluster microsoft/ mssql-linux-server
24. Indexing using Elasticsearch
25. - AKS Cluster Visualize Denormalized Data with Elasticsearch & Kibana nileshgule/ techtalksweb nileshgule/sqlclient nileshgule/techtalksapi microsoft/mssql-linux-server nileshgule/techtalksprocessor nileshgule/techtalkselkprocessor sebp/elk pivotalrabbitmq/ rabbitmq-autocluster
26. Elasticsearch And Kibana
27. - Managed SQL Server TechTalksDB AKS Cluster Service Discovery External Communication pivotalrabbitmq/ rabbitmq-autocluster nileshgule/techtalksweb nileshgule/sqlclient nileshgule/techtalksapi nileshgule/techtalksprocessor
28. Monitoring using Prometheus with Grafana
29. Container Monitoring Solution using Prometheus
30. - Summary Build & Test Phase Deploy & Run Phase
31. References ✴ Demo code - https://github.com/NileshGule/VoxxedDaysSG-2018 ✴ EshopOnContainers reference app (EventBus, CQRS) - https://github.com/dotnet-architecture/eShopOnContainers ✴ OnlineStore reference app - https://github.com/mirfanmcs/OnlineStore ✴ Docker getting started - https://docs.docker.com/get-started/ ✴ Docker Compose V3 - https://docs.docker.com/compose/compose-file/#build ✴ Docker Multi-Stage builds - https://docs.docker.com/engine/userguide/eng-image/multistage-build/ ✴ Kubernetes Basics - https://kubernetes.io/docs/tutorials/kubernetes-basics/ ✴ Azure Interactives - https://azureinteractives.azurewebsites.net ✴ Twelve Factor App - https://12factor.net
32. Thank you very much NileshGule/VoxxedDaysSG-2018 Code with Passion and Strive for Excellence
33. Q&A