Go, Swarm and DevOps vs The Mighty Monolith

Live recording: https://www.youtube.com/watch?v=IoQzscMdwzE

DevOps transformation journey talk presented at London DevOps #38 meetup.

Imagine you get hired and the first thing you see is a product which takes ages to build, test and deploy. And even if you manage to deploy, running it costs a fortune with all these bugs, downtime for multiple reasons and never ending hotfixes. The talk is going to cover how introducing Scrum, DevOps, Docker, Swarm and Go tied together into microservices based ecosystem allowed one small team to do business critical changes fast and safe.

Transcript

1. Go, Swarm and DevOps vs The Mighty Monolith Igor Karpovich
2. Disclaimer Not a true story at all All persons and events are fictitious No gophers were harmed
3. 2013: • Secure in-house infrastructure • Cloud for customer facing products • Monitoring by looking into server rooms • F&G log analysis system (F for find and G for grep) • Jenkins for tests, manual deployments • 4 products
4. 2018 • Multi cluster AWS environment • Container orchestration for all products • Logging, monitoring, tracing • CI/CD for products and infrastructure • 50+ products
5. It's been a long road... • Culture • Architecture • Platform • Legacy
6. Moving away from CDD* • No more favourite IT people to go to • Ticketing system • Change management and deployment awareness • Feedback loop • Incident management • Iterative approach with steady value delivery * Chaos Driven Development
7. DevOps ❤ Docker • Infrastructure as code • Unified environments • No more "works on my machine" • Easy debugging
8. DevOps is not: • A job title • Ops telling Dev to DIY • Giving out unlimited access to production
9. Version Zero First microservice • REST API only • Monitoring coverage • Plugged into core app • Traditional deployment • Written in PHP
10. On the way to the bright future • Maintaining stability • Security by design • Increasing velocity • Supporting innovation
11. Honouring legacy • Minimising incidents • Shutdown planning for software and infrastructure • Replacing monolith pieces with microservices calls
12. Swarm? Out of the box (2016): • Service discovery • Networking, i.e. mesh routing • Volumes (EBS/EFS in AWS) • Orchestration with Compose • Quick start, less maintenance
13. Goblin Microservice framework
14. Some of Goblin's features • Runtime (module) management • Config management • Storage & Cache • Transport (HTTP Server/Client, WS router) • Monitoring client and metrics (Gin, WS router, go runtime) • Logging (context data injection, sentry integration) • Healthchecks • Tracing
15. Healthchecks as metrics • Docker native support • Expose metrics • Push to InfluxDB • Visualisation
16. Distributed tracing Trace everything! • HTTP/WS Server • HTTP Client • SQL • NoSQL • MQ • Events • Emails
17. Platform toolkit • Clusters self-deployment (traefik, CI, monitoring, logging, tracing) • GitOps • Secret management • Service wide event broadcast • SSO, identity management (ACL, JWT, SAML) • Scheduler • Job runner • Change Audit • Multi-route email delivery • Monitoring (InfluxDB, Grafana, Slack, TV dashboards)
18. Palantir Monitoring backend and TV Dashboard featuring visual Swarm • Pulls metrics from InfluxDB • Metric format is Telegraf compatible • YAML definition for Docker Stacks • Metric templates with YAML anchors
19. Questions! Twitter: @ikarpovich