Kamal 2 – Get Out of the Cloud

Slide 1

Slide 1 text

Kamal 2 – Get out of the cloud Igor Aleksandrov https://github.com/igor-alexandrov

Slide 2

Slide 2 text

Who am I? ● On Rails since 2009 (RoR 2.2+) ● CTO and co-founder of JetRockets ● Docker Captain and Open-Source enthusiast ● Kamal contributor since 0.14.0 release

Slide 3

Slide 3 text

I live in Georgia (the country) ● Georgia has a wine tradition of more than 8000 years (Google it) ● UNESCO honors the cultural signiﬁcance of the Georgian alphabet ● Come to visit Georgia! (მობრძანდით საქართველოში!)

Slide 4

Slide 4 text

Why does it matter to talk about the deployment? ● Delivering changes to production is often a bottleneck of the development process ● Heroku, Fly.io and Render give an excellent developer experience, but cost you premium ● Hosted Kubernetes on AWS, GCP, Digital Ocean lock you to the vendor and cost you premium ● More complex infrastructure solutions require more engineers in your team

Slide 5

Slide 5 text

Develop → Deliver

Slide 6

Slide 6 text

What is Kamal? ● Essentially it is an orchestration tool around Docker ● If you know Docker – no additional knowledge is needed ● Written in Ruby on Go

Slide 7

Slide 7 text

2915 LoC Kamal (MRSK) v0.14.0, counted by https://github.com/AlDanial/cloc 9004 including tests and comments

Slide 8

Slide 8 text

5537 LoC Kamal v2.5.3, February 27, 2025 17456 including tests and comments

Slide 9

Slide 9 text

Is it a lot? ● Sidekiq – 6030 LoC ● Devise – 3445 LoC ● AASM – 2139 LoC ● Solid Queue – 1670 LoC Kamal has 5537 LoC in the v2.5.3 release.

Slide 10

Slide 10 text

The core idea of Kamal?

Slide 11

Slide 11 text

Capistrano for containers

Slide 12

Slide 12 text

Capistrano for containers It sounds obvious, but what does it mean? ● It provides zero downtime deployments ● It is imperative and simple ● It is eﬃcient

Slide 13

Slide 13 text

What are zero downtime deployments? Capistrano ● Rely on the application server (e.g. Puma phased restart) Docker ● Run Docker commands ○ Start new container and wait until it is healthy ○ Notify the proxy server about the new container ○ Tell the proxy server to stop routing requests for the old container ○ Stop old container

Slide 14

Slide 14 text

Being imperative Capistrano ● Declare commands with DSL ● Run each command and wait until it is ﬁnished Kubernetes & Docker Swarm ● You are suggesting, not insisting on your actions

Slide 15

Slide 15 text

Being eﬃcient ● Utilize Docker’s layer caching ● Not being as slow as AWS Beanstalk ● Not add any additional complexity

Slide 16

Slide 16 text

What’s new in Kamal 2?

Slide 17

Slide 17 text

Traeﬁk replaced by kamal-proxy

Slide 18

Slide 18 text

What were the reasons for this? Traefik looks like an almost perfect solution, doesn’t it? 1. Docker-ready 2. Declarative configuration ○ Uses Docker labels to define routing 3. Automatic routing ○ Traffic to the new container is routed automatically 4. Self-healing ○ If the container restarts, Traefik dynamically updates routing

Slide 19

Slide 19 text

No content

Slide 20

Slide 20 text

1. Label-based conﬁguration is challenging to maintain

Slide 21

Slide 21 text

No content

Slide 22

Slide 22 text

No content

Slide 23

Slide 23 text

2. Kamal was used for something, it wasn’t designed for

Slide 24

Slide 24 text

No content

Slide 25

Slide 25 text

3. Traeﬁk is declarative

Slide 26

Slide 26 text

- Okay… - But what does it mean?

Slide 27

Slide 27 text

No content

Slide 28

Slide 28 text

No content

Slide 29

Slide 29 text

We had no control over what was going on

Slide 30

Slide 30 text

4. Docker container’s labels are immutable

Slide 31

Slide 31 text

Container deployment plan 1. Start the “new” container while the “old” is running. 2. Wait for the “new” to become healthy 3. Stop the “old” container. Easy-peasy?

Slide 32

Slide 32 text

No content

Slide 33

Slide 33 text

Container deployment plan 1. Start the “new” container while the “old” is running. 2. Wait for the “new” to become healthy 3. Stop the “old” container.

Slide 34

Slide 34 text

It takes time for Traeﬁk to update its conﬁguration

Slide 35

Slide 35 text

Container deployment plan 1. Start the “new” container while the “old” is running. 2. Wait for the “new” to become healthy 3. “Tell” Traefik to stop routing requests to the “old” 4. Wait while Traefik will actually stop routing to the “old” container 5. Stop the “old” container. But Docker container’s labels are immutable and Traefik is declarative, we cannot “tell” him anything.

Slide 36

Slide 36 text

Workaround – plug each container with a “cord” to a socket

Slide 37

Slide 37 text

What is a cord? It is a modiﬁed health check command When starting a new container – tie the cord ● --volume ~/.kamal/cords/onetribe-web-production-cf6ccd:/tmp/kamal-cord ● --health-cmd "(curl -f http://localhost:3000/up || exit 1) && (stat /tmp/kamal-cord/cord > /dev/null || exit 1)" Before stopping the container – cut the cord ● rm -r ~/.kamal/cords/onetribe-web-production-938b7e

Slide 38

Slide 38 text

Container deployment plan 1. Create a directory with a single file on the host 2. Start the “new” container, mounting the cord file into /tmp and including a check for the file in the docker health check 3. Wait for the “new” to become healthy 4. Delete the health check file (“cut the cord”) for the “old” container 5. Wait for it to become unhealthy and give Traefik a couple of seconds to notice 6. Stop the “old” container

Slide 39

Slide 39 text

kamal-proxy eliminates these problems ● Conﬁgurable only with options ● Has Let’s Encrypt built-in ● Imperative `kamal-proxy deploy onetribe-pghero --target c67f2259dce6:8080 --host pghero.onetribe.team --tls` The command above will wait until the host is provisioned and the container is healthy. There also are `kamal-proxy remove`, `kamal-proxy list`, etc.

Slide 40

Slide 40 text

No content

Slide 41

Slide 41 text

Longest discussion about Kamal 1.x

Slide 42

Slide 42 text

Kamal 2 adds SSL with 2 lines of code The only limitation – it requires deploying to a single web server, but you can deploy queue and accessories to others.

Slide 43

Slide 43 text

Thruster

Slide 44

Slide 44 text

One more Rails trifecta ● Kamal ● kamal-proxy (ex. Parachute) ● Thruster (the most incomprehensible new thing)

Slide 45

Slide 45 text

Nginx was a common solution in the Rails stack

Slide 46

Slide 46 text

What were Nginx responsibilities? ● Reverse proxy ● Routing HTTP requests ● Static ﬁles serving ● Caching ● Compression ● Rate limiting (optional)

Slide 47

Slide 47 text

No content

Slide 48

Slide 48 text

No content

Slide 49

Slide 49 text

No content

Slide 50

Slide 50 text

Kamal kamal-proxy ● Reverse proxy ● Routing HTTP requests Puma ● Serving Static Files ● Rate Limiting (optional) Who should be responsible for `caching` and `compression`?

Slide 51

Slide 51 text

No content

Slide 52

Slide 52 text

Thruster ● Go wrapper around Puma, other app servers are not (yet?) supported ● Distributed as a gem (gem “thruster”) ● Easy to run (“./bin/thrust ./bin/rails server”) ● Handles: ○ Serving static ﬁles with “X-Sendﬁle” support ○ Caching of public assets ○ Compression

Slide 53

Slide 53 text

Kamal 2 in production

Slide 54

Slide 54 text

Harden your instance! ● `kamal setup` only installs Docker if it is missing ● No fail2ban, no ﬁrewall, hardening is a must! ● DO NOT EXPOSE the Docker daemon on port 2375 If you are using Ansible – https://github.com/dev-sec/ansible-collection-hardening

Slide 55

Slide 55 text

Learn Docker basics! To use Kamal you should be familiar with terminology: ● Container ● Image ● Registry Have any questions? User Docker docs. https://docs.docker.com/get-started/docker-concepts/the-basics/what-is-a-conta iner/

Slide 56

Slide 56 text

Keep the Docker registry close ● Kamal needs a Docker registry (at least for now) ● Every deploy – the new image is built, pushed and pulled ● ghcr.io, Docker Hub, ECR, DigitalOcean Container Registry, ● You will pay for the traﬃc

Slide 57

Slide 57 text

Make Kamal a part of CI/CD process ● Kamal can build images locally ● I insist you to use Kamal as a CD workﬂow instead: ○ Your architecture can be different from the architecture you deploy to ○ Usually ARM64 locally vs x86-64 on the server (we all love Mac). ○ Your internet connection may be slow and not stable (like in the hotel, here) ● Kamal integrates well with any CI/CD platform: GH Actions, GitLab CI, BitBucket Pipeline If you are on GitHub Actions – https://github.com/igor-alexandrov/kamal-github-actions

Slide 58

Slide 58 text

Keep your workﬂows in a good condition ● On GitHub – use composite actions ● Don’t reinvent a bicycle, use existing actions e.g., ○ https://github.com/webfactory/ssh-agent ○ https://github.com/ruby/setup-ruby Use Kamal as an action too – https://github.com/igor-alexandrov/kamal-deploy

Slide 59

Slide 59 text

Concurrent deployments handling ● The default behavior of GitHub Actions is to allow multiple jobs or workﬂow runs to run concurrently. ● Your deployments should not run in parallel for the same environment

Slide 60

Slide 60 text

Concurrent deployments handling

Slide 61

Slide 61 text

Keep your secrets safe! ● NEVER provide direct values to `.kamal/secrets` ● Two options that are suitable for a production usage: ○ Store secrets in GitHub Secrets and substitute them from environmental variables ○ Store secrets in a password manager ● I love the ﬁrst option because of no additional services

Slide 62

Slide 62 text

Run migrations once! Rails' default Dockerﬁle sets an entry point to `/rails/bin/docker-entrypoint` by default.

Slide 63

Slide 63 text

Run migrations once! If the command is `./bin/rails server`, then `./bin/rails db:prepare` is executed, which also runs DB migrations. It knows nothing about the server on which it is executed.

Slide 64

Slide 64 text

But what if your setup has multiple web containers?

Slide 65

Slide 65 text

No content

Slide 66

Slide 66 text

Run migrations once! Use `pre-deploy` hook instead. Kamal provides all the required variables and will ensure to run the command only on the primary server.

Slide 67

Slide 67 text

Logs! Logs! Logs! ● Kamal logging uses Docker logging under the hood ● Kamal limits maximum ﬁle size to 10 megabytes by default ● Don’t forget to choose the right logging driver ○ json-ﬁle (default) ○ local ○ awslogs ○ gcplogs ● You should have a log viewer (e.g. CloudWatch, Dozzle, Logdy) https://dozzle.dev OR https://logdy.dev

Slide 68

Slide 68 text

You can have multiple Kamal configs ● The most common use case – microservices in a monorepo ● Another use case – multi-tenant or multi-region deployments Kamal allows selecting the config file to use ./bin/kamal --config-file ./config/deploy.service1.yml

Slide 69

Slide 69 text

Kamal is not a silver bullet! ● Don’t forget to conﬁgure a backup for your database ● Don’t forget to set up monitoring ● Don’t forget to update your Dockerﬁle dependencies ● Keep your application up to date too ● Know what you are doing and have fun!

Slide 70

Slide 70 text

Future Roadmap

Slide 71

Slide 71 text

“Kamal is in a great place now” – DHH

Slide 72

Slide 72 text

What can be improved Kamal ● Get rid of remote image repository requirement kamal-proxy ● Maintenance mode

Slide 73

Slide 73 text

Thank you! AMA. https://x.com/igor_alexandrov https://www.linkedin.com/in/igor-alexandrov