⛔ No Cloud: how Woovi moved from the Cloud to its own servers.

Sibelius Seraphini @sseraphini in/sibeliusseraphini sibelius

Overview - Woovi CloudStack - Motivation - Woovi No Cloud - Woovi Staging Environment - High Availability - Woovi Production Environment - Migration step by step - Strategies to Migrate MongoDB - What went wrong? - Monitoring, Backup, Automation

Woovi Cloudstack

Woovi Cloudstack - AWS - S3 - SES, Mailgun - EKS (k8s) - Docker - ElasticSearch (self-hosted) - Sentry - Metabase (self-hosted) - MongoDB (self-hosted) - Mosquitto (self-hosted) - Redis (self-hosted) - N8n (self-hosted) - Nginx (self-hosted) - Thumbor (self-hosted) - Cloudflare (cloudflared, zero trust) - Bulljs

Motivation - Reduction of Cost (from 50/100k BRL to 5k BRL per month, cloud cost) - Reduction of CI/CD (from 10k BRL to 0 BRL per month) - Efficiency - Faster - Bare Metal is not so hard - Fewer abstraction, fewer virtualizations - Simpler

Woovi No Cloud Stack

Woovi No Cloud Stack - MikroTik - ProxMox - KVM, VM - Linux Containers, LXC - LVM, Ceph - MicroK8s - Internal DNS, Bind 9 - Cloudflared tunnel - Zabbix, Prometheus, Elastic APM - Disk Snapshot, S3 backup - Tekton CI/CD - ArgoCD, GitOps - Ansible

Woovi Staging Environment - 1 Dell T550 - Containers (metabase, redis, docuseal) - VM (cloudflare, mongos, elastic search, k8s node) - No high-availability

High Availability - Redundancy - Colocation - 2 electrical power grids - 2 air conditioners - 2 power generators - 2 power stabilizers - 2 internet providers - 2 power outlets - 2 routers - 5 real servers

High Availability

Woovi Production Environment - 5 Dell R750 - Containers (metabase, redis, docuseal) - VM (cloudflare, mongos, elastic search, 3 k8s node) - High-availability - Stateless k8s

Migration Step by Step - Replicate your existing environment in the datacenter - Using IaC and k8s simplify a lot, a lot of yaml - Decide in your strategy to sync your database among clusters (this is the hard part) - Stop traffic from the cloud cluster - Stop all servers and database activity - Wait for the database to finish to sync - Start all servers and traffic from the datacenter - This approach will have 5/10 minutes of downtime.

Strategies to Migrate MongoDB - Stop database activity, do a backup using mongodump, and use mongorestore, this works well if your database is small, and it will have a bigger downtime. - Mongosync - tool that sync 2 database cluster using change streams, we tried that but it lose some data. - Add another cluster mongodb node as a replicaset member, this will copy all data from another cluster, this provides the minimal downtime with the safest approach.

What went wrong? - Typo on cloudflared k8s config - Outdated docker images - Staging docker image in production for SSR - Elastic Search failing to connect - Missing s3 bucket assets

Monitoring - Zabbix - Prometheus - Elastic APM - Sentry

Backup - Databases - S3 backup - VM disk snapshot - Storage NAS - Network Attached Storage

Automation - Tekton for CI/CD - ArgoCD for GitOps - Ansible to provision VM, Container

Cloud cost April

All my content

We are hiring!