Upgrade to Pro — share decks privately, control downloads, hide ads and more …

⛔ No Cloud: how Woovi moved from the Cloud to i...

⛔ No Cloud: how Woovi moved from the Cloud to its own servers

Learn How Woovi migrated from the Cloud to its servers

Sibelius Seraphini

April 04, 2024

Transcript

21 slides
  1. Overview - Woovi CloudStack - Motivation - Woovi No Cloud

    - Woovi Staging Environment - High Availability - Woovi Production Environment - Migration step by step - Strategies to Migrate MongoDB - What went wrong? - Monitoring, Backup, Automation
  2. Woovi Cloudstack - AWS - S3 - SES, Mailgun -

    EKS (k8s) - Docker - ElasticSearch (self-hosted) - Sentry - Metabase (self-hosted) - MongoDB (self-hosted) - Mosquitto (self-hosted) - Redis (self-hosted) - N8n (self-hosted) - Nginx (self-hosted) - Thumbor (self-hosted) - Cloudflare (cloudflared, zero trust) - Bulljs
  3. Motivation - Reduction of Cost (from 50/100k BRL to 5k

    BRL per month, cloud cost) - Reduction of CI/CD (from 10k BRL to 0 BRL per month) - Efficiency - Faster - Bare Metal is not so hard - Fewer abstraction, fewer virtualizations - Simpler
  4. Woovi No Cloud Stack - MikroTik - ProxMox - KVM,

    VM - Linux Containers, LXC - LVM, Ceph - MicroK8s - Internal DNS, Bind 9 - Cloudflared tunnel - Zabbix, Prometheus, Elastic APM - Disk Snapshot, S3 backup - Tekton CI/CD - ArgoCD, GitOps - Ansible
  5. Woovi Staging Environment - 1 Dell T550 - Containers (metabase,

    redis, docuseal) - VM (cloudflare, mongos, elastic search, k8s node) - No high-availability
  6. High Availability - Redundancy - Colocation - 2 electrical power

    grids - 2 air conditioners - 2 power generators - 2 power stabilizers - 2 internet providers - 2 power outlets - 2 routers - 5 real servers
  7. Woovi Production Environment - 5 Dell R750 - Containers (metabase,

    redis, docuseal) - VM (cloudflare, mongos, elastic search, 3 k8s node) - High-availability - Stateless k8s
  8. Migration Step by Step - Replicate your existing environment in

    the datacenter - Using IaC and k8s simplify a lot, a lot of yaml - Decide in your strategy to sync your database among clusters (this is the hard part) - Stop traffic from the cloud cluster - Stop all servers and database activity - Wait for the database to finish to sync - Start all servers and traffic from the datacenter - This approach will have 5/10 minutes of downtime.
  9. Strategies to Migrate MongoDB - Stop database activity, do a

    backup using mongodump, and use mongorestore, this works well if your database is small, and it will have a bigger downtime. - Mongosync - tool that sync 2 database cluster using change streams, we tried that but it lose some data. - Add another cluster mongodb node as a replicaset member, this will copy all data from another cluster, this provides the minimal downtime with the safest approach.
  10. What went wrong? - Typo on cloudflared k8s config -

    Outdated docker images - Staging docker image in production for SSR - Elastic Search failing to connect - Missing s3 bucket assets
  11. Backup - Databases - S3 backup - VM disk snapshot

    - Storage NAS - Network Attached Storage
  12. Automation - Tekton for CI/CD - ArgoCD for GitOps -

    Ansible to provision VM, Container

More Decks by Sibelius Seraphini

Other Decks in Technology