Upgrade to Pro — share decks privately, control downloads, hide ads and more …

⛔ No Cloud: how Woovi moved from the Cloud to i...

⛔ No Cloud: how Woovi moved from the Cloud to its own servers

Learn How Woovi migrated from the Cloud to its servers

Sibelius Seraphini

April 04, 2024
Tweet

More Decks by Sibelius Seraphini

Other Decks in Technology

Transcript

  1. Overview - Woovi CloudStack - Motivation - Woovi No Cloud

    - Woovi Staging Environment - High Availability - Woovi Production Environment - Migration step by step - Strategies to Migrate MongoDB - What went wrong? - Monitoring, Backup, Automation
  2. Woovi Cloudstack - AWS - S3 - SES, Mailgun -

    EKS (k8s) - Docker - ElasticSearch (self-hosted) - Sentry - Metabase (self-hosted) - MongoDB (self-hosted) - Mosquitto (self-hosted) - Redis (self-hosted) - N8n (self-hosted) - Nginx (self-hosted) - Thumbor (self-hosted) - Cloudflare (cloudflared, zero trust) - Bulljs
  3. Motivation - Reduction of Cost (from 50/100k BRL to 5k

    BRL per month, cloud cost) - Reduction of CI/CD (from 10k BRL to 0 BRL per month) - Efficiency - Faster - Bare Metal is not so hard - Fewer abstraction, fewer virtualizations - Simpler
  4. Woovi No Cloud Stack - MikroTik - ProxMox - KVM,

    VM - Linux Containers, LXC - LVM, Ceph - MicroK8s - Internal DNS, Bind 9 - Cloudflared tunnel - Zabbix, Prometheus, Elastic APM - Disk Snapshot, S3 backup - Tekton CI/CD - ArgoCD, GitOps - Ansible
  5. Woovi Staging Environment - 1 Dell T550 - Containers (metabase,

    redis, docuseal) - VM (cloudflare, mongos, elastic search, k8s node) - No high-availability
  6. High Availability - Redundancy - Colocation - 2 electrical power

    grids - 2 air conditioners - 2 power generators - 2 power stabilizers - 2 internet providers - 2 power outlets - 2 routers - 5 real servers
  7. Woovi Production Environment - 5 Dell R750 - Containers (metabase,

    redis, docuseal) - VM (cloudflare, mongos, elastic search, 3 k8s node) - High-availability - Stateless k8s
  8. Migration Step by Step - Replicate your existing environment in

    the datacenter - Using IaC and k8s simplify a lot, a lot of yaml - Decide in your strategy to sync your database among clusters (this is the hard part) - Stop traffic from the cloud cluster - Stop all servers and database activity - Wait for the database to finish to sync - Start all servers and traffic from the datacenter - This approach will have 5/10 minutes of downtime.
  9. Strategies to Migrate MongoDB - Stop database activity, do a

    backup using mongodump, and use mongorestore, this works well if your database is small, and it will have a bigger downtime. - Mongosync - tool that sync 2 database cluster using change streams, we tried that but it lose some data. - Add another cluster mongodb node as a replicaset member, this will copy all data from another cluster, this provides the minimal downtime with the safest approach.
  10. What went wrong? - Typo on cloudflared k8s config -

    Outdated docker images - Staging docker image in production for SSR - Elastic Search failing to connect - Missing s3 bucket assets
  11. Backup - Databases - S3 backup - VM disk snapshot

    - Storage NAS - Network Attached Storage
  12. Automation - Tekton for CI/CD - ArgoCD for GitOps -

    Ansible to provision VM, Container