Мультитенантная архитектура на базе Kubernetes

Руслан Шакиров, .NET Architect

“Мультитенантная архитектура на базе Kubernetes”

Обзорный доклад о специфике B2B-проектов. Мы рассмотрим основные моменты и возможные варианты реализации требований, которые встречаются на проектах. На конкретных примерах построим гибкую модель, которую можно легко адаптировать под требования нового заказчика (даже с учетом меняющихся требований). Обязательно рассмотрим отдельные технические аспекты: контроль используемых мощностей, баланс между стоимостью, производительностью и безопасностью решений, авторизация юзеров в мультитенанси, менеджмент секретов, деплоймент новых версий сервисов. Руслан поделится методами сбора логов и метрик, а также затронет тему GitOps.

Transcript

1. Kubernetes Multi-tenancy © 2021 Quantori. Custom Software Solutions
2. quantori.com 2 About Me Шакиров Руслан .NET Architect in Quantori
3. Quantori = IT + Health Science
4. quantori.com 4 Multitenancy B2B Platform • Multitenancy • Kubernetes vs Cloud services • GitOps • Multitenancy in K8s • K8s Configuration • Multi-Tenant Databases • Identity in a multitenant app • Metrics
5. Тенант - это группа пользователей, данные которых не должны быть доступны пользователям других тенантов
6. quantori.com 6 B2B Platform Требования • Обеспечить изоляцию данных между тенантами • Каждый тенант должен быть работоспособным в любой момент времени • Независимое масштабирование тенантов • Независимый деплоймент тенантов
7. quantori.com 7 Tenancy Types Single-tenancy A single instance of the software and supporting infrastructure serve a single customer. Multi-tenancy Multiple tenants share the same physical instance of the app. Although tenants share physical resources, each tenant gets its own logical instance of the app. Pictures designed by macrovector / Freepik
8. quantori.com 8 Single-Tenancy Advantages - Security - Dependability - Customization Disadvantages - Maintenance - Setup/Management - Less efficient
9. quantori.com 9 Multi-Tenancy Advantages - Lower costs - Scalability - “Hands-free” Maintenance - Customization without coding - Improved Productivity Disadvantages - Limited Customization - Lower Security - Integrations updates/changes
10. quantori.com 10 Kubernetes vs Cloud services • Multitenancy • Kubernetes vs Cloud services • GitOps • Multitenancy in K8s • K8s Configuration • Identity in a multitenant app • Multi-Tenant Databases • Metrics
11. quantori.com 11 Cloud Providers IAM Route53 API Gateway ELB EC2 CloudWatch CloudFormation / Terraform with AWS provider Active Directory Azure DNS / Traffic Manager Azure API Management Load Balancer / App Gateway Virtual Machines Monitor ResourceManager+Azure Automation / Terraform with Azure provider + DB/Storage Cloud Identity Cloud DNS Apigee Network Load Balancing Compute Engine Cloud Trace/Debugger/Profiler Cloud Deployment Manager / Terraform with GCP provider
12. quantori.com 12 Cloud Providers IAM Route53 API Gateway ELB EC2 CloudWatch CloudFormation / Terraform with AWS provider Active Directory Azure DNS / Traffic Manager Azure API Management Load Balancer / App Gateway Virtual Machines Monitor ResourceManager+Azure Automation / Terraform with Azure provider Cloud Identity Cloud DNS Apigee Network Load Balancing Compute Engine Cloud Trace/Debugger/Profiler Cloud Deployment Manager / Terraform with GCP provider
13. quantori.com 13 On-premises software... Картинка: https://icon-library.com/icon/datacenter-icon-24.html.html
14. quantori.com 14 Cloud Providers RBAC * DNS Ingress Controller Load Balancer Service Nodes/Pods ElasticSearch + Filebeat + Kibana (EFK) / Promtail + Prometheus + Loki + Grafana (PLG)
15. quantori.com 15 Что и когда Managed Cloud Services Kubernetes As A Service DB Management Systems Storages Simple web app (AWS LightSail, Az WebApp) Unpredictable and significant load fluctuations (AWS Lambda Functions, Az Functions) Kubernetes Multiple services / web apps Predictable or moderately changing load Cloud-agnostic/Multi-cloud solution On-premises deployment
16. quantori.com 16 Kubernetes Multi-tenancy K8s - это Infrastructure as code by design
17. quantori.com 17 K8s - это язык для общения девелоперов и девопсов Kubernetes Multi-tenancy
18. quantori.com 18 GitOps • Multitenancy • Kubernetes vs Cloud services • GitOps • Multitenancy in K8s • K8s configuration • Identity in a multitenant app • Multi-Tenant Databases • Metrics
19. quantori.com 19 Kubernetes Multi-tenancy K8s - это язык для общения девелоперов и девопсов
20. quantori.com 20 Kubernetes Multi-tenancy Преимущества такого рабочего процесса • Разработчикам не нужно знать весь Kubernetes • Явный момент передачи задачи/знаний через PR • Видны все изменения в требованиях софта к окружению • Автоматически получаем историю изменений
21. quantori.com 21 Deployment - Helm charts https://cloud.redhat.com/blog/helm-and-operators-on-openshift-part-1
22. quantori.com 22 Kubernetes Multi-Tenancy Flux is based on a set of Kubernetes API extensions (“custom resources”), which control how git repositories and other sources of configuration are applied into the cluster (“synced”). For example, you create a GitRepository object to mirror configuration from a Git repository, then a Kustomization object to sync that configuration. Flux works with Kubernetes' role-based access control (RBAC), so you can lock down what any particular sync can change. It can send notifications to Slack and other like systems when configuration is synced and ready, and receive webhooks to tell it when to sync. Flux / ArgoCD
23. quantori.com 23 Helm Release CRD Kubernetes Multi-Tenancy Tenant Manager creates CRD with: • Tenant metadata (ID, Author, ...) • Deployment parameters (Domain, API Keys, ...)
24. quantori.com 24 Kubernetes Multi-Tenancy
25. quantori.com 25 Multitenancy in K8s • Multitenancy • Kubernetes vs Cloud services • GitOps • Multitenancy in K8s • K8s configuration • Identity in a multitenant app • Multi-Tenant Databases • Metrics
26. quantori.com 26 Kubernetes Multi-Tenancy Types Soft Multi-Tenancy It does not have a strict isolation of the different users, workloads, or applications. It is thus an appropriate solution for trusted and known tenants. The isolation between users is rather focused on preventing accidents and cannot prevent attacks on other tenants. Soft multi-tenancy is typically associated with simple Kubernetes namespaces that the individual tenants are working in. Hard Multi-Tenancy Hard multi-tenancy enforces stricter isolation of tenants and so also prevents negative consequences of malicious behavior from other tenants. In addition to trusted tenants, it can thus also be used for tenants you do not trust such as many unconnected users and people from different organizations. Virtual cluster - each tenant feels they own a cluster exclusively while there is only one control plane.
27. quantori.com 27 Kubernetes Multi-Tenancy Types Soft Multi-Tenancy
28. quantori.com 28 Kubernetes Multi-Tenancy Types Hard Multi-Tenancy
29. quantori.com 29 Kubernetes Soft Multi-tenancy Plain Namespaces Logical partitions of the cluster as isolated slices. Kubernetes was designed for this approach. Most of the isolation features that it provides expect you to have a separate namespace for each entity that you want to isolate. Hierarchical Namespace Controller (HNC) You can create additional namespaces under your team's namespace, even if you don't have cluster-level permission to create namespaces, and easily apply policies like RBAC and Network Policies across all namespaces in your team (e.g. a set of related microservices).
30. quantori.com 30 Kubernetes Soft Multi-tenancy 1 Tenant pool - 1 Namespace Tenant Pool represents resources allocated in the cluster for a group of tenants. Tenant’s data processing is isolated on the DAL. 1 Tenant - 1 Namespace Data processing is isolated on the K8s namespace level.
31. quantori.com 31 Tenant Pool Tiers
32. quantori.com 32 Kubernetes multi-tenancy configuration tools kiosk kiosk is an open-source multi-tenancy extension for Kubernetes. It is designed as a lightweight, pluggable, and customizable solution for any Kubernetes cluster and solves some of the multi-tenancy challenges in a simple way. Loft Loft is based on kiosk and provides a comprehensive solution for a multi-tenancy platform. Loft can be installed into any Kubernetes cluster and then lets tenants create namespaces and virtual Clusters on-demand. It cares for the user management (including SSO) and user isolation and lets the cluster admins define usage limits kubesphere.io Multi-level access control and resource quota limits underlie resource isolation in KubeSphere. The access control in KubeSphere is divided into three levels: platform, workspace and project.
33. quantori.com 33 K8s Configuration • Multitenancy • Kubernetes vs Cloud services • GitOps • Multitenancy in K8s • K8s Configuration • Identity in a multitenant app • Multi-Tenant Databases • Metrics Автор картинки: stackrox.io
34. quantori.com 34 Kubernetes Soft Multi-tenancy DNS Rule • api.domain.com • *.domain.com
35. quantori.com 35 Block traffic between namespaces Блокируем трафик между неймспейсами
36. quantori.com 36 Resource Quotas Устанавливаем лимиты на ресурсы
37. quantori.com 37 Multi-Tenant Databases • Multitenancy • Kubernetes vs Cloud services • GitOps • Multitenancy in K8s • K8s Configuration • Multi-Tenant Databases • Identity in a multitenant app • Metrics
38. quantori.com 38 Multi-Tenant Databases Measurement Standalone app Database-per-tenant Sharded multi-tenant Scale Medium 1-100s Very high 1-100,000s Unlimited 1-1,000,000s Tenant isolation Very high High Low; except for any single tenant (that is alone in an MT db). Database cost per tenant High; is sized for peaks. Low; pools used. Lowest, for small tenants in MT DBs. Performance monitoring and management Per-tenant only Aggregate + per-tenant Aggregate; although is per-tenant only for singles. Development complexity Low Low Medium; due to sharding. Operational complexity Low-High. Individually simple, complex at scale. Low-Medium. Patterns address complexity at scale. Low-High. Individual tenant management is complex. https://docs.microsoft.com/en-us/azure/azure-sql/database/saas-tenancy-app-design-patterns
39. quantori.com 39 Table structure Multi-Tenant Databases - Sharded DB SELECT tenant_id, user_id, given_name || ' ' || family_name AS name FROM tenant_user; tenant_id | user_id | name --------------------------------------+--------------------------------------+----------------- 1cf1cc14-dd34-4a7b-b87d-adf79b2c255c | d9f7d636-69a0-40d4-96d9-d429d1e1cee3 | User 1 Tenant 1 69ad9212-f5ef-456d-a724-dd8ea3c80d61 | eb7a503a-a7c6-44c0-9916-8df68dd96815 | User 1 Tenant 2 (2 rows) https://aws.amazon.com/blogs/database/multi-tenant-data-isolation-with-postgresql-row-level-security/
40. quantori.com 40 Row Level Security (RLS) Multi-Tenant Databases - RLS ALTER TABLE tenant_user ENABLE ROW LEVEL SECURITY; CREATE POLICY tenant_user_isolation_policy ON tenant_user USING (tenant_id::TEXT = CURRENT_USER); https://aws.amazon.com/blogs/database/multi-tenant-data-isolation-with-postgresql-row-level-security/
41. quantori.com 41 Current_user - user name of current execution context Multi-Tenant Databases - RLS SELECT CURRENT_USER; current_user -------------------------------------- 1cf1cc14-dd34-4a7b-b87d-adf79b2c255c (1 row) https://aws.amazon.com/blogs/database/multi-tenant-data-isolation-with-postgresql-row-level-security/
42. quantori.com 42 Row Level Security (RLS) Applied Multi-Tenant Databases - RLS SELECT * FROM tenant; tenant_id | name | status | tier --------------------------------------+----------+--------+------ 1cf1cc14-dd34-4a7b-b87d-adf79b2c255c | Tenant 1 | active | gold (1 row) SELECT * FROM tenant WHERE tenant_id = '69ad9212-f5ef-456d-a724-dd8ea3c80d61' ::UUID; tenant_id | name | status | tier -----------+------+--------+------ (0 rows) https://aws.amazon.com/blogs/database/multi-tenant-data-isolation-with-postgresql-row-level-security/
43. quantori.com 43 Row Level Security (RLS) Applied Multi-Tenant Databases - RLS UPDATE tenant SET name = 'Cross Tenant' WHERE tenant_id = 'eb7a503a-a7c6-44c0-9916-8df68dd96815' ::UUID; UPDATE 0 DELETE FROM tenant WHERE tenant_id = '69ad9212-f5ef-456d-a724-dd8ea3c80d61' ::UUID; DELETE 0 INSERT INTO tenant (name) VALUES ('Tenant 3'); ERROR: new row violates row-level security policy for table "tenant" https://aws.amazon.com/blogs/database/multi-tenant-data-isolation-with-postgresql-row-level-security/
44. quantori.com 44 Alternative Option Multi-Tenant Databases - RLS CREATE POLICY tenant_user_isolation_policy ON tenant_user USING (tenant_id = current_setting('app.current_tenant'):: TEXT); Before a command executing: SET app.current_tenant = '1cf1cc14-dd34-4a7b-b87d-adf79b2c255c' ::UUID; https://aws.amazon.com/blogs/database/multi-tenant-data-isolation-with-postgresql-row-level-security/
45. quantori.com 45 Alternative Option Multi-Tenant Databases - RLS
46. quantori.com 46 Identity in a multitenant app • Multitenancy • Kubernetes vs Cloud services • GitOps • Multitenancy in K8s • Request processing • Multi-Tenant Databases • Identity in a multitenant app • Metrics
47. quantori.com 47 Identity in a multitenant app Имеет смысл, когда у тенантов есть специфичные требования к процессу аутентификации (MFA, СКУД). Юзер вынужден логиниться в каждый тенант отдельно. Однажды залогинившись, юзеры могут работать с разными тенантами, переключаясь между ними. User Pool per Tenant Global User Pool
48. quantori.com 48 Global User Pool JWT
49. quantori.com 49 Metrics • Multitenancy • Kubernetes vs Cloud services • GitOps • Multitenancy in K8s • Multi-Tenant Databases • K8s configuration • Identity in a multitenant app • Metrics
50. quantori.com 50 Metrics Prometheus CPU usage by namespace: sum(rate(container_cpu_usage_seconds_total{container_name!="POD",namespace!=""}[5m])) by (namespace) Memory usage by namespace: sum(rate(container_memory_usage_bytes{container_name!="POD",namespace!=""}[5m])) by (namespace) CPU requests by namespace: sum(rate(container_cpu_usage_seconds_total{container_name!="POD",namespace!=""}[5m])) by (namespace) Memory requests by namespace: sum(rate(container_memory_usage_bytes{container_name!="POD",namespace!=""}[5m])) by (namespace)
51. quantori.com 51 Метрики Grafana
52. quantori.com 52 Вывод Kubernetes • Cloud-agnostic solution • GitOps native • Boosts team performance • Supports several tenant isolation approaches • Namespace- and pod-level observability
53. Спасибо!