Scaling Multi-tenant Applications Using the Django ORM & Postgres | PyCon Canada 2018 | Sai Srirampur

Sai Srirampur | PyConCA 2018
Scaling Multi-Tenant
Applications Using the
Django ORM & Postgres
Sai Srirampur
PyCon Canada | Toronto | Nov 2018

View Slide

• Sai Srirampur a.k.a Sai
• Engineer at Citus Data
• Joined Citus to make it so
developers never have to
worry about scaling their
database
• Creator django multi-tenant
• Follow me @saisrirampur
@citusdata

View Slide


View Slide

Talk about my favorite things

View Slide

PYTHON &
DJANGO
POSTGRES SCALING
MULTI-TENANT
APPLICATIONS

View Slide

Why Postgres? TLDR;
Open source
Constraints
Extensions
PostGIS / Geospatial
HLL, TopN, Citus
Foreign data wrappers
Rich SQL
CTEs
Window functions
Full text search
Datatypes
JSONB

View Slide

Today… Scaling Multi-Tenant Apps
Using the Django ORM & Postgres
3 architectures to build Multi-tenant apps
django_multitenant
Scaling multi-tenant apps with distributed
postgres

View Slide

• Multiple customers
(“tenants”)
• Each with own data
• SaaS
• Shopify, Salesforce
Multi-Tenant Apps

View Slide

Why talk
about scalable
multi-tenant
applications?
tens
100’s
1000’s

View Slide

Architectures
To
Build
Multi-Tenant
Applications
3

View Slide

[1] One database per
tenant

View Slide

What defines a Database?
• Organized collection of interrelated data
• Don’t share resources.
• Username and password
• Connections
• Memory

View Slide

One database per tenant
Tenant 1251 Tenant 1252
Tenant 5
[1]

View Slide

CREATE DATABASE tenant_100;
./manage.py migrate --database=tenant_100;
One database per tenant / Onboarding
database_routing: db_for_read, db_for_write etc.

View Slide

● Start quickly
● Isolate customer (tenant) data
● Compliance is a bit easier
● Time for DBA/developer to manage
● Maintain consistency (ex: create
index across all databases)
● Longer running migrations
● Performance degrades as #
customers (tenants) goes up
PROS CONS
[1] One Database Per Tenant

View Slide

[2] One schema per
tenant

View Slide

What defines a database Schema?
• Logical namespaces to hold a set of tables
• Share resources:
• Username and password
• Connections
• Memory

View Slide

One schema per tenant
Tenant 5 Tenant 1251
Tenant 1252
Database
[2]

View Slide

CREATE schema tenant_100;
./manage.py migrate --schema=tenant_100;
app_code for schema routing based on tenant
One schema per tenant / Onboarding

View Slide

● Better resource utilization vs. one
database per tenant
● Start quickly
● Logical isolation
● Hard to manage (ex: add column
across all schemas)
● Longer running migrations
● Performance degrades as #
customers (tenants) goes up
PROS CONS
[2] One Schema Per Tenant

View Slide

[3] Shared table
architecture

View Slide

Accounts
Shared table architecture
Campaigns
Leads TenantId
5
5
5
1251
Database
[3]

View Slide

Example

View Slide

● Easy maintenance
● Faster running migrations
● Best resource utilization
● Faster performance
● Scales to 1k-100k tenants
● Application code to guarantee
isolation
● Make sure ORM calls are always
scoped to a single tenant
PROS CONS
[3] Shared Table Architecture :-)

View Slide

Comparing the 3 architectures for scaling
multi-tenant Django applications
ONE
DATABASE
PER TENANT
ONE
SCHEMA
PER TENANT
SHARED
TABLE
ARCHITECTURE
Database
Database
[1]
[2]
[3]
start quickly & isolation guarantees,
bad resource utilization & not scalable
better resource util, logical isolation
not scalable
Scales to over 100K tenants!
explicitly handle isolation

View Slide


View Slide

django_multitenant to the rescue

View Slide

django_multitenant
Automates all ORM calls
to be scoped to a single tenant

View Slide

Today...
Purchase.objects.filter(id=1)
<=>
"SELECT* from purchase where id=1"

View Slide

With django_multitenant
Purchase.objects.filter(id=1)
<=>
"SELECT* from purchase where id=1 and
store_id="

View Slide

Components of
django_multitenant

View Slide

django_multitenant usage — 3 steps
1. Inherit all models with TenantModel
2. Change ForeignKey to TenantForeignKey
3. Define tenant scoping: set_current_tenant(t)

View Slide

TenantModel defines new Manager

View Slide

TenantForeignKey mimics composite
foreign key behavior
• Adds tenant_id filter to referenced model. Handles:
• Reference lookups — ex: (Purchase.product.name)
• select_related() / prefetch_related()
• Explicit Joins (product__name)

View Slide

set_current_tenant(t)
• Specifies which tenant the APIs should be scoped to
• Set at authentication logic via middleware
• Set explicitly at top of function (ex. view, external
tasks/jobs)

View Slide

Example Code

View Slide

Models without django_multitenant
class Purchase(models.Model):
store = models.ForeignKey(Store)
product_purchased = models.ForeignKey(Product)
ordered_at = models.DateTimeField(default=timezone.now)
billing_address = models.TextField()

View Slide

Post django_multitenant
class Purchase(TenantModel):
store = models.ForeignKey(Store)
product_purchased = TenantForeignKey(Product)
ordered_at = models.DateTimeField(default=timezone.now)
billing_address = models.TextField()

View Slide

Setting the tenant at authentication
class SetCurrentTenantFromUser(object):
def process_request(self, request):
if not hasattr(self, 'authenticator'):
from rest_framework_jwt.authentication import
JSONWebTokenAuthentication
self.authenticator = JSONWebTokenAuthentication()
try:
user, _ = self.authenticator.authenticate(request)
except:
return
try:
#Assuming your app has a function to get the tenant associated
for a user
current_tenant = get_tenant_for_user(user)
except:
# TODO: handle failure
return
set_current_tenant(current_tenant)

View Slide


View Slide

Benefits of django_multitenant
Drop-in implementation of shared tables architecture
Guarantees isolation
Ready to scale with distributed Postgres (Citus)

View Slide


View Slide

Infinite scale
with Citus

View Slide

django_multitenant with Postgres

View Slide

django_multitenant with Citus

View Slide

also one of my favorites: Citus

View Slide

Scale out Django!
github.com/citusdata/django-multitenant
@saisrirampur @citusdata
[email protected]
citusdata.com/newsletter
Thank You

View Slide

Scaling Multi-tenant Applications Using the Django ORM & Postgres | PyCon Canada 2018 | Sai Srirampur

Scaling Multi-tenant Applications Using the Django ORM & Postgres | PyCon Canada 2018 | Sai Srirampur

Citus Data

More Decks by Citus Data

Other Decks in Technology

Featured

Transcript