Pro Yearly is on sale from $80 to $50! »

Scaling Multi-tenant Applications Using the Django ORM & Postgres | PyCon Canada 2018 | Sai Srirampur

024d6a0dd14fb31c804969a57a06dfbe?s=47 Citus Data
November 11, 2018

Scaling Multi-tenant Applications Using the Django ORM & Postgres | PyCon Canada 2018 | Sai Srirampur

In the real-world there are 10000s of B2B companies. Their app-stack fits the multi-tenant model - each tenant(customer) deals with it’s own data. It is super critical to build scalable applications which gives the company leeway to grow as more customers get on-boarded. Let’s learn how to do that!

024d6a0dd14fb31c804969a57a06dfbe?s=128

Citus Data

November 11, 2018
Tweet

Transcript

  1. Sai Srirampur | PyConCA 2018 Sai Srirampur | PyConCA 2018

    Scaling Multi-Tenant Applications Using the Django ORM & Postgres Sai Srirampur PyCon Canada | Toronto | Nov 2018
  2. Sai Srirampur | PyConCA 2018 • Sai Srirampur a.k.a Sai

    • Engineer at Citus Data • Joined Citus to make it so developers never have to worry about scaling their database • Creator django multi-tenant • Follow me @saisrirampur @citusdata
  3. Sai Srirampur | PyConCA 2018

  4. Sai Srirampur | PyConCA 2018 Talk about my favorite things

    Sai Srirampur | PyConCA 2018
  5. Sai Srirampur | PyConCA 2018 Sai Srirampur | PyConCA 2018

    PYTHON & DJANGO POSTGRES SCALING MULTI-TENANT APPLICATIONS
  6. Sai Srirampur | PyConCA 2018 Sai Srirampur | PyConCA 2018

    Why Postgres? TLDR; Open source Constraints Extensions PostGIS / Geospatial HLL, TopN, Citus Foreign data wrappers Rich SQL CTEs Window functions Full text search Datatypes JSONB
  7. Sai Srirampur | PyConCA 2018 Today… Scaling Multi-Tenant Apps Using

    the Django ORM & Postgres 3 architectures to build Multi-tenant apps django_multitenant Scaling multi-tenant apps with distributed postgres
  8. Sai Srirampur | PyConCA 2018 • Multiple customers (“tenants”) •

    Each with own data • SaaS • Shopify, Salesforce Multi-Tenant Apps Sai Srirampur | PyConCA 2018
  9. Sai Srirampur | PyConCA 2018 Sai Srirampur | PyConCA 2018

    Why talk about scalable multi-tenant applications? tens 100’s 1000’s
  10. Sai Srirampur | PyConCA 2018 Sai Srirampur | PyConCA 2018

    Architectures To Build Multi-Tenant Applications 3
  11. Sai Srirampur | PyConCA 2018 [1] One database per tenant

    Sai Srirampur | PyConCA 2018
  12. Sai Srirampur | PyConCA 2018 What defines a Database? •

    Organized collection of interrelated data • Don’t share resources. • Username and password • Connections • Memory
  13. Sai Srirampur | PyConCA 2018 One database per tenant Tenant

    1251 Tenant 1252 Tenant 5 [1]
  14. Sai Srirampur | PyConCA 2018 Sai Srirampur | PyConCA 2018

    CREATE DATABASE tenant_100; ./manage.py migrate --database=tenant_100; One database per tenant / Onboarding database_routing: db_for_read, db_for_write etc.
  15. Sai Srirampur | PyConCA 2018 • Start quickly • Isolate

    customer (tenant) data • Compliance is a bit easier • Time for DBA/developer to manage • Maintain consistency (ex: create index across all databases) • Longer running migrations • Performance degrades as # customers (tenants) goes up PROS CONS [1] One Database Per Tenant Sai Srirampur | PyConCA 2018
  16. Sai Srirampur | PyConCA 2018 [2] One schema per tenant

    Sai Srirampur | PyConCA 2018
  17. Sai Srirampur | PyConCA 2018 What defines a database Schema?

    • Logical namespaces to hold a set of tables • Share resources: • Username and password • Connections • Memory
  18. Sai Srirampur | PyConCA 2018 One schema per tenant Tenant

    5 Tenant 1251 Tenant 1252 Database [2]
  19. Sai Srirampur | PyConCA 2018 Sai Srirampur | PyConCA 2018

    CREATE schema tenant_100; ./manage.py migrate --schema=tenant_100; app_code for schema routing based on tenant One schema per tenant / Onboarding
  20. Sai Srirampur | PyConCA 2018 • Better resource utilization vs.

    one database per tenant • Start quickly • Logical isolation • Hard to manage (ex: add column across all schemas) • Longer running migrations • Performance degrades as # customers (tenants) goes up PROS CONS [2] One Schema Per Tenant Sai Srirampur | PyConCA 2018
  21. Sai Srirampur | PyConCA 2018 [3] Shared table architecture Sai

    Srirampur | PyConCA 2018
  22. Sai Srirampur | PyConCA 2018 Accounts Shared table architecture Campaigns

    Leads TenantId 5 5 5 1251 Database [3]
  23. Sai Srirampur | PyConCA 2018 Example

  24. Sai Srirampur | PyConCA 2018 • Easy maintenance • Faster

    running migrations • Best resource utilization • Faster performance • Scales to 1k-100k tenants • Application code to guarantee isolation • Make sure ORM calls are always scoped to a single tenant PROS CONS [3] Shared Table Architecture :-) Sai Srirampur | PyConCA 2018
  25. Sai Srirampur | PyConCA 2018 Comparing the 3 architectures for

    scaling multi-tenant Django applications ONE DATABASE PER TENANT ONE SCHEMA PER TENANT SHARED TABLE ARCHITECTURE Database Database [1] [2] [3] start quickly & isolation guarantees, bad resource utilization & not scalable better resource util, logical isolation not scalable Scales to over 100K tenants! explicitly handle isolation
  26. Sai Srirampur | PyConCA 2018

  27. Sai Srirampur | PyConCA 2018 django_multitenant to the rescue Sai

    Srirampur | PyConCA 2018
  28. Sai Srirampur | PyConCA 2018 Sai Srirampur | PyConCA 2018

    django_multitenant Automates all ORM calls to be scoped to a single tenant
  29. Sai Srirampur | PyConCA 2018 Today... Purchase.objects.filter(id=1) <=> "SELECT* from

    purchase where id=1"
  30. Sai Srirampur | PyConCA 2018 Sai Srirampur | PyConCA 2018

    With django_multitenant Purchase.objects.filter(id=1) <=> "SELECT* from purchase where id=1 and store_id=<current_tenant>"
  31. Sai Srirampur | PyConCA 2018 Components of django_multitenant Sai Srirampur

    | PyConCA 2018
  32. Sai Srirampur | PyConCA 2018 django_multitenant usage — 3 steps

    1. Inherit all models with TenantModel 2. Change ForeignKey to TenantForeignKey 3. Define tenant scoping: set_current_tenant(t)
  33. Sai Srirampur | PyConCA 2018 TenantModel defines new Manager Sai

    Srirampur | PyConCA 2018
  34. Sai Srirampur | PyConCA 2018 TenantForeignKey mimics composite foreign key

    behavior • Adds tenant_id filter to referenced model. Handles: • Reference lookups — ex: (Purchase.product.name) • select_related() / prefetch_related() • Explicit Joins (product__name)
  35. Sai Srirampur | PyConCA 2018 set_current_tenant(t) • Specifies which tenant

    the APIs should be scoped to • Set at authentication logic via middleware • Set explicitly at top of function (ex. view, external tasks/jobs)
  36. Sai Srirampur | PyConCA 2018 Example Code Sai Srirampur |

    PyConCA 2018
  37. Sai Srirampur | PyConCA 2018 Models without django_multitenant class Purchase(models.Model):

    store = models.ForeignKey(Store) product_purchased = models.ForeignKey(Product) ordered_at = models.DateTimeField(default=timezone.now) billing_address = models.TextField() Sai Srirampur | PyConCA 2018
  38. Sai Srirampur | PyConCA 2018 Post django_multitenant class Purchase(TenantModel): store

    = models.ForeignKey(Store) product_purchased = TenantForeignKey(Product) ordered_at = models.DateTimeField(default=timezone.now) billing_address = models.TextField() Sai Srirampur | PyConCA 2018
  39. Sai Srirampur | PyConCA 2018 Setting the tenant at authentication

    class SetCurrentTenantFromUser(object): def process_request(self, request): if not hasattr(self, 'authenticator'): from rest_framework_jwt.authentication import JSONWebTokenAuthentication self.authenticator = JSONWebTokenAuthentication() try: user, _ = self.authenticator.authenticate(request) except: return try: #Assuming your app has a function to get the tenant associated for a user current_tenant = get_tenant_for_user(user) except: # TODO: handle failure return set_current_tenant(current_tenant) Sai Srirampur | PyConCA 2018
  40. Sai Srirampur | PyConCA 2018

  41. Sai Srirampur | PyConCA 2018 Benefits of django_multitenant Drop-in implementation

    of shared tables architecture Guarantees isolation Ready to scale with distributed Postgres (Citus)
  42. Sai Srirampur | PyConCA 2018

  43. Sai Srirampur | PyConCA 2018 Infinite scale with Citus

  44. Sai Srirampur | PyConCA 2018 django_multitenant with Postgres

  45. Sai Srirampur | PyConCA 2018 django_multitenant with Citus

  46. Sai Srirampur | PyConCA 2018 Sai Srirampur | PyConCA 2018

    django_multitenant with Citus
  47. Sai Srirampur | PyConCA 2018 also one of my favorites:

    Citus
  48. Scale out Django! github.com/citusdata/django-multitenant @saisrirampur @citusdata sai@citusdata.com citusdata.com/newsletter Thank You