Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Scaling Django with Distributed Systems
Search
Andrew Godwin
April 07, 2017
Programming
3
2.2k
Scaling Django with Distributed Systems
A talk I gave at PyCon Ukraine 2017.
Andrew Godwin
April 07, 2017
Tweet
Share
More Decks by Andrew Godwin
See All by Andrew Godwin
Reconciling Everything
andrewgodwin
1
270
Django Through The Years
andrewgodwin
0
170
Writing Maintainable Software At Scale
andrewgodwin
0
410
A Newcomer's Guide To Airflow's Architecture
andrewgodwin
0
330
Async, Python, and the Future
andrewgodwin
2
620
How To Break Django: With Async
andrewgodwin
1
690
Taking Django's ORM Async
andrewgodwin
0
690
The Long Road To Asynchrony
andrewgodwin
0
620
The Scientist & The Engineer
andrewgodwin
1
720
Other Decks in Programming
See All in Programming
SwiftUI移行のためのインプレッショントラッキング基盤の構築
kokihirokawa
0
170
Lambdaの監視、できてますか?Datadogを用いてLambdaを見守ろう
nealle
2
720
Jakarta EE meets AI
ivargrimstad
0
600
PEPCは何を変えようとしていたのか
ken7253
3
300
Kotlinの開発でも AIをいい感じに使いたい / Making the Most of AI in Kotlin Development
kohii00
5
1.8k
お前もAI鬼にならないか?👹Bolt & Cursor & Supabase & Vercelで人間をやめるぞ、ジョジョー!👺
taishiyade
7
4.2k
生成AIで加速するテスト実装 - ロリポップ for Gamersの事例と 生成AIエディタの活用
kinosuke01
0
140
Learning Kotlin with detekt
inouehi
1
190
複数のAWSアカウントから横断で 利用する Lambda Authorizer の作り方
tc3jp
0
120
1年目の私に伝えたい!テストコードを怖がらなくなるためのヒント/Tips for not being afraid of test code
push_gawa
1
650
CSS Linter による Baseline サポートの仕組み
ryo_manba
1
160
クリーンアーキテクチャから見る依存の向きの大切さ
shimabox
5
1.1k
Featured
See All Featured
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
eileencodes
33
2.1k
How to Think Like a Performance Engineer
csswizardry
22
1.4k
Stop Working from a Prison Cell
hatefulcrawdad
268
20k
Side Projects
sachag
452
42k
The Psychology of Web Performance [Beyond Tellerrand 2023]
tammyeverts
46
2.4k
Learning to Love Humans: Emotional Interface Design
aarron
273
40k
How to Ace a Technical Interview
jacobian
276
23k
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
30
2.2k
Visualization
eitanlees
146
15k
StorybookのUI Testing Handbookを読んだ
zakiyama
28
5.5k
Docker and Python
trallard
44
3.3k
The Cost Of JavaScript in 2023
addyosmani
47
7.4k
Transcript
None
Andrew Godwin Hi, I'm Django core developer Senior Software Engineer
at Used to complain about migrations a lot
Distributed Systems
c = 299,792,458 m/s
Early CPUs c = 60m propagation distance Clock ~2cm 5
MHz
Modern CPUs c = 10cm propagation distance 3 GHz
Distributed systems are made of independent components
They are slower and harder to write than synchronous systems
But they can be scaled up much, much further
Trade-offs
There is never a perfect solution.
Fast Good Cheap
None
Load Balancer WSGI Worker WSGI Worker WSGI Worker
Load Balancer WSGI Worker WSGI Worker WSGI Worker Cache
Load Balancer WSGI Worker WSGI Worker WSGI Worker Cache Cache
Cache
Load Balancer WSGI Worker WSGI Worker WSGI Worker Database
CAP Theorem
Partition Tolerant Consistent Available
PostgreSQL: CP Consistent everywhere Handles network latency/drops Can't write if
main server is down
Cassandra: AP Can read/write to any node Handles network latency/drops
Data can be inconsistent
It's hard to design a product that might be inconsistent
But if you take the tradeoff, scaling is easy
Otherwise, you must find other solutions
Read Replicas (often called master/slave) Load Balancer WSGI Worker WSGI
Worker WSGI Worker Replica Replica Main
Replicas scale reads forever... But writes must go to one
place
If a request writes to a table it must be
pinned there, so later reads do not get old data
When your write load is too high, you must then
shard
Vertical Sharding Users Tickets Events Payments
Horizontal Sharding Users 0 - 2 Users 3 - 5
Users 6 - 8 Users 9 - A
Both Users 0 - 2 Users 3 - 5 Users
6 - 8 Users 9 - A Events 0 - 2 Events 3 - 5 Events 6 - 8 Events 9 - A Tickets 0 - 2 Tickets 3 - 5 Tickets 6 - 8 Tickets 9 - A
Both plus caching Users 0 - 2 Users 3 -
5 Users 6 - 8 Users 9 - A Events 0 - 2 Events 3 - 5 Events 6 - 8 Events 9 - A Tickets 0 - 2 Tickets 3 - 5 Tickets 6 - 8 Tickets 9 - A User Cache Event Cache Ticket Cache
Teams have to scale too; nobody should have to understand
eveything in a big system.
Services allow complexity to be reduced - for a tradeoff
of speed
Users 0 - 2 Users 3 - 5 Users 6
- 8 Users 9 - A Events 0 - 2 Events 3 - 5 Events 6 - 8 Events 9 - A Tickets 0 - 2 Tickets 3 - 5 Tickets 6 - 8 Tickets 9 - A User Cache Event Cache Ticket Cache User Service Event Service Ticket Service
User Service Event Service Ticket Service WSGI Server
Each service is its own, smaller project, managed and scaled
separately.
But how do you communicate between them?
Service 2 Service 3 Service 1 Direct Communication
Service 2 Service 3 Service 1 Service 4 Service 5
Service 2 Service 3 Service 1 Service 4 Service 5
Service 6 Service 7 Service 8
Service 2 Service 3 Service 1 Message Bus Service 2
Service 3 Service 1
A single point of failure is not always bad -
if the alternative is multiple, fragile ones
Channels and ASGI provide a standard message bus built with
certain tradeoffs
Backing Store e.g. Redis, RabbitMQ ASGI (Channel Layer) Channels Library
Django Django Channels Project
Backing Store e.g. Redis, RabbitMQ ASGI (Channel Layer) Pure Python
Failure Mode At most once Messages either do not arrive,
or arrive once. At least once Messages arrive once, or arrive multiple times
Guarantees vs. Latency Low latency Messages arrive very quickly but
go missing more Low loss rate Messages are almost never lost but arrive slower
Queuing Type First In First Out Consistent performance for all
users First In Last Out Hides backlogs but makes them worse
Queue Sizing Finite Queues Sending can fail Infinite queues Makes
problems even worse
You must understand what you are making (This is surprisingly
uncommon)
Design as much as possible around shared-nothing
Per-machine caches On-demand thumbnailing Signed cookie sessions
Has to be shared? Try to split it
Has to be shared? Try sharding it.
Django's job is to be slowly replaced by your code
Just make sure you match the API contract of what
you're replacing!
Don't try to scale too early; you'll pick the wrong
tradeoffs.
Thanks. Andrew Godwin @andrewgodwin channels.readthedocs.io