Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Python and Relational/Non-relational Databases
Search
Andrew Godwin
October 22, 2010
Programming
0
130
Python and Relational/Non-relational Databases
A talk I gave at PyCon Ukraine 2010.
Andrew Godwin
October 22, 2010
Tweet
Share
More Decks by Andrew Godwin
See All by Andrew Godwin
Reconciling Everything
andrewgodwin
1
330
Django Through The Years
andrewgodwin
0
220
Writing Maintainable Software At Scale
andrewgodwin
0
460
A Newcomer's Guide To Airflow's Architecture
andrewgodwin
0
370
Async, Python, and the Future
andrewgodwin
2
680
How To Break Django: With Async
andrewgodwin
1
740
Taking Django's ORM Async
andrewgodwin
0
740
The Long Road To Asynchrony
andrewgodwin
0
680
The Scientist & The Engineer
andrewgodwin
1
780
Other Decks in Programming
See All in Programming
Pythonスレッドとは結局何なのか? CPython実装から見るNoGIL時代の変化
curekoshimizu
5
1.7k
Django Ninja による API 開発効率化とリプレースの実践
kashewnuts
0
1.2k
登壇は dynamic! な営みである / speech is dynamic
da1chi
0
240
オープンソースソフトウェアへの解像度🔬
utam0k
11
2.3k
開発生産性を上げるための生成AI活用術
starfish719
3
360
Serena MCPのすすめ
wadakatu
4
950
クラシルを支える技術と組織
rakutek
0
200
なぜあの開発者はDevRelに伴走し続けるのか / Why Does That Developer Keep Running Alongside DevRel?
nrslib
3
390
高度なUI/UXこそHotwireで作ろう Kaigi on Rails 2025
naofumi
4
3.8k
overlayPreferenceValue で実現する ピュア SwiftUI な AdMob ネイティブ広告
uhucream
0
170
GraphQL×Railsアプリのデータベース負荷分散 - 月間3,000万人利用サービスを無停止で
koxya
1
1.2k
詳しくない分野でのVibe Codingで困ったことと学び/vibe-coding-in-unfamiliar-area
shibayu36
3
4.8k
Featured
See All Featured
It's Worth the Effort
3n
187
28k
Performance Is Good for Brains [We Love Speed 2024]
tammyeverts
12
1.2k
JavaScript: Past, Present, and Future - NDC Porto 2020
reverentgeek
52
5.6k
Agile that works and the tools we love
rasmusluckow
331
21k
The Art of Delivering Value - GDevCon NA Keynote
reverentgeek
15
1.7k
The Pragmatic Product Professional
lauravandoore
36
6.9k
ReactJS: Keep Simple. Everything can be a component!
pedronauck
667
120k
Building a Scalable Design System with Sketch
lauravandoore
462
33k
Raft: Consensus for Rubyists
vanstee
139
7.1k
How to Create Impact in a Changing Tech Landscape [PerfNow 2023]
tammyeverts
54
3k
RailsConf 2023
tenderlove
30
1.2k
Sharpening the Axe: The Primacy of Toolmaking
bcantrill
45
2.5k
Transcript
Relational / Non-relational Databases Python and Andrew Godwin
Introduction Python for 5 years Django core developer Data modelling
/ visualisation
""Andrew speaks English like a machine gun speaks bullets."" Reinout
van Rees
If I speak too fast - tell me!
What is a relational database?
A relational database is a “collection of relations”
It's what a lot of people are used to.
Relational Databases PostgreSQL MySQL SQLite
Let's pick PostgreSQL (it's a good choice)
Usage conn = psycopg2.connect( host="localhost", user="postgres" ) cursor = conn.cursor()
cursor.execute('SELECT * FROM users WHERE username = "andrew";') for row in cursor.fetchall(): print row
You've probably seen all that before.
Now, to introduce some non-relational databases
Document Databases MongoDB CouchDB
Key-Value Stores Redis Cassandra
Message Queues AMQP Celery
Various Others Graph databases Filesystems VCSs
Redis and MongoDB are two good examples here
Redis: Key-value store with strings, lists, sets, channels and atomic
operations.
Redis Example conn = redis.Redis(host="localhost") print conn.get("top_value") conn.set("last_user", "andrew") conn.inc("num_runs")
conn.sadd("users", "andrew") conn.sadd("users", "martin") for item in conn.smembers("users"): print item
MongoDB: Document store with indexing and a wide range of
query filters.
MongoDB Example conn = pymongo.Connection("localhost") db = conn['mongo_example'] coll =
db['users'] coll.insert({ "username": "andrew", "uid": 1000, }) for entry in coll.find({"username": "andrew"}): print entry
These all solve different problems - you can't easily replace
one with the other.
""When all you have is a hammer, everything looks like
a nail"" Abraham Manslow (paraphrased)
JOIN - your best friend, and your worst enemy.
Denormalising your data speeds up reads, and slows down writes.
Schemaless != Denormalised
Atomic operations are nice. conn.incrby("num_users', 2)
But SQL can do some of them. UPDATE foo SET
bar = bar + 1 WHERE baz;
Redis, the datastructures server. SETNX, GETSET, EXPIRES and friends
Locks / Semaphores conn.setnx("lock:foo", time.time() + 3600) val = conn.decr("sem:foo")
if val >= 0: ... else: conn.incr("sem:foo")
Queues conn.lpush("myqueue", "workitem") todo = conn.lpop("myqueue") (or publish/subscribe)
Priority Queues conn.zadd("myqueue", "handle-meltdown", 1) conn.zadd("myqueue", "feed-cats", 5) todo =
conn.zrange("myqueue", 0, 1) conn.zrem(todo)
Lock-free linked lists! new_id = "bgrdsd" old_end = conn.getset(":end", new_id)
conn.set("%s:next" % old_end, new_id)
Performance-wise, the less checks/integrity the faster it goes.
Maturity can sometimes be an issue, but new features can
appear rapidly.
You can also use databases for the wrong thing -
it often only matters ""at scale""
But how does this all relate to Python?
Most databases - even new ones - have good Python
bindings
Postgres: PsycoPG2 Redis: redis-py MongoDB: pymongo (and more - neo4j,
VCSen, relational, etc.)
Some databases have Python available inside (Postgres has it as
an option)
Document databases map really well to Python dicts
You may find non-relational databases a nicer way to store
state - for any app
Remember, you might still need transactions/reliability. (Business logic is probably
better off on mature systems for now)
Overall? Just keep all the options in mind. Don't get
caught by trends, and don't abuse your relational store
Thanks. Andrew Godwin @andrewgodwin http://aeracode.org