Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Python and Relational/Non-relational Databases
Search
Andrew Godwin
October 22, 2010
Programming
0
130
Python and Relational/Non-relational Databases
A talk I gave at PyCon Ukraine 2010.
Andrew Godwin
October 22, 2010
Tweet
Share
More Decks by Andrew Godwin
See All by Andrew Godwin
Reconciling Everything
andrewgodwin
1
310
Django Through The Years
andrewgodwin
0
200
Writing Maintainable Software At Scale
andrewgodwin
0
440
A Newcomer's Guide To Airflow's Architecture
andrewgodwin
0
350
Async, Python, and the Future
andrewgodwin
2
660
How To Break Django: With Async
andrewgodwin
1
720
Taking Django's ORM Async
andrewgodwin
0
720
The Long Road To Asynchrony
andrewgodwin
0
650
The Scientist & The Engineer
andrewgodwin
1
760
Other Decks in Programming
See All in Programming
Result型で“失敗”を型にするPHPコードの書き方
kajitack
4
530
データの民主化を支える、透明性のあるデータ利活用への挑戦 2025-06-25 Database Engineering Meetup#7
y_ken
0
330
Deep Dive into ~/.claude/projects
hiragram
10
2k
#kanrk08 / 公開版 PicoRubyとマイコンでの自作トレーニング計測装置を用いたワークアウトの理想と現実
bash0c7
1
640
Systèmes distribués, pour le meilleur et pour le pire - BreizhCamp 2025 - Conférence
slecache
0
110
Benchmark
sysong
0
280
WebViewの現在地 - SwiftUI時代のWebKit - / The Current State Of WebView
marcy731
0
100
今ならAmazon ECSのサービス間通信をどう選ぶか / Selection of ECS Interservice Communication 2025
tkikuc
20
3.8k
0626 Findy Product Manager LT Night_高田スライド_speaker deck用
mana_takada
0
130
なぜ適用するか、移行して理解するClean Architecture 〜構造を超えて設計を継承する〜 / Why Apply, Migrate and Understand Clean Architecture - Inherit Design Beyond Structure
seike460
PRO
1
710
PicoRuby on Rails
makicamel
2
110
XP, Testing and ninja testing
m_seki
3
220
Featured
See All Featured
How to train your dragon (web standard)
notwaldorf
94
6.1k
GraphQLの誤解/rethinking-graphql
sonatard
71
11k
4 Signs Your Business is Dying
shpigford
184
22k
BBQ
matthewcrist
89
9.7k
Documentation Writing (for coders)
carmenintech
72
4.9k
Fantastic passwords and where to find them - at NoRuKo
philnash
51
3.3k
[RailsConf 2023] Rails as a piece of cake
palkan
55
5.6k
GitHub's CSS Performance
jonrohan
1031
460k
Why Our Code Smells
bkeepers
PRO
337
57k
Imperfection Machines: The Place of Print at Facebook
scottboms
267
13k
Fireside Chat
paigeccino
37
3.5k
RailsConf 2023
tenderlove
30
1.1k
Transcript
Relational / Non-relational Databases Python and Andrew Godwin
Introduction Python for 5 years Django core developer Data modelling
/ visualisation
""Andrew speaks English like a machine gun speaks bullets."" Reinout
van Rees
If I speak too fast - tell me!
What is a relational database?
A relational database is a “collection of relations”
It's what a lot of people are used to.
Relational Databases PostgreSQL MySQL SQLite
Let's pick PostgreSQL (it's a good choice)
Usage conn = psycopg2.connect( host="localhost", user="postgres" ) cursor = conn.cursor()
cursor.execute('SELECT * FROM users WHERE username = "andrew";') for row in cursor.fetchall(): print row
You've probably seen all that before.
Now, to introduce some non-relational databases
Document Databases MongoDB CouchDB
Key-Value Stores Redis Cassandra
Message Queues AMQP Celery
Various Others Graph databases Filesystems VCSs
Redis and MongoDB are two good examples here
Redis: Key-value store with strings, lists, sets, channels and atomic
operations.
Redis Example conn = redis.Redis(host="localhost") print conn.get("top_value") conn.set("last_user", "andrew") conn.inc("num_runs")
conn.sadd("users", "andrew") conn.sadd("users", "martin") for item in conn.smembers("users"): print item
MongoDB: Document store with indexing and a wide range of
query filters.
MongoDB Example conn = pymongo.Connection("localhost") db = conn['mongo_example'] coll =
db['users'] coll.insert({ "username": "andrew", "uid": 1000, }) for entry in coll.find({"username": "andrew"}): print entry
These all solve different problems - you can't easily replace
one with the other.
""When all you have is a hammer, everything looks like
a nail"" Abraham Manslow (paraphrased)
JOIN - your best friend, and your worst enemy.
Denormalising your data speeds up reads, and slows down writes.
Schemaless != Denormalised
Atomic operations are nice. conn.incrby("num_users', 2)
But SQL can do some of them. UPDATE foo SET
bar = bar + 1 WHERE baz;
Redis, the datastructures server. SETNX, GETSET, EXPIRES and friends
Locks / Semaphores conn.setnx("lock:foo", time.time() + 3600) val = conn.decr("sem:foo")
if val >= 0: ... else: conn.incr("sem:foo")
Queues conn.lpush("myqueue", "workitem") todo = conn.lpop("myqueue") (or publish/subscribe)
Priority Queues conn.zadd("myqueue", "handle-meltdown", 1) conn.zadd("myqueue", "feed-cats", 5) todo =
conn.zrange("myqueue", 0, 1) conn.zrem(todo)
Lock-free linked lists! new_id = "bgrdsd" old_end = conn.getset(":end", new_id)
conn.set("%s:next" % old_end, new_id)
Performance-wise, the less checks/integrity the faster it goes.
Maturity can sometimes be an issue, but new features can
appear rapidly.
You can also use databases for the wrong thing -
it often only matters ""at scale""
But how does this all relate to Python?
Most databases - even new ones - have good Python
bindings
Postgres: PsycoPG2 Redis: redis-py MongoDB: pymongo (and more - neo4j,
VCSen, relational, etc.)
Some databases have Python available inside (Postgres has it as
an option)
Document databases map really well to Python dicts
You may find non-relational databases a nicer way to store
state - for any app
Remember, you might still need transactions/reliability. (Business logic is probably
better off on mature systems for now)
Overall? Just keep all the options in mind. Don't get
caught by trends, and don't abuse your relational store
Thanks. Andrew Godwin @andrewgodwin http://aeracode.org