Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Python and Relational/Non-relational Databases
Search
Andrew Godwin
October 22, 2010
Programming
0
120
Python and Relational/Non-relational Databases
A talk I gave at PyCon Ukraine 2010.
Andrew Godwin
October 22, 2010
Tweet
Share
More Decks by Andrew Godwin
See All by Andrew Godwin
Reconciling Everything
andrewgodwin
1
300
Django Through The Years
andrewgodwin
0
190
Writing Maintainable Software At Scale
andrewgodwin
0
430
A Newcomer's Guide To Airflow's Architecture
andrewgodwin
0
340
Async, Python, and the Future
andrewgodwin
2
650
How To Break Django: With Async
andrewgodwin
1
720
Taking Django's ORM Async
andrewgodwin
0
710
The Long Road To Asynchrony
andrewgodwin
0
640
The Scientist & The Engineer
andrewgodwin
1
740
Other Decks in Programming
See All in Programming
カウシェで Four Keys の改善を試みた理由
ike002jp
1
140
ぽちぽち選択するだけでOSSを読めるVSCode拡張機能
ymbigo
14
6.6k
AWS Summit Hong Kong 2025: Reinventing Programming - How AI Transforms Our Enterprise Coding Approach
dwchiang
0
150
型安全なDrag and Dropの設計を考える
yudppp
3
140
監視 やばい
syossan27
12
10k
最速Green Tea 🍵 Garbage Collector
kuro_kurorrr
1
160
Ruby で作る RISC-V CPU エミュレーター / RISC-V CPU emulator made with Ruby
hayaokimura
5
1.2k
SpringBootにおけるオブザーバビリティのなにか
irof
0
180
一緒に働きたくなるプログラマの思想 #QiitaConference
mu_zaru
84
21k
MySQL初心者が311個のカラムにNot NULL制約を追加していってALTER TABLEについて学んだ話
hatsu38
2
150
Road to Ruby for A Linguistics Nerd
hayat01sh1da
PRO
0
390
ドメイン駆動設計とXPで支える子どもの未来 / Domain-Driven Design and XP Supporting Children's Future
nrslib
0
330
Featured
See All Featured
Art, The Web, and Tiny UX
lynnandtonic
298
21k
GitHub's CSS Performance
jonrohan
1031
460k
Rebuilding a faster, lazier Slack
samanthasiow
81
9k
Let's Do A Bunch of Simple Stuff to Make Websites Faster
chriscoyier
507
140k
Dealing with People You Can't Stand - Big Design 2015
cassininazir
367
26k
Building a Modern Day E-commerce SEO Strategy
aleyda
40
7.3k
Side Projects
sachag
453
42k
Being A Developer After 40
akosma
91
590k
Fontdeck: Realign not Redesign
paulrobertlloyd
84
5.5k
Stop Working from a Prison Cell
hatefulcrawdad
268
20k
Building a Scalable Design System with Sketch
lauravandoore
462
33k
Refactoring Trust on Your Teams (GOTO; Chicago 2020)
rmw
34
3k
Transcript
Relational / Non-relational Databases Python and Andrew Godwin
Introduction Python for 5 years Django core developer Data modelling
/ visualisation
""Andrew speaks English like a machine gun speaks bullets."" Reinout
van Rees
If I speak too fast - tell me!
What is a relational database?
A relational database is a “collection of relations”
It's what a lot of people are used to.
Relational Databases PostgreSQL MySQL SQLite
Let's pick PostgreSQL (it's a good choice)
Usage conn = psycopg2.connect( host="localhost", user="postgres" ) cursor = conn.cursor()
cursor.execute('SELECT * FROM users WHERE username = "andrew";') for row in cursor.fetchall(): print row
You've probably seen all that before.
Now, to introduce some non-relational databases
Document Databases MongoDB CouchDB
Key-Value Stores Redis Cassandra
Message Queues AMQP Celery
Various Others Graph databases Filesystems VCSs
Redis and MongoDB are two good examples here
Redis: Key-value store with strings, lists, sets, channels and atomic
operations.
Redis Example conn = redis.Redis(host="localhost") print conn.get("top_value") conn.set("last_user", "andrew") conn.inc("num_runs")
conn.sadd("users", "andrew") conn.sadd("users", "martin") for item in conn.smembers("users"): print item
MongoDB: Document store with indexing and a wide range of
query filters.
MongoDB Example conn = pymongo.Connection("localhost") db = conn['mongo_example'] coll =
db['users'] coll.insert({ "username": "andrew", "uid": 1000, }) for entry in coll.find({"username": "andrew"}): print entry
These all solve different problems - you can't easily replace
one with the other.
""When all you have is a hammer, everything looks like
a nail"" Abraham Manslow (paraphrased)
JOIN - your best friend, and your worst enemy.
Denormalising your data speeds up reads, and slows down writes.
Schemaless != Denormalised
Atomic operations are nice. conn.incrby("num_users', 2)
But SQL can do some of them. UPDATE foo SET
bar = bar + 1 WHERE baz;
Redis, the datastructures server. SETNX, GETSET, EXPIRES and friends
Locks / Semaphores conn.setnx("lock:foo", time.time() + 3600) val = conn.decr("sem:foo")
if val >= 0: ... else: conn.incr("sem:foo")
Queues conn.lpush("myqueue", "workitem") todo = conn.lpop("myqueue") (or publish/subscribe)
Priority Queues conn.zadd("myqueue", "handle-meltdown", 1) conn.zadd("myqueue", "feed-cats", 5) todo =
conn.zrange("myqueue", 0, 1) conn.zrem(todo)
Lock-free linked lists! new_id = "bgrdsd" old_end = conn.getset(":end", new_id)
conn.set("%s:next" % old_end, new_id)
Performance-wise, the less checks/integrity the faster it goes.
Maturity can sometimes be an issue, but new features can
appear rapidly.
You can also use databases for the wrong thing -
it often only matters ""at scale""
But how does this all relate to Python?
Most databases - even new ones - have good Python
bindings
Postgres: PsycoPG2 Redis: redis-py MongoDB: pymongo (and more - neo4j,
VCSen, relational, etc.)
Some databases have Python available inside (Postgres has it as
an option)
Document databases map really well to Python dicts
You may find non-relational databases a nicer way to store
state - for any app
Remember, you might still need transactions/reliability. (Business logic is probably
better off on mature systems for now)
Overall? Just keep all the options in mind. Don't get
caught by trends, and don't abuse your relational store
Thanks. Andrew Godwin @andrewgodwin http://aeracode.org