Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up for free
Python and Relational/Non-relational Databases
Andrew Godwin
October 22, 2010
Programming
0
85
Python and Relational/Non-relational Databases
A talk I gave at PyCon Ukraine 2010.
Andrew Godwin
October 22, 2010
Tweet
Share
More Decks by Andrew Godwin
See All by Andrew Godwin
A Newcomer's Guide To Airflow's Architecture
andrewgodwin
0
86
Async, Python, and the Future
andrewgodwin
1
360
How To Break Django: With Async
andrewgodwin
1
280
Taking Django's ORM Async
andrewgodwin
0
330
The Long Road To Asynchrony
andrewgodwin
0
380
The Scientist & The Engineer
andrewgodwin
1
370
Pioneering Real-Time
andrewgodwin
0
150
Just Add Await: Retrofitting Async Into Django
andrewgodwin
2
1.1k
Terrain, Art, Python and LiDAR
andrewgodwin
1
230
Other Decks in Programming
See All in Programming
Is Rust a great language for building Kubernetes ecosystem
deepu105
0
130
microCMS × Shopifyで、ECサイトがリニューアル後急成長した話
microcms
0
440
Node.js 最新動向 TFCon 2022
yosuke_furukawa
PRO
4
2.3k
Micro Frontends with Module Federation: Beyond the Basics
manfredsteyer
PRO
0
300
型パラメータが使えるようになったのでLINQを実装してみた
makki_d
2
490
Milestoner
bkuhlmann
1
200
Micro Frontends with Module Federation: Beyond the Basics @jax2022
manfredsteyer
PRO
0
240
Untangling Coroutine Testing (Android Makers 2022)
zsmb
0
380
既存のプロジェクトにKMMを導入するための対応策
martysuzuki
2
280
Git Rebase
bkuhlmann
7
1k
バンドル最適化マニアクス at tfconf
mizchi
3
1.3k
ゼロから作る Protocol Buffer のパーサーとレキサー / Writing Protocol Buffer Parser/Lexer in Go from scratch
yoheimuta
1
150
Featured
See All Featured
ReactJS: Keep Simple. Everything can be a component!
pedronauck
655
120k
From Idea to $5000 a Month in 5 Months
shpigford
372
44k
ピンチをチャンスに:未来をつくるプロダクトロードマップ #pmconf2020
aki_i
21
14k
Clear Off the Table
cherdarchuk
79
280k
Making Projects Easy
brettharned
98
4.3k
Designing with Data
zakiwarfel
91
3.8k
The Art of Programming - Codeland 2020
erikaheidi
31
5.8k
10 Git Anti Patterns You Should be Aware of
lemiorhan
638
52k
Web development in the modern age
philhawksworth
197
9.3k
CSS Pre-Processors: Stylus, Less & Sass
bermonpainter
349
27k
Unsuck your backbone
ammeep
659
55k
The Mythical Team-Month
searls
208
39k
Transcript
Relational / Non-relational Databases Python and Andrew Godwin
Introduction Python for 5 years Django core developer Data modelling
/ visualisation
""Andrew speaks English like a machine gun speaks bullets."" Reinout
van Rees
If I speak too fast - tell me!
What is a relational database?
A relational database is a “collection of relations”
It's what a lot of people are used to.
Relational Databases PostgreSQL MySQL SQLite
Let's pick PostgreSQL (it's a good choice)
Usage conn = psycopg2.connect( host="localhost", user="postgres" ) cursor = conn.cursor()
cursor.execute('SELECT * FROM users WHERE username = "andrew";') for row in cursor.fetchall(): print row
You've probably seen all that before.
Now, to introduce some non-relational databases
Document Databases MongoDB CouchDB
Key-Value Stores Redis Cassandra
Message Queues AMQP Celery
Various Others Graph databases Filesystems VCSs
Redis and MongoDB are two good examples here
Redis: Key-value store with strings, lists, sets, channels and atomic
operations.
Redis Example conn = redis.Redis(host="localhost") print conn.get("top_value") conn.set("last_user", "andrew") conn.inc("num_runs")
conn.sadd("users", "andrew") conn.sadd("users", "martin") for item in conn.smembers("users"): print item
MongoDB: Document store with indexing and a wide range of
query filters.
MongoDB Example conn = pymongo.Connection("localhost") db = conn['mongo_example'] coll =
db['users'] coll.insert({ "username": "andrew", "uid": 1000, }) for entry in coll.find({"username": "andrew"}): print entry
These all solve different problems - you can't easily replace
one with the other.
""When all you have is a hammer, everything looks like
a nail"" Abraham Manslow (paraphrased)
JOIN - your best friend, and your worst enemy.
Denormalising your data speeds up reads, and slows down writes.
Schemaless != Denormalised
Atomic operations are nice. conn.incrby("num_users', 2)
But SQL can do some of them. UPDATE foo SET
bar = bar + 1 WHERE baz;
Redis, the datastructures server. SETNX, GETSET, EXPIRES and friends
Locks / Semaphores conn.setnx("lock:foo", time.time() + 3600) val = conn.decr("sem:foo")
if val >= 0: ... else: conn.incr("sem:foo")
Queues conn.lpush("myqueue", "workitem") todo = conn.lpop("myqueue") (or publish/subscribe)
Priority Queues conn.zadd("myqueue", "handle-meltdown", 1) conn.zadd("myqueue", "feed-cats", 5) todo =
conn.zrange("myqueue", 0, 1) conn.zrem(todo)
Lock-free linked lists! new_id = "bgrdsd" old_end = conn.getset(":end", new_id)
conn.set("%s:next" % old_end, new_id)
Performance-wise, the less checks/integrity the faster it goes.
Maturity can sometimes be an issue, but new features can
appear rapidly.
You can also use databases for the wrong thing -
it often only matters ""at scale""
But how does this all relate to Python?
Most databases - even new ones - have good Python
bindings
Postgres: PsycoPG2 Redis: redis-py MongoDB: pymongo (and more - neo4j,
VCSen, relational, etc.)
Some databases have Python available inside (Postgres has it as
an option)
Document databases map really well to Python dicts
You may find non-relational databases a nicer way to store
state - for any app
Remember, you might still need transactions/reliability. (Business logic is probably
better off on mature systems for now)
Overall? Just keep all the options in mind. Don't get
caught by trends, and don't abuse your relational store
Thanks. Andrew Godwin @andrewgodwin http://aeracode.org