Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Thoughts About Normal and Abnormal Data (PyCon ...
Search
Markus H
October 27, 2017
Technology
0
13k
Thoughts About Normal and Abnormal Data (PyCon UK 2017)
Speaker notes at
https://markusholtermann.eu/2017/10/thoughts-about-normal-and-abnormal-data/
Markus H
October 27, 2017
Tweet
Share
More Decks by Markus H
See All by Markus H
🐍 ❤️ 🦀 — Python loves Rust
markush
0
220
Knock! Knock! Who's There?
markush
0
63
An Introduction To Kubernetes ☸
markush
0
88
Writing Safe Database Migrations (DjangoCon Europe 2021)
markush
0
14k
A Pony On The Move: How Migrations Work In Django 🐎
markush
0
13k
All Hands on Deck — Handling Security Issues
markush
0
13k
Logging Rethought 2: The Actions of Frank Taylor Jr. (PyCon UK 2019)
markush
0
57
Logging Rethought 2: The Actions of Frank Taylor Jr. (PyCon Australia 2019)
markush
1
200
Logging Rethought 2: The Actions of Frank Taylor Jr. (DjangoCon Europe 2019)
markush
0
13k
Other Decks in Technology
See All in Technology
cdk initで生成されるあのファイル達は何なのか/cdk-init-generated-files
tomoki10
1
670
衛星運用をソフトウェアエンジニアに依頼したときにできあがるもの
sankichi92
1
1k
AI Ready API ─ AI時代に求められるAPI設計とは?/ AI-Ready API - Designing MCP and APIs in the AI Era
yokawasa
6
1.9k
IPA&AWSダブル全冠が明かす、人生を変えた勉強法のすべて
iwamot
PRO
2
230
american aa airlines®️ USA Contact Numbers: Complete 2025 Support Guide
aaguide
0
500
Figma Dev Mode MCP Serverを用いたUI開発
zoothezoo
0
230
[SRE NEXT] ARR150億円_エンジニア140名_27チーム_17プロダクトから始めるSLO.pdf
satos
5
3k
OpenTelemetryセマンティック規約の恩恵とMackerel APMにおける活用例 / SRE NEXT 2025
mackerelio
3
2k
SREのためのeBPF活用ステップアップガイド
egmc
2
1.3k
Introduction to Bill One Development Engineer
sansan33
PRO
0
260
サービスを止めるな! DDoS攻撃へのスマートな備えと最前線の事例
coconala_engineer
1
180
AI エージェントと考え直すデータ基盤
na0
20
7.9k
Featured
See All Featured
How GitHub (no longer) Works
holman
314
140k
GraphQLの誤解/rethinking-graphql
sonatard
71
11k
Principles of Awesome APIs and How to Build Them.
keavy
126
17k
Java REST API Framework Comparison - PWX 2021
mraible
31
8.7k
Into the Great Unknown - MozCon
thekraken
40
1.9k
Building Better People: How to give real-time feedback that sticks.
wjessup
367
19k
CSS Pre-Processors: Stylus, Less & Sass
bermonpainter
357
30k
Evolution of real-time – Irina Nazarova, EuRuKo, 2024
irinanazarova
8
830
Fantastic passwords and where to find them - at NoRuKo
philnash
51
3.3k
The Language of Interfaces
destraynor
158
25k
Rails Girls Zürich Keynote
gr2m
95
14k
Easily Structure & Communicate Ideas using Wireframe
afnizarnur
194
16k
Transcript
Thoughts About Normal and Abnormal Data Markus Holtermann @m_holtermann markusholtermann.eu
@m_holtermann I am Markus Holtermann • Senior Software Engineer at
LaterPay • Django Core Developer
@m_holtermann How do we store our data?
@m_holtermann Files CC-BY-NC 2.0 by Tim Gee https://flic.kr/p/rZm63
@m_holtermann Document Stores CC-BY-SA 4.0 by Susan Gerbic https://commons.wikimedia.org/wiki/File%3AArchive_Room.JPG
@m_holtermann Copyright Geek Batman https://www.youtube.com/watch?v=gPDx_IwdYMY
@m_holtermann Name Home planet Gender Padmé Naboo Female Luke Tatooine
Male Leia Alderaan, Naboo Female
@m_holtermann First Normal Form (1NF)
@m_holtermann PersonID Name Home planet Gender 1 Padmé Naboo Female
2 Luke Tatooine Male 3 Leia Alderaan Female 3 Leia Naboo Female
@m_holtermann PersonID Name Home planet Gender 3 Leia Alderaan Female
3 Leia Naboo Male Update Anomalies
@m_holtermann Second Normal Form (2NF)
@m_holtermann PersonID Name Home planet Gender 1 Padmé Naboo Female
2 Luke Tatooine Male 3 Leia Alderaan Female 3 Leia Naboo Female
@m_holtermann PersonID Planet Name 1 Naboo 2 Tatooine 3 Alderaan
3 Naboo PersonID Name Gender 1 Padmé Female 2 Luke Male 3 Leia Female
@m_holtermann PersonID Planet Name 1 Naboo 2 Tatooine 3 Alderaan
3 Naboo ??? Dagobah Insert Anomalies
@m_holtermann Deletion Anomalies PersonID Planet Name 1 Naboo 2 Tatooine
3 Alderaan 3 Naboo PersonID Name Gender 1 Padmé Female 2 Luke Male 3 Leia Female
@m_holtermann Third Normal Form (3NF)
@m_holtermann PlanetID Name Water 10 Naboo 85% 11 Tatooine 1%
12 Alderaan 78% 13 Dagobah 88% PersonID Name Gender 1 Padmé Female 2 Luke Male 3 Leia Female PersonID PlanetID 1 10 2 11 3 10 3 12
@m_holtermann Database normalization is great!
@m_holtermann Always?
@m_holtermann Yet Another Wiki
@m_holtermann Page + PageID Name Slug Revision + RevisionID PageID
Text Date Database Schema
@m_holtermann Task 1: Fetch a single page and its current
revision
@m_holtermann Task 2: Fetch all page titles and the date
of their current revision
Task 1: Fetch a single page SELECT * FROM page
INNER JOIN revision ON page.page_id = revision.page_id WHERE page.slug = 'some-slug' ORDER BY revision.date DESC LIMIT 1;
Task 2: Fetch all pages SELECT page.name, last_revs.date FROM page
INNER JOIN ( SELECT revision.page_id, MAX(revision.date) date FROM revision GROUP BY revision.page_id ) last_revs ON page.page_id = last_revs.page_id;
@m_holtermann Benchmark Environment • Intel i7-6600U, 2.60GHz • 8 GB
Memory • PostgreSQL 9.6.5 • 10k pages, 6m revisions
@m_holtermann Task 1: Fetch a single page Concurrent queries 10
Pages per connection 1000 Queries per page 10 Queries total 100000
@m_holtermann Task 2: Fetch all pages Concurrent queries 1 Queries
per connection 10 Queries total 10
@m_holtermann Task 1: Fetch a single page
@m_holtermann Task 2: Fetch all pages
@m_holtermann Rae Knowler https://speakerdeck.com/bellisk/unsafe-at-any-speed-pycon-uk-26th-october-2017
@m_holtermann Database Schema Page + PageID Name Slug LastRevision Revision
+ RevisionID PageID Text Date
Task 1: Fetch a single page SELECT * FROM page
INNER JOIN revision ON page.last_revision_id = revision.revision_id WHERE page.slug = 'some-slug';
Task 2: Fetch all pages SELECT page.name, revision.date FROM page
INNER JOIN revision ON page.last_revision_id = revision.revision_id;
@m_holtermann Task 1: Fetch a single page
@m_holtermann Task 2: Fetch all pages
@m_holtermann Conclusion
Thanks Markus Holtermann @m_holtermann markusholtermann.eu