Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Thoughts About Normal and Abnormal Data (PyCon ...
Search
Markus H
October 27, 2017
Technology
0
13k
Thoughts About Normal and Abnormal Data (PyCon UK 2017)
Speaker notes at
https://markusholtermann.eu/2017/10/thoughts-about-normal-and-abnormal-data/
Markus H
October 27, 2017
Tweet
Share
More Decks by Markus H
See All by Markus H
🐍 ❤️ 🦀 — Python loves Rust
markush
0
190
Knock! Knock! Who's There?
markush
0
59
An Introduction To Kubernetes ☸
markush
0
77
Writing Safe Database Migrations (DjangoCon Europe 2021)
markush
0
14k
A Pony On The Move: How Migrations Work In Django 🐎
markush
0
13k
All Hands on Deck — Handling Security Issues
markush
0
13k
Logging Rethought 2: The Actions of Frank Taylor Jr. (PyCon UK 2019)
markush
0
49
Logging Rethought 2: The Actions of Frank Taylor Jr. (PyCon Australia 2019)
markush
1
180
Logging Rethought 2: The Actions of Frank Taylor Jr. (DjangoCon Europe 2019)
markush
0
13k
Other Decks in Technology
See All in Technology
Helm , Kustomize に代わる !? 次世代 k8s パッケージマネージャー Glasskube 入門 / glasskube-entry
parupappa2929
0
250
君も受託系GISエンジニアにならないか
sudataka
2
430
偶然 × 行動で人生の可能性を広げよう / Serendipity × Action: Discover Your Possibilities
ar_tama
1
1.1k
あれは良かった、あれは苦労したB2B2C型SaaSの新規開発におけるCloud Spanner
hirohito1108
2
580
プロダクトエンジニア構想を立ち上げ、プロダクト志向な組織への成長を続けている話 / grow into a product-oriented organization
hiro_torii
1
170
スタートアップ1人目QAエンジニアが QAチームを立ち上げ、“個”からチーム、 そして“組織”に成長するまで / How to set up QA team at reiwatravel
mii3king
2
1.5k
開発スピードは上がっている…品質はどうする? スピードと品質を両立させるためのプロダクト開発の進め方とは #DevSumi #DevSumiB / Agile And Quality
nihonbuson
2
2.9k
組織貢献をするフリーランスエンジニアという生き方
n_takehata
1
1.3k
7日間でハッキングをはじめる本をはじめてみませんか?_ITエンジニア本大賞2025
nomizone
2
1.8k
Classmethod AI Talks(CATs) #16 司会進行スライド(2025.02.12) / classmethod-ai-talks-aka-cats_moderator-slides_vol16_2025-02-12
shinyaa31
0
110
なぜ私は自分が使わないサービスを作るのか? / Why would I create a service that I would not use?
aiandrox
0
730
関東Kaggler会LT: 人狼コンペとLLM量子化について
nejumi
3
580
Featured
See All Featured
Fireside Chat
paigeccino
34
3.2k
Building Your Own Lightsaber
phodgson
104
6.2k
Bash Introduction
62gerente
611
210k
Six Lessons from altMBA
skipperchong
27
3.6k
GraphQLの誤解/rethinking-graphql
sonatard
68
10k
Building a Scalable Design System with Sketch
lauravandoore
461
33k
Let's Do A Bunch of Simple Stuff to Make Websites Faster
chriscoyier
507
140k
Intergalactic Javascript Robots from Outer Space
tanoku
270
27k
Embracing the Ebb and Flow
colly
84
4.6k
VelocityConf: Rendering Performance Case Studies
addyosmani
328
24k
Side Projects
sachag
452
42k
Building a Modern Day E-commerce SEO Strategy
aleyda
38
7.1k
Transcript
Thoughts About Normal and Abnormal Data Markus Holtermann @m_holtermann markusholtermann.eu
@m_holtermann I am Markus Holtermann • Senior Software Engineer at
LaterPay • Django Core Developer
@m_holtermann How do we store our data?
@m_holtermann Files CC-BY-NC 2.0 by Tim Gee https://flic.kr/p/rZm63
@m_holtermann Document Stores CC-BY-SA 4.0 by Susan Gerbic https://commons.wikimedia.org/wiki/File%3AArchive_Room.JPG
@m_holtermann Copyright Geek Batman https://www.youtube.com/watch?v=gPDx_IwdYMY
@m_holtermann Name Home planet Gender Padmé Naboo Female Luke Tatooine
Male Leia Alderaan, Naboo Female
@m_holtermann First Normal Form (1NF)
@m_holtermann PersonID Name Home planet Gender 1 Padmé Naboo Female
2 Luke Tatooine Male 3 Leia Alderaan Female 3 Leia Naboo Female
@m_holtermann PersonID Name Home planet Gender 3 Leia Alderaan Female
3 Leia Naboo Male Update Anomalies
@m_holtermann Second Normal Form (2NF)
@m_holtermann PersonID Name Home planet Gender 1 Padmé Naboo Female
2 Luke Tatooine Male 3 Leia Alderaan Female 3 Leia Naboo Female
@m_holtermann PersonID Planet Name 1 Naboo 2 Tatooine 3 Alderaan
3 Naboo PersonID Name Gender 1 Padmé Female 2 Luke Male 3 Leia Female
@m_holtermann PersonID Planet Name 1 Naboo 2 Tatooine 3 Alderaan
3 Naboo ??? Dagobah Insert Anomalies
@m_holtermann Deletion Anomalies PersonID Planet Name 1 Naboo 2 Tatooine
3 Alderaan 3 Naboo PersonID Name Gender 1 Padmé Female 2 Luke Male 3 Leia Female
@m_holtermann Third Normal Form (3NF)
@m_holtermann PlanetID Name Water 10 Naboo 85% 11 Tatooine 1%
12 Alderaan 78% 13 Dagobah 88% PersonID Name Gender 1 Padmé Female 2 Luke Male 3 Leia Female PersonID PlanetID 1 10 2 11 3 10 3 12
@m_holtermann Database normalization is great!
@m_holtermann Always?
@m_holtermann Yet Another Wiki
@m_holtermann Page + PageID Name Slug Revision + RevisionID PageID
Text Date Database Schema
@m_holtermann Task 1: Fetch a single page and its current
revision
@m_holtermann Task 2: Fetch all page titles and the date
of their current revision
Task 1: Fetch a single page SELECT * FROM page
INNER JOIN revision ON page.page_id = revision.page_id WHERE page.slug = 'some-slug' ORDER BY revision.date DESC LIMIT 1;
Task 2: Fetch all pages SELECT page.name, last_revs.date FROM page
INNER JOIN ( SELECT revision.page_id, MAX(revision.date) date FROM revision GROUP BY revision.page_id ) last_revs ON page.page_id = last_revs.page_id;
@m_holtermann Benchmark Environment • Intel i7-6600U, 2.60GHz • 8 GB
Memory • PostgreSQL 9.6.5 • 10k pages, 6m revisions
@m_holtermann Task 1: Fetch a single page Concurrent queries 10
Pages per connection 1000 Queries per page 10 Queries total 100000
@m_holtermann Task 2: Fetch all pages Concurrent queries 1 Queries
per connection 10 Queries total 10
@m_holtermann Task 1: Fetch a single page
@m_holtermann Task 2: Fetch all pages
@m_holtermann Rae Knowler https://speakerdeck.com/bellisk/unsafe-at-any-speed-pycon-uk-26th-october-2017
@m_holtermann Database Schema Page + PageID Name Slug LastRevision Revision
+ RevisionID PageID Text Date
Task 1: Fetch a single page SELECT * FROM page
INNER JOIN revision ON page.last_revision_id = revision.revision_id WHERE page.slug = 'some-slug';
Task 2: Fetch all pages SELECT page.name, revision.date FROM page
INNER JOIN revision ON page.last_revision_id = revision.revision_id;
@m_holtermann Task 1: Fetch a single page
@m_holtermann Task 2: Fetch all pages
@m_holtermann Conclusion
Thanks Markus Holtermann @m_holtermann markusholtermann.eu