Slide 1

Slide 1 text

To Index Or Not That’s Not The Question @m_holtermann • github.com/MarkusH • markusholtermann.eu

Slide 2

Slide 2 text

Some Background Theory

Slide 3

Slide 3 text

No content

Slide 4

Slide 4 text

How Database Lookups Work

Slide 5

Slide 5 text

id name age 1 Emma 3 2 Mia 37 3 Ben 41 4 Jonas 11 5 Anna 19 6 Lena 5 7 Paul 13

Slide 6

Slide 6 text

id name age 1 Emma 3 2 Mia 37 3 Ben 41 4 Jonas 11 5 Anna 19 6 Lena 5 7 Paul 13 SELECT * FROM people WHERE age = 19;

Slide 7

Slide 7 text

id name age 1 Emma 3 2 Mia 37 3 Ben 41 4 Jonas 11 5 Anna 19 6 Lena 5 7 Paul 13 SELECT * FROM people WHERE age = 19;

Slide 8

Slide 8 text

What are Indexes?

Slide 9

Slide 9 text

B-Tree / B+ Tree

Slide 10

Slide 10 text

No content

Slide 11

Slide 11 text

11 37 < 11 ≥ 11 and < 37 ≥ 37

Slide 12

Slide 12 text

11 37 3 5 11 13 19 37 41 < 11 ≥ 11 and < 37 ≥ 37

Slide 13

Slide 13 text

11 37 3 5 11 13 19 37 41 < 11 ≥ 11 and < 37 ≥ 37 3 5 11 13 37 41 19 age =

Slide 14

Slide 14 text

11 37 3 5 11 13 19 37 41 < 11 ≥ 11 and < 37 ≥ 37 3 5 11 13 37 41 19 age =

Slide 15

Slide 15 text

id name age 1 Emma 3 2 Mia 37 3 Ben 41 4 Jonas 11 5 Anna 19 6 Lena 5 7 Paul 13 3 5 11 13 19 37 41

Slide 16

Slide 16 text

SELECT * FROM people WHERE age = 19;

Slide 17

Slide 17 text

11 37 3 5 11 13 19 37 41 id name age … 5 Anna 19 …

Slide 18

Slide 18 text

SELECT COUNT(*) FROM people WHERE age BETWEEN 5 AND 13; SELECT COUNT(*) FROM people WHERE age >= 5 AND age <= 13;

Slide 19

Slide 19 text

11 37 3 5 11 13 19 37 41

Slide 20

Slide 20 text

Indexes Are Awesome

Slide 21

Slide 21 text

• db_index=True

Slide 22

Slide 22 text

• db_index=True • index_together=(('name', 'age'),)

Slide 23

Slide 23 text

• db_index=True • index_together=(('name', 'age'),) • ForeignKey() / OneToOneField()

Slide 24

Slide 24 text

• db_index=True • index_together=(('name', 'age'),) • ForeignKey() / OneToOneField() • primary_key=True

Slide 25

Slide 25 text

2016

Slide 26

Slide 26 text

Google Summer of Code 2016

Slide 27

Slide 27 text

Introducing Index(fields, name)

Slide 28

Slide 28 text

from django.db import models class Person(models.Model): name = models.CharField(max_length=200) class Meta: indexes = [ models.Index( fields=['name'], name='name_idx', ), ]

Slide 29

Slide 29 text

from django.db import models class Person(models.Model): name = models.CharField(max_length=200) age = models.PositiveSmallIntegerField() class Meta: indexes = [ models.Index( fields=['name', 'age'], name='name_age_idx', ), ]

Slide 30

Slide 30 text

from django.contrib.postgres.fields import JSONField from django.contrib.postgres.indexes import GinIndex from django.db import models class Doc(models.Model): data = JSONField() class Meta: indexes = [ GinIndex( fields=['data'], name='data_gin', ), ]

Slide 31

Slide 31 text

$ python manage.py sqlmigrate someapp 0001 BEGIN; -- -- Create model Doc -- CREATE TABLE "someapp_doc" ( "id" serial NOT NULL PRIMARY KEY, "data" jsonb NOT NULL); -- -- Create index data_gin on field(s) data of model doc -- CREATE INDEX "data_gin" ON "someapp_doc" USING gin ("data"); COMMIT;

Slide 32

Slide 32 text

Feature Ideas

Slide 33

Slide 33 text

Functional Indexes from django.db import models class Author(models.Model): name = models.CharField(max_length=200) class Meta: indexes = [ FuncIndex( expression=Lower('name'), name='name_lower_idx', ), ]

Slide 34

Slide 34 text

db_index= from django.db import models class Author(models.Model): name = models.CharField( max_length=200, db_index=HashIndex )

Slide 35

Slide 35 text

Field.default_index_class from django.contrib.postgres.fields import JSONField from django.contrib.postgres.indexes import GinIndex from django.db import models # Somewhere in Django’s JSONField implementation: # JSONField.default_index_class = GinIndex class Document(models.Model): data = JSONField(db_index=True)

Slide 36

Slide 36 text

Refactor index_together and db_index

Slide 37

Slide 37 text

Add GiSTIndex

Slide 38

Slide 38 text

Sprints April 6th - 7th

Slide 39

Slide 39 text

Thank You

Slide 40

Slide 40 text

I’m Markus Holtermann @m_holtermann • github.com/MarkusH • markusholtermann.eu @laterpay • laterpay.net • Django Core Developer • Software Engineer at

Slide 41

Slide 41 text

I’m Markus Holtermann @m_holtermann • github.com/MarkusH • markusholtermann.eu @laterpay • laterpay.net • Django Core Developer • Software Engineer at W e’re hiring

Slide 42

Slide 42 text

Where We’re At

Slide 43

Slide 43 text

Functional Indexes from django.db import models class Author(models.Model): name = models.CharField(max_length=200) class Meta: indexes = [ FuncIndex( expression=Lower(Col('name', None)), name='name_lower_idx', ), ]