Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
On The Look-out For Your Data (DjangoCon Europe...
Search
Markus H
May 24, 2018
Technology
0
13k
On The Look-out For Your Data (DjangoCon Europe 2018)
My talk from DjangoCon Europe 2018
Markus H
May 24, 2018
Tweet
Share
More Decks by Markus H
See All by Markus H
🐍 ❤️ 🦀 — Python loves Rust
markush
0
240
Knock! Knock! Who's There?
markush
0
64
An Introduction To Kubernetes ☸
markush
0
93
Writing Safe Database Migrations (DjangoCon Europe 2021)
markush
0
14k
A Pony On The Move: How Migrations Work In Django 🐎
markush
0
13k
All Hands on Deck — Handling Security Issues
markush
0
14k
Logging Rethought 2: The Actions of Frank Taylor Jr. (PyCon UK 2019)
markush
0
60
Logging Rethought 2: The Actions of Frank Taylor Jr. (PyCon Australia 2019)
markush
1
210
Logging Rethought 2: The Actions of Frank Taylor Jr. (DjangoCon Europe 2019)
markush
0
13k
Other Decks in Technology
See All in Technology
あなたの知らない Linuxカーネル脆弱性の世界
recruitengineers
PRO
3
160
AWS DMS で SQL Server を移行してみた/aws-dms-sql-server-migration
emiki
0
230
初めてのDatabricks Apps開発
taka_aki
1
400
Databricks AI/BI Genie の「値ディクショナリー」をAmazonの奥地(S3)まで見に行く
kameitomohiro
1
410
20251024_TROCCO/COMETAアップデート紹介といくつかデモもやります!_#p_UG 東京:データ活用が進む組織の作り方
soysoysoyb
0
100
OTEPsで知るOpenTelemetryの未来 / Observability Conference Tokyo 2025
arthur1
0
240
Okta Identity Governanceで実現する最小権限の原則 / Implementing the Principle of Least Privilege with Okta Identity Governance
tatsumin39
0
170
ゼロコード計装導入後のカスタム計装でさらに可観測性を高めよう
sansantech
PRO
1
420
会社を支える Pythonという言語戦略 ~なぜPythonを主要言語にしているのか?~
curekoshimizu
3
670
20251027_findyさん_音声エージェントLT
almondo_event
2
430
ハノーファーメッセ2025で見た生成AI活用ユースケース.pdf
hamadakoji
1
470
IBC 2025 動画技術関連レポート / IBC 2025 Report
cyberagentdevelopers
PRO
2
160
Featured
See All Featured
CSS Pre-Processors: Stylus, Less & Sass
bermonpainter
359
30k
Agile that works and the tools we love
rasmusluckow
331
21k
Put a Button on it: Removing Barriers to Going Fast.
kastner
60
4k
Refactoring Trust on Your Teams (GOTO; Chicago 2020)
rmw
35
3.2k
Why You Should Never Use an ORM
jnunemaker
PRO
59
9.6k
Faster Mobile Websites
deanohume
310
31k
Reflections from 52 weeks, 52 projects
jeffersonlam
353
21k
Easily Structure & Communicate Ideas using Wireframe
afnizarnur
194
16k
How GitHub (no longer) Works
holman
315
140k
Designing for Performance
lara
610
69k
Connecting the Dots Between Site Speed, User Experience & Your Business [WebExpo 2025]
tammyeverts
10
620
Building a Modern Day E-commerce SEO Strategy
aleyda
44
7.8k
Transcript
On The Look-Out For Your Data @m_holtermann #djangocon Europe 2018
I’m Markus Holtermann @m_holtermann • github.com/MarkusH • markusholtermann.eu @laterpay •
laterpay.net • Django Contributor • Software Engineer at
What Is Search?
How To Search In Django?
What Is Search?
What Is Search? Try to find something by looking or
otherwise seeking carefully and thoroughly. — Oxford English Dictionary
What Is Search? Try to find something by looking or
otherwise seeking carefully and thoroughly. — Oxford English Dictionary
What Is Search? Try to find something by looking or
otherwise seeking carefully and thoroughly. — Oxford English Dictionary
Search Is Hard
Searching In Django?
from django.shortcuts import get_object_or_404, render from blog.models import Article def
article_view(request, pk): article = get_object_or_404(Article, pk=pk) return render( request, 'article.html', context={'article': article}, )
Searching Text
from django.shortcuts import get_list_or_404, render from blog.models import Article def
article_view(request): articles = get_list_or_404( Article, text__icontains=request.GET.get('query', ''), ) return render( request, 'articles.html', context={'articles': articles}, )
SELECT * FROM blog_article WHERE text ILIKE '%Looking for text%'
Trigrams
-- As superuser # CREATE EXTENSION pg_trgm; # SELECT show_trgm('I
love Django'); show_trgm ---------------------------------------------- - {" d", " i", " l", " dj", " i ", " lo", ang, dja, "go ", jan, lov, ngo, ove, "ve "}
from django.contrib.postgres.indexes import GistIndex class TrigramGistIndex(GistIndex): suffix = 'trgm_gist' sql
= 'CREATE INDEX %(name)s ON %(table)s %(using)s \ (UPPER(%(columns)s) gist_trgm_ops)%(extra)s' def create_sql(self, model, schema_editor, using=''): statement = super().create_sql(model, schema_editor, using=using) statement.template = self.sql return statement
Searching Text
Full-text Search
Word order doesn’t matter “Django Migrations” = “Migrations Django”
Stemming computer, compute, computation = comput
Ignoring Stopwords “Django is the best” = “Django best”
__search & PostgreSQL https://docs.djangoproject.com/en/ 2.0/ref/contrib/postgres/search/
External Search Tools
None
from django.db import models, transaction class Article(models.Model): # ... def
save(self, *args, **kwargs): with transaction.atomic(): transaction.on_commit( lambda: update_search(self.pk)) super().save(*args, **kwargs)
from django.db import models, transaction class Article(models.Model): # ... def
delete(self, *args, **kwargs): pk = self.pk with transaction.atomic(): transaction.on_commit( lambda: delete_search(pk)) return super().delete( *args, **kwargs)
Maintain A Complete Search Index
What Is Search? Try to find something by looking or
otherwise seeking carefully and thoroughly. — Oxford English Dictionary
• Example: https://github.com/MarkusH/talk-django-search • Search in Django: https://docs.djangoproject.com/en/2.0/topics/db/search/ • Choosing
a PostgreSQL text search method: https://blog.2ndquadrant.com/text-search-strategies-in-postgresql/ • Trigram Extension: https://www.postgresql.org/docs/10/static/pgtrgm.html • Full-text search: https://www.postgresql.org/docs/10/static/textsearch-tables.html
Thank you! @m_holtermann
import blog.indexes from django.contrib.postgres.operations import TrigramExtension from django.db import migrations
class Migration(migrations.Migration): dependencies = [('blog', '0002_auto_20180503_1925')] operations = [ TrigramExtension(), migrations.AddIndex( model_name='entry', index=blog.indexes.TrigramGistIndex( fields=['body'], name='body_trgm_gist')), ]
-- Creates extension pg_trgm CREATE EXTENSION IF NOT EXISTS "pg_trgm";
-- Create index body_trgm_gist on field(s) -- body of model entry CREATE INDEX "body_trgm_gist" ON "blog_entry" USING gist (UPPER("body") gist_trgm_ops);