Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
On The Look-out For Your Data (DjangoCon Europe...
Search
Markus H
May 24, 2018
Technology
0
13k
On The Look-out For Your Data (DjangoCon Europe 2018)
My talk from DjangoCon Europe 2018
Markus H
May 24, 2018
Tweet
Share
More Decks by Markus H
See All by Markus H
🐍 ❤️ 🦀 — Python loves Rust
markush
0
190
Knock! Knock! Who's There?
markush
0
59
An Introduction To Kubernetes ☸
markush
0
77
Writing Safe Database Migrations (DjangoCon Europe 2021)
markush
0
14k
A Pony On The Move: How Migrations Work In Django 🐎
markush
0
13k
All Hands on Deck — Handling Security Issues
markush
0
13k
Logging Rethought 2: The Actions of Frank Taylor Jr. (PyCon UK 2019)
markush
0
49
Logging Rethought 2: The Actions of Frank Taylor Jr. (PyCon Australia 2019)
markush
1
180
Logging Rethought 2: The Actions of Frank Taylor Jr. (DjangoCon Europe 2019)
markush
0
13k
Other Decks in Technology
See All in Technology
Data-centric AI入門第6章:Data-centric AIの実践例
x_ttyszk
1
400
開発スピードは上がっている…品質はどうする? スピードと品質を両立させるためのプロダクト開発の進め方とは #DevSumi #DevSumiB / Agile And Quality
nihonbuson
2
2.9k
MC906491 を見据えた Microsoft Entra Connect アップグレード対応
tamaiyutaro
1
540
運用しているアプリケーションのDBのリプレイスをやってみた
miura55
1
720
飲食店予約台帳を支えるインタラクティブ UI 設計と実装
siropaca
7
1.8k
次世代KYC活動報告 / 20250219-BizDay17-KYC-nextgen
oidfj
0
250
ハッキングの世界に迫る~攻撃者の思考で考えるセキュリティ~
nomizone
13
5.2k
2/18/25: Java meets AI: Build LLM-Powered Apps with LangChain4j
edeandrea
PRO
0
110
N=1から解き明かすAWS ソリューションアーキテクトの魅力
kiiwami
0
130
RECRUIT TECH CONFERENCE 2025 プレイベント【高橋】
recruitengineers
PRO
0
160
管理者しか知らないOutlookの裏側のAIを覗く#AzureTravelers
hirotomotaguchi
2
390
分解して理解する Aspire
nenonaninu
1
120
Featured
See All Featured
ReactJS: Keep Simple. Everything can be a component!
pedronauck
666
120k
Designing Experiences People Love
moore
140
23k
A Modern Web Designer's Workflow
chriscoyier
693
190k
Making the Leap to Tech Lead
cromwellryan
133
9.1k
Being A Developer After 40
akosma
89
590k
Build your cross-platform service in a week with App Engine
jlugia
229
18k
Writing Fast Ruby
sferik
628
61k
Designing Dashboards & Data Visualisations in Web Apps
destraynor
231
53k
Site-Speed That Sticks
csswizardry
4
380
Rails Girls Zürich Keynote
gr2m
94
13k
Testing 201, or: Great Expectations
jmmastey
42
7.2k
Speed Design
sergeychernyshev
27
790
Transcript
On The Look-Out For Your Data @m_holtermann #djangocon Europe 2018
I’m Markus Holtermann @m_holtermann • github.com/MarkusH • markusholtermann.eu @laterpay •
laterpay.net • Django Contributor • Software Engineer at
What Is Search?
How To Search In Django?
What Is Search?
What Is Search? Try to find something by looking or
otherwise seeking carefully and thoroughly. — Oxford English Dictionary
What Is Search? Try to find something by looking or
otherwise seeking carefully and thoroughly. — Oxford English Dictionary
What Is Search? Try to find something by looking or
otherwise seeking carefully and thoroughly. — Oxford English Dictionary
Search Is Hard
Searching In Django?
from django.shortcuts import get_object_or_404, render from blog.models import Article def
article_view(request, pk): article = get_object_or_404(Article, pk=pk) return render( request, 'article.html', context={'article': article}, )
Searching Text
from django.shortcuts import get_list_or_404, render from blog.models import Article def
article_view(request): articles = get_list_or_404( Article, text__icontains=request.GET.get('query', ''), ) return render( request, 'articles.html', context={'articles': articles}, )
SELECT * FROM blog_article WHERE text ILIKE '%Looking for text%'
Trigrams
-- As superuser # CREATE EXTENSION pg_trgm; # SELECT show_trgm('I
love Django'); show_trgm ---------------------------------------------- - {" d", " i", " l", " dj", " i ", " lo", ang, dja, "go ", jan, lov, ngo, ove, "ve "}
from django.contrib.postgres.indexes import GistIndex class TrigramGistIndex(GistIndex): suffix = 'trgm_gist' sql
= 'CREATE INDEX %(name)s ON %(table)s %(using)s \ (UPPER(%(columns)s) gist_trgm_ops)%(extra)s' def create_sql(self, model, schema_editor, using=''): statement = super().create_sql(model, schema_editor, using=using) statement.template = self.sql return statement
Searching Text
Full-text Search
Word order doesn’t matter “Django Migrations” = “Migrations Django”
Stemming computer, compute, computation = comput
Ignoring Stopwords “Django is the best” = “Django best”
__search & PostgreSQL https://docs.djangoproject.com/en/ 2.0/ref/contrib/postgres/search/
External Search Tools
None
from django.db import models, transaction class Article(models.Model): # ... def
save(self, *args, **kwargs): with transaction.atomic(): transaction.on_commit( lambda: update_search(self.pk)) super().save(*args, **kwargs)
from django.db import models, transaction class Article(models.Model): # ... def
delete(self, *args, **kwargs): pk = self.pk with transaction.atomic(): transaction.on_commit( lambda: delete_search(pk)) return super().delete( *args, **kwargs)
Maintain A Complete Search Index
What Is Search? Try to find something by looking or
otherwise seeking carefully and thoroughly. — Oxford English Dictionary
• Example: https://github.com/MarkusH/talk-django-search • Search in Django: https://docs.djangoproject.com/en/2.0/topics/db/search/ • Choosing
a PostgreSQL text search method: https://blog.2ndquadrant.com/text-search-strategies-in-postgresql/ • Trigram Extension: https://www.postgresql.org/docs/10/static/pgtrgm.html • Full-text search: https://www.postgresql.org/docs/10/static/textsearch-tables.html
Thank you! @m_holtermann
import blog.indexes from django.contrib.postgres.operations import TrigramExtension from django.db import migrations
class Migration(migrations.Migration): dependencies = [('blog', '0002_auto_20180503_1925')] operations = [ TrigramExtension(), migrations.AddIndex( model_name='entry', index=blog.indexes.TrigramGistIndex( fields=['body'], name='body_trgm_gist')), ]
-- Creates extension pg_trgm CREATE EXTENSION IF NOT EXISTS "pg_trgm";
-- Create index body_trgm_gist on field(s) -- body of model entry CREATE INDEX "body_trgm_gist" ON "blog_entry" USING gist (UPPER("body") gist_trgm_ops);