Search from the Ground Up

A practical hands-on guide to implementing search in any Django web application. From basic search to full-text search with PostgreSQL.

Talk given at DjangoCon US 2019.

Transcript

1. DjangoCon US 2019 | William Vincent Search From the Ground Up William Vincent DjangoCon US 2019
2. DjangoCon US 2019 | William Vincent Just add search...
3. DjangoCon US 2019 | William Vincent /wsvincent/djangocon2019-search
4. DjangoCon US 2019 | William Vincent Who am I?
5. DjangoCon US 2019 | William Vincent Search Options Basic Filters Q Objects Full Text Search Hosted Service
6. DjangoCon US 2019 | William Vincent Search Magic City Search Website Massachusetts City Search Website City Search Website • Boston, MA • Worcester, MA • Springfield, MA Form Results + Magic =
7. DjangoCon US 2019 | William Vincent Start a New Project
8. DjangoCon US 2019 | William Vincent Installation / Set Up $ pipenv install django==2.2.5 $ pipenv shell (env) $ django-admin startproject citysearch_project . (end) $ python manage.py migrate (end) $ python manage.py startapp cities
9. DjangoCon US 2019 | William Vincent INSTALLED_APPS # citysearch_project/settings.py INSTALLED_APPS = [ ... 'cities.apps.CitiesConfig', # new ]
10. DjangoCon US 2019 | William Vincent Models # cities/models.py from django.db import models class City(models.Model): name = models.CharField(max_length=100) state = models.CharField(max_length=100) class Meta: verbose_name_plural = 'cities'
11. DjangoCon US 2019 | William Vincent Migrations (env) $ python manage.py makemigrations cities (env) $ python manage.py migrate (env) $ python manage.py createsuperuser
12. DjangoCon US 2019 | William Vincent Admin # cities/admin.py from django.contrib import admin from .models import City class CityAdmin(admin.ModelAdmin): list_display = ('name', 'state') admin.site.register(City, CityAdmin)
13. DjangoCon US 2019 | William Vincent
14. DjangoCon US 2019 | William Vincent Django’s 4-Way Dance urls.py templates.html views.py models.py WEBPAGE!!!
15. DjangoCon US 2019 | William Vincent URLs (part 1) # citysearch_project/urls.py from django.contrib import admin from django.urls import path, include # new urlpatterns = [ path('admin/', admin.site.urls), path('', include('cities.urls')), # new ]
16. DjangoCon US 2019 | William Vincent URLs (part 2) # cities/urls.py from django.urls import path from .views import SearchResultsView, HomepageView urlpatterns = [ path('search/', SearchResultsView.as_view(), name='search_results'), path('', HomepageView.as_view(), name='home'), ]
17. DjangoCon US 2019 | William Vincent Views # cities/views.py from django.views.generic import TemplateView, ListView from .models import City class HomepageView(TemplateView): template_name = 'home.html' class SearchResultsView(ListView): model = City template_name = 'search_results.html'
18. DjangoCon US 2019 | William Vincent Template Structure Default (in app) citysearch_project |_ cities |_ templates |_ cities |_ home.html Custom (project-level) citysearch_project cities templates |_ home.html
19. DjangoCon US 2019 | William Vincent Template DIRS # citysearch_projects/settings.py TEMPLATES = [ { ... 'DIRS': [os.path.join(BASE_DIR, 'templates')], # new } ]
20. DjangoCon US 2019 | William Vincent Homepage Template # templates/home.html <h1>HomePage</h1>
21. DjangoCon US 2019 | William Vincent Search Template # templates/search_results.html <h1>Search Results</h1> <ul> {% for city in object_list %} <li> {{ city.name }}, {{ city.state }} </li> {% endfor %} </ul>
22. DjangoCon US 2019 | William Vincent Basic Search
23. DjangoCon US 2019 | William Vincent Filtering # cities/views.py class SearchResultsView(ListView): model = City template_name = 'search_results.html' def get_queryset(self): # new return City.objects.filter(name__icontains='Boston')
24. DjangoCon US 2019 | William Vincent Chaining Filters (ANDs) # cities/views.py class SearchResultsView(ListView): model = City template_name = 'search_results.html' def get_queryset(self): # new return City.objects.filter( name__icontains='Boston' ).exclude( state__icontains='NY' )
25. DjangoCon US 2019 | William Vincent Q Objects (ORs) # cities/views.py from django.db.models import Q # new ... class SearchResultsView(ListView): model = City template_name = 'search_results.html' def get_queryset(self): # new return City.objects.filter( Q(name__icontains='Boston') | Q(state__icontains='NY') )
26. DjangoCon US 2019 | William Vincent Forms san diego GET POST
27. DjangoCon US 2019 | William Vincent HTML Form # templates/home.html <h1>HomePage</h1> <form action={% url 'search_results' %} method='get'> <input name='q' type='text' placeholder='Search...'> </form>
28. DjangoCon US 2019 | William Vincent Query not currently passed into View
29. DjangoCon US 2019 | William Vincent Add q to Queryset # cities/views.py class SearchResultsView(ListView): ... def get_queryset(self): # new query = self.request.GET.get('q') object_list = City.objects.filter( Q(name__icontains=query) | Q(state__icontains=query) ) return object_list
30. DjangoCon US 2019 | William Vincent Django Forms # cities/forms.py from django import forms class SearchForm(forms.Form): q = forms.CharField(label='Search', max_length=50)
31. DjangoCon US 2019 | William Vincent View with Django Form # cities/views.py from django.views.generic import FormView, ListView from .forms import SearchForm from .models import City class HomepageView(FormView): template_name = 'home.html' form_class = SearchForm ...
32. DjangoCon US 2019 | William Vincent Template with Django Form # templates/home.html <h1>Homepage</h1> <form action={% url 'search_results' %} method='get'> {{ form }} </form>
33. DjangoCon US 2019 | William Vincent Full-Text Search
34. DjangoCon US 2019 | William Vincent Search Options Basic Filters Q Objects Full Text Search Hosted Service
35. DjangoCon US 2019 | William Vincent Full-Text Search Search Documents that satisfy a Query and sort by Relevance. • PostgreSQL support since 8.3 (2008) • Django support since version 1.10 (2016) led by Marc Tamlyn
36. DjangoCon US 2019 | William Vincent Full Text Search Features • Rankings • Indexing • Phrase search • Stop words • Stemming • Accent / multiple language support • JSON[B] support
37. DjangoCon US 2019 | William Vincent Pre-Processing Documents -> Tokens -> Lexemes
38. DjangoCon US 2019 | William Vincent PostgreSQL Data Types tsvector: preprocessed documents tsquery: processed queries
39. DjangoCon US 2019 | William Vincent tsvector Document: 'The quick brown fox jumps over the lazy dog.' tsvector ------------------------------------------------------------------------------------- 'brown':3 'dog':9 'fox':4 'jump':5 'lazy':8 'quick:2'
40. DjangoCon US 2019 | William Vincent tsquery List of words checked against normalized tsvector 'The quick brown fox jumps over the lazy dog.' Query Match? (using @@ operator) 'dog' true 'dogs' true 'dogfood' false 'jumping' true
41. DjangoCon US 2019 | William Vincent tsquery operators 'The quick brown fox jumps over the lazy dog.' Operator tsquery Result? @@ (match) 'dogs foxes' true & (and) 'dog & fox' true | (or) 'dog | cat' true ! (negation) '!pony ' true
42. DjangoCon US 2019 | William Vincent django.contrib.postgres.search SearchVector = search multiple fields SearchQuery = add stemming and/or stop words SearchRank = apply rank to result of documents Weighted Queries = add weights SearchVectorField = performance upgrade with manual trigger
43. DjangoCon US 2019 | William Vincent SearchVector # cities/views.py from django.contrib.postgres.search import SearchVector ... class SearchResultsView(ListView): ... def get_queryset(self): query = self.request.GET.get('q') object_list = City.objects.annotate( search=SearchVector('name', 'state'), ).filter(search=query) return object_list
44. DjangoCon US 2019 | William Vincent SearchQuery # cities/views.py from django.contrib.postgres.search import SearchVector, SearchQuery def get_queryset(self): query = self.request.GET.get('q') object_list = City.objects.annotate( search=SearchVector('name', 'state'), ).filter(search=SearchQuery(query)) return object_list
45. DjangoCon US 2019 | William Vincent SearchRank # cities/views.py from django.contrib.postgres.search import SearchVector, SearchQuery, SearchRank def get_queryset(self): query = self.request.GET.get('q') vector = SearchVector('state') search_query = SearchQuery(query) object_list = City.objects.annotate( rank=SearchRank(vector, search_query) ).order_by('-rank') return object_list
46. DjangoCon US 2019 | William Vincent Resources Source Code https://github.com/wsvincent/djangocon2019-search Slides https://tinyurl.com/djangocon2019-search PostgreSQL Docs: Full Text Search https://www.postgresql.org/docs/current/textsearch.html