Django as your data management framework

Slides from Python Porto meetup #11. Examples can be found here: https://github.com/imankulov/django-data-management-example

If you come to Python to work with data, most likely you are intimately familiar with its data science ecosystem, which includes Anaconda, Jupyter and Pandas, among others. Probably though you are less familiar with the other side of Python, including frameworks, ORMs and template engines, traditionally used to create web applications.

This talk aims at blurring the border between data science and web development ecosystems. I will show how Django, extremely popular framework for creating web applications, can be successfully applied to some common tasks of data processing and data analysis and how it can be used seamlessly with your tools of choice.

The talk can be interesting for Python data analysts who want to extend their knowledge beyond the traditional DS/ML stack. It can also be helpful for beginners interested in web development with Django.

Transcript

1. Django as your data management framework Roman Imankulov Python Porto #11 in Blip | 30 Jan 2019
2. Motivation
3. What is Django
4. What can Django give to me? • Project structure • Object-relational mapping (ORM) • Admin panel • Model-view-controller framework (MVC)
5. Example github.com/imankulov/django-data-management-example
6. What can Django give to me? • Project structure • Object-relational mapping (ORM) • Admin panel • Model-view-controller framework (MVC)
7. project manage.py settings.py app app app urls.py
8. project manage.py settings.py app app app urls.py admin.py models.py views.py templates …
9. $ pip install Django $ django-admin startproject explore
11. What can Django give to me? • Project structure • Object-relational mapping (ORM) • Admin panel • Model-view-controller framework (MVC)
12. CREATE TABLE groups ( id BIGINT PRIMARY KEY AUTOINCREMENT, name TEXT, urlname TEXT, description TEXT, created DATETIME, … ); > communities/models.py class Group(models.Model): name = models.CharField(…) urlname = models.CharField(…) description = models.TextField() created = models.DateTimeField() … class Meta: db_table = ‘groups’
14. $ ./manage.py makemigrations
15. $ ./manage.py makemigrations $ ./manage.py migrate
16. $ ./manage.py makemigrations $ ./manage.py migrate $ ./manage.py inspectdb
18. Get a single object sql> SELECT * FROM groups WHERE id = 42
19. Get a single object sql> SELECT * FROM groups WHERE id = 42 >>> Group.objects.get(id=42) <Group: Data Science Unplugged>
20. Get a queryset sql> SELECT * FROM groups ... WHERE members > 400 ... AND created >= ‘2018-01-01’ ... ORDER BY members DESC ... LIMIT 10;
21. Get a queryset sql> SELECT * FROM groups ... WHERE members > 400 ... AND created >= ‘2018-01-01’ ... ORDER BY members DESC ... LIMIT 10; >>> Group.objects.filter( members__gt=400, created__gt=datetime(2018, 1, 1) ).order_by(‘-members')[:10]
 <QuerySet [<Group: AI Lisbon>, …]>
22. Convert to DataFrame queryset = Group.objects.filter(…) df = pd.DataFrame.from_records(groups.values())
23. Convert to DataFrame queryset = Group.objects.filter(…) df = pd.DataFrame.from_records(groups.values()) city country ... urlname who 0 Lisbon PT ... ironhack-lisbon Ironhackers 1 Lisbon PT ... AI-Lisbon Members 2 Lisbon PT ... Lisbon-Artificial-Intelligence-Meetup Members 3 Lisbon PT ... Lisbon-Virtual-Currency-Meetup Members 4 Lisbon PT ... Zalando-Tech-Events-Lisbon Members 5 Lisbon PT ... Lisbon-Kaggle Kagglers 6 Lisbon PT ... Sketch-Design-Lisbon Members [7 rows x 13 columns]
25. What can Django give to me? • Project structure • Object-relational mapping (ORM) • Admin panel • Model-view-controller framework (MVC)
27. > communities/admin.py @admin.register(Group) class GroupAdmin(admin.ModelAdmin): list_display = ['name', ‘city’, ...] list_filter = ['country', 'city'] search_fields = ['name', 'organizer_name']
31. What can Django give to me? • Project structure • Object-relational mapping (ORM) • Admin panel • Model-view-controller framework (MVC)
32. What can Django give to me? • Project structure • Object-relational mapping (ORM) • Admin panel • Model-view-controller framework (MVC) Model-template-view framework (MTV)
35. > urls.py urlpatterns = [ path('simple/', views.simple_dashboard), path('input/', views.input_dashboard), … ] GET /input/?limit=20&order_by=… <HttpRequest: GET /input/?limit=20&order_by=…> > views.py def simple_dashboard(request): …
36. <HttpRequest: GET /input/?limit=20&order_by=…> > views.py def simple_dashboard(request): … return HttpResponse(…) <HttpResponse status_code=200, "text/html; charset=utf-8">
37. <HttpRequest: GET /input/?limit=20&order_by=…> > views.py @staff_member_required def simple_dashboard(request): … return HttpResponse(…) <HttpResponse status_code=200, "text/html; charset=utf-8">
38. <HttpRequest: GET /input/?limit=20&order_by=…> > views.py @staff_member_required def simple_dashboard(request): … return HttpResponse(…) <HttpResponse status_code=200, "text/html; charset=utf-8">
39. <HttpRequest: GET /input/?limit=20&order_by=…> > views.py def input_dashboard(request): form = InputForm(…) … df = pd.DataFrame.from_values(…) return render( request, ‘dashboard.html’, {…}) <HttpResponse status_code=200, "text/html; charset=utf-8">
40. <HttpRequest: GET /input/?limit=20&order_by=…> > views.py def input_dashboard(request): form = InputForm(…) … df = pd.DataFrame.from_values(…) return render( request, ‘dashboard.html’, {…}) <HttpResponse status_code=200, "text/html; charset=utf-8"> > templates/dashboard.html <form method="GET" action="."> <table> {{ form.as_table }} </table> <input type="submit" value="Filter"> </form> {{ df.to_html|safe }}
42. What can Django give to me? • Project structure • Object-relational mapping (ORM) • Admin panel • Model-view-controller framework (MVC)
43. Django hints for data people
44. Django hints for data people • Can I use Django from Jupyter Notebook? • You can, but you need to initialize your environment properly to make it explicit which project we are working with right now
45. Django hints for data people • Can I use Django from Jupyter Notebook? • You can, but you need to initialize your environment properly to make it explicit which project we are working with right now
46. Django hints for data people • What is the easiest way to display a plot in my Django view? • Use Pygal (https://pygal.org) which draws plots to SVG and put them right in your templates
47. Django hints for data people • What is the easiest way to display a plot in my Django view? • Use Pygal (https://pygal.org) which draws plots to SVG and put them right in your templates > views.py @staff_member_required def plot_dashboard(request): groups = Group.objects.order_by('-members')[:10] df = pd.DataFrame.from_records(groups.values()) bar = pygal.Bar(…) bar.x_labels = df.name bar.add('Members', df.members) return render(request, 'plot_dashboard.html', { 'df': df, 'bar': bar.render(is_unicode=True), })
48. Django hints for data people • What is the easiest way to display a plot in my Django view? • Use Pygal (https://pygal.org) which draws plots to SVG and put them right in your templates > views.py @staff_member_required def plot_dashboard(request): groups = Group.objects.order_by('-members')[:10] df = pd.DataFrame.from_records(groups.values()) bar = pygal.Bar(…) bar.x_labels = df.name bar.add('Members', df.members) return render(request, 'plot_dashboard.html', { 'df': df, 'bar': bar.render(is_unicode=True), }) > templates/plot_dashboard.html {{ bar|safe }} {{ df.to_html|safe }}
49. Django hints for data people • What is the easiest way to display a plot in my Django view? • Use Pygal (https://pygal.org) which draws plots to SVG and put them right in your templates > views.py @staff_member_required def plot_dashboard(request): groups = Group.objects.order_by('-members')[:10] df = pd.DataFrame.from_records(groups.values()) bar = pygal.Bar(…) bar.x_labels = df.name bar.add('Members', df.members) return render(request, 'plot_dashboard.html', { 'df': df, 'bar': bar.render(is_unicode=True), }) > templates/plot_dashboard.html {{ bar|safe }} {{ df.to_html|safe }}
50. Django hints for data people • Can I serve my Jupyter Notebooks with Django • It’s not straightforward, but you can convert them to HTML with nbconvert and serve this way
51. Django hints for data people • Can I serve my Jupyter Notebooks with Django • It’s not straightforward, but you can convert them to HTML with nbconvert and serve this way > views.py class StringWriter(WriterBase): content = None def write(self, output, resources, **kw): self.content = output @staff_member_required def notebook(request): exporter = HTMLExporter( exclude_input=True, exclude_output_prompt=True) writer = StringWriter() app = NbConvertApp(writer=writer, exporter=exporter) app.convert_single_notebook(‘filename.ipynb’) return HttpResponse(writer.content)
52. Django hints for data people • Can I serve my Jupyter Notebooks with Django • It’s not straightforward, but you can convert them to HTML with nbconvert and serve this way > views.py class StringWriter(WriterBase): content = None def write(self, output, resources, **kw): self.content = output @staff_member_required def notebook(request): exporter = HTMLExporter( exclude_input=True, exclude_output_prompt=True) writer = StringWriter() app = NbConvertApp(writer=writer, exporter=exporter) app.convert_single_notebook(‘filename.ipynb’) return HttpResponse(writer.content)
53. Django hints for data people • How can I initiate long-running tasks from Django views? • Offload these tasks to workers. For example, use Celery 
 (http://celeryproject.org/)
54. Questions?