Wednesday, September 5, 12

Alex Gaynor, DjangoCon 2012 Take 2 If I got to do Django all over again Wednesday, September 5, 12

Thanks rdio! We’re hiring! Wednesday, September 5, 12 Thanks to my employer, rdio, for paying for me to be here today. Also we’re hiring, for just about everything, ops, frontend, backend, come talk to me if you’re interested. Also, use our product, it’s awesome!

About Me Django core developer Django Software Foundation board member Frameworks and compiler nerd Wednesday, September 5, 12 If I’m up here proposing a bunch of changes for django, and why I think everything is wrong with it, it’s only fair that I have some qualifications to say these things. I’m one of the people who helps to build this thing, I try to serve the community, so my hatred and my critique, they come from a place of love, I wouldn’t be here if I didn’t think django had the most potential of anything I’d worked with.

What is this talk? What is Django? What’s wrong with Django? How can we fix it? Wednesday, September 5, 12 This talk is basically 3 parts. What’s is Django, which is not a question I think presenters ask very frequently at DjangoCon. What’s wrong with it, and how do we fix those things? I guess there’s an implicit “can we fix it?” in there, but the mere fact I’m up here should tell you I think the answer is yes. My goal here today is to get you thinking more about framework design, and more about how Django works. There’s a lot of critique about individual APIs in Django, but I don’t think we spend enough time talking about the overall design of the Django “system”, so if you do any more of that because of this talk, I think I’ll have suceeded.

Why isn’t this just a .diff? No regard for backwards compatibility. No regard for ease of use for new users. No attempts to build real apps on this. Wednesday, September 5, 12 So I’m a Django developer, I have things I want to change in it. Why am I giving a talk? Shouldn’t I be in a back room coding somewhere furiously? Historically when people have given “what’s wrong with Django talks” audience members have gone and turned each item into the talk into a ticket for Django. Please don’t do that. What I’m about to propose has no regard for backwards compatibility, I haven’t thought about how easy this would be for new users to grasp, AT ALL, and I haven’t tried to build anything real with this proposed architecture. That last one is pretty important. I think it’s pretty well established, empirically, that the best frameworks are extracted from people building real projects. The next Django, or Django 2.0, probably won’t come from me ponding the design of frameworks, or anyone else doing so, it will, I hope, come from people thinking about these things, trying to use them, seeing why their wrong, and taking them to a place where real projects can be built on them. That’s the key step, and these ideas aren’t right for a project as big as Django without that step.

What is Django? Wednesday, September 5, 12 So, without further ado, What is Django? Not a question people ask here much. It’s a web framework... duh.

An abstraction over WSGI A command line application runner A set of libraries that are useful for web sites ORM Templates i18n Wednesday, September 5, 12 A web framework isn’t a thing though, just look at the space of web frameworks, they aren’t all one thing. Django, for example, is about three things. A request/response abstraction over WSGI, a command like runner, manage.py, and then a whole lot of libraries. So really it’s like 20 things.

What’s the problem? Wednesday, September 5, 12 So Django is all these things, but what’s wrong with it? Sure, I have complaints about a lot of the individual components, but I’m here to talk about a larger, broader issue. I have 2 real complaints, one of which I will spend almost no time talking about, I just want you to think about it, and the other which will be the rest of this talk.

“How do I do X in Django?” “Does Django have something for X?” Wednesday, September 5, 12 My first complaint is, I heard the first question way too much. How do you do X in Django? The same way as any other web framework, you read the POST variable, you write the SQL query, there’s no special magic to doing these things in Django, in the absence of Django having a helper for something, the way to solve a problem is the same as in any other Python code. Once you think of Django as a toolbox, and not a strait jacket, you can get a lot further with it.

Testing in Django Sucks Wednesday, September 5, 12 Testing in Django sucks. Sounds kind of underwhelming for what I consider to be the fundamental flaw in Django right? But I assure you, this is it, if testing were better in Django every single other thing would fall into place. In the likely event there is someone in the room not sold on testing, I’m going to give you my 30 second “Why, if you aren’t testing I never want to work on your project” pitch, then I’ll show you why testing sucks in Django, and how we can fix it.

Why testing? The alternative is manual testing. You are going to introduce bugs Testing changes the way you write software. Wednesday, September 5, 12

Testing changes the way you write software Wednesday, September 5, 12 This is the big one. When you write tests, suddenly you can’t just slam together anything that throws HTML on a page. You have to think about how different components of your project work with each other. That is, you have to do something that almost approaches being a real engineering discipline, instead of the shit show that is a lot of software projects. There aren’t well tested software projects that actually have atrocious code bases. It just doesn’t happen.

from django.test import TestCase from .models import Track class TrackTests(TestCase): def test_can_stream_no_rights(self): t = Track.objects.create() self.assertFalse(t.can_stream()) Wednesday, September 5, 12 Decent test right? Relatively isolated, tests a pretty isolated block of code. One class, one method. Hands up if you think this is a pretty good test?

The problem Wednesday, September 5, 12 That test is fine. The problem is the test runner. That test has no state. The test runner, has a metric ton of state. In my view, the key to successful testing is state isolation. You control the inputs, you check the inputs. The universe is an input to a test. In Haskell they try to stick the universe inside of a monad. I don’t know what that means, but I do know that if you don’t control what state the universe is for your tests, you’re going to have a bad time.

The Universe Database is created with every single app. Tons of signal handlers Fixtures Probably other stuff Wednesday, September 5, 12 The database gets created, with every model and app in your project. If you have a big enough project this takes a crap load of time. I’ve worked on projects where this is 30 seconds. Then a bunch of signal handlers get run, Django create contenttypes, permissions, plus anything in your project’s apps. initial_data fixtures get loaded And god only knows what I missed, basically the universe is huge, and you control none of it.

And one more thing... django.conf.settings Wednesday, September 5, 12 Oh yeah, global variables are a part of the universe too. So settings are a giant magical global, which influences tons of other things (default media storage for example). And again, besides monkey patching them, you don’t control them, and monkey patching them doesn’t even work most of the time

How it should work You tell the test what you need. The test prepares that (creates the DB tables, uses the settings you provided, etc.) No global state is used. Wednesday, September 5, 12 Instead of you having a project, with a bunch of settings that describe how the tests will be run. The tests should declare what they need setup, essentially their own mini-settings.py, and that should be setup. This should be a part of that tests setup, no global test setup.

from django.test import TestConfig from .models import Track class TrackTests(object): test_config = TestConfig( INSTALLED_APPS=["library"], ) def test_can_stream_no_rights(self): t = Track.objects.create() self.assertFalse(t.can_stream()) Wednesday, September 5, 12 So this is what a test might look like under such a system. Your test declares the config it needs. There’s a pretty critical issue in this though. If we no longer have any global state, that means no django.conf.settings, how does Track.objects.create() know what database to use? The answer: it can’t. So we need to go get back to basics and think about how we’d design an API

Killing the globals Wednesday, September 5, 12 If we want to get rid of globals in Django, we need everything to be a local variable. To do that, we need to start from the beginning, and look at the entry points Django has.

Where does Django begin? wsgi.py manage.py Wednesday, September 5, 12 Django has two primary entrypoints. wsgi.py, and manage.py. When you run django-admin start project, you get these files generated for you, both of the work by setting the DJANGO_SETTINGS_MODULE enviroment variable to the settings file that was generated for you. So, to get started, I want to kill django generating files for you. Instead, when you create a new project you’ll write something like this

from django import DjangoApplication app = DjangoApplication() if __name__ == "__main__": app.run_cli() Wednesday, September 5, 12 Pretty simple, a DjangoApplication is the core of what we currently call a project. It holds any options, any configuration, and acts as the CLI runner and WSGI application. So if you had a file like this you could already point gunicorn at it to serve your site, or run this like we run manage.py. Now that we have any empty site, let’s start building up a real site with it, first thing we need is a URL.

from django import DjangoApplication from .urls import url_patterns app = DjangoApplication( root_view=url_patterns, ) if __name__ == "__main__": app.run_cli() Wednesday, September 5, 12 Simon Willison had this really cool idea a few years ago, of Django being “turtles all the way down”, what he meant by that was “everything is a view”. What is a view? In Django parlance, a view is a function that takes a request, and parameters from the URL, and returns a response. Pretty simple contract. So here, you’ll notice we aren’t giving our DjangoApplications a urlconf parameter, we’re giving it a root_view parameter, because any view would work here. We could pass a view function, a middleware class, anything.

from django import URLPatterns, url from . import views url_patterns = URLPatterns( url(r"^artist/(?P[\w_]+)/$", views.artist_page), ) Wednesday, September 5, 12 So this is what a urls.py looks like. I changed some names around, but really it’s all the same. URLPatterns is just a class that takes a sequence of URLs, and returns a view that dispatches to them based on the path of the request. You also might note that I’m not using the “dotted string path” form of the providing the URL. That’s because passing python import paths as strings sucks. It doesn’t make code any easier to write, and just means that if you ever have an import error, the traceback is going to be obscure as hell. Like I said, no thought, whatsoever, has been given to backwards compatibility.

from django.shortcuts import render from .models import Artist def artist_page(request, slug): artist = Artist.objects.get(slug=slug) return render(request, "artist.html", { "artist": artist }) Wednesday, September 5, 12 Now we need to write a view, and here’s where stuff gets interesting. This is the old way of writing a view. Pretty simple, really elegant in my opinion. We have our parameters from the URL, we fetch our data, we dispatch to a template. It’s kind of beautiful, makes me want to shed a tear. Unfortunately it’s also ugly as all hell, because this has about, and this is just a back-of-the-napkin estimate, 4 bajillion globals. What are they? Artist.objects.get somehow magically knows what your databases are, and which to query. render somehow knows what template loaders to use, where your templates live, what your template context processors are; and thats to say nothing of all the magic inside of the template, like what where your templatetag libraries live. So basically I love this and hate this. Can we retain the beauty, and get rid of all the ugly?

from .models import Artist def artist_page(request, slug): artist = request.app.query(Artist).get(slug=slug) return request.app.render(request, "artist.html", { "artist": artist }) Wednesday, September 5, 12 Yes, I think we can keep the beauty, and toss out the ugly. So you’re probably noticing that this is a little more verbose. You’re also probably wondering what the hell request.app is, and that these query and render functions are. Here’s the core idea: everywhere you want to do something regarding your django project, you need to have an app objects, and app object is an explicit way to pass around all the state that used to be global. So the query function? That takes a model and returns to you a manager that’s bound to a specific app. So now you’ve explicitly told the manager, and therefore the queryset, what databases you’re querying against, what database routers you’re using, and all the other info you need to actually generate and execute queries. The render function? It looks at the current app, figures out where to load templates from, and then renders them.

from django import DjangoApplication from .models import Track class TrackTests(object): django_app = DjangoApplication( installed_apps=[ "library", ], ) def test_can_stream_no_rights(self, app): track = app.query(Track).create() self.assertFalse(track.can_stream()) Wednesday, September 5, 12 So now we obviously need to go back and get our test working. This is what it might look like. This should all be pretty obvious, except one bit. Now our test function is taking this “app” argument. Where does it come from? There’s a test runner I really love called py.test, and it has this really fantastic feature called “funcargs”. Which are basically these things where, your function declares what funcargs it takes, and then py.test provides them as arguments. Funcargs are allowed to have their own setup and teardown logic. So an example funcarg included in py.test is tempdir. On setup it creates a temporary directory on your hard drive, your function then takes that directory as an argument, and on teardown the directory is deleted for you, all automatically. So in this case, django would provide a funcarg named app, it would look on your test case for the django_app, it would run whatever setup is needed, probably creating a few tables, give you the app to run queries against, and then it would do the teardown logic for you.

Solved Problems Running multiple django sites in a single process. Faster tests. Better reusable apps. Wednesday, September 5, 12 So, in changing the entire architecture of django, we’ve solved some real problems. Now, because there’s no global settings, you can run multiple sites in one web server process. Your tests run faster because you have more limited setup/teardown. You can write better reusable apps, instead of an app providing integration by saying “include these URLs” you can now provide anything you want, that can be customized however you want, just so long as you can call it with a request and it returns a response.

And that’s all there is to it! Wednesday, September 5, 12 So that’s it right. This is how I think django should look basically.

Recap No auto generated files. No global state. Everything is a view. Isolated tests. Wednesday, September 5, 12 These are the four key principles. Of course, there are all sorts of other things, I want to change, but when you’re building on this foundation, I think solving other problems becomes a lot easier.

Other things I hate INSTALLED_APPs sucks The ORM’s API sucks We should probably just replace the template engine with Jinja2 templatetags suck Wednesday, September 5, 12 So, just in case you were curious, I still hate a ton of other things. I’m pretty sure INSTALLED_APPS is a terrible idea, and the idea of “apps” shouldn’t even be a thing in Django. The ORM sucks, I’ve given an entire other talk on that. templatetags are awful, and there’s no reason for us to maintain our own template language at this point.

I still <3 Django. Wednesday, September 5, 12 No really, I promise I still love Django. But Django won’t be around forever, every generation of frameworks has been replace by something better. One of the Meteor guys is giving a keynote, where he’s probably going to tell us why frameworks like Django and rails are obsolete in the face of new realtime Javascript stuff. But I’m pretty sure the thing that’s going to replace Django is going to be a better Django, not something entirely different.

https:/ /speakerdeck.com/u/alex Thank you! Questions? Comments? Thinly veiled insults? Wednesday, September 5, 12 Thank you guys for listening! These slides are going to be online at that URL. We’ve now got a few minutes for questions, comments, or you guys to insult me a bit. Please come up to the mic to ask your questions and/or insult me so that everyone can hear. If we run out of time, please come find me in the halls or around, to talk with me.