Letters from the Battlefield

Slide 1

Slide 1 text

Letters from the Battlefield Armin @mitsuhiko Ronacher

Slide 2

Slide 2 text

No content

Slide 3

Slide 3 text

No content

Slide 4

Slide 4 text

No content

Slide 5

Slide 5 text

I like to review code and design APIs

Slide 6

Slide 6 text

design for maintainability and security

Slide 7

Slide 7 text

“if I could do it again …”

Slide 8

Slide 8 text

so here are some lessons learned

Slide 9

Slide 9 text

the thing about overengineering PREFACE overengineering |ˈōvərˌenjəˈniriNG| noun the designing of a product to be more robust or complicated than is necessary for its application

Slide 10

Slide 10 text

a lot of what's in this talk is often seen as “unnecessary”

Slide 11

Slide 11 text

developers are afraid of complexity and initial overhead

Slide 12

Slide 12 text

but the right solutions were often already created; use them

Slide 13

Slide 13 text

being afraid of changes PROLOGUE afraid |əˈfrād| adjective worried that something undesirable will occur or be done: he was afraid that the farmer would send the dog after them

Slide 14

Slide 14 text

changes • developers should never feel afraid of code changes • developers should not be afraid of the ﬁrst change • developers should feel comfortable doing big changes • developers should not accidentally produce security problems

Slide 15

Slide 15 text

bite size chunks • write code so that developers are never overwhelmed • neither on making new features • nor on changing existing code • simpliﬁes code review

Slide 16

Slide 16 text

the goal is to make developers confident and happy

Slide 17

Slide 17 text

where is the state? CHAPTER 1 state |stāt| noun the particular condition that someone or something is in at a specific time

Slide 18

Slide 18 text

state in programming • Most prominent languages are rich in state • But poor in explicitly managing it • Most programmers do not know how their own state works • No rules when mutable state becomes assumed constant state

Slide 19

Slide 19 text

why is that a problem? • Most prominent languages are rich in state • But poor in explicitly managing it • Most programmers do not know how their own state works

Slide 20

Slide 20 text

practical example from functools import update_wrapper from django.conf import settings def might_debug(f): def new_func(*args, **kwargs):  if settings.DEBUG: do_some_debug_stuff() return f(*args, **kwargs) return update_wrapper(new_func, f)

Slide 21

Slide 21 text

is ‘settings’ mutable? • it's python, so the answer is yes • however at which point is it safe to modify them? • what if people drag out state to an unsafe scope?

Slide 22

Slide 22 text

decision made from functools import update_wrapper from django.conf import settings if settings.DEBUG: def might_debug(f): def new_func(*args, **kwargs):  do_some_debug_stuff() return f(*args, **kwargs) return update_wrapper(new_func, f) else: might_debug = lambda x: x

Slide 23

Slide 23 text

module state in python • imports are stateful • module scope is stateful • this inﬂuences code we write in Python • modules in Python are giant singletons • the scope of state can be hidden

Slide 24

Slide 24 text

hidden state from flask import request def is_api_request(): return bool(request.headers.get('Authorization'))

Slide 25

Slide 25 text

“Every once a while the error messages are Spanish”

Slide 26

Slide 26 text

decisions made from hidden state >>> from django.utils.translation import ugettext >>> ugettext('Hmmmm') u'Hmmmm'

Slide 27

Slide 27 text

decisions made from hidden state from django.utils.translation import ugettext class LoginForm(…): ERROR = ugettext(u"Could not sign in")

Slide 28

Slide 28 text

decisions made from hidden state def handle_request(request): endpoint, args, kwargs = match_request(request) func = import_view_function(endpoint) return func(*args, **kwargs)

Slide 29

Slide 29 text

shackle the state! CHAPTER 2 shackle |ˈSHak(ə)l| verb restrain; limit: they seek to shackle the oil and gas companies by imposing new controls.

Slide 30

Slide 30 text

stateful APIs suck • nobody likes stateful APIs • in particular nobody likes APIs that randomly change behavior

Slide 31

Slide 31 text

ideal state management • create scope • set up initial working conditions (modify here) • execute code • clean up state • destroy scope

Slide 32

Slide 32 text

prevent access • If something is not there, say so, not not fall back • translations should not silently become idempotent calls

Slide 33

Slide 33 text

raise if accessed in bad scope >>> from flask import request >>> request.headers Traceback (most recent call last): … RuntimeError: Working outside of request context.

Slide 34

Slide 34 text

prevent modifications with settings.transaction() as t: t.CONFIG_VALUE = 42 settings.close()

Slide 35

Slide 35 text

prevent stupid code >>> settings.transaction() Traceback (most recent call last): File "", line 1, in RuntimeError: Settings are closed. No more modifications

Slide 36

Slide 36 text

import madness CHAPTER 3 madness |ˈmadnəs| noun the state of being mentally ill, especially severely.

Slide 37

Slide 37 text

the art of importing • import all • upfront • do not import at runtime • there be many evil backstabbing dragons

Slide 38

Slide 38 text

import all stuﬀ from werkzeug.utils import find_modules def import_all(pkg): for module in find_modules(pkg, recursive=True): __import__(module) import_all(__name__.split('.')[0])

Slide 39

Slide 39 text

why? • importing requires locks; imports can be recursive • imports have side effects, let's get it done early • both those things are bad • once it's imported, it's cached • after that things become much, much more predictable

Slide 40

Slide 40 text

circular dependencies • good luck with that ;-) • I do not have a good response to this.

Slide 41

Slide 41 text

make it searchable CHAPTER 4 search |sərCH| verb try to find something by looking or otherwise seeking carefully and thoroughly: I searched among the rocks, but there was nothing

Slide 42

Slide 42 text

why? • new developers need to understand context • when you have conceptional security issues you need to ﬁnd things • aids code review

Slide 43

Slide 43 text

what's ‘searchable’ • assume your only tool is grep • write code so that you can grep/full text search it • it will be worth it

Slide 44

Slide 44 text

things that are easily grep-able • decorators! • explicit and clear function and class names • special methods • avoid funky operator overloads if they do something non-standard

Slide 45

Slide 45 text

predict common behavior CHAPTER 5 predict |prəˈdikt| verb say or estimate that (a specified thing) will happen in the future or will be a consequence of something: he predicts that the trend will continue

Slide 46

Slide 46 text

my least favorite code import json from django.http import HttpResponse def view_function(request): some_data = generate_some_data(…) return HttpResponse(json.dumps(some_data), mimetype='application/json')

Slide 47

Slide 47 text

what about this? from myproject.api import ApiResponse def view_function(): some_data = generate_some_data(…) return ApiResponse(some_data)

Slide 48

Slide 48 text

why? • we establish “request context” • we deﬁne a clear common case of “this is the result of an API” • we can transform and handle data on the way out

Slide 49

Slide 49 text

what do we gain? • JSON encode security issues? One clear point to handle it • Need to support a custom mimetype? Change all in one go • Instrumentation? One common object

Slide 50

Slide 50 text

convert common values def handle_request(request): rv = dispatch_request(request) if isinstance(rv, ApiResponse): rv = Response(json.dumps(rv), mimetype='application/json', status=rv.status_code)  return rv

Slide 51

Slide 51 text

define context CHAPTER 6 context |ˈkäntekst| noun the circumstances that form the setting for an event, statement, or idea, and in terms of which it can be fully understood and assessed

Slide 52

Slide 52 text

what is context • runtime context (“scopes”) • data context (“transfer encodings”) • security context (“who is the actor?”)

Slide 53

Slide 53 text

context behavior • what happens based on context? • how does data look like? • how does context inﬂuence what is happening?

Slide 54

Slide 54 text

examples of scoped context • current language • current http request • current authenticated user • current access restrictions

Slide 55

Slide 55 text

implied context >>> from myapp.i18n import ugettext, set_language >>> with set_language("en_US"): ... ugettext("Sign in") ... u"Sign in" >>> with set_language("de_DE"): ... ugettext("Sign in") ... u"Anmelden"

Slide 56

Slide 56 text

context for data • object in string context • object in HTML context • object serialization

Slide 57

Slide 57 text

data in context >>> from markupsafe import Markup, escape >>> unicode(my_user) u"Peter Doe" >>> escape(my_user) u'Peter Doe' >>> Markup("%s") % my_user u'Peter Doe' >>> print json.dumps(my_user) {"username": "Peter Doe", "id": 42}

Slide 58

Slide 58 text

prevent misuse CHAPTER 7 misuse |ˌmisˈyo͞os| noun the wrong or improper use of something: a misuse of power.

Slide 59

Slide 59 text

context for improved security from myapp.db import Model, Query from myapp.access import get_available_organizations class Project(Model): … @property def query(self):  org_query = get_available_organizations() return Query(self).filter( Project.organization.in_(org_query))

Slide 60

Slide 60 text

automatic escaping • Template engines escape data automatically by HTML rules • However HTML is complex in behavior (script tags, attributes etc.) • It becomes possible to accidentally misuse things • People will get it wrong, so worth investigating the options

Slide 61

Slide 61 text

JSON in HTML • Common case to send JSON to HTML • Two areas of concern: HTML attributes and tags • How to escape in those. Common case? Can we make one function for both?

Slide 62

Slide 62 text

example escaping >>> from flask.json import htmlsafe_dumps >>> print htmlsafe_dumps("var x = 'foo';") "\u003cem\u003evar x = \u0027foo\u0027;\u003c/em\u003e"

Slide 63

Slide 63 text

result of this exercise • does not produce any HTML entities • now works in … • … as well as single quoted attributes • falls over very obviously in double quoted attributes • it's pretty clear how it's supposed to work and hard to misuse