API Design Tips

Given at DjangoCon 2012. A collection of ideas (built through the pain of experience) on how to improve the code you write.

1. API Design Tips
2. Who?
3. Who? •Daniel Lindsley
4. Who? •Daniel Lindsley •Toast Driven ( ) is my business
5. Who? •Daniel Lindsley •Toast Driven ( ) is my business •Wrote a couple Django apps •Haystack •Tastypie
6. What?
7. What? •Not HTTP APIs
8. What? •Not HTTP APIs •(though I love those. seriously. come talk to me about them...)
9. What? •Programmatic APIs
10. What? •Programmatic APIs •Think libraries •Especially ones you hand off to other people...
11. Why?
12. Why? •Think about how many times you’ve started with someone else’s library...
13. Why? •Think about how many times you’ve started with someone else’s library... •Used it some...
14. Why? •Think about how many times you’ve started with someone else’s library... •Used it some... •Then wanted to strangle them for the hoops they made you jump through
15. Why? •Other people use your code all the time.
16. Why? •Other people use your code all the time. •They might be wanting to strangle you!
17. Why? •Other people use your code all the time. •They might be wanting to strangle you! •It might even be future you wanting to strangle past you.
18. Why? •Nice APIs beget Happiness
19. Why? •Nice APIs beget Happiness •Happiness begets Recommendations
20. Why? •Nice APIs beget Happiness •Happiness begets Recommendations •Recommendations beget Users/ Community
21. Philosophy
22. You can’t make everyone happy by default. You should still have sane defaults.
23. But more people will be happy if they can tweak it. You can bet they’ll need to.
24. And most people will be happy if it’s easy to tweak. I don’t have a clever followup here.
25. No copy-paste should be needed. Boilerplate sucks.
26. They shouldn’t have to constantly refer to the docs. Especially not for things they use all the freaking time.
27. Good docs matter. Saves you (support) & them (implementation) time. Everyone wins.
28. Real World™ use is the best sanity check. Cleverness goes here.
29. Someone is going to do something weird/insane with your code. It’s inevitable, so design for it up front.
30. Approaches on Design
31. Approaches On Design •Common Methodologies:
32. Approaches On Design •Common Methodologies: •Bottom-up
33. Approaches On Design •Common Methodologies: •Bottom-up •Top-down
34. Approaches On Design •Common Methodologies: •Bottom-up •Top-down •WARNING: OPINIONS FOLLOW
35. Approaches On Design •Bottom-up sucks.
36. Approaches On Design •Bottom-up sucks. •Sure, you built little pieces that work.
37. Approaches On Design •Bottom-up sucks. •Sure, you built little pieces that work. •But do they work well together?
38. Approaches On Design •Bottom-up sucks. •Sure, you built little pieces that work. •But do they work well together? •Likely not. •HOW IS BABBY PHP FORMED
39. Approaches On Design •Top-down feels better.
40. Approaches On Design •Top-down feels better. •Everything fits together right.
41. Approaches On Design •Top-down feels better. •Everything fits together right. •Less duplication.
42. Approaches On Design •Top-down feels better. •Everything fits together right. •Less duplication. •Resist the urge to duct tape.
43. Bonus to Top-Down?
44. Instant TDD
45. Approaches On Design •With (some) instant TDD, you get your tests started from the get-go.
46. Approaches On Design •With (some) instant TDD, you get your tests started from the get-go. •Fewer massive, painful refactorings down the road.
47. Things You Should Do
48. Small Components Worked for UNIX, it’ll work for you.
49. Small Components class DoEverything(object): # ... def update_cache(self, timeout=600): # Stuff... def post_to_twitter(self, user): # More stuff... def make_me_a_sandwich(self, sudo=False): # Even more stuff...
50. Small Components class Cachable(object): timeout = 600 def update(self): # Cache myself... class SocialPosts(object): def twitter(self, user): # Post a tweet class SandwichMaker(object): def __init__(self, sudo=False): self.sudo = sudo def make(self): # Maybe yes, maybe no.
51. Composition >= Inheritance Why do the work yourself when you can delegate?
52. Composition >= Inheritance class CachedView(View): def get(self, request, **kwargs): # Cache the output & serve. class GhettoAPIView(View): def get(self, request, **kwargs): # Return either JSON or HTML. # Now try to use both. :/ class MyView(CachedView, GhettoAPIView): # Oh crap, they stomp on each other. pass
53. Composition >= Inheritance class MyView(BetterView): def get(self, **kwargs): response = self.preprocess() # Do stuff. return self.render('template.html', {}) def preprocess(self): # Delegate! cached = ViewCache() return cached.get(self, self.request) def render(self, template, context): output = GhettoAPI(self.request) return output.render(template, context)
54. Reflection If data can flow one way, add the opposite direction as well.
55. Reflection class FakeDate(BadDate): def to_iso8801(self): return '-'.join( self.year, self.month, self.day )
56. Reflection class FakeDate(BadDate): def to_iso8801(self): return '-'.join( self.year, self.month, self.day ) def from_iso8801(self, date_string): date_bits = date_string.split('-') self.year = date_bits[0] self.month = date_bits[1] self.day = date_bits[2]
57. Broad Familiarity If it’s a similar task to something else they’ll know, mimic that something else.
58. Broad Familiarity def go_search(engine_url, query, limit=10 offset=0, just_values=False): # Hit the search engine... return results def crazy_raw_query(engine_url, raw_query): # Hope & pray the query is OK. return raw_results # Usage: kwargs = { 'limit': 20, 'offset': 20, } results = go_search('http://...', 'banana')
59. Broad Familiarity class SearchQuerySet(object): # Make it look like ``QuerySet``. def __getitem__(self, key): # Handle the slice here. def filter(self, **kwargs): # Apply filters. def raw(self, query): # Do a raw query. # Usage: sqs = SearchQuerySet().filter(text='banana') results = sqs[20:20]
60. Narrow Familiarity Call signatures matter.
61. Narrow Familiarity def go_search(engine_url, query, limit=10 offset=0, just_values=False): # Do stuff. return results def execute_raw_query(raw_query, offset, engine_url): # Do a raw query here. return { 'search_results': results, }
62. Narrow Familiarity def query(engine_url, query, limit=10 offset=0, just_values=False): # Do stuff. return { 'results': results, 'total': total_count, } def raw_query(engine_url, query, limit=10 offset=0, just_values=False): # Do a raw query here. return { 'results': results, 'total': total_count, }
63. Protocol Gently “suggest” things behave the same way. With a stick.
64. Protocol class CoolKidBackend(object): # Whatever, as if we care. pass class RedisBackend(CoolKidBackend): def get(self, key): return self.conn.hget(key) class RiakBackend(CoolKidBackend): def get_hash(self, bucket, key): return self._conn.get(bucket, key) # You can't transparently switch backends. # Welp, time to go rewrite everything.
65. Protocol class CoolKidBackend(object): def __init__(self, conn_string): # Parse the bucket out of # the conn_string, if present. def get(self, key): raise NotImplementedError('...') class RedisBackend(CoolKidBackend): def get(self, key): return self.conn.hget(key) class RiakBackend(CoolKidBackend): def get(self, key): return self._conn.get(...)
66. Assume The Worst™! Don’t code for just the easy case.
67. Assume The Worst™! def parse_name(name): # It'll be fine. Names are always # "<first name> <single last name>" # and ASCII, right? return name.split(' ')
68. Assume The Worst™! def parse_name(name, separator=' ', middlename_present=False, allowed_suffixes=SUFFIXES): if ',' in name: maybe_last_name = name.split(',', 1) if maybe_last_name in allowed_suffixes: # ... return { 'original': name, 'confidence': confidence_level * 100, 'first_name': first, # ... }
69. Things You Should NOT Do
70. Stop At A Low Level “This Low Level API ought to be good enough.” Yeah, right. And 640Kb was totally enough for everyone.
71. Stop At A Low Level def get_page(socket, timeout): # Use the open socket to fetch # the web page. It's a open # socket to the correct place, # right?
72. Stop At A Low Level def get_raw_page(socket, timeout=60): # Do the actual fetch. return headers, content def get_page(url, timeout=None, fetcher=get_raw_page): # Validate the URL. # Open a socket. head, content = fetcher(my_socket) if head.status == 500: raise OhNoes('...') # ...
73. Wildly Different Return Values “Hm, does this return an integer or a dictionary?”
74. Wildly Different Return Values def query(engine_url, query, limit=10): # Make the user deal with the raw data. results = [res for res in results if res] return results, total def raw_query(engine_url, query, limit=10): # Make the user deal with the raw data. return { 'search_results': results, 'count': total, }
75. Wildly Different Return Values def query(engine_url, query, limit=10): # We're doing a search query. # Return sane, similar structures. return { 'results': results, 'total': total_count, } def raw_query(engine_url, query, limit=10): # We're doing a search query. # Return sane, similar structures. return { 'results': results, 'total': total_count, }
76. Useless “Implementation” Code If it can’t do anything for itself, it’s code without purpose.
77. Useless “Implementation” Code class BaseBackend(object): # Full implementation. def __init__(self, url): raise NotImplementedError('...') def save(self, filename, data): raise NotImplementedError('...') def load(self, filename): raise NotImplementedError('...')
78. Useless “Implementation” Code class FileBackend(object): # Does useful things. Subclasses can # override less. def __init__(self, url): self.url = url def save(self, filename, data): the_file = open(filename, 'w') return json.dump(the_file, data) def load(self, filename): data = open(filename) return json.load(data)
79. If It’s Difficult To Test... ...you screwed up. But maybe you can fix it.
80. If It’s Difficult To Test... def current_temperature(station): # Oh yeah, this totally works BTW. url = 'http://w1.weather.gov/obhistory/' + \ '%s.html' % station resp = requests.get(url) data = pq(resp.content) return float(data('table').eq(3)\ .find('tr').eq(4)\ .find('td').eq(6).text())
81. If It’s Difficult To Test... class Weather(object): def build_url(self, station): return 'http://w1.weather.gov/obhistory/' + \ '%s.html' % station def fetch_page(self, url): return requests.get(url).content def parse_page(self, content): return pq(resp.content) def last_conditions(self, data): return data('table').eq(3).find('tr').eq(4) def current_temperature(self, station): url = self.build_url(station) content = self.fetch_page(url) data = self.parse_page(content) return float(self.last_conditions(data)\ .find('td').eq(6).text())
82. django-Specific Topics
83. Pluggable Backend ALL THE THINGS
84. Internationalize ALL THE THINGS
85. Dynamically Loadable Classes/Code 60% more error-handling, every time.
86. Declarative Syntax Metaclasses: Call your doctor if headaches last more than 4 hours.
87. Don’t metaclass ALL THE THINGS
88. The ORM Love it or hate it, there’s some great learning opportunities there to be had.
89. Other Ideas
90. Very Little Global State # Only defaults or things that absolute # have to be module-level. BASE_URL = 'http://w1.weather.gov/obhistory/' TIMEOUT = 10 # Make them defaults but allow passing # different in their place. def fetch_page(self, url=BASE_URL, timeout=TIMEOUT): return requests.get( url, timeout=timeout ).content
91. Decrease Reliance On ``self`` class Weather(object): def build_url(self, station): # Make a URL. def fetch_page(self, url): # Get a page def parse_page(self, content): # Use the page content. def current_temperature(self, station): url = self.build_url(station) content = self.fetch_page(url) data = self.parse_page(content) # You could be using ``self.<variable>`` on # these, but by allowing data to be passed # in, you make it easier to test.
92. Resist The Urge To Use Magic Be explicit first, then add shortcuts. (which can be a little more magical)
93. So...
94. So... •Use the Golden Rule.
95. So... •Use the Golden Rule. •Consistency is key.
96. So... •Use the Golden Rule. •Consistency is key. •Plan for the worst & include sweet shortcuts for the best.
97. So... •Use the Golden Rule. •Consistency is key. •Plan for the worst & include sweet shortcuts for the best. •Write a thing you’d to use...
98. So... •Use the Golden Rule. •Consistency is key. •Plan for the worst & include sweet shortcuts for the best. •Write a thing you’d to use... •...then make it even better.
99. Thanks!
100. I’m Daniel Lindsley of Toast Driven @toastdriven http://toastdriven.com/