Crafting [Better] API Clients

This talk is primarily for web developers. It's about understanding how to write an API client that is testable and sensible without being more opinionated than necessary.

A Python-centric talk given at PyTennessee 2015 in Nashville, TN.

Transcript

1. Crafting [Better] API Clients PyTennesee 2015- Ben Lopatin
2. Partner and developer @ Wellfire Interactive @bennylope
4. Clients, not Services
5. Why?
6. A great API is necessary but insufficient to be useful.
7. This is the right way.
8. This is the right way. This is some good ideas.
9. Acknowledge that you’re making design decisions.
10. Some assumptions
11. Data Model
12. Why?
13. Database API Real world
14. Layers of abstraction!
15. Connections & data
16. Designing the Python data interface
17. Explicit data attributes or implicit collection?
18. class DataResource:
 def __init__(self, id, first_name, last_name):
 self.id = id
 self.first_name = first_name
 self.last_name = last_name
 
 
 resource.first_name
19. resource = {
 'id': 1,
 'first_name': 'Ben',
 'last_name': 'Lopatin',
 }
 
 resource['first_name']
20. class DataResource(dict):
 
 def update(self, client):
 # Do something cool here
 client.update(self.id, self.first_name, self.last_name)

21. Data or Resources? • If you think about everything as a resource, then shouldn’t it have the same methods? • Or is it data and the API is just providing some • You’re not obliged to represent 1:1
22. Be mindful of what you discard
23. Second, we wanted to give you a heads up that we're announcing a new feature soon -- additional fields. This will allow people to get congressional districts, state legislative districts, timezones, and school districts with a forward or reverse lookup. We are looking to add more additional fields in the future (namely Census data).
24. Errors
26. Why?
27. What?
28. API application errors Authentication errors Account errors …
29. If only there were a concept to encapsulate errors in Python…
30. …that would be exceptional!
31. Exceptions for obvious flow control
32. try: 
 response = client.request()
 except APIPaymentError:
 # Email corporate accounts payable
 raise
 except APIAuthError:
 # Redirect to account settings
 raise
33. Map exceptions to API errors
35. class SmartyStreetsError(Exception):
 """Unknown SmartyStreets error"""
 
 def __str__(self):
 return self.__doc__
 
 
 class SmartyStreetsInputError(SmartyStreetsError):
 """HTTP 400 Bad input. Required fields missing from input or are malformed."""
 
 
 class SmartyStreetsAuthError(SmartyStreetsError):
 """HTTP 401 Unauthorized. Authentication failure; invalid credentials"""
 
 
 class SmartyStreetsPaymentError(SmartyStreetsError):
 """HTTP 402 Payment required. No active subscription found."""
 
 
 class SmartyStreetsServerError(SmartyStreetsError):
 """HTTP 500 Internal server error. General service failure; retry request."""
36. ERROR_CODES = {
 400: SmartyStreetsInputError,
 401: SmartyStreetsAuthError,
 402: SmartyStreetsPaymentError,
 500: SmartyStreetsServerError,
 }
37. …maybe not all errors
39. Errors aren’t necessarily limited to HTTP status codes
40. Exceptions + Error codes
41. Where?
42. Logging
43. Why?
44. What?
45. Verify requests issued
46. Request/response count
47. API performance
48. API errors
49. How?
50. @decorators?
51. @log_requests
 def _req(self, method='get', verb=None, headers={}, params={}, data={}): 
 url = self.BASE_URL.format(verb=verb)
 request_headers = {'content-type': 'application/json'}
 request_params = {'api_key': self.API_KEY}
 request_headers.update(headers)
 request_params.update(params)
 return getattr(requests, method)(url, params=request_params,
 headers=request_headers, data=data)
52. 
 def log_requests(func):
 def decorator(*args, **kwargs):
 request_id = str(uuid.uuid4())
 logger.info("Requesting %s" % request_id)
 try:
 resp = func(*args, **kwargs)
 except:
 logger.exception("Request error %s" % request_id)
 raise
 logger.info("Response %s" % request_id)
 return resp
 return decorator
53. Testing
54. Why?
56. How?
57. Live API Mock servers Mocked responses
59. That’s not nice (or fun)
60. Mock server?
61. Live testing’s hard, let’s go mocking!
62. Mocking requests
63. responses HTTPretty
64. responses HTTPretty betamax
65. @responses.activate def test_auth_error(self): responses.add(responses.POST, 'https://api.smartystreets.com/street-address', body='', status=401, content_type='application/json') self.assertRaises(SmartyStreetsAuthError, self.client.street_addresses, [{}, {}]) @responses.activate def test_payment_error(self): responses.add(responses.POST, 'https://api.smartystreets.com/street-address', body='', status=402, content_type='application/json') self.assertRaises(SmartyStreetsPaymentError, self.client.street_addresses, [{}, {}])
66. @httpretty.activate def test_auth_error(self): """Ensure an HTTP 403 code raises GeocodioAuthError""" httpretty.register_uri(httpretty.GET, “http://api.geocod.io/v1/parse", body="This does not matter", status=403) self.assertRaises(GeocodioAuthError, self.client.parse, "")
67. There can be benefits to some live testing
69. Performance
70. Connections
71. Batching
72. The right API methods
73. Data structures
74. class APIData:
 def init(self, id, name, image):
 self.id = id
 self.name = name
 self.image = image

75. 
 class APIData:
 __slots__ = ['id', 'name', 'image']
 
 def init(self, id, name, image):
 self.id = id
 self.name = name
 self.image = image

76. Concurrency
77. response_iter = (
 grequests.post(
 url=url,
 data=json.dumps(data_chunk),
 params=params,
 headeresponse_iter=headeresponse_iter,
 ) for data_chunk in chunker(data, 100)
 )
 
 responses = grequests.imap(response_iter, size=parallelism)
 status_codes = {}
 addresses = AddressCollection([])
 for response in responses:
 if response.status_code not in status_codes.keys():
 status_codes[response.status_code] = 1
 else:
 status_codes[response.status_code] += 1
 
 if response.status_code == 200:
 addresses[0:0] = AddressCollection(response.json())
78. session = FuturesSession(max_workers=parallelism)
 session.headers = headers
 session.params = params
 
 futures = [
 session.post(url, data=json.dumps(data_chunk)) for data_chunk in chunker(data, 100)
 ]
 
 while not all([f.done() for f in futures]):
 continue
 
 status_codes = {}
 responses = [f.result() for f in futures]
 
 addresses = AddressCollection([])
 for response in responses:
 if response.status_code not in status_codes.keys():
 status_codes[response.status_code] = 1
 else:
 status_codes[response.status_code] += 1
 
 if response.status_code == 200:
 addresses[0:0] = AddressCollection(response.json())
79. You know Twisted can do that, right?
80. Security
81. HTTPS
82. Python 2.7 • pyOpenSSL • ndg-httpsclient • pyasn1
83. Authenticating
84. Docs
85. Django
86. No?
87. Python library Anything Django related
88. • Create the client *first* then Django integration is a bonus. • Mapping models to a distant API. • It’ll be easier for you to maintain and test • Easier for other people to use for other things
89. class SomeModel(models.Model):
 my_field = models.CharField(max_length=100)
 
 def sync(self, client=None):
 if client is None:
 client = APIClient()
 client.update(id=self.id, name=self.my_field)
90. Code Gen
91. Starting from (DSL) API documentation, automatically generate matching client code
93. Crank ‘em out
94. In [hypothetical] practice?
95. Suitability?
96. The End