Crafting [Better] API Clients

A3b1bb5e498495de407c0a2547982139?s=47 Ben Lopatin
February 08, 2015

Crafting [Better] API Clients

This talk is primarily for web developers. It's about understanding how to write an API client that is testable and sensible without being more opinionated than necessary.

A Python-centric talk given at PyTennessee 2015 in Nashville, TN.

A3b1bb5e498495de407c0a2547982139?s=128

Ben Lopatin

February 08, 2015
Tweet

Transcript

  1. Crafting [Better] API Clients PyTennesee 2015- Ben Lopatin

  2. Partner and developer @ Wellfire Interactive @bennylope

  3. None
  4. Clients, not Services

  5. Why?

  6. A great API is necessary but insufficient to be useful.

  7. This is the right way.

  8. This is the right way. This is some good ideas.

  9. Acknowledge that you’re making design decisions.

  10. Some assumptions

  11. Data Model

  12. Why?

  13. Database API Real world

  14. Layers of abstraction!

  15. Connections & data

  16. Designing the Python data interface

  17. Explicit data attributes or implicit collection?

  18. class DataResource:
 def __init__(self, id, first_name, last_name):
 self.id = id


    self.first_name = first_name
 self.last_name = last_name
 
 
 resource.first_name
  19. resource = {
 'id': 1,
 'first_name': 'Ben',
 'last_name': 'Lopatin',
 }


    
 resource['first_name']
  20. class DataResource(dict):
 
 def update(self, client):
 # Do something cool

    here
 client.update(self.id, self.first_name, self.last_name)

  21. Data or Resources? • If you think about everything as

    a resource, then shouldn’t it have the same methods? • Or is it data and the API is just providing some • You’re not obliged to represent 1:1
  22. Be mindful of what you discard

  23. Second, we wanted to give you a heads up that

    we're announcing a new feature soon -- additional fields. This will allow people to get congressional districts, state legislative districts, timezones, and school districts with a forward or reverse lookup. We are looking to add more additional fields in the future (namely Census data).
  24. Errors

  25. None
  26. Why?

  27. What?

  28. API application errors Authentication errors Account errors …

  29. If only there were a concept to encapsulate errors in

    Python…
  30. …that would be exceptional!

  31. Exceptions for obvious flow control

  32. try: 
 response = client.request()
 except APIPaymentError:
 # Email corporate

    accounts payable
 raise
 except APIAuthError:
 # Redirect to account settings
 raise
  33. Map exceptions to API errors

  34. None
  35. class SmartyStreetsError(Exception):
 """Unknown SmartyStreets error"""
 
 def __str__(self):
 return self.__doc__


    
 
 class SmartyStreetsInputError(SmartyStreetsError):
 """HTTP 400 Bad input. Required fields missing from input or are malformed."""
 
 
 class SmartyStreetsAuthError(SmartyStreetsError):
 """HTTP 401 Unauthorized. Authentication failure; invalid credentials"""
 
 
 class SmartyStreetsPaymentError(SmartyStreetsError):
 """HTTP 402 Payment required. No active subscription found."""
 
 
 class SmartyStreetsServerError(SmartyStreetsError):
 """HTTP 500 Internal server error. General service failure; retry request."""
  36. ERROR_CODES = {
 400: SmartyStreetsInputError,
 401: SmartyStreetsAuthError,
 402: SmartyStreetsPaymentError,
 500:

    SmartyStreetsServerError,
 }
  37. …maybe not all errors

  38. None
  39. Errors aren’t necessarily limited to HTTP status codes

  40. Exceptions + Error codes

  41. Where?

  42. Logging

  43. Why?

  44. What?

  45. Verify requests issued

  46. Request/response count

  47. API performance

  48. API errors

  49. How?

  50. @decorators?

  51. @log_requests
 def _req(self, method='get', verb=None, headers={}, params={}, data={}): 
 url

    = self.BASE_URL.format(verb=verb)
 request_headers = {'content-type': 'application/json'}
 request_params = {'api_key': self.API_KEY}
 request_headers.update(headers)
 request_params.update(params)
 return getattr(requests, method)(url, params=request_params,
 headers=request_headers, data=data)
  52. 
 def log_requests(func):
 def decorator(*args, **kwargs):
 request_id = str(uuid.uuid4())
 logger.info("Requesting

    %s" % request_id)
 try:
 resp = func(*args, **kwargs)
 except:
 logger.exception("Request error %s" % request_id)
 raise
 logger.info("Response %s" % request_id)
 return resp
 return decorator
  53. Testing

  54. Why?

  55. None
  56. How?

  57. Live API Mock servers Mocked responses

  58. None
  59. That’s not nice (or fun)

  60. Mock server?

  61. Live testing’s hard, let’s go mocking!

  62. Mocking requests

  63. responses HTTPretty

  64. responses HTTPretty betamax

  65. @responses.activate def test_auth_error(self): responses.add(responses.POST, 'https://api.smartystreets.com/street-address', body='', status=401, content_type='application/json') self.assertRaises(SmartyStreetsAuthError, self.client.street_addresses,

    [{}, {}]) @responses.activate def test_payment_error(self): responses.add(responses.POST, 'https://api.smartystreets.com/street-address', body='', status=402, content_type='application/json') self.assertRaises(SmartyStreetsPaymentError, self.client.street_addresses, [{}, {}])
  66. @httpretty.activate def test_auth_error(self): """Ensure an HTTP 403 code raises GeocodioAuthError"""

    httpretty.register_uri(httpretty.GET, “http://api.geocod.io/v1/parse", body="This does not matter", status=403) self.assertRaises(GeocodioAuthError, self.client.parse, "")
  67. There can be benefits to some live testing

  68. None
  69. Performance

  70. Connections

  71. Batching

  72. The right API methods

  73. Data structures

  74. class APIData:
 def init(self, id, name, image):
 self.id = id


    self.name = name
 self.image = image

  75. 
 class APIData:
 __slots__ = ['id', 'name', 'image']
 
 def

    init(self, id, name, image):
 self.id = id
 self.name = name
 self.image = image

  76. Concurrency

  77. response_iter = (
 grequests.post(
 url=url,
 data=json.dumps(data_chunk),
 params=params,
 headeresponse_iter=headeresponse_iter,
 ) for

    data_chunk in chunker(data, 100)
 )
 
 responses = grequests.imap(response_iter, size=parallelism)
 status_codes = {}
 addresses = AddressCollection([])
 for response in responses:
 if response.status_code not in status_codes.keys():
 status_codes[response.status_code] = 1
 else:
 status_codes[response.status_code] += 1
 
 if response.status_code == 200:
 addresses[0:0] = AddressCollection(response.json())
  78. session = FuturesSession(max_workers=parallelism)
 session.headers = headers
 session.params = params
 


    futures = [
 session.post(url, data=json.dumps(data_chunk)) for data_chunk in chunker(data, 100)
 ]
 
 while not all([f.done() for f in futures]):
 continue
 
 status_codes = {}
 responses = [f.result() for f in futures]
 
 addresses = AddressCollection([])
 for response in responses:
 if response.status_code not in status_codes.keys():
 status_codes[response.status_code] = 1
 else:
 status_codes[response.status_code] += 1
 
 if response.status_code == 200:
 addresses[0:0] = AddressCollection(response.json())
  79. You know Twisted can do that, right?

  80. Security

  81. HTTPS

  82. Python 2.7 • pyOpenSSL • ndg-httpsclient • pyasn1

  83. Authenticating

  84. Docs

  85. Django

  86. No?

  87. Python library Anything Django related

  88. • Create the client *first* then Django integration is a

    bonus. • Mapping models to a distant API. • It’ll be easier for you to maintain and test • Easier for other people to use for other things
  89. class SomeModel(models.Model):
 my_field = models.CharField(max_length=100)
 
 def sync(self, client=None):
 if

    client is None:
 client = APIClient()
 client.update(id=self.id, name=self.my_field)
  90. Code Gen

  91. Starting from (DSL) API documentation, automatically generate matching client code

  92. None
  93. Crank ‘em out

  94. In [hypothetical] practice?

  95. Suitability?

  96. The End