Crafting [Better] API Clients

Transcript

1. Crafting [Better] API Clients PyTennesee 2015- Ben Lopatin

2. Partner and developer @ Wellfire Interactive @bennylope

3. None

4. Clients, not Services

5. Why?

6. A great API is necessary but insufficient to be useful.

7. This is the right way.

8. This is the right way. This is some good ideas.

9. Acknowledge that you’re making design decisions.

10. Some assumptions

11. Data Model

12. Why?

13. Database API Real world

14. Layers of abstraction!

15. Connections & data

16. Designing the Python data interface

17. Explicit data attributes or implicit collection?

18. class DataResource:
 def __init__(self, id, first_name, last_name):
 self.id = id


    self.first_name = first_name
 self.last_name = last_name
 
 
 resource.first_name

19. resource = {
 'id': 1,
 'first_name': 'Ben',
 'last_name': 'Lopatin',
 }


    
 resource['first_name']

20. class DataResource(dict):
 
 def update(self, client):
 # Do something cool

    here
 client.update(self.id, self.first_name, self.last_name)


21. Data or Resources? • If you think about everything as

    a resource, then shouldn’t it have the same methods? • Or is it data and the API is just providing some • You’re not obliged to represent 1:1

22. Be mindful of what you discard

23. Second, we wanted to give you a heads up that

    we're announcing a new feature soon -- additional fields. This will allow people to get congressional districts, state legislative districts, timezones, and school districts with a forward or reverse lookup. We are looking to add more additional fields in the future (namely Census data).

24. Errors

25. None

26. Why?

27. What?

28. API application errors Authentication errors Account errors …

29. If only there were a concept to encapsulate errors in

    Python…

30. …that would be exceptional!

31. Exceptions for obvious flow control

32. try: 
 response = client.request()
 except APIPaymentError:
 # Email corporate

    accounts payable
 raise
 except APIAuthError:
 # Redirect to account settings
 raise

33. Map exceptions to API errors

34. None

35. class SmartyStreetsError(Exception):
 """Unknown SmartyStreets error"""
 
 def __str__(self):
 return self.__doc__


    
 
 class SmartyStreetsInputError(SmartyStreetsError):
 """HTTP 400 Bad input. Required fields missing from input or are malformed."""
 
 
 class SmartyStreetsAuthError(SmartyStreetsError):
 """HTTP 401 Unauthorized. Authentication failure; invalid credentials"""
 
 
 class SmartyStreetsPaymentError(SmartyStreetsError):
 """HTTP 402 Payment required. No active subscription found."""
 
 
 class SmartyStreetsServerError(SmartyStreetsError):
 """HTTP 500 Internal server error. General service failure; retry request."""

36. ERROR_CODES = {
 400: SmartyStreetsInputError,
 401: SmartyStreetsAuthError,
 402: SmartyStreetsPaymentError,
 500:

    SmartyStreetsServerError,
 }

37. …maybe not all errors

38. None

39. Errors aren’t necessarily limited to HTTP status codes

40. Exceptions + Error codes

41. Where?

42. Logging

43. Why?

44. What?

45. Verify requests issued

46. Request/response count

47. API performance

48. API errors

49. How?

50. @decorators?

51. @log_requests
 def _req(self, method='get', verb=None, headers={}, params={}, data={}): 
 url

    = self.BASE_URL.format(verb=verb)
 request_headers = {'content-type': 'application/json'}
 request_params = {'api_key': self.API_KEY}
 request_headers.update(headers)
 request_params.update(params)
 return getattr(requests, method)(url, params=request_params,
 headers=request_headers, data=data)

52. 
 def log_requests(func):
 def decorator(*args, **kwargs):
 request_id = str(uuid.uuid4())
 logger.info("Requesting

    %s" % request_id)
 try:
 resp = func(*args, **kwargs)
 except:
 logger.exception("Request error %s" % request_id)
 raise
 logger.info("Response %s" % request_id)
 return resp
 return decorator

53. Testing

54. Why?

55. None

56. How?

57. Live API Mock servers Mocked responses

58. None

59. That’s not nice (or fun)

60. Mock server?

61. Live testing’s hard, let’s go mocking!

62. Mocking requests

63. responses HTTPretty

64. responses HTTPretty betamax

65. @responses.activate def test_auth_error(self): responses.add(responses.POST, 'https://api.smartystreets.com/street-address', body='', status=401, content_type='application/json') self.assertRaises(SmartyStreetsAuthError, self.client.street_addresses,

    [{}, {}]) @responses.activate def test_payment_error(self): responses.add(responses.POST, 'https://api.smartystreets.com/street-address', body='', status=402, content_type='application/json') self.assertRaises(SmartyStreetsPaymentError, self.client.street_addresses, [{}, {}])

66. @httpretty.activate def test_auth_error(self): """Ensure an HTTP 403 code raises GeocodioAuthError"""

    httpretty.register_uri(httpretty.GET, “http://api.geocod.io/v1/parse", body="This does not matter", status=403) self.assertRaises(GeocodioAuthError, self.client.parse, "")

67. There can be benefits to some live testing

68. None

69. Performance

70. Connections

71. Batching

72. The right API methods

73. Data structures

74. class APIData:
 def init(self, id, name, image):
 self.id = id


    self.name = name
 self.image = image


75. 
 class APIData:
 __slots__ = ['id', 'name', 'image']
 
 def

    init(self, id, name, image):
 self.id = id
 self.name = name
 self.image = image


76. Concurrency

77. response_iter = (
 grequests.post(
 url=url,
 data=json.dumps(data_chunk),
 params=params,
 headeresponse_iter=headeresponse_iter,
 ) for

    data_chunk in chunker(data, 100)
 )
 
 responses = grequests.imap(response_iter, size=parallelism)
 status_codes = {}
 addresses = AddressCollection([])
 for response in responses:
 if response.status_code not in status_codes.keys():
 status_codes[response.status_code] = 1
 else:
 status_codes[response.status_code] += 1
 
 if response.status_code == 200:
 addresses[0:0] = AddressCollection(response.json())

78. session = FuturesSession(max_workers=parallelism)
 session.headers = headers
 session.params = params
 


    futures = [
 session.post(url, data=json.dumps(data_chunk)) for data_chunk in chunker(data, 100)
 ]
 
 while not all([f.done() for f in futures]):
 continue
 
 status_codes = {}
 responses = [f.result() for f in futures]
 
 addresses = AddressCollection([])
 for response in responses:
 if response.status_code not in status_codes.keys():
 status_codes[response.status_code] = 1
 else:
 status_codes[response.status_code] += 1
 
 if response.status_code == 200:
 addresses[0:0] = AddressCollection(response.json())

79. You know Twisted can do that, right?

80. Security

81. HTTPS

82. Python 2.7 • pyOpenSSL • ndg-httpsclient • pyasn1

83. Authenticating

84. Docs

85. Django

86. No?

87. Python library Anything Django related

88. • Create the client *first* then Django integration is a

    bonus. • Mapping models to a distant API. • It’ll be easier for you to maintain and test • Easier for other people to use for other things

89. class SomeModel(models.Model):
 my_field = models.CharField(max_length=100)
 
 def sync(self, client=None):
 if

    client is None:
 client = APIClient()
 client.update(id=self.id, name=self.my_field)

90. Code Gen

91. Starting from (DSL) API documentation, automatically generate matching client code

92. None

93. Crank ‘em out

94. In [hypothetical] practice?

95. Suitability?

96. The End