Crafting [Better] API Clients

This talk is primarily for web developers. It's about understanding how to write an API client that is testable and sensible without being more opinionated than necessary.

A Python-centric talk given at PyTennessee 2015 in Nashville, TN.

Transcript

1. Crafting [Better]
   API Clients
   PyTennesee 2015- Ben Lopatin
   

   View Slide

2. Partner and developer @
   Wellfire Interactive
   @bennylope
   

   View Slide

3. View Slide

4. Clients, not Services
   

   View Slide

5. Why?
   

   View Slide

6. A great API is necessary
   but insufficient to be
   useful.
   

   View Slide

7. This is the right way.
   

   View Slide

8. This is the right way.
   This is some good ideas.
   

   View Slide

9. Acknowledge that you’re
   making design decisions.
   

   View Slide

10. Some assumptions
    

    View Slide

11. Data Model
    

    View Slide

12. Why?
    

    View Slide

13. Database
    API
    Real world
    

    View Slide

14. Layers of abstraction!
    

    View Slide

15. Connections & data
    

    View Slide

16. Designing the Python
    data interface
    

    View Slide

17. Explicit data attributes
    or implicit collection?
    

    View Slide

18. class DataResource:

    def __init__(self, id, first_name,
    last_name):

    self.id = id

    self.first_name = first_name

    self.last_name = last_name

    

    

    resource.first_name
    

    View Slide

19. resource = {

    'id': 1,

    'first_name': 'Ben',

    'last_name': 'Lopatin',

    }

    

    resource['first_name']
    

    View Slide

20. class DataResource(dict):

    

    def update(self, client):

    # Do something cool here

    client.update(self.id,
    self.first_name,
    self.last_name)

    

    View Slide

21. Data or Resources?
    • If you think about everything as a resource, then
    shouldn’t it have the same methods?
    • Or is it data and the API is just providing some
    • You’re not obliged to represent 1:1
    

    View Slide

22. Be mindful of what you
    discard
    

    View Slide

23. Second, we wanted to give you a heads up that
    we're announcing a new feature soon -- additional
    fields. This will allow people to get congressional
    districts, state legislative districts, timezones, and
    school districts with a forward or reverse lookup. We
    are looking to add more additional fields in the
    future (namely Census data).
    

    View Slide

24. Errors
    

    View Slide

25. View Slide

26. Why?
    

    View Slide

27. What?
    

    View Slide

28. API application errors
    Authentication errors
    Account errors
    …
    

    View Slide

29. If only there were a
    concept to encapsulate
    errors in Python…
    

    View Slide

30. …that would be
    exceptional!
    

    View Slide

31. Exceptions for obvious
    flow control
    

    View Slide

32. try: 

    response = client.request()

    except APIPaymentError:

    # Email corporate accounts payable

    raise

    except APIAuthError:

    # Redirect to account settings

    raise
    

    View Slide

33. Map exceptions to API
    errors
    

    View Slide

34. View Slide

35. class SmartyStreetsError(Exception):

    """Unknown SmartyStreets error"""

    

    def __str__(self):

    return self.__doc__

    

    

    class SmartyStreetsInputError(SmartyStreetsError):

    """HTTP 400 Bad input. Required fields missing from input or are malformed."""

    

    

    class SmartyStreetsAuthError(SmartyStreetsError):

    """HTTP 401 Unauthorized. Authentication failure; invalid credentials"""

    

    

    class SmartyStreetsPaymentError(SmartyStreetsError):

    """HTTP 402 Payment required. No active subscription found."""

    

    

    class SmartyStreetsServerError(SmartyStreetsError):

    """HTTP 500 Internal server error. General service failure; retry request."""
    

    View Slide

36. ERROR_CODES = {

    400: SmartyStreetsInputError,

    401: SmartyStreetsAuthError,

    402: SmartyStreetsPaymentError,

    500: SmartyStreetsServerError,

    }
    

    View Slide

37. …maybe not all errors
    

    View Slide

38. View Slide

39. Errors aren’t necessarily
    limited to HTTP status
    codes
    

    View Slide

40. Exceptions
    +
    Error codes
    

    View Slide

41. Where?
    

    View Slide

42. Logging
    

    View Slide

43. Why?
    

    View Slide

44. What?
    

    View Slide

45. Verify requests issued
    

    View Slide

46. Request/response
    count
    

    View Slide

47. API performance
    

    View Slide

48. API errors
    

    View Slide

49. How?
    

    View Slide

50. @decorators?
    

    View Slide

51. @log_requests

    def _req(self, method='get', verb=None, headers={},
    params={}, data={}):
    

    url = self.BASE_URL.format(verb=verb)

    request_headers = {'content-type': 'application/json'}

    request_params = {'api_key': self.API_KEY}

    request_headers.update(headers)

    request_params.update(params)

    return getattr(requests, method)(url, params=request_params,

    headers=request_headers, data=data)
    

    View Slide

52. 

    def log_requests(func):

    def decorator(*args, **kwargs):

    request_id = str(uuid.uuid4())

    logger.info("Requesting %s" % request_id)

    try:

    resp = func(*args, **kwargs)

    except:

    logger.exception("Request error %s" %
    request_id)

    raise

    logger.info("Response %s" % request_id)

    return resp

    return decorator
    

    View Slide

53. Testing
    

    View Slide

54. Why?
    

    View Slide

55. View Slide

56. How?
    

    View Slide

57. Live API
    Mock servers
    Mocked responses
    

    View Slide

58. View Slide

59. That’s not nice (or fun)
    

    View Slide

60. Mock server?
    

    View Slide

61. Live testing’s hard,
    let’s go mocking!
    

    View Slide

62. Mocking requests
    

    View Slide

63. responses
    HTTPretty
    

    View Slide

64. responses
    HTTPretty
    betamax
    

    View Slide

65. @responses.activate
    def test_auth_error(self):
    responses.add(responses.POST,
    'https://api.smartystreets.com/street-address',
    body='', status=401, content_type='application/json')
    self.assertRaises(SmartyStreetsAuthError,
    self.client.street_addresses, [{}, {}])
    @responses.activate
    def test_payment_error(self):
    responses.add(responses.POST,
    'https://api.smartystreets.com/street-address',
    body='', status=402, content_type='application/json')
    self.assertRaises(SmartyStreetsPaymentError,
    self.client.street_addresses, [{}, {}])
    

    View Slide

66. @httpretty.activate
    def test_auth_error(self):
    """Ensure an HTTP 403 code raises GeocodioAuthError"""
    httpretty.register_uri(httpretty.GET,
    “http://api.geocod.io/v1/parse",
    body="This does not matter", status=403)
    self.assertRaises(GeocodioAuthError,
    self.client.parse, "")
    

    View Slide

67. There can be benefits
    to some live testing
    

    View Slide

68. View Slide

69. Performance
    

    View Slide

70. Connections
    

    View Slide

71. Batching
    

    View Slide

72. The right API methods
    

    View Slide

73. Data structures
    

    View Slide

74. class APIData:

    def init(self, id, name, image):

    self.id = id

    self.name = name

    self.image = image

    

    View Slide

75. 

    class APIData:

    __slots__ = ['id', 'name', 'image']

    

    def init(self, id, name, image):

    self.id = id

    self.name = name

    self.image = image

    

    View Slide

76. Concurrency
    

    View Slide

77. response_iter = (

    grequests.post(

    url=url,

    data=json.dumps(data_chunk),

    params=params,

    headeresponse_iter=headeresponse_iter,

    ) for data_chunk in chunker(data, 100)

    )

    

    responses = grequests.imap(response_iter, size=parallelism)

    status_codes = {}

    addresses = AddressCollection([])

    for response in responses:

    if response.status_code not in status_codes.keys():

    status_codes[response.status_code] = 1

    else:

    status_codes[response.status_code] += 1

    

    if response.status_code == 200:

    addresses[0:0] = AddressCollection(response.json())
    

    View Slide

78. session = FuturesSession(max_workers=parallelism)

    session.headers = headers

    session.params = params

    

    futures = [

    session.post(url, data=json.dumps(data_chunk))
    for data_chunk in chunker(data, 100)

    ]

    

    while not all([f.done() for f in futures]):

    continue

    

    status_codes = {}

    responses = [f.result() for f in futures]

    

    addresses = AddressCollection([])

    for response in responses:

    if response.status_code not in status_codes.keys():

    status_codes[response.status_code] = 1

    else:

    status_codes[response.status_code] += 1

    

    if response.status_code == 200:

    addresses[0:0] = AddressCollection(response.json())
    

    View Slide

79. You know Twisted can
    do that, right?
    

    View Slide

80. Security
    

    View Slide

81. HTTPS
    

    View Slide

82. Python 2.7
    • pyOpenSSL
    • ndg-httpsclient
    • pyasn1
    

    View Slide

83. Authenticating
    

    View Slide

84. Docs
    

    View Slide

85. Django
    

    View Slide

86. No?
    

    View Slide

87. Python library
    Anything Django related
    

    View Slide

88. • Create the client *first* then Django integration is
    a bonus.
    • Mapping models to a distant API.
    • It’ll be easier for you to maintain and test
    • Easier for other people to use for other things
    

    View Slide

89. class SomeModel(models.Model):

    my_field = models.CharField(max_length=100)

    

    def sync(self, client=None):

    if client is None:

    client = APIClient()

    client.update(id=self.id, name=self.my_field)
    

    View Slide

90. Code Gen
    

    View Slide

91. Starting from (DSL) API
    documentation, automatically
    generate matching client code
    

    View Slide

92. View Slide

93. Crank ‘em out
    

    View Slide

94. In [hypothetical]
    practice?
    

    View Slide

95. Suitability?
    

    View Slide

96. The End
    

    View Slide