Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Crafting [Better] API Clients

Ben Lopatin
February 08, 2015

Crafting [Better] API Clients

This talk is primarily for web developers. It's about understanding how to write an API client that is testable and sensible without being more opinionated than necessary.

A Python-centric talk given at PyTennessee 2015 in Nashville, TN.

Ben Lopatin

February 08, 2015
Tweet

More Decks by Ben Lopatin

Other Decks in Programming

Transcript

  1. Crafting [Better]
    API Clients
    PyTennesee 2015- Ben Lopatin

    View Slide

  2. Partner and developer @
    Wellfire Interactive
    @bennylope

    View Slide

  3. View Slide

  4. Clients, not Services

    View Slide

  5. Why?

    View Slide

  6. A great API is necessary
    but insufficient to be
    useful.

    View Slide

  7. This is the right way.

    View Slide

  8. This is the right way.
    This is some good ideas.

    View Slide

  9. Acknowledge that you’re
    making design decisions.

    View Slide

  10. Some assumptions

    View Slide

  11. Data Model

    View Slide

  12. Why?

    View Slide

  13. Database
    API
    Real world

    View Slide

  14. Layers of abstraction!

    View Slide

  15. Connections & data

    View Slide

  16. Designing the Python
    data interface

    View Slide

  17. Explicit data attributes
    or implicit collection?

    View Slide

  18. class DataResource:

    def __init__(self, id, first_name,
    last_name):

    self.id = id

    self.first_name = first_name

    self.last_name = last_name



    resource.first_name

    View Slide

  19. resource = {

    'id': 1,

    'first_name': 'Ben',

    'last_name': 'Lopatin',

    }


    resource['first_name']

    View Slide

  20. class DataResource(dict):


    def update(self, client):

    # Do something cool here

    client.update(self.id,
    self.first_name,
    self.last_name)


    View Slide

  21. Data or Resources?
    • If you think about everything as a resource, then
    shouldn’t it have the same methods?
    • Or is it data and the API is just providing some
    • You’re not obliged to represent 1:1

    View Slide

  22. Be mindful of what you
    discard

    View Slide

  23. Second, we wanted to give you a heads up that
    we're announcing a new feature soon -- additional
    fields. This will allow people to get congressional
    districts, state legislative districts, timezones, and
    school districts with a forward or reverse lookup. We
    are looking to add more additional fields in the
    future (namely Census data).

    View Slide

  24. Errors

    View Slide

  25. View Slide

  26. Why?

    View Slide

  27. What?

    View Slide

  28. API application errors
    Authentication errors
    Account errors

    View Slide

  29. If only there were a
    concept to encapsulate
    errors in Python…

    View Slide

  30. …that would be
    exceptional!

    View Slide

  31. Exceptions for obvious
    flow control

    View Slide

  32. try: 

    response = client.request()

    except APIPaymentError:

    # Email corporate accounts payable

    raise

    except APIAuthError:

    # Redirect to account settings

    raise

    View Slide

  33. Map exceptions to API
    errors

    View Slide

  34. View Slide

  35. class SmartyStreetsError(Exception):

    """Unknown SmartyStreets error"""


    def __str__(self):

    return self.__doc__



    class SmartyStreetsInputError(SmartyStreetsError):

    """HTTP 400 Bad input. Required fields missing from input or are malformed."""



    class SmartyStreetsAuthError(SmartyStreetsError):

    """HTTP 401 Unauthorized. Authentication failure; invalid credentials"""



    class SmartyStreetsPaymentError(SmartyStreetsError):

    """HTTP 402 Payment required. No active subscription found."""



    class SmartyStreetsServerError(SmartyStreetsError):

    """HTTP 500 Internal server error. General service failure; retry request."""

    View Slide

  36. ERROR_CODES = {

    400: SmartyStreetsInputError,

    401: SmartyStreetsAuthError,

    402: SmartyStreetsPaymentError,

    500: SmartyStreetsServerError,

    }

    View Slide

  37. …maybe not all errors

    View Slide

  38. View Slide

  39. Errors aren’t necessarily
    limited to HTTP status
    codes

    View Slide

  40. Exceptions
    +
    Error codes

    View Slide

  41. Where?

    View Slide

  42. Logging

    View Slide

  43. Why?

    View Slide

  44. What?

    View Slide

  45. Verify requests issued

    View Slide

  46. Request/response
    count

    View Slide

  47. API performance

    View Slide

  48. API errors

    View Slide

  49. How?

    View Slide

  50. @decorators?

    View Slide

  51. @log_requests

    def _req(self, method='get', verb=None, headers={},
    params={}, data={}):

    url = self.BASE_URL.format(verb=verb)

    request_headers = {'content-type': 'application/json'}

    request_params = {'api_key': self.API_KEY}

    request_headers.update(headers)

    request_params.update(params)

    return getattr(requests, method)(url, params=request_params,

    headers=request_headers, data=data)

    View Slide


  52. def log_requests(func):

    def decorator(*args, **kwargs):

    request_id = str(uuid.uuid4())

    logger.info("Requesting %s" % request_id)

    try:

    resp = func(*args, **kwargs)

    except:

    logger.exception("Request error %s" %
    request_id)

    raise

    logger.info("Response %s" % request_id)

    return resp

    return decorator

    View Slide

  53. Testing

    View Slide

  54. Why?

    View Slide

  55. View Slide

  56. How?

    View Slide

  57. Live API
    Mock servers
    Mocked responses

    View Slide

  58. View Slide

  59. That’s not nice (or fun)

    View Slide

  60. Mock server?

    View Slide

  61. Live testing’s hard,
    let’s go mocking!

    View Slide

  62. Mocking requests

    View Slide

  63. responses
    HTTPretty

    View Slide

  64. responses
    HTTPretty
    betamax

    View Slide

  65. @responses.activate
    def test_auth_error(self):
    responses.add(responses.POST,
    'https://api.smartystreets.com/street-address',
    body='', status=401, content_type='application/json')
    self.assertRaises(SmartyStreetsAuthError,
    self.client.street_addresses, [{}, {}])
    @responses.activate
    def test_payment_error(self):
    responses.add(responses.POST,
    'https://api.smartystreets.com/street-address',
    body='', status=402, content_type='application/json')
    self.assertRaises(SmartyStreetsPaymentError,
    self.client.street_addresses, [{}, {}])

    View Slide

  66. @httpretty.activate
    def test_auth_error(self):
    """Ensure an HTTP 403 code raises GeocodioAuthError"""
    httpretty.register_uri(httpretty.GET,
    “http://api.geocod.io/v1/parse",
    body="This does not matter", status=403)
    self.assertRaises(GeocodioAuthError,
    self.client.parse, "")

    View Slide

  67. There can be benefits
    to some live testing

    View Slide

  68. View Slide

  69. Performance

    View Slide

  70. Connections

    View Slide

  71. Batching

    View Slide

  72. The right API methods

    View Slide

  73. Data structures

    View Slide

  74. class APIData:

    def init(self, id, name, image):

    self.id = id

    self.name = name

    self.image = image


    View Slide


  75. class APIData:

    __slots__ = ['id', 'name', 'image']


    def init(self, id, name, image):

    self.id = id

    self.name = name

    self.image = image


    View Slide

  76. Concurrency

    View Slide

  77. response_iter = (

    grequests.post(

    url=url,

    data=json.dumps(data_chunk),

    params=params,

    headeresponse_iter=headeresponse_iter,

    ) for data_chunk in chunker(data, 100)

    )


    responses = grequests.imap(response_iter, size=parallelism)

    status_codes = {}

    addresses = AddressCollection([])

    for response in responses:

    if response.status_code not in status_codes.keys():

    status_codes[response.status_code] = 1

    else:

    status_codes[response.status_code] += 1


    if response.status_code == 200:

    addresses[0:0] = AddressCollection(response.json())

    View Slide

  78. session = FuturesSession(max_workers=parallelism)

    session.headers = headers

    session.params = params


    futures = [

    session.post(url, data=json.dumps(data_chunk))
    for data_chunk in chunker(data, 100)

    ]


    while not all([f.done() for f in futures]):

    continue


    status_codes = {}

    responses = [f.result() for f in futures]


    addresses = AddressCollection([])

    for response in responses:

    if response.status_code not in status_codes.keys():

    status_codes[response.status_code] = 1

    else:

    status_codes[response.status_code] += 1


    if response.status_code == 200:

    addresses[0:0] = AddressCollection(response.json())

    View Slide

  79. You know Twisted can
    do that, right?

    View Slide

  80. Security

    View Slide

  81. HTTPS

    View Slide

  82. Python 2.7
    • pyOpenSSL
    • ndg-httpsclient
    • pyasn1

    View Slide

  83. Authenticating

    View Slide

  84. Docs

    View Slide

  85. Django

    View Slide

  86. No?

    View Slide

  87. Python library
    Anything Django related

    View Slide

  88. • Create the client *first* then Django integration is
    a bonus.
    • Mapping models to a distant API.
    • It’ll be easier for you to maintain and test
    • Easier for other people to use for other things

    View Slide

  89. class SomeModel(models.Model):

    my_field = models.CharField(max_length=100)


    def sync(self, client=None):

    if client is None:

    client = APIClient()

    client.update(id=self.id, name=self.my_field)

    View Slide

  90. Code Gen

    View Slide

  91. Starting from (DSL) API
    documentation, automatically
    generate matching client code

    View Slide

  92. View Slide

  93. Crank ‘em out

    View Slide

  94. In [hypothetical]
    practice?

    View Slide

  95. Suitability?

    View Slide

  96. The End

    View Slide