RuhrJS 2016: The dirty little secrets of building large, highly available, scalable HTTP APIs

When you first start building an API for a new product you mostly focus on getting an MVP ready, with the goal of shipping as soon as possible so you can get feedback from customers. If you are lucky enough, your product will be successful and you will have to start worrying about things like authentication, authorization, documentation, validation, rate limiting, geo-redundancy, and no downtime deployments. In this talk I will go over some real life examples of our experience evolving our APIs at Auth0 and some of the tools we use for that.

Transcript

1. The dirty little secrets
   of building large, highly available, scalable HTTP APIs
   by @dschenkelman
   

   View Slide

2. Topic 1
   

   View Slide

3. Topic 2
   

   View Slide

4. Topic 3
   

   View Slide

5. Why?
   var express = require('express');
   var app = express();
   app.get('/', function (req, res) {
   res.send('Hello World!');
   });
   var server = app.listen(3000, function () {
   var host = server.address().address;
   var port = server.address().port;
   console.log('Example app listening at http://%s:%s', host, port);
   });
   

   View Slide

6. Developers
   

   View Slide

7. Evolution
   

   View Slide

8. Pillars
   

   View Slide

9. Validation
   

   View Slide

10. enabled: true
    

    View Slide

11. enabled: 1
    

    View Slide

12. enabled: 1.0
    

    View Slide

13. enabled: "false"
    

    View Slide

14. View Slide

15. JSON Schemas
    "enabled": {
    "type": "boolean"
    }
    

    View Slide

16. Errors for developers
    {
    "statusCode": 400,
    "error": "Bad Request",
    "message": "Payload validation error:
    'Expected type boolean but found type
    string' on property enabled
    (true if the rule is
    enabled, false
    otherwise).",
    "errorCode": "invalid_body"
    }
    

    View Slide

17. ratify
    const ZSchemaErrors = require('z-schema-errors');
    const errorReporters = ['headers', 'query', 'path',
    'payload'].reduce((current, part) => {
    current[part] = ZSchemaErrors.init({/*...*/});
    return current;
    }, {});
    plugins.push({
    register: require('ratify'),
    options: {
    errorReporters: errorReporters,
    /*...*/
    }
    });
    

    View Slide

18. Documentation
    

    View Slide

19. Old API Explorer
    

    View Slide

20. Auto-generate docs
    

    View Slide

21. Swagger
    

    View Slide

22. Leverage validation
    "enabled": {
    "type": "boolean",
    "description": "true if
    the connection is enabled,
    false (default) otherwise"
    }
    

    View Slide

23. Customize
    

    View Slide

24. AuthN & AuthZ
    

    View Slide

25. Token Exchange
    API Client API Server
    1. Credentials
    2. API Token
    3. Requests
    

    View Slide

26. Granular Security
    

    View Slide

27. Decentralized Issuance
    

    View Slide

28. Expiration Control
    

    View Slide

29. Debuggability
    

    View Slide

30. JSON Web Token
    JWT
    

    View Slide

31. Authenticate
    For all endpoints
    

    View Slide

32. Authorize
    Per endpoint
    

    View Slide

33. Route
    {
    method: 'GET',
    path: '/api/v2/clients/{id}',
    config: {
    auth: {
    strategy: 'jwt',
    scope: ['read:clients', 'read:client_keys']
    },
    }
    }
    

    View Slide

34. Blacklisting
    

    View Slide

35. Badgers
    

    View Slide

36. Stress tests
    

    View Slide

37. Faith
    unsigned long income;
    

    View Slide

38. Engineers Stress
    

    View Slide

39. What to do…
    

    View Slide

40. Rate Limiting
    

    View Slide

41. Token Bucket
    

    View Slide

42. limitd
    #port to listen on
    port: 9001
    #db path
    db: /var/limitd/database
    #define the bucket types
    buckets:
    customers:
    size: 10
    per_second: 10
    

    View Slide

43. 429
    X-RateLimit-Limit
    Maximum amount
    X-RateLimit-Remaining
    How many there are left
    X-RateLimit-Reset
    When will bucket be full again
    

    View Slide

44. patova
    plugins.push({
    register: require('patova'),
    options: {
    event: 'onPostAuth', // when to perform the limit check
    type: 'tenant', // bucket
    address: env.LIMITD_SERVER,
    extractKey: function(request, reply, done){
    const key = request.auth.credentials.__tenant;
    done(null, key);
    }
    }
    });
    

    View Slide

45. Geo Redundancy
    

    View Slide

46. Data Center Failure
    

    View Slide

47. Natural Disasters
    

    View Slide

48. Cloud Provider Failure
    

    View Slide

49. Our Setup
    

    View Slide

50. Our Setup
    App Instances
    MongoDB
    Elastic Search
    App Instances
    MongoDB
    Elastic Search
    MongoDB
    

    View Slide

51. The switch
    

    View Slide

52. Changes
    

    View Slide

53. Experiments
    

    View Slide

54. feature-change
    const feature_change = require('feature-change');
    var options = {
    expected: function(cb){
    mongo_search(mongo_opts, cb);
    },
    actual: function(cb){
    es_search(es_opts, cb);
    },
    logAction: function(current_result, new_result){
    // invoked when there is a difference in the results
    // (useful for logging)
    }
    };
    feature_change(options, function(err, result){
    // this is the original callback you were using for mongo
    // err and result always come from mongo_search
    });
    

    View Slide

55. Iron out differences
    

    View Slide

56. Wrap up
    

    View Slide

57. Shipping is important
    

    View Slide

58. Remember: evolve
    

    View Slide

59. Validation & Docs
    • http://json-schema.org
    • http://swagger.io
    • https://auth0.com/docs/api/v2 (example of
    branded auto-generated docs)
    • https://github.com/mac-/ratify
    

    View Slide

60. AuthN & AuthZ
    • https://tools.ietf.org/html/rfc7519
    • http://jwt.io/
    • https://auth0.com/blog/2014/12/02/using-json-
    web-tokens-as-api-keys/
    • https://auth0.com/blog/2015/03/10/blacklist-json-
    web-token-api-keys/
    • https://github.com/auth0/hapi-auth-jwt
    

    View Slide

61. Rate limiting
    • http://tools.ietf.org/html/rfc6585
    • https://github.com/limitd
    • https://github.com/dschenkelman/patova
    

    View Slide

62. BaaS
    • http://security.stackexchange.com/a/83382
    • http://www.brendangregg.com/FlameGraphs/
    cpuflamegraphs.html
    • https://github.com/thlorenz/v8-perf/issues/4
    • https://github.com/auth0/node-baas
    • http://docs.aws.amazon.com/AutoScaling/latest/
    DeveloperGuide/US_SetUpASLBApp.html
    

    View Slide

63. Geo Redundancy
    • http://highscalability.com/blog/2014/12/1/auth0-
    architecture-running-in-multiple-cloud-providers-
    and-r.html
    • https://auth0.com/availability-trust
    • http://docs.mongodb.org/master/tutorial/deploy-
    geographically-distributed-replica-set/
    

    View Slide

64. Feature Changes
    • http://zachholman.com/talk/move-fast-break-
    nothing/
    • https://auth0.com/blog/2015/10/27/feature-
    changes-at-auth0/
    • https://github.com/dschenkelman/feature-
    change
    

    View Slide

65. Thanks
    https://github.com/dschenkelman/api-secrets-talk
    @dschenkelman
    npm i dschenkelman
    

    View Slide