NodeConfBR 2016: The dirty little secrets of building large, highly available, scalable HTTP APIs

When you first start building an API for a new product you mostly focus on getting an MVP ready, with the goal of shipping as soon as possible so you can get feedback from customers. If you are lucky enough, your product will be successful and you will have to start worrying about things like authentication, authorization, documentation, validation, rate limiting, geo-redundancy, and no downtime deployments. In this talk I will go over some real life examples of our experience evolving our APIs at Auth0 and some of the tools we use for that.

Transcript

1. The dirty little secrets
   of building large, highly available, scalable HTTP APIs
   by @dschenkelman
   

   View Slide

2. Christophers talk:
   Part 2
   

   View Slide

3. Why?
   var express = require('express');
   var app = express();
   app.get('/', function (req, res) {
   res.send('Hello World!');
   });
   var server = app.listen(3000, function () {
   var host = server.address().address;
   var port = server.address().port;
   console.log('Example app listening at http://%s:%s', host, port);
   });
   

   View Slide

4. Developers
   

   View Slide

5. Evolution
   

   View Slide

6. Pillars
   

   View Slide

7. Validation
   

   View Slide

8. enabled: true
   

   View Slide

9. enabled: 1
   

   View Slide

10. enabled: 1.0
    

    View Slide

11. enabled: "false"
    

    View Slide

12. View Slide

13. JSON Schemas
    "enabled": {
    "type": "boolean"
    }
    

    View Slide

14. Errors for developers
    {
    "statusCode": 400,
    "error": "Bad Request",
    "message": "Payload validation error:
    'Expected type boolean but found type
    string' on property enabled
    (true if the rule is
    enabled, false
    otherwise).",
    "errorCode": "invalid_body"
    }
    

    View Slide

15. ratify
    const ZSchemaErrors = require('z-schema-errors');
    const errorReporters = ['headers', 'query', 'path',
    'payload'].reduce((current, part) => {
    current[part] = ZSchemaErrors.init({/*...*/});
    return current;
    }, {});
    plugins.push({
    register: require('ratify'),
    options: {
    errorReporters: errorReporters,
    /*...*/
    }
    });
    

    View Slide

16. Documentation
    

    View Slide

17. Old API Explorer
    

    View Slide

18. Auto-generate docs
    

    View Slide

19. Swagger
    

    View Slide

20. Leverage validation
    "enabled": {
    "type": "boolean",
    "description": "true if
    the connection is enabled,
    false (default) otherwise"
    }
    

    View Slide

21. Customize
    

    View Slide

22. AuthN & AuthZ
    

    View Slide

23. Token Exchange
    API Client API Server
    1. Credentials
    2. API Token
    3. Requests
    

    View Slide

24. Granular Security
    

    View Slide

25. Decentralized Issuance
    

    View Slide

26. Expiration Control
    

    View Slide

27. Debuggability
    

    View Slide

28. JSON Web Token
    JWT
    

    View Slide

29. Authenticate
    For all endpoints
    

    View Slide

30. Authorize
    Per endpoint
    

    View Slide

31. Route
    {
    method: 'GET',
    path: '/api/v2/clients/{id}',
    config: {
    auth: {
    strategy: 'jwt',
    scope: ['read:clients', 'read:client_keys']
    },
    }
    }
    

    View Slide

32. Blacklisting
    

    View Slide

33. Interval
    

    View Slide

34. View Slide

35. View Slide

36. View Slide

37. View Slide

38. View Slide

39. 2nd half
    

    View Slide

40. Stress tests
    

    View Slide

41. Optimism
    unsigned long income;
    

    View Slide

42. Ops Stress
    

    View Slide

43. What to do…
    

    View Slide

44. Rate Limiting
    

    View Slide

45. Token Bucket
    

    View Slide

46. limitd
    #port to listen on
    port: 9001
    #db path
    db: /var/limitd/database
    #define the bucket types
    buckets:
    customers:
    size: 10
    per_second: 10
    

    View Slide

47. 429
    X-RateLimit-Limit
    Maximum amount
    X-RateLimit-Remaining
    How many there are left
    X-RateLimit-Reset
    When will bucket be full again
    

    View Slide

48. patova
    plugins.push({
    register: require('patova'),
    options: {
    event: 'onPostAuth', // when to perform the limit check
    type: 'tenant', // bucket
    address: env.LIMITD_SERVER,
    extractKey: function(request, reply, done){
    const key = request.auth.credentials.__tenant;
    done(null, key);
    }
    }
    });
    

    View Slide

49. Geo Redundancy
    

    View Slide

50. Data Center Failure
    

    View Slide

51. Natural Disasters
    

    View Slide

52. Cloud Provider Failure
    

    View Slide

53. Our Setup
    

    View Slide

54. Our Setup
    App Instances
    MongoDB
    Elastic Search
    App Instances
    MongoDB
    Elastic Search
    MongoDB
    

    View Slide

55. The switch
    

    View Slide

56. Changes
    

    View Slide

57. Experiments
    

    View Slide

58. feature-change
    const feature_change = require('feature-change');
    var options = {
    expected: cb => mongo_search(mongo_opts, cb),
    actual: cb => es_search(es_opts, cb),
    logAction: (current_result, new_result) => {
    // invoked when there is a difference in the results
    // (useful for logging)
    }
    };
    feature_change(options, (err, result) => {
    // this is the original callback you were using for mongo
    // err and result always come from mongo_search
    });
    

    View Slide

59. Iron out differences
    

    View Slide

60. Wrap up
    

    View Slide

61. Shipping is important
    

    View Slide

62. Remember: evolve
    

    View Slide

63. Validation & Docs
    • http://json-schema.org
    • http://swagger.io
    • https://auth0.com/docs/api/v2 (example of
    branded auto-generated docs)
    • https://github.com/mac-/ratify
    

    View Slide

64. AuthN & AuthZ
    • https://tools.ietf.org/html/rfc7519
    • http://jwt.io/
    • https://auth0.com/blog/2014/12/02/using-json-
    web-tokens-as-api-keys/
    • https://auth0.com/blog/2015/03/10/blacklist-json-
    web-token-api-keys/
    • https://github.com/auth0/hapi-auth-jwt
    

    View Slide

65. Rate limiting
    • http://tools.ietf.org/html/rfc6585
    • https://github.com/limitd
    • https://github.com/dschenkelman/patova
    

    View Slide

66. BaaS
    • http://security.stackexchange.com/a/83382
    • http://www.brendangregg.com/FlameGraphs/
    cpuflamegraphs.html
    • https://github.com/thlorenz/v8-perf/issues/4
    • https://github.com/auth0/node-baas
    • http://docs.aws.amazon.com/AutoScaling/latest/
    DeveloperGuide/US_SetUpASLBApp.html
    

    View Slide

67. Geo Redundancy
    • http://highscalability.com/blog/2014/12/1/auth0-
    architecture-running-in-multiple-cloud-providers-
    and-r.html
    • https://auth0.com/availability-trust
    • http://docs.mongodb.org/master/tutorial/deploy-
    geographically-distributed-replica-set/
    

    View Slide

68. Feature Changes
    • http://zachholman.com/talk/move-fast-break-
    nothing/
    • https://auth0.com/blog/2015/10/27/feature-
    changes-at-auth0/
    • https://github.com/dschenkelman/feature-
    change
    

    View Slide

69. Thanks
    https://github.com/dschenkelman/api-secrets-talk
    @dschenkelman
    npm i dschenkelman
    

    View Slide