Scaleconf 2016: The dirty little secrets of building large, highly available, scalable HTTP APIs

When you first start building an API for a new product you mostly focus on getting an MVP ready, with the goal of shipping as soon as possible so you can get feedback from customers. If you are lucky enough, your product will be successful and you will have to start worrying about things like authentication, authorization, documentation, validation, rate limiting, geo-redundancy, and no downtime deployments. In this talk I will go over some real life examples of our experience evolving our APIs at Auth0 and some of the tools we use for that.

Transcript

1. The dirty little secrets
   of building large, highly available, scalable HTTP APIs
   by @dschenkelman
   

   View Slide

2. Why?
   var express = require('express');
   var app = express();
   app.get('/', function (req, res) {
   res.send('Hello World!');
   });
   var server = app.listen(3000, function () {
   var host = server.address().address;
   var port = server.address().port;
   console.log('Example app listening at http://%s:%s', host, port);
   });
   

   View Slide

3. Developers
   

   View Slide

4. Evolution
   

   View Slide

5. Pillars
   

   View Slide

6. Validation
   

   View Slide

7. enabled: true
   

   View Slide

8. enabled: 1
   

   View Slide

9. enabled: 1.0
   

   View Slide

10. enabled: "false"
    

    View Slide

11. JSON Schemas
    "enabled": {
    "type": "boolean"
    }
    

    View Slide

12. Errors for developers
    {
    "statusCode": 400,
    "error": "Bad Request",
    "message": "Payload validation error:
    'Expected type boolean but found type
    string' on property enabled
    (true if the rule is
    enabled, false
    otherwise).",
    "errorCode": "invalid_body"
    }
    

    View Slide

13. ratify
    const ZSchemaErrors = require('z-schema-errors');
    const errorReporters = ['headers', 'query', 'path',
    'payload'].reduce((current, part) => {
    current[part] = ZSchemaErrors.init({/*...*/});
    return current;
    }, {});
    plugins.push({
    register: require('ratify'),
    options: {
    errorReporters: errorReporters,
    /*...*/
    }
    });
    

    View Slide

14. Documentation
    

    View Slide

15. Old API Explorer
    

    View Slide

16. Auto-generate docs
    

    View Slide

17. Swagger
    

    View Slide

18. Leverage validation
    "enabled": {
    "type": "boolean",
    "description": "true if
    the connection is enabled,
    false (default) otherwise"
    }
    

    View Slide

19. Customize
    

    View Slide

20. AuthN & AuthZ
    

    View Slide

21. Token Exchange
    API Client API Server
    1. Credentials
    2. API Token
    3. Requests
    

    View Slide

22. Granular Security
    

    View Slide

23. Decentralized Issuance
    

    View Slide

24. Expiration Control
    

    View Slide

25. Debuggability
    

    View Slide

26. JSON Web Token
    JWT
    

    View Slide

27. Authenticate
    For all endpoints
    

    View Slide

28. Authorize
    Per endpoint
    

    View Slide

29. Route
    {
    method: 'GET',
    path: '/api/v2/clients/{id}',
    config: {
    auth: {
    strategy: 'jwt',
    scope: ['read:clients', 'read:client_keys']
    },
    }
    }
    

    View Slide

30. Blacklisting
    

    View Slide

31. T-shirt time
    

    View Slide

32. Stress tests
    

    View Slide

33. Devops Stress
    

    View Slide

34. What to do…
    

    View Slide

35. Rate Limiting
    

    View Slide

36. Token Bucket
    

    View Slide

37. limitd
    #port to listen on
    port: 9001
    #db path
    db: /var/limitd/database
    #define the bucket types
    buckets:
    customers:
    size: 10
    per_second: 10
    

    View Slide

38. 429
    X-RateLimit-Limit
    Maximum amount
    X-RateLimit-Remaining
    How many there are left
    X-RateLimit-Reset
    When will bucket be full again
    

    View Slide

39. patova
    plugins.push({
    register: require('patova'),
    options: {
    event: 'onPostAuth', // when to perform the limit check
    type: 'tenant', // bucket
    address: env.LIMITD_SERVER,
    extractKey: function(request, reply, done){
    const key = request.auth.credentials.__tenant;
    done(null, key);
    }
    }
    });
    

    View Slide

40. By Example
    

    View Slide

41. Admit the problem
    

    View Slide

42. Response times
    

    View Slide

43. Flame graphs
    

    View Slide

44. Sunburst
    

    View Slide

45. Password Hashing
    app.use('/login', (req, res) => {
    // fetch the user by req.username from db
    db.users.findOne({ email: req.body.email }, (err, user) =>
    {
    ...
    // compare bcrypt for req.password to db hash
    const success = bcrypt.compare(req.body.password,
    user.passwordHash);
    res.send(success ? 200 : 401);
    });
    });
    

    View Slide

46. Faster Hash
    

    View Slide

47. Caching
    

    View Slide

48. Scale up
    

    View Slide

49. Multiple IdP instances
    ELB
    IdP
    IdP
    IdP
    IdP
    

    View Slide

50. BaaS
    app.use('/login', (req, res) => {
    // fetch the user by req.username from db
    db.users.findOne({ email: req.body.email }, (err, user) =>
    {
    ...
    // compare bcrypt for req.password to db hash
    baas.compare(req.body.password, user.passwordHash).then((e
    success) => {
    res.send(err || !success ? 401 : 200);
    });
    });
    });
    

    View Slide

51. Auto Scaling
    Identity
    provider
    ELB
    BaaS
    BaaS
    BaaS
    BaaS
    

    View Slide

52. Cost comparison
    Price / (1M req)
    #req per sec /
    vCPU
    t2-micro $0.36 10.00
    t2-medium $0.76 9.50
    c4-large $1.53 10.00
    c3-8xlarge $1.64 8.88
    

    View Slide

53. Fail gracefully
    

    View Slide

54. Geo Redundancy
    

    View Slide

55. Data Center Failure
    

    View Slide

56. Natural Disasters
    

    View Slide

57. Cloud Provider Failure
    

    View Slide

58. Our Setup
    

    View Slide

59. Our Setup
    App Instances
    MongoDB
    Elastic Search
    App Instances
    MongoDB
    Elastic Search
    MongoDB
    

    View Slide

60. The switch
    

    View Slide

61. Changes
    

    View Slide

62. Experiments
    

    View Slide

63. feature-change
    const feature_change = require('feature-change');
    var options = {
    expected: function(cb){
    mongo_search(mongo_opts, cb);
    },
    actual: function(cb){
    es_search(es_opts, cb);
    },
    logAction: function(current_result, new_result){
    // invoked when there is a difference in the results
    // (useful for logging)
    }
    };
    feature_change(options, function(err, result){
    // this is the original callback you were using for mongo
    // err and result always come from mongo_search
    });
    

    View Slide

64. Iron out differences
    

    View Slide

65. Wrap up
    

    View Slide

66. Shipping is important
    

    View Slide

67. Remember: evolve
    

    View Slide

68. Validation & Docs
    • http://json-schema.org
    • http://swagger.io
    • https://auth0.com/docs/api/v2 (example of
    branded auto-generated docs)
    • https://github.com/mac-/ratify
    

    View Slide

69. AuthN & AuthZ
    • https://tools.ietf.org/html/rfc7519
    • http://jwt.io/
    • https://auth0.com/blog/2014/12/02/using-json-
    web-tokens-as-api-keys/
    • https://auth0.com/blog/2015/03/10/blacklist-json-
    web-token-api-keys/
    • https://github.com/auth0/hapi-auth-jwt
    

    View Slide

70. Rate limiting
    • http://tools.ietf.org/html/rfc6585
    • https://github.com/limitd
    • https://github.com/dschenkelman/patova
    

    View Slide

71. BaaS
    • http://security.stackexchange.com/a/83382
    • http://www.brendangregg.com/FlameGraphs/
    cpuflamegraphs.html
    • https://github.com/thlorenz/v8-perf/issues/4
    • https://github.com/auth0/node-baas
    • http://docs.aws.amazon.com/AutoScaling/latest/
    DeveloperGuide/US_SetUpASLBApp.html
    

    View Slide

72. Geo Redundancy
    • http://highscalability.com/blog/2014/12/1/auth0-
    architecture-running-in-multiple-cloud-providers-
    and-r.html
    • https://auth0.com/availability-trust
    • http://docs.mongodb.org/master/tutorial/deploy-
    geographically-distributed-replica-set/
    

    View Slide

73. Feature Changes
    • http://zachholman.com/talk/move-fast-break-
    nothing/
    • https://auth0.com/blog/2015/10/27/feature-
    changes-at-auth0/
    • https://github.com/dschenkelman/feature-
    change
    

    View Slide

74. Thanks
    https://github.com/dschenkelman/api-secrets-talk
    @dschenkelman
    npm i dschenkelman
    

    View Slide